Elamur Research⌕
☰
home / papers / 2016 / Unsupervised Stemmer for Arabic Tweets

Unsupervised Stemmer for Arabic Tweets

Paper · 2016 · Fahad Albogamy and Allan Ramsay

Summary

The paper tackles stemming for Arabic tweets. This text type differs from Modern Standard Arabic (MSA) in vocabulary and syntax and keeps coining new words, so dictionary-based analysers such as BAMA or the Khoja stemmer fit it poorly. The authors also had no labelled data for training a statistical model. Their answer is a light stemmer written in Python. A small hand-written grammar describes which prefixes and suffixes may combine with nouns and verbs, and in what order. For each word, the stemmer first lists every segmentation the grammar allows. It then keeps the shortest candidate as the stem, subject to a minimum length. To test it, the authors manually annotated 1250 nouns and 692 verbs drawn from 390 tweets of a Gulf-region Twitter corpus. The best setting requires noun stems of at least 3 letters and verb stems of at least 2, and it reaches 77.91% overall accuracy. This is slightly above the figures previously reported for three known Arabic stemmers. The result suggests that a cheap stemmer with no lexicon can cope with noisy, open-vocabulary social media text.

Key points

  • Data: The source corpus has about 10 million Twitter tokens (Albogamy and Ramsay, 2015), collected through the Twitter Stream API using coordinates for the Arabian Peninsula. The test set is 390 sampled tweets, hand-annotated into a gold standard of 1250 nouns and 692 verbs.

  • Method: The stemmer works in two phases.

    • Phase 1 runs the word against a noun/verb affix-combination grammar and lists all possible stems.
    • Phase 2 picks the shortest stem. No root dictionary or pattern list is used.

  • Accuracy by minimum stem length:

    SettingNounsVerbsOverall
    Stems ≥3 letters78.16%67.19%74.25%
    Stems ≥2 letters76.8%77.45%77.03%
    Nouns ≥3, verbs ≥2——77.91%

  • Comparison: The baselines are Al-Kabi et al. (2015) at 75.03%, Khoja at 74.03% and Ghwanmeh et al. at 67.40%. These figures are taken from Al-Kabi et al. (2015), not re-run on this test set.

  • Error types:

    • under-stemming: an affix is left on the stem;
    • over-stemming: a letter of the stem is mistaken for an affix, e.g. an initial "w" read as a conjunction;
    • orthography: the stem is right, but its final letter has the wrong form.

Interesting ideas

  • I1. The stemmer first enumerates every analysis an affix grammar permits and then applies one explicit tie-break rule (shortest stem), so each output can be traced to a single grammar path and a single choice.
  • I2. The minimum stem length is tuned separately per word class, because one global threshold helped nouns but hurt verbs.
  • I3. Affixes are modelled as ordered slots with combination constraints, such as the future prefix being allowed only on imperfective verb stems. This prunes impossible segmentations before any selection step runs.
  • I4. Errors are split into under-stemming, over-stemming and orthographic repair failures. The third type separates finding the right boundary from restoring the correct surface form of the stem.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en crate could use this for out-of-vocabulary or noisy words. It would generate all suffix-stripping candidates from a small English affix grammar, then log which rule selected the lemma, keeping the step explainable.
  • I2 (tokens): The English fallback lemmatizer could use part-of-speech-specific stripping limits, for example stricter limits for nouns than for verbs. This would avoid trading accuracy in one class for gains in another.
  • I3 (tokens): Clitics like 's, n't and 'll, plus stacked derivational suffixes, could be encoded as slotted, constrained templates. The tokenizer would then reject illegal splits by rule instead of by statistics.
  • I4 (learn): Evaluations of the lemmatizer could tag each error with these three labels. Cases like "mak" versus "make" would then be counted separately and could seed targeted self-induced repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Stemming · 17 works

Cite

Cite as Fahad Albogamy and Allan Ramsay. 2016. Unsupervised Stemmer for Arabic Tweets. Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{albogamy2016unsupervised,
  title = {{Unsupervised Stemmer for Arabic Tweets}},
  author = {Fahad Albogamy and Allan Ramsay},
  booktitle = {Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)},
  year = {2016},
  url = {https://aclanthology.org/W16-3912/}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. GH Al-Gaphari and M Al-Yadoumi. 2010. A method to convert sana’ani accent to modern standard Arabic. International Journal of Information Science & Management, 8(1). find
  2. Mohammed N Al-Kabi, Saif A Kazakzeh, Belal M Abu Ata, Saif A Al-Rababah, and Izzat M Alsmadi. 2015. A novel root based Arabic stemmer. Journal of King Saud University-Computer and Information Sciences, 27(2):94–103. ↗
  3. Hasan Al-Serhan and Aladdin Ayesh. 2006. A triliteral word roots extraction using neural network for Arabic. In 2006 International Conference on Computer Engineering and Systems, pages 436–440. IEEE. find
  4. Yahya Alamlahi and Fateh Ahmed. 2007. Sanaani dialect to modern standard Arabic: rule-based direct machine translation. Computer Science Dep., Sanaa University, Sanaa, Yemen. find
  5. Fahad Albogamy and Allan Ramsay. 2015. POS tagging for Arabic tweets. RECENT ADVANCES IN Natural Language Processing, page 1.
  6. Mohammed Algarni, Brent Martin, Tim Bell, and Kourosh Neshatian. 2014. Simple Arabic stemmer. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM ’14, pages 1803–1806, New York, NY, USA. ACM. find
  7. Tim Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. ldc catalog number ldc2004l02. Technical report, ISBN 1-58563-324-0. find
  8. Kareem Darwish and Douglas W Oard. 2007. Adapting morphology for Arabic information retrieval. In Arabic Computational Morphology, pages 245–262. Springer. ↗
  9. Sajib Dasgupta and Vincent Ng. 2007. High-performance, language-independent morphological segmentation. In NAACL HLT 2007: Proceedings of the Main Conference, pages 155–163. find
  10. Anne N De Roeck and Waleed Al-Fares. 2000. A morphologically sensitive clustering algorithm for identifying Arabic roots. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 199–206. Association for Computational Linguistics.
  11. Joseph Dichy and Ali Farghaly. 2003. Roots & patterns vs. stems plus grammar-lexis specifications: on what basis should a multilingual lexical database centred on Arabic be built. In The MT-Summit IX workshop on Machine Translation for Semitic Languages, New Orleans. find
  12. W. B. Frakes. 1992. Information retrieval. chapter Stemming Algorithms, pages 131–160. Prentice-Hall, Inc., Upper Saddle River, NJ, USA. find
  13. Sameh Ghwanmeh, Ghassan Kanaan, Riyad Al-Shalabi, and Saif Rabab’ah. 2009. Enhanced algorithm for extracting the root of Arabic words. In Computer Graphics, Imaging and Visualization, 2009. CGIV’09. Sixth International Conference on, pages 388–391. IEEE. find
  14. John Goldsmith. 2001. Unsupervised learning of the morphology of a natural language. Computational linguistics, 27(2):153–198. ↗
  15. Akshay Java, Xiaodan Song, Tim Finin, and Belle Tseng. 2007. Why we Twitter: Understanding microblogging usage and communities. In Proceedings of the 9th WebKDD, pages 56–65, New York, NY, USA. ACM. find
  16. S. Khoja and R. Garside. 1999. Stemming Arabic text. Computing Department, Lancaster University, Lancaster, UK. find
  17. Chris D. Paice. 1994. An evaluation method for stemming algorithms. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’94, pages 42–50, New York, NY, USA. Springer-Verlag New York, Inc. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/albogamy-2016-unsupervised-stemmer-arabic-tweets/