Elamur Research⌕
☰
home / papers / 2019 / An Accuracy-Enhanced Stemming Algorithm for Arabic Information Retrieval

An Accuracy-Enhanced Stemming Algorithm for Arabic Information Retrieval

Summary

The paper addresses a core problem in Arabic information retrieval. Arabic derivational morphology is rich, and text is usually written without vowels, so exact string matching between a query and documents misses many relevant texts. The authors propose ESAIR (Enhanced Stemmer for Arabic Information Retrieval), which has three stages: normalization, lexical analysis and index generation. Analysis is driven by dictionaries of triliteral roots, templates (patterns), stop words and grammatical words.

A word is segmented in two layers:

  • First into proclitic + base + enclitic.
  • Then the base is split into prefix + core + suffix.

Each layer is checked against an incompatibility table. The core is matched to a template of equal length whose fixed letters sit at the same positions. Deleting those letters leaves the root.

Evaluation used the EASC corpus with 25 queries and manually extracted roots. ESAIR found the correct root about 96% of the time. It raised average precision and recall well above a no-stemming baseline. The work shows that template-based root extraction reduces retrieval "silence" (missed relevant documents) for a highly inflected language.

Key points

  • Two-level segmentation. Words are first split as proclitic + base 1 + enclitic, and base 1 is then split as prefix + base 2 + suffix. The clitic list follows Darwish. Any decomposition whose concatenated proclitic+enclitic string appears in an incompatibility table is discarded.

  • Template matching. A template applies when its length equals the word's length and its infix letters occur at the listed positions. For example, MAFATIH matches template MAFAIIL with infix code 135, and removing positions 1, 3 and 5 yields the root FTH.

  • Illustration text. A 162-word text shrank to 72 words after stop-word removal (44%) and to 43 index terms after ESAIR (26%). On that text, root accuracy was 69/72 ≈ 0.958, and one sample query gave P = 0.64, R = 0.75 and silence = 0.25.

  • EASC corpus. It contains 153 articles from the "Alwatan" and "Alrai" newspapers, with 59,548 words and about 389 words per document.

  • Retrieval results over 25 queries.

    MethodPrecisionRecall
    ESAIR0.57320.6916
    NoStem0.43280.4152

    The authors report the difference as statistically significant, and overall root accuracy reached up to 96%.

Interesting ideas

  • I1. Segmentation candidates are pruned by a compact blacklist of incompatible affix pairs, stored as concatenated strings, rather than by enumerating all legal combinations. This is cheap, transparent and easy to audit. — For Mova (tokens): the en lemmatizer could keep a small explicit table of forbidden suffix/clitic stacks (e.g. impossible contraction or -ness/-ly chains) to reject bad splits with a readable reason.
  • I2. A morphological pattern is represented as a length plus a list of fixed-letter positions, so matching and root extraction reduce to positional checks and deletions, which makes every analysis directly explainable. — For Mova (tokens): irregular or derivational English families (sing/sang/sung, -ation/-ative stems) could be encoded as positional templates whose matched slots are reported as the justification for a lemma.
  • I3. Every valid decomposition is generated and kept, then compared against the stored ones before one reliable analysis is chosen, instead of committing greedily to the first split. — For Mova (tokens): the morphology step could emit all licensed analyses with their rule traces and let later context choose, keeping the losing analyses visible for debugging.
  • I4. Index compression is shown on two contrasting texts: one packed with words from a single root (reduction near 98%) and one with no shared roots (near 0%). Together they act as a built-in sanity check for a conflation step. — For Mova (learn): lemma/frame-merging stages could be tested on paired positive and negative control texts where the expected collapse ratio is known in advance.
  • I5. The paper tracks "silence" (the share of relevant items missed) as a separate quantity alongside precision and recall. This puts the cost of under-matching in plain view. — For Mova (answer): QA over fables and tales could log silence, meaning relevant story facts never retrieved for a question, to spot where surface-form mismatches block answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Morphology · Semitic languages · Information retrieval & search

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.14311/nnw.2014.24.007 · arXiv 1911.08249

@misc{bessou2019accuracy,
  title = {{An Accuracy-Enhanced Stemming Algorithm for Arabic Information Retrieval}},
  author = {Bessou, Sadik and Touahria, Mohamed},
  year = {2019},
  doi = {10.14311/nnw.2014.24.007},
  eprint = {1911.08249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.14311/nnw.2014.24.007}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sophoclis N., Abdeen M., El-Horbaty E., Yagoub M.: A Novel Approach for Indexing Arabic Documents through GPU Computing. 25th IEEE Canadian Conference on Electrical and Computer Engineering, 2012, pp. 1-4. find
  2. Berners-Lee T., Hendler J., Lassila O.: The Semantic Web. Scientific American, 2001, pp. 28-37. find
  3. Zweigenbaum P., Demner-Fushman D., Yu H., Cohen K. B.: Frontiers of Biomedical Text Mining: Current Progress. Briefings in Bioinformatics, 2007, pp. 358-375. find
  4. Chalabi A.: MT-Based Transparent Arabization of the Internet tarjim.com. 4th Conference of the Association for Machine Translation in the Americas, 2000, pp. 189-191. find
  5. Daimi K.: Identifying Syntactic Ambiguities in Single-Parse Arabic Sentence. Computer and Humanities, 2001, pp. 333-349. find
  6. Farag A., Nurnberger A.: Arabic/English Word Translation Disambiguation Approach Based on Naive Bayesian Classifier. International Multiconference on Computer Science and Information Technology, 2008, pp. 331-338. find
  7. Attia M.: An Ambiguity-Controlled Morphological Analyzer for Modern Standard Arabic Modelling Finite State Networks. Challenge of Arabic for NLP/MT Conference, 2006, pp. 48-67. find
  8. Al-Jedady A. A., Alsmadi I. M., Al-Shawakfa E. M., Al-Kabi M. N.: Enhancing Query Retrieval Efficiency Using BGIT Coding, Bigram Based Index Term Coding Applied to Arabic Language. International Conference on Computer, Information and Telecommunication Systems, 2012, pp. 1-5. find
  9. Otair M., Al-Sardi R., Al-Gialain S.: An Arabic Retrieval System with Native Language rather than SQL Queries. 1st International Conference on the Applications of Digital Information and Web Technologies, 2008, pp. 84-89. find
  10. Shaalan K., Al-Sheikh S., Oroumchian F.: Query Expansion Based-on Similarity of Terms for Improving Arabic information Retrieval. Intelligent Information Processing, 2012, pp. 167-176. find
  11. Ghwanmeh S., Rabab’ha S.: Enhanced Algorithm for Extracting the Root of Arabic Words. 6th International Conference on Computer Graphics, Imaging and Visualization, 2009, pp. 388-391. find
  12. Douzidia F., Lapalme G.: Lakhas, an Arabic Summarization System. Document Understanding Conference, 2004, pp. 128-135. find
  13. Attia M.: Developing Robust Arabic Morphological Transducer Using Finite State Technology. 8th Annual CLUK Research Colloquium, 2005, pp. 9-18. find
  14. Mohamadi T., Mokhnache S.: Design and Development of Arabic Speech Synthesis. World Scientific and Engineering Academy and Society, 2002, pp. 25-28. find
  15. Larkey L., Ballesteros L., Connell M.: Improving Stemming for Arabic Information Retrieval: Light Stemming and Co-occurrence Analysis. 25th Annual International Conference on Research and Development in Information Retrieval, 2002, pp. 275-282. find
  16. Aljlayl M., Frieder O.: On Arabic Search: Improving the Retrieval Effectiveness via a Light Stemming Approach. 11th International Conference on Information and Knowledge Management, 2002, pp. 340-347. find
  17. Eldesouki M., Darwish K. M., Arafa W. M.: Stemming Techniques of Arabic Language: Comparative Study from the Information Retrieval Perspective. The Egyptian Computer Journal, 2009, pp. 30-49. find
  18. Chen A., Gey F.: Building an Arabic Stemmer for Information Retrieval. 11th Text Retrieval Conference, 2002, pp. 631-640. find
  19. Habash N., Rambow O., Kiraz G.: Morphological Analysis and Generation for Arabic Dialects. ACL Workshop on Computational Approaches to Semitic Languages, 2005, pp. 17-24. find our card
  20. Bessou S., Louail M., Refoufi A., Kadem Z., Touahria M.: Un système de lemmatisation pour les applications de TALN. Colloque international sur le traitement automatique de la langue arabe, 2007, pp. 35-51. find
  21. Sawalha C., Atwell E.: Comparative Evaluation of Arabic Language Morphological Analysers and Stemmers. 22nd International Conference on Computational Linguistics, 2008, pp. 107-110. find
  22. Koster C. H. A., Beney J., Verberne S., Vogel M.: Phrase-Based Document Categorization, Current Challenges in Patent Information Retrieval. Springer, the Information Retrieval Series, 2011, pp. 263-286. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bessou-2019-accuracy-enhanced-stemming-algorithm-arabic/