Elamur Research⌕
☰
home / papers / 2014 / AraNLP: a Java-based Library for the Processing of Arabic Text.

AraNLP: a Java-based Library for the Processing of Arabic Text.

Paper · 2014 · Maha Althobaiti, Udo Kruschwitz and Massimo Poesio

Summary

Arabic has rich morphology: clitics, affixes, root-and-pattern derivation, and inconsistent spelling. Nearly every Arabic NLP application therefore needs heavy preprocessing. Existing tools are scattered and often hard to combine, so researchers keep rebuilding their own pipelines. The authors respond with AraNLP, a free, platform-independent Java library that bundles the main preprocessing steps behind one interface. Some components are written from scratch, others are adapted from existing work:

  • a maximum-entropy sentence boundary detector trained on Arabic Wikipedia text
  • a simple tokenizer
  • light stemmers in the style of Larkey et al. and an adapted Khoja root stemmer
  • wrappers around Stanford's Arabic tools for segmenting words and tagging parts of speech
  • a normalizer with configurable levels, plus punctuation and diacritic removal

The modules can run alone or be chained in any order, with each one's output feeding the next. The only quantitative result is for the sentence detector, which reached 0.97 precision and about 0.98 recall on held-out data. The library matters because it lowers the barrier to building reproducible Arabic pipelines, and it has already been used to prepare a corpus in later work by the first author.

Key points

  • The sentence detector is a maximum-entropy model. It uses context features around candidate end characters: the neighbouring tokens and the spacing between them. It was trained on 1,838 sentences from 59 Arabic Wikipedia documents and tested on 871 sentences.
  • The Khoja root stemmer was adapted for the library. Its 168 Arabic stop words are now skipped by stemming instead of being deleted. Several light-stemming variants follow Larkey et al. (2002).
  • Word segmentation uses the Stanford CRF-based segmenter, which splits off the three PATB clitic classes: conjunctions, prepositions and pronouns.
    • POS tagging uses the Stanford maximum-entropy tagger.
    • Segmentation runs automatically before tagging, because the authors saw better tagging on segmented text.
  • Diacritic removal covers three types: vowel marks, nunation and shadda. The punctuation list is editable.
  • Orthographic normalization is user-selectable and covers four operations: hamzated alif to bare alif, taa marbutah to haa, alif maksura to yaa, and tatweel removal. The paper cites that such spelling variation affects 11% of PATB words.

Interesting ideas

  • I1. Treat sentence splitting as a small supervised classifier over local context rather than a fixed list of punctuation rules, and report precision and recall for it; the authors note almost nobody evaluates this step even though later stages depend on it.
  • I2. When stemming, leave function words untouched instead of dropping them, so downstream steps still see every word of the input.
  • I3. Make a prerequisite step run implicitly inside the component that needs it, here segmentation before tagging. Users then cannot accidentally feed the tagger unprepared input.
  • I4. Offer normalization as a set of separately switchable operations, so each application picks how aggressively spelling variants are merged.
  • I5. Naive splitting on punctuation can break names that contain hyphens or digits (product models, domain-specific identifiers). The tokenizer therefore needs to know about such entity patterns.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Mova's sentence splitter could be scored on a held-out set of tricky boundaries (abbreviations, quotes, decimals), with every boundary decision traceable to the features that triggered it.
  • I2 (tokens): the lemmatizer should never discard closed-class words, since determiners, auxiliaries and prepositions carry the arcs the UD parser needs.
  • I3 (tokens): the tagger could always apply contraction and clitic splitting (e.g. possessive 's, n't) itself, which guarantees the tokenization the treebank-trained models expect.
  • I4 (tokens): quote, dash, spelling-variant and casing normalizations could be individual flags that are logged per token, keeping the original surface form recoverable for explanations and for generation.
  • I5 (world): entity tracking would benefit if tokens such as model numbers were kept whole or re-joined, so one product name maps to one entity in the world of the text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EssexColchester · United Kingdom34 works · 3,537 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Maha Althobaiti, Udo Kruschwitz and Massimo Poesio. 2014. AraNLP: a Java-based Library for the Processing of Arabic Text. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{althobaiti2014aranlp,
  title = {{AraNLP: a Java-based Library for the Processing of Arabic Text.}},
  author = {Maha Althobaiti, Udo Kruschwitz and Massimo Poesio},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/2r58j2tkkx7s},
  url = {https://doi.org/10.63317/2r58j2tkkx7s}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Hamid, A. and Darwish, K. (2010). Simplified feature set for Arabic Named Entity Recognition. In Proceedings of the 2010 Named Entities Workshop, pages 110–115. Association for Computational Linguistics.
  2. Aljlayl, M. and Frieder, O. (2002). On Arabic search: improving the retrieval effectiveness via a light stemming approach. In Proceedings of the eleventh international conference on Information and knowledge management, pages 340–347. ACM. find
  3. Althobaiti, M., Kruschwitz, U., and Poesio, M. (2012). Identifying Named Entities on a University Intranet. In Computer Science and Electronic Engineering Conference (CEEC), 2012 4th, pages 94–99. IEEE. find
  4. Althobaiti, M., Kruschwitz, U., and Poesio, M. (2013). A Semi-supervised Learning Approach to Arabic Named Entity Recognition. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 32–40, Hissar, Bulgaria, September. INCOMA Ltd. Shoumen, BULGARIA.
  5. Croft, W., Metzler, D., and Strohman, T. (2010). Search Engines: Information Retrieval in Practice. Alternative Etext Formats. ADDISON WESLEY Publishing Company Incorporated. find
  6. de Roeck, A. and Al-Fares, W. (2000). A morphologically sensitive clustering algorithm for identifying Arabic roots. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 199– 206. Association for Computational Linguistics.
  7. Diab, M., Ghoneim, M., and Habash, N. (2007). Arabic diacritization in the context of statistical machine translation. In Proceedings of MT-Summit.
  8. El-Beltagy, S. and Rafea, A. (2011). An accuracy-enhanced light stemmer for Arabic text. ACM Transactions on Speech and Language Processing (TSLP), 7(2):2. ↗
  9. Green, S. and DeNero, J. (2012). A class-based agreement model for generating accurately inflected translations. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 146–155. Association for Computational Linguistics. find
  10. Grefenstette, G. and Tapanainen, P. (1994). What is a Word, what is a Sentence?: Problems of Tokenisation. Rank Xerox Research Centre. find
  11. Habash, N. and Sadat, F. (2006). Arabic preprocessing schemes for statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL., pages 49–52. ACL.
  12. Habash, N. Y. (2010). Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  13. Khoja, S. and Garside, R. (1999). Stemming Arabic Text. Lancaster, UK, Computing Department, Lancaster University. http://zeus.cs.pacificu.edu/shereen/research.htm. ↗
  14. Kroeger, P. (2005). Analyzing Grammar: An Introduction. Cambridge University Press. find
  15. Larkey, L. S. and Connell, M. E. (2006). Arabic information retrieval at UMass in TREC-10. Technical report, DTIC Document. ↗
  16. Larkey, L. S., Ballesteros, L., and Connell, M. E. (2002). Improving stemming for Arabic Information Retrieval: light stemming and co-occurrence analysis. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 275–282. ACM. find
  17. Larkey, L., Ballesteros, L., and Connell, M. (2007). Light Stemming for Arabic Information Retrieval. In Soudi, A., Bosch, A. d., and Neumann, G., editors, Arabic Computational Morphology, volume 38 of Text, Speech and Language Technology, pages 221–243. Springer Netherlands. ↗
  18. Loos, E., Anderson, S., Day, D., Jordan, P., and Wingate, J. (2004). Glossary of linguistic terms, volume 29. SIL International. find
  19. Ouersighni, R. (2001). A major offshoot of the DIINAR-MBC project: AraParse, a morphosyntactic analyzer for unvowelled Arabic texts. In ACL 39th Annual Meeting, pages 9–16. find
  20. Payne, T. E. (2006). Exploring language structure: a student’s guide. Cambridge University Press. find
  21. Pirkola, A. (2001). Morphological typology of languages for IR. volume 57, pages 330–348. MCB UP Ltd. find
  22. Ritchey, T. (1998). General morphological analysis. In 16th EURO Conference on Operational Analysis. find
  23. Shaalan, K. and Raza, H. (2009). NERA: Named entity recognition for Arabic. volume 60, pages 1652–1663. Wiley Online Library. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/althobaiti-2014-aranlp-java-based-library-processing/