Elamur Research⌕
☰
home / papers / 2012 / The Floating Arabic Dictionary: An Automatic Method for Updating a Lexical Database through the Detection and Lemmatization of Unknown Words

The Floating Arabic Dictionary: An Automatic Method for Updating a Lexical Database through the Detection and Lemmatization of Unknown Words

Paper · 2012 · Mohammed Attia, Younes Samih, Khaled Shaalan et al.

Summary

Arabic NLP tools struggle with out-of-vocabulary words. In a corpus of about 1.09 billion words, roughly 29% of word types could not be analysed by the SAMA morphological analyser, so checking new words by hand is not feasible. The authors define an unknown word as one that SAMA rejects but the Microsoft Spell Checker accepts. They then build a pipeline that detects such words, lemmatizes them and ranks them for inclusion in a lexicon.

The pipeline works in two steps:

  • Context tagging. MADA, an SVM-based contextual tagger, first assigns a POS tag and morpho-syntactic features to each token.
  • Guessing. A rule-based finite-state guesser then peels off affixes and clitics in every possible way. From its many candidate analyses, the one whose features best match MADA's tags is kept.

Lemmas are scored by how many distinct surface forms they have, their combined frequency and their POS. Proper nouns are pushed down because news makes them temporarily frequent.

The method was evaluated on a hand-annotated gold standard and beat the baseline. It produced a free lexicon of 18,399 lemmatized, tagged and weighted entries. The work shows that a highly ambiguous guesser can still be useful when context narrows its choices.

Key points

  • Corpus: 1,089,111,204 words in total. It combines the Arabic Gigaword Fourth Edition (925M words) with 163.6M words crawled from Al-Jazeera news.
  • Unknown words: 2,116,180 of 7,348,173 types were unanalysed by SAMA. Unknowns make up 2–9% of tokens, depending on whether spelling variants are counted.
  • Base forms: only 45% of unknown types in the test set occur in their base form, versus about 86% for new English words (Lindén). 36% never appear in base form anywhere in the corpus.
  • Ranking weight: ((sister forms × 800) + cumulative frequency of the sister forms) / 2, plus a POS bonus. Verbs get 50, nouns and adjectives get 30, and proper nouns get 0.
  • Evaluation and output: the gold standard contains 1,310 types. From 40,255 unknown words the pipeline built a lexicon of 18,399 entries, released at arabic-unknowns.sourceforge.net.

Interesting ideas

  • I1. Let a guesser over-generate every possible analysis, then let a separate context-based tagger choose the candidate that agrees on the most features; this splits recall and precision between two simple components.
  • I2. Call a word "unknown" only when two resources disagree: the analyser rejects it but a spell checker accepts it. This separates genuinely new words from typos.
  • I3. Group surface variants under one lemma first, then rank by how many distinct forms the lemma takes, not just raw counts. Productive inflection is better evidence of a core word than a burst of frequency.
  • I4. Treat proper nouns as time-bound and down-weight them, because news coverage inflates their counts temporarily.
  • I5. Judge a ranking by two checks: how many relevant items reach the top 100, and how few remain in the bottom 100.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): an OOV lemmatizer in crate en could enumerate suffix-stripping hypotheses and keep the one most consistent with the tagger's features, logging why it won.
  • I2 (learn): disagreement between independent witnesses could flag true lexical gaps, as distinct from noise, before they enter silver data.
  • I3 (world): when deciding which new terms to promote into compiled knowledge, inflectional spread could be weighted above raw frequency.
  • I4 (world): names could stay as text-local entities unless they persist across many documents, which keeps the ontology from filling with transient names.
  • I5 (learn): self-induced repair rules or candidate lexicon entries could be scored this way, so a ranking is not rewarded only for its top.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Lemmatization · 20 works

Cite

Cite as Mohammed Attia, Younes Samih, Khaled Shaalan and Josef van Genabith. 2012. The Floating Arabic Dictionary: An Automatic Method for Updating a Lexical Database through the Detection and Lemmatization of Unknown Words. Proceedings of COLING 2012.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{attia2012floating,
  title = {{The Floating Arabic Dictionary: An Automatic Method for Updating a Lexical Database through the Detection and Lemmatization of Unknown Words}},
  author = {Mohammed Attia, Younes Samih, Khaled Shaalan and Josef {van Genabith}},
  booktitle = {Proceedings of COLING 2012},
  year = {2012},
  url = {https://aclanthology.org/C12-1006/}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adler, M., Goldberg, Y., Gabay, D. and Elhadad, M. )2008(. Unsupervised Lexicon-Based Resolution of Unknown Words for Full Morpholological Analysis. In: Proceedings of Association for Computational Linguistics (ACL), Columbus, Ohio. find
  2. Attia, M. )2006(. An Ambiguity-Controlled Morphological Analyzer for Modern Standard Arabic Modelling Finite State Networks. In: Challenges of Arabic for NLP/MT Conference, The British Computer Society, London, UK. find
  3. Attia, Mohammed, Jennifer Foster, Deirdre Hogan, Joseph Le Roux, Lamia Tounsi and Josef van Genabith. )2010(. 'Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French'. First Workshop on Statistical Parsing of Morphologically Rich Languages (SPMRL 2010), NAACL HLT. Los Angeles, CA.
  4. Attia, Mohammed, Pavel Pecina, Lamia Tounsi, Antonio Toral, Josef van Genabith. (2011). An Open-Source Finite State Morphological Transducer for Modern Standard Arabic. International Workshop on Finite State Methods and Natural Language Processing (FSMNLP). Blois, France.
  5. Beesley, K. R. )2001(. Finite-State Morphological Analysis and Generation of Arabic at Xerox Research: Status and Plans in 2001. In: The ACL 2001 Workshop on Arabic Language Processing: Status and Prospects, Toulouse, France. find
  6. Beesley, K. R., and Karttunen, L.. )2003(. Finite State Morphology: CSLI studies in computational linguistics. Stanford, Calif.: Csli. find
  7. Buckwalter, T. )2004(. Buckwalter Arabic Morphological Analyzer (BAMA) Version 2.0. Linguistic Data Consortium (LDC) catalogue number LDC2004L02, ISBN1-58563-324-0 find
  8. Crystal, D. )1980(. A First Dictionary of Linguistics and Phonetics. London: Deutsch. find
  9. Diab, Mona, Kadri Hacioglu and Daniel Jurafsky. (2004). Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks. Proceedings of Human Language Technology-North American Association for Computational Linguistics (HLT-NAACL)
  10. Dichy, J. )2001(. On lemmatization in Arabic, A formal definition of the Arabic entries of multilingual lexical databases. ACL/EACL 2001 Workshop on Arabic Language Processing: Status and Prospects. Toulouse, France. find
  11. Dichy, J., and Farghaly, A. )2003(. Roots & Patterns vs. Stems plus Grammar-Lexis Specifications: on what basis should a multilingual lexical database centred on Arabic be built? In: The MT-Summit IX workshop on Machine Translation for Semitic Languages, New Orleans. find
  12. Erjavec, T., and Džerosk, S. )2004(. Machine Learning of Morphosyntactic Structure: Lemmatizing Unknown Slovene Words. Applied Artificial Intelligence, 18:17–41. ↗
  13. Huang, Chung-chi, Ho-ching Yen and Jason S. Chang. (2010). Using Sublexical Translations to Handle the OOV Problem in MT. in Proceedings of The Ninth Conference of the Association for Machine Translation in the Americas(AMTA). find
  14. Hulden, M. )2009(. Foma: a finite-state compiler and library. In: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL '09). Stroudsburg, PA, USA. find
  15. Kiraz, G. A. )2001(. Computational Nonlinear Morphology: With Emphasis on Semitic Languages. Cambridge University Press. find
  16. Lindén, K. )2008(. A Probabilistic Model for Guessing Base Forms of New Words by Analogy. In CICling-2008, 9th International Conference on Intelligent Text Processing and Computational Linguistics, Haifa, Israel, pp. 106-116. find
  17. Maamouri, M., Graff, D., Bouziri, B., Krouna, S., and Kulick, S. )2010(. LDC Standard Arabic Morphological Analyzer (SAMA) v. 3.1. LDC Catalog No. LDC2010L01. ISBN: 1-58563-555-3. find
  18. Mohamed, Emad; Sandra Kübler (2010). Arabic Part of Speech Tagging. Proceedings of LREC 2010, Valetta, Malta.
  19. Parker, R., Graff, D., Chen, K., Kong, J., and Maeda, K. )2009(. Arabic Gigaword Fourth Edition. LDC Catalog No. LDC2009T30. ISBN: 1-58563-532-4. find
  20. Roth, R., Rambow, O., Habash, N., Diab, M., and Rudin, C. )2008(. Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking. In: Proceedings of Association for Computational Linguistics (ACL), Columbus, Ohio.
  21. Sinclair, J. M. (ed.). )1987(. Looking Up: An Account of the COBUILD Project in Lexical Computing. London: Collins. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/attia-2012-floating-arabic-dictionary-automatic-method/