Elamur Research⌕
☰
home / papers / 2010 / Arabic Word Segmentation for Better Unit of Analysis

Arabic Word Segmentation for Better Unit of Analysis

Paper · 2010 · Yassine Benajiba and Imed Zitouni

Summary

Arabic builds words by stacking prefixes, a stem and suffixes, so raw whitespace tokens are very sparse. This hurts learning and retrieval. The authors build two segmenters on Weighted Finite State Transducers. One produces full morphological segmentation, splitting every affix including gender markers. The other produces the lighter Arabic TreeBank (ATB) scheme, which splits off only pieces that head their own phrasal constituent. Each segmenter composes three machines: an affix-expansion lattice, a stem dictionary and a Kneser-Ney trigram model over morphemes. Each is first trained on hand-segmented ATB data and then grown with stems discovered in automatically segmented Arabic Gigaword text.

On ATB data, the ATB segmenter reaches 99.4% accuracy and the morphological one 98.1%. The authors then train a mention detection (MD) system on each output and on unsegmented text. Full morphological segmentation gives the best MD scores, even though its segmenter makes more errors. Unsegmented text is worst by more than 6 F points. The main lesson is to choose the segmentation scheme by its effect on the downstream task.

Key points

  • Segmenter accuracy: ATB 99.4% (42,380 correct vs 274 wrong); morphological 98.1% (41,850 correct vs 804 wrong).
  • Decoding picks the most probable segmentation path using a Kneser-Ney back-off trigram model over segments.
  • Unknown words get a character-unigram model, but an empirically tuned penalty dominates it.
  • Lexicon bootstrapping:
    • A character n-gram segmenter labels part of a 155M-word Arabic Gigaword corpus.
    • A stem or affix is added to the vocabulary when its count exceeds two in the supervised data or exceeds ten in the automatically segmented text.
  • Overall MD F-measure with the richest (syntactic) feature set:
    • 75.5 with morphological segmentation,
    • 72.1 with ATB segmentation,
    • 69.0 on unsegmented text.
  • Morphological segmentation also leaves fewer OOV tokens (1,518 vs 2,464). ATB segmentation helps somewhat on long-span mentions.

Interesting ideas

  • I1. The segmenter that scores better on its own benchmark (ATB) gives the weaker downstream model, so a unit of analysis should be judged by the task it feeds.
  • I2. The segmenter lists every affix split allowed by a lookup table as a lattice, then lets an n-gram model over morphemes choose the path in context, as in deciding "and in" vs "faithful".
  • I3. The ATB scheme splits a piece off only if it projects its own phrasal constituent, which gives a principled, syntax-driven definition of a token.
  • I4. A character-level model, which handles unseen words better, auto-labels a large corpus to discover new stems. Separate frequency thresholds apply to trusted and auto-labeled data.
  • I5. Unknown words get an explicit model whose weight is a tuned penalty, so the decoder does not cheaply explain everything as new words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When choosing between tokenization or UD dialect variants, compare them on frame and answer accuracy, not only on tokenizer or parser scores.
  • I2 (tokens): English clitics and contractions ('s, n't, 'd) and hyphenated forms could be split the same way, keeping the rejected options inspectable for explainability.
  • I3 (tree): This rule can justify which English splits become separate UD syntactic words (e.g. possessive 's) when converting treebank dialects.
  • I4 (learn): Silver data from weaker but broader-coverage witnesses can grow the lexicon if auto-labeled evidence must pass a stricter count threshold than gold.
  • I5 (tokens): A tunable cost for "unknown lemma" analyses in the morphology could stop the tagger from over-using the out-of-vocabulary path.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations🏢IBMArmonk · United States85 works · 66,118 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 1,485 worksTokenization & segmentation · 77 works

Cite

Cite as Yassine Benajiba and Imed Zitouni. 2010. Arabic Word Segmentation for Better Unit of Analysis. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{benajiba2010arabic,
  title = {{Arabic Word Segmentation for Better Unit of Analysis}},
  author = {Yassine Benajiba and Imed Zitouni},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/3fgnn63q26d9},
  url = {https://doi.org/10.63317/3fgnn63q26d9}
}

References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Benajiba and I. Zitouni. 2009. Morphology-based segmentation combination for arabic mention detection. Special Issue on Arabic Natural Language Processing of ACM Transactions on Asian Language Information Processing (TALIP), 8(4). ↗
  2. Y. Benajiba, P. Rosso, and J.M. Gomez. 2007. Adapting jirs passage retrieval system to the arabic. In CI-CLing’07, pages 530–541. find
  3. Y. Benajiba, M. Diab, and P. Rosso. 2008. Arabic named entity recognition using optimized feature sets. In Proceedings of EMNLP’08, pages 284–293. find
  4. Y. Benajiba, M. Diab, and P. Rosso. 2009. Arabic named entity recognition: A feature-driven study. In the special issue on Processing Morphologically Rich Languages of the IEEE Transaction on Audio, Speech and Language. ↗
  5. T. Buckwalter. 2005. Buckwalter arabic morphological analyzer. http://www.ldc.upenn.edu/. LDC Catalog number LDC2002L49. ↗
  6. S. F. Chen and J. Goodman. 1998. An empirical study of smoothing techinques for language modeling. Technical Report TR-10-98, Center for Research in Computing Technology, Harvard University, Cambridge, Massachusettes, August. find
  7. K. Darwish. 2002. Building a shallow arabic morphological analyser in one day. In ACL02 Workshop on Computational Approaches to Semitic Languages.
  8. M. Diab, K. Hacioglu, and D. Jurafsky. 2004. Automatic tagging of arabic text: from raw text to base phrase chunks. In HLT/NAACL.
  9. M. Diab, K. Hacioglu, and D. Jurafsky, 2007. Arabic Computational Morphology: Knowledge-based and Empirical Methods, chapter 9. Springer. find
  10. R. Florian, H. Hassan, A. Ittycheriah, H. Jing, N. Kambhatla, X. Luo, N Nicolov, and S Roukos. 2004. A statistical model for multilingual entity detection and tracking. In Proceedings of HLT-NAACL 2004, pages 1–8. ↗
  11. D. Graff. 2003. Arabic gigaword. http://www.ldc.upenn.edu/, June. LDC Catalog number LDC2003T12. ↗
  12. N. Habash and O. Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proc. of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  13. N. Habash and F. Sadat. 2006. Combination of arabic preprocessing schemes for statistical machine translation. In Proceedings of ACL’06, pages 1–8, Sydney, Australia.
  14. L.S. Larkey, L. Ballesteros, and M.E. Connell. 2002. mproving stemming for arabic information retrieval: light stemming and co-occurrence analysis. In SI-GIR’02, pages 275–282. find
  15. Y.-S. Lee, K. Papineni, S. Roukos, O. Emam, and H. Hassan. 2003. Language model based Arabic word segmentation. In Proceedings of the ACL’03, pages 399–406. find
  16. M. Maamouri, A. Bies, T. Buckwalter, and W. Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. In Proceedings of NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
  17. M. Maamouri, A. Bies, T. Buckwalter, and H. Jin. 2005. Arabic treebank: Part 1 v 3.0. http://www.ldc.upenn.edu/, February. LDC Catalog number LDC2005T02. ↗
  18. M. Mohri, F. Pereira, and M. Riley. 2002. Weighted finite-state transducers in speech recognition. Computer Speech and Language, 16(1):69–88. ↗
  19. NIST. 2007. The ACE evaluation plan. www.nist.gov/speech/tests/ace/index.htm. find
  20. L. Ramshaw and M. Marcus. 1994. Exploring the statistical derivation of transformational rule sequences for part-of-speech tagging. In The Balancing Act: Proceedings of the ACL Workshop on Combining Symbolic and Statistical Approaches to Language, pages 128–135, New Mexico State University, July. ↗
  21. L. Ramshaw and M. Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora, ACL.
  22. E. F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2002, pages 155– 158. Taipei, Taiwan. ↗
  23. I. Zitouni and R. Florian. 2008. Mention detection crossing the language barrier. In Proceedings of EMNLP’08, Honolulu, Hawaii, October. find
  24. I. Zitouni, J. Sorensen, X. Luo, and R. Florian. 2005. The impact of morphological stemming on arabic mention detection and coreference resolution. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 63–70, Ann Arbor, June. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/benajiba-2010-arabic-word-segmentation-better-unit/