📄 Clitics in Arabic Language: A Statistical Study

What it is about

Arabic clitics (prefixes such as the article and the conjunction, and suffixes such as the pronouns) are written without a separator, so one white-space word can contain several tokens. The authors tokenize a 600-million-word Arabic newswire corpus with AMIRA 2.0 in its fast mode (about 300 hours on a quad-processor PC) and count unique types before and after the split. The unique main tokens go from 2,207,637 to 1,665,899 sub-tokens, a reduction of 24.54%, and the clitic frequencies are compared with the manually tokenized Arabic Treebank part 3. The tool produces false clitics (a letter A taken as a prefix, and others), and the paper lists them in a table with about 100,000 cases in total. The authors recommend a lexicon of sub-tokens for speech recognition, because it is smaller and covers more of the language. The paper reports the lexicon size as a function of corpus size, from 10 to 600 million tokens.

Ideas

• I1. Measure the type reduction by splitting, and the growth curve by corpus size, for Hebrew and Aramaic text in the same way. The count is deterministic and needs only open text, and it says how big a segmenter lexicon must be before it stops growing.
• I2. Keep the paper's list of false clitics as a negative control for the segmenter: a split into a prefix that is not a clitic must be rejected. Concretely, a split is allowed only when the remainder is a known lexeme (checked in lex and pre of tools/hbo).
• I3. Count clitic frequencies on the MACULA trees and compare them with the automatic split on the same words. The gap, not the raw frequencies, tells where the segmenter tables are missing entries.

For Mova

I1 is a measurement for tools/hbo (the segmenter lexicon, lex/pre tables) and for languages/hbo/10-lexicon.md. I2 is a negative test for the same tables. I3 belongs to tools/macula and to the clitic inventory in languages/hbo/05-pronouns-particles.md. The same method applies to languages/arc/ once there is a corpus.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{alotaiby2010clitics,
  title = {{Clitics in Arabic Language: A Statistical Study}},
  author = {Alotaiby, Fahad and Foda, Salah and Alkharashi, Ibrahim},
  booktitle = {Proceedings of the 24th Pacific Asia Conference on Language, Information and Computation},
  year = {2010},
  url = {https://aclanthology.org/Y10-1068/}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alotaiby, F., I. Alkharashi and S. Foda. 2009. Processing large Arabic text corpora: Preliminary analysis and results. Proceedings of the Second International Conference on Arabic Language Resources and Tools, Cairo, Egypt, pp. 78-82. find
2. Attia, M. 2007. Arabic tokenization system. Proceedings of the 2007 Workshop on Computational Approaches To Semitic Languages: Common Issues and Resources, Prague, Czech Republic. Association for Computational Linguistics, pp. 65-72. find
3. Buckwalter, T. 2004a. Issues in Arabic Orthography and Morphology Analysis. Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages, Geneva, Switzerland. find our card
4. Buckwalter, T. 2004b. Buckwalter Arabic Morphological Analyzer Version 2.0, Linguistic Data Consortium (LDC) catalogue number LDC2004L02, ISBN 1-58563-324-0, Philadelphia. find
5. Diab, M. 2009. Second generation AMIRA tools for Arabic processing: Fast and robust tokenization, POS tagging, and phrase chunking. Proceedings of the Second International Conference on Arabic Language Resources and Tools, Cairo, Egypt, pp. 285-288. find
6. Diab, M., K. Hacioglu and D. Jurafsky. 2004. Automatic tagging of Arabic text: From raw text to base phrase chunks. 5th Meeting of the North American Chapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL04), Boston, MA. find
7. Graff, D. 2007. Arabic Gigaword Third Edition. Linguistic Data Consortium. Philadelphia, USA. ↗
8. Graff, D., J. Kong, K. Chen and K. Maeda. 2007. English Gigaword Third Edition. Linguistic Data Consortium. Philadelphia, USA. ↗
9. Habash, N., O. Rambow and R. Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. Proceedings of the Second International Conference on Arabic Language Resources and Tools, Cairo, Egypt, pp. 102-109. find
10. Maamouri, M., A. Bies, S. Kulick, F. Gaddeche, and W. Mekki. 2007. Arabic Treebank: Part 3(a) v. 2.6. Linguistic Data Consortium. Philadelphia, USA, Catalog ID: LDC2007E65. find
11. Manning, C., and H. Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, Massachusetts.. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aclbulk-2010-alotaiby-clitics-arabic-language-statistical-study/