Elamur Research⌕
☰
home / papers / 2014 / Arabizi Detection and Conversion to Arabic

Arabizi Detection and Conversion to Arabic

Paper · 2014 · Kareem Darwish

Summary

The paper tackles two problems with Arabizi, which is Arabic (MSA or dialect) typed in Latin letters and digits and often mixed with English on social media. The first problem is finding which words in a tweet are Arabizi. The second is rewriting those words in Arabic script so that ordinary Arabic NLP tools can process them.

For detection, the author built a Twitter dataset. He started from a few frequent Arabizi seed words, then pulled the other tweets of the users who wrote them. Each token was hand-labelled as Arabizi, English or other. A CRF++ sequence labeller then classified each word in context, using word-level and sequence-level features such as character language-model scores. This reached 98.5% word accuracy.

For conversion, character mappings were learned from Arabizi–Arabic word pairs. These mappings produce up to ten Arabic candidates per word, and a trigram language model covering both MSA and dialects picks the best sequence. This gave 88.7% accuracy.

The work matters because Arabizi has no fixed spelling, and normalizing it unlocks search and existing tools.

Key points

  • Detection data:
    • Seed queries were "e7na", "3shan" and "la2a". Expanding through their authors yielded 265 users, 16,507 tweets and 132,236 words.
    • Training used 522 tweets (5,207 tokens) and testing used 101 tweets (3,491 tokens).
    • Labels were a (Arabizi), e (English) and o (other).
  • Detection model:
    • CRF++ with default settings, using features of the current, previous and next word.
    • Character LMs were built with BerkeleyLM from 9.4M English words and 1,000 Arabizi words.
    • Word-level accuracy was 98.5%.
  • Conversion model:
    • It reuses the El-Kahky et al. (2011) baseline transliteration miner, trained on 3,452 Arabizi–Arabic pairs.
    • The top 10 candidates are ranked by Bayes rule, and an IRSTLM trigram LM chooses among them in context.
  • Conversion results, on 127 tweets (1,385 words):
    • Out of context, MRR was 0.84. The correct form ranked first 77.1% of the time and was missing from the list 4.9% of the time, which caps reachable accuracy at 95.1%.
    • In context, accuracy was 88.7%, compared with 77.1% for a top-1 baseline.
  • Error breakdown: of the incorrect outputs, 91 bore no relation to the reference word, while 46 differed from it only in spelling or morphology.

Interesting ideas

  • I1. Collect a rare register by snowballing: query a few marker words, then fetch everything else those authors wrote, which surfaces texts that lack the seed words
  • I2. Decide a word's language or kind by its neighbours, not by dictionary lookup, because ambiguous words like "men" or "Mohamed" only resolve in context; real-valued scores are binned so a nominal CRF can use them
  • I3. Normalize by generate-then-select: produce a short, scored list of lexicon-valid candidates from character mappings, then let a word-sequence model choose jointly over the sentence
  • I4. Evaluate the candidate generator on its own (MRR plus a "not found" rate that sets an accuracy ceiling) before evaluating in-context selection
  • I5. Split errors into near-misses (spelling or affix variants of the truth) and truly unrelated outputs

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): this could harvest informal or nonstandard English for silver data without hand-picking every document.
  • I2 (tokens): a context labeller could flag shorthand tokens like "2", "u" or "cuz" before tagging, with the bin and feature that fired kept as an explanation.
  • I3 (tokens): misspellings and stretched words ("cooooool") could map to lemma candidates whose transliteration and context scores remain inspectable.
  • I4 (learn): any candidate-producing stage, such as lemma or frame candidates, could report its oracle coverage separately so failures are traced to the right step.
  • I5 (learn): separating benign variant errors from real mistakes would give a more honest picture of where repair rules are worth writing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝Qatar FoundationDoha · Qatar17 works · 2,111 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Kareem Darwish. 2014. Arabizi Detection and Conversion to Arabic. Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3629

@inproceedings{darwish2014arabizi,
  title = {{Arabizi Detection and Conversion to Arabic}},
  author = {Kareem Darwish},
  booktitle = {Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)},
  year = {2014},
  doi = {10.3115/v1/w14-3629},
  url = {https://doi.org/10.3115/v1/w14-3629}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Alex, A. Dubey, and F. Keller. 2007. Using foreign inclusion detection to improve parsing performance. In Proceedings of EMNLP-CoNLL K. Beesley. 1988. Language Identifier: A computer program for automatic natural-language identification of on-line text. Proceedings of the 29th Annual Conference of the American Translators Association, 4754. find
  2. Kareem Darwish, Walid Magdy, and Ahmed Mourad. 2012. Language processing for arabic microblog retrieval. Proceedings of the 21st ACM international conference on Information and knowledge management. find
  3. T. Dunning. 1994. Statistical identification of language. Technical Report, CRL MCCS-94-273, New Mexico State University. find
  4. Y. Ehara, K. Tanaka-Ishii. 2008. Multilingual text entry using automatic language detection. In Proceedings of IJCNLP-2008. find
  5. Ali El-Kahky, Kareem Darwish, Ahmed Saad Aldein, Mohamed Abd El-Wahab, Ahmed Hefny, and Waleed Ammar. 2001. Improved transliteration mining using graph reinforcement. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1384-1393. Association for Computational Linguistics, 2011. find
  6. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: an open source toolkit for handling large scale language models. Proceedings of Interspeech. 2008. find
  7. Nizar Habash. 2010. Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies 3.1 (2010): 1-187. ↗
  8. Nizar Habash, Mona T. Diab, Owen Rambow. 2012. Conventional Orthography for Dialectal Arabic. LREC, pp. 711-718. 2012.
  9. Sittichai Jiampojamarn, Kenneth Dwyer, Shane Bergsma, Aditya Bhargava, Qing Dou, Mi-Young Kim and Grzegorz Kondrak. 2010. Transliteration Generation and Mining with Limited Training Resources. ACL NEWS workshop 2010. find
  10. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, Evan Herbst, Moses: Open Source Toolkit for Statistical Machine Translation, Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session, Prague, Czech Republic, June 2007. find
  11. Jin-Shea Kuo, Haizhou Li, Ying-Kuei Yang. 2006. Learning Transliteration Lexicons from the Web. COLING-ACL 2006, Sydney, Australia, 11291136. find
  12. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, In Proc. of ICML, pp.282-289, 2001. find
  13. Kavi Narayana Murthy and G. Bharadwaja Kumar. 2006. Language identification from small text samples. Journal of Quantitative Linguistics 13.01 (2006): 57-80. ↗
  14. Sara Noeman and Amgad Madkour. 2010. Language Independent Transliteration Mining System Using Finite State Automata Framework. ACL NEWS workshop 2010. find
  15. Hassan Sajjad, Alexander Fraser, Helmut Schmid. 2012. A Statistical Model for Unsupervised and Semi-supervised Transliteration Mining. ACL (1) 2012: 469-477 find
  16. F. Sha and F. Pereira. 2003. Shallow parsing with conditional random fields, In Proc. of HLT/NAACL 2003 find
  17. Raghavendra Udupa, K. Saravanan, Anton Bakalov, Abhijit Bhole. 2009. ”They Are Out There, If You Know Where to Look”: Mining Transliterations of OOV Query Terms for Cross-Language Information Retrieval. ECIR-2009, Toulouse, France, 2009. find
  18. Raghavendra Udupa, Shaishav Kumar. 2010. Hashing-based Approaches to Spelling Correction of Personal Names. EMNLP 2010. find
  19. A. Xafopoulos, C. Kotropoulos, G. Almpanidis, I. Pitas. 2004. Language identification in web documents using discrete hidden Markov models. Pattern Recognition, 37 (3), 583594 find
  20. Min Zhang, A Kumaran, Haizhou Li. 2011. Whitepaper of NEWS 2012 Shared Task on Machine Transliteration. IJCNLP-2011 NEWS workshop. find
  21. Min Zhang, Haizhou Li, Ming Liu, A Kumaran. 2012. Whitepaper of NEWS 2012 Shared Task on Machine Transliteration. ACL-2012 NEWS workshop. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/darwish-2014-arabizi-detection-conversion-arabic/