Elamur Research⌕
☰
home / papers / 2015 / POS-tagging of Tunisian Dialect Using Standard Arabic Resources and Tools

POS-tagging of Tunisian Dialect Using Standard Arabic Resources and Tools

Paper · 2015 · Ahmed Hamdi, Alexis Nasr, Nizar Habash et al.

Summary

Tunisian Arabic (TUN) is a spoken dialect with about twelve million speakers, and it has almost no annotated data. Modern Standard Arabic (MSA), by contrast, has mature tools. The authors build a TUN part-of-speech tagger without a TUN training corpus. Their method rewrites dialect text into an approximation of MSA, which they call "pseudo MSA". This intermediate form is not meant for human readers, only for MSA tools.

The conversion has three steps:

  • morphological analysis of TUN words with MAGEAD,
  • lexical transfer through TUN–MSA lexica, some of them derived automatically,
  • optional generation of MSA forms.

The output is a lattice. An HMM tagger trained on MSA chooses a path through it, and the tags are projected back onto the original tokens.

Tagging works best on undiacritized lemmas ("lmms"). This setting reaches 89.1% accuracy, against 69% for running the MSA tagger directly on TUN. The work matters because it offers a reusable recipe for Arabic dialects that avoids the cost of transcribing and annotating new corpora.

Key points

  • The tagger is trained on Penn Arabic Treebank (PATB) Part 3, converted to the CATiB representation. This gives about 24K sentences and 330K tokens, with a coarse tagset of six tags. Trigram HMMs did slightly better than bigram ones.
  • The evaluation set has 805 transcribed TUN sentences (10,746 tokens, 2,455 types). Sources include TV series, political debates, a theater play and railway-desk dialogues.
  • The baseline of running the MSA tagger directly on TUN forms reaches 69.04%, with 26.9% OOV. Gold diacritized lemmas do even worse (67.41%) because TUN and MSA diacritize lemmas differently.
  • After conversion, tagging improves at each level of input:
InputAccuracyOOV
Forms82.5%13.5%
Predicted lemmas86.9%6.2%
Predicted lmms89.1%4.9%
  • Letting the tagger choose a path through the lattice (82.5%) beat committing early to a language model's best path (77.2%).
  • The authors examined the first 100 errors:
    • 34 came from conversion,
    • 49 came from disambiguation,
    • only 17 came from the tagger itself.

Interesting ideas

  • I1. Instead of translating a low-resource variety, map it into a crude machine-only approximation of a well-resourced variety, run the existing tool, and project the labels back to the original tokens. This reuses mature tools without retraining them.
  • I2. Tag at an abstracted level, using lemmas stripped of variety-specific marks, rather than at surface forms. This greatly shrinks the OOV rate when two lexicons differ mostly in spelling detail.
  • I3. Keep ambiguity as a lattice and let the downstream model resolve it, rather than trusting an upstream language model's single best output.
  • I4. Attribute a sample of errors to individual pipeline stages to see where the real losses are. Here the tagger itself was a minor error source.
  • I5. Always measure the untouched source tool on the target data as a baseline, so the conversion's own contribution is isolated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Nonstandard English (tweets, dialectal or archaic text) could be normalized into "pseudo-standard" input for the UD parsers. The arcs would then be projected back onto the original tokens, which keeps each step explainable.
  • I2 (tokens): The en tagger could fall back to a normalized lemma key, with spelling variants and archaic endings folded, when a surface form is unseen.
  • I3 (tree): The tokenizer and morphology could pass several candidate analyses to the parser, so that syntactic evidence makes the choice.
  • I4 (learn): Evaluation reports could routinely label the first N errors by stage (tokens, tree, frames, world). Training effort would then go to the stage that actually causes failures.
  • I5 (learn): Any dialect-conversion or repair rule could be judged against the raw-parser baseline, which acts as a natural negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States219 works · 103,487 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates124 works · 4,709 citations🎓Aix-Marseille UniversitéMarseille · France25 works · 39,241 citations🔬Laboratoire d’Informatique Fondamentale de MarseilleMarseille · France4 works · 358 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksPart-of-speech tagging · 126 works

Cite

Cite as Ahmed Hamdi, Alexis Nasr, Nizar Habash and Núria Gala. 2015. POS-tagging of Tunisian Dialect Using Standard Arabic Resources and Tools. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3207

@inproceedings{hamdi2015pos,
  title = {{POS-tagging of Tunisian Dialect Using Standard Arabic Resources and Tools}},
  author = {Ahmed Hamdi, Alexis Nasr, Nizar Habash and Núria Gala},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3207},
  url = {https://doi.org/10.18653/v1/w15-3207}
}

References

46 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rania Al-Sabbagh and Roxana Girju. 2010. Mining the web for the induction of a dialectical Arabic lexicon. In LREC.
  2. Rania Al-Sabbagh and Roxana Girju. 2012. A supervised pos tagger for written Arabic social networking corpora. In Proceedings of KONVENS, pages 39–52. find
  3. Mohamed Altantawy, Nizar Habash, Owen Rambow, and Ibrahim Saleh. 2010. Morphological Analysis and Generation of Arabic Nouns: A Morphemic Functional Approach. In Proceedings of the seventh International Conference on Language Resources and Evaluation (LREC), Valletta, Malta.
  4. Delphine Bernhard, Anne-Laure Ligozat, et al. 2013. Hassle-free pos-tagging for the alsatian dialects. Non-Standard Data Sources in Corpus Based-Research. find
  5. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic.
  6. Rahma Boujelbane, Siwar BenAyed, and Lamia Hadrich Belguith. 2013. Building bilingual lexicon to create dialect Tunisian corpora and adapt language model. ACL 2013, page 88. find
  7. Rahma Boujelbane, Mariem Mallek, Mariem Ellouze, and Lamia Hadrich Belguith. 2014. Fine-grained pos tagging of spoken Tunisian dialect corpora. In Natural Language Processing and Information Systems, pages 59–62. Springer. find
  8. David Chiang, Mona T Diab, Nizar Habash, Owen Rambow, and Safiullah Shareef. 2006. Parsing Arabic dialects. In EACL.
  9. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 600–609. Association for Computational Linguistics.
  10. E Dhouib. 2007. El makki w zakiyya. d’Ãl’dition manshuwrat manara, Tunis. find
  11. Maison Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. Ldc Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
  12. Kevin Duh and Katrin Kirchhoff. 2005. Pos tagging of dialectal Arabic: a minimally supervised approach. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 55– 62. Association for Computational Linguistics.
  13. Long Duong, Paul Cook, Steven Bird, and Pavel Pecina. 2013. Simpler unsupervised pos tagging with bilingual projections. In ACL (2), pages 634– 639. find
  14. Heba Elfardy and Mona T Diab. 2012. Simplified guidelines for the creation of large scale dialectal Arabic annotations. In LREC, pages 371–378.
  15. Heba Elfardy and Mona T Diab. 2013. Sentence level dialect identification in Arabic. In ACL (2), pages 456–461.
  16. Anna Feldman, Jirka Hana, and Chris Brew. 2006. A cross-language approach to rapid creation of new morpho-syntactically annotated resources. In Proceedings of LREC, pages 549–554. find
  17. Charles Albert Ferguson. 1959. Diglossia. WORD-JOURNAL OF THE INTERNATIONAL LINGUIS-TIC ASSOCIATION, 15(2):325–340. find
  18. David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  19. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 573–580. Association for Computational Linguistics.
  20. Nizar Habash and Owen Rambow. 2006. Magead: a morphological analyzer and generator for the Arabic dialects. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 681–688. Association for Computational Linguistics. ↗
  21. Nizar Habash and Ryan M Roth. 2009. Catib: The columbia Arabic treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224. Association for Computational Linguistics. ↗
  22. Nizar Habash, Owen Rambow, and George Kiraz. 2005. Morphological analysis and generation for Arabic dialects. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 17–24. Association for Computational Linguistics.
  23. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  24. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of Arabic dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world, pages 49–53. find
  25. Nizar Habash, Mona T Diab, and Owen Rambow. 2012. Conventional orthography for dialectal Arabic. In LREC, pages 711–718.
  26. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological analysis and disambiguation for dialectal Arabic. In HLT-NAACL, pages 426–432. Citeseer.
  27. Ahmed Hamdi, Rahma Boujelbane, Nizar Habash, and Alexis Nasr. 2013. The effects of factorizing root and pattern mapping in bidirectional Tunisian - standard Arabic machine translation. In MT Summit, Nice.
  28. Ahmed Hamdi, Núria Gala, and Alexis Nasr. 2014. Automatically building a Tunisian lexicon for deverbal nouns. COLING 2014, page 95. find
  29. S Harrat, K Meftouh, M Abbas, and K Smaili. 2014. Building resources for algerian Arabic dialects. Corpus (sentences), 4000(6415):2415. find
  30. George Anton Kiraz. 2000. Multitiered nonlinear morphology using multitape finite automata: a case study on Syriac and Arabic. Computational Linguistics, 26(1):77–105, March. ↗
  31. Shen Li, Joao V Graça, and Ben Taskar. 2012. Wiki-ly supervised part-of-speech tagging. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1389–1398. Association for Computational Linguistics. find
  32. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn Arabic treebank: Building a large-scale annotated Arabic corpus. In NEMLAR conference on Arabic language resources and tools, pages 102–109. find
  33. S. Mejri, S. Mosbah, and I. Sfar. 2009. Pluringuisme et diglossie en tunisie. Synergies Tunisie n 1, pages 53–74. find
  34. Emad Mohamed, Behrang Mohit, and Kemal Oflazer. 2012. Annotating and learning morphological segmentation of egyptian colloquial Arabic. In LREC, pages 873–877.
  35. Arfath Pasha, Mohamed Al-Badrashiny, Ahmed El Kholy, Ramy Eskander, Mona Diab, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In In Proceedings of the 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland.
  36. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 117–120. Association for Computational Linguistics.
  37. Hassan Sajjad, Kareem Darwish, and Yonatan Belinkov. 2013. Translating dialectal Arabic to english. In ACL (2), pages 1–6. find
  38. Wael Salloum and Nizar Habash. 2011. Dialectal to standard Arabic paraphrasing to improve Arabic-english statistical machine translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21. Association for Computational Linguistics. ↗
  39. Hassan Sawaf. 2010. Arabic dialect handling in hybrid machine translation. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas.
  40. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1–12. ↗
  41. Marianne Vergez-Couret. 2013. Tagging occitan using french and castillan tree tagger. In Proceedings of 6th Language & Technology Conference. find
  42. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the first international conference on Human language technology research, pages 1– 8. Association for Computational Linguistics.
  43. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  44. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59. Association for Computational Linguistics.
  45. Inès Zribi, Mariem Ellouze Khemakhem, and Lamia Hadrich Belguith. 2013. Morphological analysis of Tunisian dialect. In Proceeding of International Joint Conference on Natural Language Processing (IJCNLP 2013), Nagoya, Japan. ↗
  46. Inès Zribi, Rahma Boujelbane, Abir Masmoudi, Mariem Ellouze, Lamia Belguith, and Nizar Habash. 2014. A conventional orthography for Tunisian Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hamdi-2015-pos-tagging-tunisian-dialect-standard/