Elamur Research⌕
☰
home / papers / 2022 / Improving POS Tagging for Arabic Dialects on Out-of-Domain Texts

Improving POS Tagging for Arabic Dialects on Out-of-Domain Texts

Paper · 2022 · Noor Abo Mokh, Daniel Dakota and Sandra Kübler

Summary

The authors want a POS tagger that stays reliable across four Arabic dialects (Gulf, Levantine, Egyptian, Maghrebi) when the test text comes from a different domain than the training text. Their larger goal is to compare dialect syntax on the parallel MADAR corpus, which has no linguistic annotation. They train the Bi-LSTM tagger of Darwish et al. on the tweet-based Darwish corpus. For evaluation, they build a small hand-corrected gold set from MADAR's travel-phrase sentences. They test three ways to close the gap:

  • giving the target dialect more weight in a joint model through upsampling,
  • cleaning up inconsistent tags across dialects with little effort,
  • adding fastText embeddings pretrained on a large dialectal Arabic corpus.

Harmonizing the annotations gives large gains on its own. After that cleanup, upsampling helps little and sometimes hurts. Pretrained embeddings then raise accuracy for every dialect. Overall accuracy improves by roughly 20 percentage points on average. This makes automatic tagging of MADAR practical as a first step toward cross-dialect syntactic analysis and a future dialect parser.

Key points

  • Training data: the Darwish corpus is Twitter text with 350 sentences per dialect, about 71k–84k training words per dialect. Its tagset has 18 MSA tags plus two dialect-specific tags, Prog_Part and Neg_Part.
  • Gold test set: 100 MADAR sentences per dialect, 2,846 words in total. Only one city translation of each original sentence was kept. The sentences were pre-tagged automatically and then corrected by two speakers per dialect. Agreement was 95% for EGY, 90% for LEV and GLF, and 85% for MAG, followed by an adjudication pass that used the Camel guidelines.
  • Annotation cleanup: making tags consistent raised accuracy over the baseline by 11.9 points (EGY) up to 22.8 points (GLF). For GLF, the cleaned data without upsampling reached 82.3%, beating the upsampled baseline at 72.8%. Adding upsampling on top of the cleaned data dropped GLF to 73.6%.
  • Pretrained embeddings: the authors used Erdmann et al.'s fastText vectors, trained on 1.1M–1.7M sentences per dialect. With these, accuracy went from 82.3 to 83.2 (GLF), 75.2 to 84.3 (LEV), 80.2 to 87.9 (EGY) and 73.8 to 78.9 (MAG). The authors note a caveat: MADAR was part of the embedding training data.
  • Remaining errors: numbers are still the hardest tag. In MAG they are mostly tagged NOUN, because number spelling varies across dialects. Noun/adjective ambiguity drags down adjective precision.

Interesting ideas

  • I1. When pooling several sub-corpora, a quick pass that makes the same word or function receive the same tag everywhere can matter more than extra data or a cleverer model.
  • I2. The benefit of rebalancing data (upsampling one sub-domain) depended on annotation quality and even reversed after cleanup, so data-mixing choices should be re-tested whenever the labels change.
  • I3. A gold set was built cheaply by letting an existing tagger pre-annotate, then having two people correct the output, measuring their agreement and adjudicating the differences.
  • I4. The test sample kept only one variant of each parallel source sentence, which forces lexical and syntactic diversity instead of near-duplicates.
  • I5. The authors flag that their test corpus appeared in the embedding training data, a reminder that pretrained resources can leak into the evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Before merging UD treebanks or converted dialects for the perceptron parser and gparse, audit high-frequency function words for inconsistent labels across sources and normalize them first.
  • I2 (learn): After each silver-data repair round, re-run the mixing and weighting ablations instead of keeping the old ratios fixed.
  • I3 (learn): Witness-parser output could seed small hand-checked gold sets, with reported agreement, for each new genre such as fables or word problems.
  • I4 (answer): When building QA or school-test evaluation sets from paraphrased or translated items, keep one variant per original so scores are not inflated by repetition.
  • I5 (learn): Track which corpora each pretrained resource or LLM teacher has seen, and add negative-control test sets known to be outside them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana UniversityBloomington · United States41 works · 3,663 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksPart-of-speech tagging · 126 works

Cite

Cite as Noor Abo Mokh, Daniel Dakota and Sandra Kübler. 2022. Improving POS Tagging for Arabic Dialects on Out-of-Domain Texts. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.22

@inproceedings{mokh2022improving,
  title = {{Improving POS Tagging for Arabic Dialects on Out-of-Domain Texts}},
  author = {Noor Abo Mokh, Daniel Dakota and Sandra Kübler},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.22},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.22}
}

References

44 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, CA.
  2. Faisal Al-Shargi, Aidan Kaplan, Ramy Eskander, Nizar Habash, and Owen Rambow. 2016. Morphologically annotated corpora and morphological analyzers for Moroccan and Sanaani Yemeni Arabic. In Proceedings of the 10th Language Resources and Evaluation Conference (LREC), Portorož, Slovenia.
  3. Abdullah I. Alharbi and Mark Lee. 2020. BhamNLP at SemEval-2020 task 12: An ensemble of different word embeddings and emotion transfer learning for Arabic offensive language identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1532–1538, Barcelona (online). International Committee for Computational Linguistics.
  4. Randah Alharbi, Walid Magdy, Kareem Darwish, Ahmed Abdelali, and Hamdy Mubarak. 2018. Part-of-speech tagging for Arabic Gulf dialect using Bi-LSTM. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan.
  5. Khalid Almeman and Mark Lee. 2013. Automatic building of Arabic multi dialect text corpora by bootstrapping dialect words. In 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6, Sharjah, UAE. find
  6. Mohammed Attia and Ali Elkahky. 2019. Segmentation for domain adaptation in Arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP), pages 119–129, Florence, Italy.
  7. Shabnam Behzad and Amir Zeldes. 2020. A cross-genre ensemble approach to robust Reddit part of speech tagging. In Proceedings of the 12th Web as Corpus Workshop, pages 50–56, Marseille, France. European Language Resources Association. ↗
  8. Bernd Bohnet, Ryan McDonald, Goncalo Simoes, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta-BiLSTM model over context sensitive token encodings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2642–2652, Melbourne, Australia. find
  9. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  10. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan. find
  11. Kareem Darwish, Mohammed Attia, Hamdy Mubarak, Younes Samih, Ahmed Abdelali, Lluís Màrquez, Mohamed Eldesouki, and Laura Kallmeyer. 2020. Effective multi-dialectal Arabic POS tagging. Natural Language Engineering, 26(6):677–690. ↗
  12. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pages 1070–1074, Portorož, Slovenia.
  13. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, and Mohamed Eldesouki. 2017. Arabic POS tagging: Don’t abandon feature engineering just yet. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 130–137, Valencia, Spain.
  14. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, Mohamed Eldesouki, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, and Laura Kallmeyer. 2018. Multi-dialect Arabic POS tagging: A CRF approach. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan.
  15. Mona Diab and Nizar Habash. 2007. Arabic dialect processing tutorial. In Proceedings of the Human Language Technology Conference of the NAACL, pages 5–6, New York, NY. find
  16. Kevin Duh and Katrin Kirchhoff. 2005. POS tagging of dialectal Arabic: A minimally supervised approach. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 55– 62, Ann Arbor, MI.
  17. Mohamed Eldesouki, Younes Samih, Ahmed Abdelali, Mohammed Attia, Hamdy Mubarak, Kareem Darwish, and Laura Kallmeyer. 2017. Arabic multi-dialect segmentation: bi-LSTM-CRF vs. SVM. arXiv preprint arXiv:1708.05891. ↗
  18. Ashraf Elnagar, Sane M Yagi, Ali Bou Nassif, Ismail Shahin, and Said A Salloum. 2021. Systematic literature review of dialectal Arabic: Identification and detection. IEEE Access, 9:31010–31042. ↗
  19. Alexander Erdmann, Salam Khalifa, Mai Oudah, Nizar Habash, and Houda Bouamor. 2019. A little linguistics goes a long way: Unsupervised segmentation with limited language specific guidance. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 113–124, Florence, Italy. find
  20. Alexander Erdmann, Nasser Zalmout, and Nizar Habash. 2018. Addressing noise in multidialectal word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 558–565, Melbourne, Australia. find
  21. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
  22. Nizar Habash, Houda Bouamor, Hazem Hajj, Walid Magdy, Wajdi Zaghouani, Fethi Bougares, Nadi Tomeh, Ibrahim Abu Farha, and Samia Touileb, editors. 2021. Proceedings of the Sixth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, Kyiv, Ukraine (Virtual).
  23. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological analysis and disambiguation for dialectal Arabic. In Proceedings of theConference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 426–432, Atlanta, GA.
  24. Benjamin Heinzerling and Michael Strube. 2019. Sequence tagging with contextual and non-contextual subword representations: A multilingual evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273– 291, Florence, Italy. find
  25. Tobias Horsmann and Torsten Zesch. 2015. Effectiveness of domain adaptation approaches for social media PoS tagging. In Proceeding of the Second Italian Conference on Computational Linguistics, pages 166– 170, Trento, Italy. find
  26. Mustafa Jarrar, Nizar Habash, Diyam Akra, and Nasser Zalmout. 2014. Building a corpus for Palestinian Arabic: A preliminary study. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 18–27, Doha, Qatar.
  27. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. A large scale corpus of Gulf Arabic. arXiv preprint arXiv:1609.02960. ↗
  28. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of Emirati Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
  29. Sandra Kübler and Eric Baucom. 2011. Fast domain adaptation for part of speech tagging for dialogues. In Proceedings of the International Conference on Recent Advances in NLP (RANLP), Hissar, Bulgaria. find
  30. Prathyusha Kuncham, Chandu Khyathi Raghavi, Kovida Nelakuditi, and Dipti Misra Sharma. 2014. Domain adaptation in morphological analysis. International Journal of Languages, Literature and Linguistics, 1(2). ↗
  31. Atreyee Mukherjee, Sandra Kübler, and Matthias Scheutz. 2017. Creating POS tagging and dependency parsing experts via topic modeling. In Proceedings of Fifteenth Conference of the European Chapter of the ACL (EACL), Valencia, Spain. find
  32. Melanie Neunerdt, Michael Reyer, and Rudolf Mathar. 2014. Efficient training data enrichment and unknown token handling for POS tagging of non-standardized texts. In Conference on Natural Language Processing (KONVENS), pages 186–192, Hildesheim, Germany. find
  33. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.
  34. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin, Germany. Association for Computational Linguistics. ↗
  35. Younes Samih, Mohammed Attia, Mohamed Eldesouki, Ahmed Abdelali, Hamdy Mubarak, Laura Kallmeyer, and Kareem Darwish. 2017a. A neural architecture for dialectal Arabic segmentation. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 46–54, Valencia, Spain.
  36. Younes Samih, Mohamed Eldesouki, Mohammed Attia, Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Laura Kallmeyer. 2017b. Learning from relatives: Unified dialectal Arabic segmentation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 432–441, Vancouver, Canada.
  37. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China.
  38. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. International Journal of Computational Linguistics & Chinese Language Processing: Special Issue on Invited Papers from ISCSLP 2006, 12(3):303–324. find
  39. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Automated concatenation of embeddings for structured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2643–2660, Online. ↗
  40. Michihiro Yasunaga, Jungo Kasai, and Dragomir Radev. 2018. Robust multilingual part-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 976–986, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  41. Omar Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: An annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland OR.
  42. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59, Montréal, Canada.
  43. Imed Zitouni, Muhammad Abdul-Mageed, Houda Bouamor, Fethi Bougares, Mahmoud El-Haj, Nadi Tomeh, and Wajdi Zaghouani, editors. 2020. Proceedings of the Fifth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, Barcelona, Spain (Online). find
  44. Inès Zribi, Mariem Ellouze, Lamia Hadrich Belguith, and Philippe Blache. 2017. Morphological disambiguation of Tunisian dialect. Journal of King Saud University - Computer and Information Sciences, 29(2):147–155. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mokh-2022-improving-pos-tagging-arabic-dialects/