Elamur Research⌕
☰
home / papers / 2020 / Multi-Task Sequence Prediction For Tunisian Arabizi Multi-Level Annotation

Multi-Task Sequence Prediction For Tunisian Arabizi Multi-Level Annotation

Paper · 2020 · Elisa Gugliotta, Marco Dinarelli and Olivier Kraif

Summary

The paper tackles the problem of annotating Tunisian Arabish, which is Tunisian dialect written in Latin letters and digits on the web. Labelled data and tools for this variety are scarce. The authors build a recurrent multi-task sequence predictor for Fairseq. One encoder reads the Arabish input, and a chain of decoders produces the annotation levels one after another. The levels are word classification (arabizi / foreign / emotag), conversion to Arabic script in CODA*, clitic tokenization and PoS tagging. Each decoder can draw on the outputs of the decoders before it. The architecture is first checked on the TIGER German corpus, recast as a multi-level task, and it performs on par with published systems. The model is then used to pre-annotate the Tunisian Arabish Corpus (TArC) block by block, with a linguist correcting each block before retraining. On TArC it reaches above 97% on classification, about 91% on tokenization and up to 76.38% on PoS. This gives an open, multi-level dialect resource and a reusable tool for bootstrapping annotation in low-resource settings.

Key points

  • TArC has four annotation levels:
    • word class (arabizi, foreign, emotag);
    • CODA* Arabic-script orthography;
    • D3 BWFORM tokenization, in which clitics and the article are split off;
    • PoS tags following PATB guidelines.
  • Annotation was bootstrapped with the MADAR corpus, which is well-formed Arabic script. TArC blocks were then added step by step:
    • at step 3, tokenization reached 91.47%;
    • the best classification score on TArC was 97.21%;
    • the best PoS accuracy was 76.38%.
  • Moving from MADAR text to spontaneous CMC text caused clear accuracy drops. Adding the extra Arabish→CODA* task at step 4 lowered the results again.
  • "Smart-init" improved every task except classification at steps 4 and 5. It means initialising the parameters from the step-0 MADAR model, even though that model has a different architecture.
  • The code and data are public (a GitLab repository for the system, GitHub for TArC). The system is designed so it can be reused for other sequence prediction problems.

Interesting ideas

  • I1. The tasks are chained as a cascade of decoders over a shared encoder. Each later decoder sees the encoder states plus the hidden states of earlier decoders, and the losses are summed, so lower levels can guide the disambiguation of higher ones.
  • I2. Rich composite PoS tags are broken into component pieces, predicted, and reassembled afterwards to reduce sparsity, while accuracy is still scored on the full rebuilt tag.
  • I3. The corpus is annotated in growing blocks: the model pre-labels a block, a human corrects it, the model retrains, and a fresh validation split is drawn each round because the blocks come from different sources.
  • I4. A model trained on clean, well-formed data is used to initialise a model with a different architecture for noisy data. This recovered accuracy lost when moving to informal text.
  • I5. Before applying the architecture to the low-resource target, it is validated on a well-studied corpus (TIGER) recast as a multi-level task. This separates "does the architecture work" from "is the data hard".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The crate en tagger could predict UPOS first and feed its states into the FEATS and lemma predictors, so that earlier decisions explicitly condition later ones.
  • I2 (tokens): UD FEATS bundles could be predicted feature-by-feature and recombined, which keeps rare combinations learnable and lets each feature error be traced.
  • I3 (learn): Silver data could be grown in rounds with a newly sampled dev set per source, so that a stale dev set does not hide drift between genres.
  • I4 (tree): gparse could be warm-started from a parser trained on a clean UD treebank before fine-tuning on web or dialogue text, even if the heads or label sets differ.
  • I5 (learn): New multi-stage components could first be tested on a standard English treebank converted into the same task chain, which works as a control before evaluating on difficult in-house data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy60 works · 10,105 citations🎓Université Grenoble AlpesSaint-Martin-d'Hères · France24 works · 8,259 citations🔬Laboratoire d'Informatique de GrenobleGrenoble · France24 works · 689 citations🔬Laboratoire de Linguistique et Didactique des Langues Etrangères et MaternellesGrenoble · France7 works · 113 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Elisa Gugliotta, Marco Dinarelli and Olivier Kraif. 2020. Multi-Task Sequence Prediction For Tunisian Arabizi Multi-Level Annotation. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gugliotta2020multi,
  title = {{Multi-Task Sequence Prediction For Tunisian Arabizi Multi-Level Annotation}},
  author = {Elisa Gugliotta, Marco Dinarelli and Olivier Kraif},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.16/}
}

References

65 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. This resulted in a quite task-specific parameter initialisation. ↗
  2. Nasser Berjaoui. 2001. Aspects of the Moroccan Arabic orthography with preliminary insights from the Moroccan computer-mediated communication. na. find
  3. Ann Bies, Zhiyi Song, Mohamed Maamouri, Stephen Grimes, Haejoong Lee, Jonathan Wright, Stephanie Strassel, Nizar Habash, Ramy Eskander, and Owen Rambow. 2014. Transliteration of arabizi into arabic orthography: Developing a parallel annotated arabizi-arabic script sms/chat corpus. In Proceedings of the EMNLP 2014 workshop on Arabic natural language processing (ANLP), pages 93–103.
  4. Charles Boberg, John A Nerbonne, and Dominic James Landon Watt. 2018. The handbook of dialectology. Wiley Online Library. find
  5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  6. Rahma Boujelbane, Mariem Ellouze Khemekhem, Siwar BenAyed, and Lamia Hadrich Belguith. 2013. Building bilingual lexicon to create dialect tunisian corpora and adapt language model. In Proceedings of the Second Workshop on Hybrid Approaches to Translation, pages 88–93. find
  7. Rahma Boujelbane, Mariem Mallek, Mariem Ellouze, and Lamia Hadrich Belguith. 2014. Fine-grained pos tagging of spoken tunisian dialect corpora. In International Conference on Applications of Natural Language to Data Bases/Information Systems, pages 59–62. Springer. find
  8. Sabine Brants, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther Konig, Wolfgang Lezius, Christian Rohrer, George Smith, and Hans Uszkoreit. 2004. TIGER: Linguistic interpretation of a german corpus. Research on Language and Computation, 2(4):597–620, dec. ↗
  9. Jorina Brysbaert and Karen Lahousse. 2019. Computer-mediated versus non-computer-mediated corpora of informal french: Differences in politeness and intensification in the expression of contrast by au contraire. Social Media Corpora for the Humanities (CMC-Corpora2019), page 48. find
  10. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer (bama) version 2.0. Linguistic Data Consortium (LDC), University of Pennsylvania, Philadelphia, PA, USA. find
  11. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078. ↗
  12. Ronan Collobert and Jason Weston. 2007. Fast Semantic Extraction Using a Novel Neural Network Architecture. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 560–567, Prague, Czech Republic, June. Association for Computational Linguistics. find
  13. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 160–167, New York, NY, USA. ACM. find
  14. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12, November.
  15. Kareem Darwish. 2014. Arabizi detection and conversion to arabic. In In the Arabic Natural Language Processing Workshop, EMNLP. ↗
  16. Chiang David, Mona Diab, Nizar Habash, Owen Rambow, and Safiullah Shareef. 2006. Parsing arabic dialects. In Proceedings of EACL, pages 369–376. ↗
  17. Marco Dinarelli and Loı̈c Grobol. 2019. Hybrid neural models for sequence modelling: The best of three worlds. CoRR. arXiv preprint 1909.07102. ↗
  18. Kevin Duh and Katrin Kirchhoff. 2005. Pos tagging of dialectal arabic: a minimally supervised approach. In Proceedings of the acl workshop on computational approaches to semitic languages, pages 55–62.
  19. Rehab M Duwairi, Mosab Alfaqeh, Mohammad Wardat, and Areen Alrabadi. 2016. Sentiment analysis for arabizi text. In 2016 7th International Conference on Information and Communication Systems (ICICS), pages 127– 132. IEEE. find
  20. Helmut Feldweg, Ralf Kibiger, and Christine Thielen. 1995. Zum sprachgebrauch in deutschen newsgruppen. Osnabrücker Beiträge zur Sprachtheorie, 50:143–154. find
  21. Chayma Fourati, Abir Messaoudi, and Hatem Haddad. 2020. Tunizi: a tunisian arabizi sentiment analysis dataset. arXiv preprint arXiv:2004.14303. ↗
  22. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. of ICML. ↗
  23. Elisa Gugliotta and Marco Dinarelli. 2020. Tarc: Incrementally and semi-automatically collecting a tunisian arabish corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6279–6286. ↗
  24. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, et al. 2018. Unified guidelines and resources for arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  25. Nizar Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  26. Ahmed Hamdi, Alexis Nasr, Nizar Habash, and Núria Gala. 2015. Pos-tagging of tunisian dialect using standard arabic resources and tools. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 59–68. Association for Computational Linguistics (ACL), July.
  27. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780, November. ↗
  28. Go Inoue, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Joint prediction of morphosyntactic categories for fine-grained arabic part-of-speech tagging exploiting tag dictionary information. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 421–431.
  29. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the palestinian arabic dialect. Language Resources and Evaluation, 51(3):745–775. ↗
  30. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. ↗
  31. William Labov. 1972. Sociolinguistic Patterns. Conduct and Communication. University of Pennsylvania Press, Incorporated. find
  32. Thomas Lavergne and François Yvon. 2017. Learning the structure of variable-order crfs: a finite-state perspective. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 433–439. Association for Computational Linguistics. find
  33. Cédric Lopez, Sarah Zenasni, Eric Kergosien, Ioannis Partalas, Mathieu Roche, Maguelonne Teisseire, and Rachel Panckhurst. 2018. Extracting absolute spatial entities from sms: comparing a supervised and an unsupervised approach. Language and the new (instant) media. find
  34. Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In International Conference on Learning Representations. ↗
  35. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466–467. Cairo. find
  36. Mohamed Maamouri, Tim Buckwalter, Dave Graff, and Hubert Jin. 2007. Fisher levantine arabic conversational telephone speech. Linguistic Data Consortium, University of Pennsylvania, LDC Catalog No.: LDC2007S02. find
  37. Mohamed Maamouri, Ann Bies, Sondos Krouna, Fatma Gaddeche, and Basma Bouziri. 2009. Penn arabic treebank guidelines. Linguistic Data Consortium. find
  38. Abir Masmoudi, Nizar Habash, Mariem Ellouze, Yannick Estève, and Lamia Hadrich Belguith. 2015. Arabic transliteration of romanized tunisian dialect text: A preliminary investigation. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 608–619. Springer. find
  39. Abir Masmoudi, Mariem Ellouze Khmekhem, Mourad Khrouf, and Lamia Hadrich Belguith. 2019. Transliteration of arabizi into arabic script for tunisian dialect. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(2):1–21. ↗
  40. Deepthi Mave, Suraj Maharjan, and Thamar Solorio. 2018. Language identification and analysis of code-switched social media text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 51–61. find
  41. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium, October-November. Association for Computational Linguistics. ↗
  42. Etienne Morel and Simona Pekarek Doehler. 2013. Les ‘textos’ plurilingues: l’alternance codique comme ressource d’affiliation à une communauté globalisée. Revue française de linguistique appliquée, 18(2):29–43. ↗
  43. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. ↗
  44. Rachel Panckhurst. 2017. Entre linguistique et informatique. Des outils de traitement automatique du langage naturel écrit (TALNE) à l’analyse du discours numérique médié (DNM). Ph.D. thesis, Université Paris-Est, Paris, France. find
  45. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. Understanding the exploding gradient problem. CoRR, abs/1211.5063. ↗
  46. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland, May. European Language Resources Association (ELRA).
  47. Monica A Riordan and Roger J Kreuz. 2010. Emotion encoding and interpretation in computer-mediated communication: Reasons for use. Computers in human behavior, 26(6):1667–1673. ↗
  48. Oren Soffer. 2010. “silent orality”: toward a conceptualization of the digital oral features in cmc and sms texts. Communication Theory, 20(4):387–404. find
  49. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958.
  50. Natalie Sullivan. 2017. Writing Arabizi: Orthographic Variation in Romanized Lebanese Arabic on Twitter. Ph.D. thesis, The University of Texas at Austin, USA. find
  51. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS, Cambridge, MA, USA. MIT Press. ↗
  52. Benedikt Szmrecsanyi and Lieselotte Anderwald. 2018. Corpus-Based Approaches to Dialect Study. In The Handbook of Dialectology, pages 300–313. John Wiley & Sons, Ltd. Section: 17 eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118827628.ch17. ↗
  53. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. find
  54. Yasmin Tantawi and Mary Beth Rosson. 2019. The paralinguistic function of emojis in twitter communication. Social Media Corpora for the Humanities (CMC-Corpora2019), page 68. ↗
  55. Sunisa Todla. 1999. Patterns of communicative behaviour in internet chatrooms. Unpublished master’s thesis, Chulalongkorn University. find
  56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems.
  57. Fangzhao Wu and Yongfeng Huang. 2015. Collaborative multi-domain sentiment classification. In 2015 IEEE International Conference on Data Mining, pages 459–468. find
  58. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations. ↗
  59. Martin Wynne. 2005. Developing linguistic corpora: A guide to good practice. Oxbow Books Limited. find
  60. Simeon J. Yates. 1996. Oral and written linguistic aspects of computer conferencing. Pragmatics and beyond New Series, pages 29–46. ↗
  61. Jihen Younes, Hadhemi Achour, and Emna Souissi. 2015. Constructing linguistic resources for the tunisian dialect using textual user-generated contents on the social web. In International Conference on Web Engineering, pages 3–14. Springer. find
  62. Jihene Younes, Hadhemi Achour, Emna Souissi, and Ahmed Ferchichi. 2020. Romanized tunisian dialect transliteration using sequence labelling techniques. Journal of King Saud University-Computer and Information Sciences. ↗
  63. Nasser Zalmout and Nizar Habash. 2019. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. arXiv preprint arXiv:1910.02267. ↗
  64. Inès Zribi, Mariem Ellouze Khemekhem, and Lamia Hadrich Belguith. 2013. Morphological analysis of tunisian dialect. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 992–996. ↗
  65. Inès Zribi, Mariem Ellouze, Lamia Hadrich Belguith, and Philippe Blache. 2017. Morphological disambiguation of tunisian dialect. Journal of king Saud University-computer and information sciences, 29(2):147–155. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gugliotta-2020-multi-task-sequence-prediction-tunisian/