Elamur Research⌕
☰
home / papers / 2023 / An Empirical Analysis of Task Relations in the Multi-Task Annotation of an Arabizi Corpus

An Empirical Analysis of Task Relations in the Multi-Task Annotation of an Arabizi Corpus

Paper · 2023 · Elisa Gugliotta and Marco Dinarelli

Summary

Arabic dialects lack standard spelling and detailed grammatical descriptions, so annotated corpora for them must usually be built from scratch. The authors look for the most effective way to semi-automatically annotate such a corpus. They reuse two existing resources: the Tunisian Arabish Corpus (TArC), which is written in Arabizi (an informal Latin-letter spelling of Tunisian Arabic), and a multi-task encoder-decoder they released earlier. That network has one LSTM decoder per annotation level, and the decoders form a cascade. Each decoder attends to the encoder and to every earlier decoder, and the whole model is trained end-to-end on the sum of the task losses.

Five annotation levels are involved. Through many single-task and multi-task runs, the authors vary which tasks are combined, which level serves as input, and the order of the tasks.

The main findings are:

  • POS-tagging is the hardest task and harms the other tasks.
  • Lemmatisation is easy, benefits from tokenization and POS, and limits that harm when placed between them.
  • Task difficulty, more than linguistic relatedness, appears to decide the best order.

These findings give practical guidance for ordering multi-level annotation pipelines in low-resource settings.

Key points

  • Data: TArC, the Tunisian Arabish Corpus of spontaneous Arabizi text, annotated on five levels:
    • word-level code-switching classification (Arabizi vs. foreign words),
    • normalization into CODA*,
    • tokenization,
    • POS-tagging,
    • lemmatisation.
  • Model: the authors' released multi-task system can train LSTM or Transformer models. They chose LSTM because earlier TArC experiments showed a large gap in its favour on these partly structured outputs.
  • Evidence of negative transfer: adding POS-tagging degraded other tasks, for example tokenization in one multi-task configuration.
  • With CODA* input, POS scored best when paired with lemmatisation rather than tokenization. In that setup, lemmatisation accuracy rose by about 1 point over its single-task result.
  • Recommended order: tokenization, then lemmatisation, then POS last. The authors also observed a general tendency for results to improve when tasks needing more model capacity come early in the cascade. Ordering tasks by linguistic intuition alone in single-task setups proved unreliable.

Interesting ideas

  • I1. An easy intermediate task, here lemmatisation, can serve as a buffer that absorbs the damage a hard task (POS) does to earlier tasks in a shared model.
  • I2. In a cascade, every level attends to the encoder and to all previous levels, so each prediction sees the full stack of earlier analyses, and one summed loss trains everything together.
  • I3. Measure each task in isolation and in every multi-task combination and order, so that negative transfer becomes visible instead of being assumed away.
  • I4. Order tasks by how hard they are for the model, not only by linguistic intuition about which level feeds which.
  • I5. Classify words as native or foreign before deeper annotation, so that later levels are not forced to analyse borrowed tokens.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Try an explicit lemmatizer stage between tokenizer and tagger in crate en. Check whether the tagger's errors stop leaking back into the tokenizer and morphology when they are trained jointly.
  • I2 (tree): gparse could stack tagging and morphology heads under the dependency head, with attention to each earlier head. This keeps intermediate layers inspectable while letting the parser use them.
  • I3 (learn): Treat the single-task model as a mandatory control whenever an auxiliary task or witness signal is added. Accept the addition only if it beats that baseline.
  • I4 (learn): When designing training curricula or multi-head schedules, rank stages by measured difficulty and test early placement of the hard ones.
  • I5 (tokens): A per-token flag for non-English words, such as names, quotes or code-switches, could stop the tagger and lemmatizer from producing spurious English analyses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Grenoble AlpesSaint-Martin-d'Hères · France20 works · 8,059 citations🔬Laboratoire d'Informatique de GrenobleGrenoble · France21 works · 688 citations🔬Laboratoire de Linguistique et Didactique des Langues Etrangères et MaternellesGrenoble · France7 works · 113 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHuman-Computer Interaction · 141 works

Cite

Cite as Elisa Gugliotta and Marco Dinarelli. 2023. An Empirical Analysis of Task Relations in the Multi-Task Annotation of an Arabizi Corpus. Proceedings of the 4th Conference on Language, Data and Knowledge.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gugliotta2023empirical,
  title = {{An Empirical Analysis of Task Relations in the Multi-Task Annotation of an Arabizi Corpus}},
  author = {Elisa Gugliotta and Marco Dinarelli},
  booktitle = {Proceedings of the 4th Conference on Language, Data and Knowledge},
  year = {2023},
  url = {https://aclanthology.org/2023.ldk-1.14/}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elisa Gugliotta and Marco Dinarelli. 2020. TArC: Incrementally and Semi-Automatically Collecting a Tunisian Arabish Corpus. In Proceedings of the Twelfth LREC, pages 6279–6286. ↗
  2. Arfan Ahmed et al. 2022. Free and accessible Arabic corpora: A scoping review. Computer Methods and Programs in Biomedicine Update, page 100049. find
  3. Elisa Gugliotta and Marco Dinarelli. 2022. TArC: Tunisian Arabish Corpus first complete release. In Proceedings of the Thirteenth International Conference on LREC.
  4. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. arXiv preprint arXiv:1707.09861. ↗
  5. Elisa Gugliotta, Marco Dinarelli, and Olivier Kraif. 2020. Multi-task sequence prediction for Tunisian Arabizi multi-level annotation. arXiv preprint arXiv:2011.05152. ↗
  6. Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. 2018. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270–287. find
  7. Nizar Habash et al. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Eleventh International Conference on LREC.
  8. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the ACL, 8:156–171. ↗
  9. Salima Harrat, Karima Meftouh, and Kamel Smaïli. 2018. Maghrebi Arabic dialect processing: an overview. Journal of International Science and General Applications, 1. find
  10. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. ↗
  11. Alexander M Rush, Roi Reichart, Michael Collins, and Amir Globerson. 2012. Improved parsing and pos tagging using inter-sentence consistency constraints. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pages 1434–1444. find
  12. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the ACL (Volume 2: Short Papers), pages 231–235.
  13. T Standley, AR Zamir, D Chen, L Guibas, J Malik, and S Savarese. 2019. Which tasks should be learned together in multi-task learning? arxiv e-prints. arXiv preprint arXiv:1905.07553. ↗
  14. Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. 2020. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33:8728–8740. ↗
  15. Benjamin Hary. 1996. The Importance of the Language Continuum in Arabic Multiglossia. Understanding Arabic: essays in contemporary Arabic linguistics in honor of El-Said Badawi, pages 69–90. find
  16. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. ↗
  17. Tu Vu et al. 2020. Exploring and predicting transferability across nlp tasks. arXiv preprint arXiv:2005.00770. ↗
  18. Lev Semenovich Vygotsky and Michael Cole. 1978. Mind in society: Development of higher psychological processes. Harvard university press. find
  19. Go Inoue, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Joint prediction of morphosyntactic categories for fine-grained Arabic part-of-speech tagging exploiting tag dictionary information. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 421–431.
  20. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the ACL (Vol 2: Short Papers), pages 740–745, Melbourne, Australia. ACL.
  21. James A Kole and Alice F Healy. 2007. Using prior knowledge to minimize interference when learning large amounts of information. Memory & Cognition, 35(1):124–137. ↗
  22. Joseph Worsham and Jugal Kalita. 2020. Multi-task learning for natural language processing in the 2020s: where are we going? Pattern Recognition Letters, 136:120–126. ↗
  23. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnika. 2018. A lexical distance study of Arabic dialects. Procedia computer science, 142:2–13. find
  24. Nasser Zalmout and Nizar Habash. 2019. Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling. arXiv preprint arXiv:1910.12702. ↗
  25. Alexander Magidow. 2021. The old and the new: Considerations in Arabic historical dialectology. Languages, 6(4):163. ↗
  26. Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. 2018. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712–3722. find
  27. Jean Piaget. 2003. Routledge. The psychology of intelligence. find
  28. Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. 2022. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. arXiv preprint arXiv:2204.03508. ↗
  29. Chatrine Qwaider, Stergios Chatzikyriakidis, and Simon Dobnik. 2019. Can modern standard Arabic approaches be used for Arabic dialects? sentiment analysis as a case study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 40–50.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gugliotta-2023-empirical-analysis-task-relations-multi/