Elamur Research⌕
☰
home / papers / 2024 / BabelBot at AraFinNLP2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling

BabelBot at AraFinNLP2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling

Paper · 2024 · Murhaf Fares and Samia Touileb

Summary

The authors took part in sub-task 1 of the AraFinNLP 2024 shared task. The task asks systems to assign banking customer queries to intent classes when the queries may be written in many Arabic varieties. The training resource, ArBanking77, covers only MSA and Palestinian (PAL). It was produced by machine-translating the English Banking77 set and then having native speakers correct the output.

The team audited this data and found several problems:

  • English typos that were carried into the Arabic translations
  • recurring mistranslations
  • labels inherited incorrectly from the English source
  • awkward renderings of newer concepts

They then pooled the MSA and PAL portions. They also used Command R+ to generate synthetic queries in Moroccan, Tunisian and Saudi, plus fresh MSA translations made directly from the English originals.

For classification, they took the encoder of a T5 model that had already been tuned to convert dialects into MSA and added a classification head. The final submission combined three such models by voting. It placed third of eleven teams with micro-F1 0.8709. The work shows that LLM-generated dialect data helps when gold dialect coverage is thin. It also shows that careful data inspection can expose leakage and noise.

Key points

  • ArBanking77 size: about 10.7k MSA and 10.8k PAL training queries, with roughly 1.2k development queries per variety.
  • Recurring mistranslation: the English word "charge" was rendered with the Arabic word for "accusation" in 17 queries.
  • Synthetic data from Command R+: 8,694 Moroccan, 8,067 Tunisian and 7,885 Saudi queries, plus 4,014 MSA queries re-translated from English. This gives 28,660 training and 2,804 development items. Two light filters were applied:
    • discarding outputs that match the source string
    • a length check to catch repetitive output
  • Base model: the encoder of AraT5-MSAizer (Fares 2024), which is a fine-tuned AraT5 v2, with a dense–dropout–output head on top.
    • Training on Synthetic+Joint gave the best development scores: 0.9424 on Joint-Dev and 0.8973 on Synthetic-Dev.
    • A model trained on MSA alone lost about 7 points on PAL-Dev.
  • Ensemble: majority vote over three models, with ties broken by the highest prediction score. It reached 0.8709 F1 on test, against 0.8773 and 0.8762 for the top two teams. Each single model reached about 0.86, so the authors question whether the extra compute is worthwhile.

Interesting ideas

  • I1. When several varieties are translated from the same source sentence, merging them can secretly put development items into training under a different surface form; splits should be made by source item, not by variety.
  • I2. Translation-derived datasets inherit the source corpus's typos and wrong labels, and a single lexical mistranslation can repeat across many items; an error audit grouped by error type catches these systematically.
  • I3. Very cheap sanity filters on generated data work well: drop items identical to their input and items whose length signals self-repetition.
  • I4. Reusing the encoder of a model already trained to map non-standard varieties onto the standard language gives a classifier a head start on dialect robustness.
  • I5. Reporting each ensemble member's individual score next to the vote result shows whether combining models truly pays off relative to its running cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data from parallel versions of a sentence (original, UD dialect-converted, paraphrased), keep all versions of one source on the same side of the train/eval split so evaluation numbers are honest.
  • I2 (learn): Audit LLM-teacher outputs by grouping errors per lexeme or construction, so one bad sense choice repeated across dozens of examples is found and fixed once.
  • I3 (learn): Apply similar no-op and degenerate-repetition checks before LLM-produced paraphrases or annotations enter silver training sets.
  • I4 (tree): gparse could initialise from an encoder previously trained to normalise informal or regional English toward standard English, which may help parsing of noisy input.
  • I5 (learn): When several witness parsers vote on silver trees, always log each witness's solo accuracy, and use confidence as the tie-breaker, to judge whether the extra witnesses earn their compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BergenBergen · Norway9 works · 583 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Murhaf Fares and Samia Touileb. 2024. BabelBot at AraFinNLP2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.40

@inproceedings{fares2024babelbot,
  title = {{BabelBot at AraFinNLP2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling}},
  author = {Murhaf Fares and Samia Touileb},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.40},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.40}
}

References

17 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient Intent Detection with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38– 45, Online. Association for Computational Linguistics. ↗
  3. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, Wassim El-Hajj, et al. 2021. A panoramic survey of natural language processing in the arab world. Communications of the ACM, 64(4):72–81. ↗
  4. Heejin Do, Yunsu Kim, and Gary Lee. 2024. Autoregressive Score Generation for Multi-trait Essay Scoring. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1659–1666, St. Julian’s, Malta. Association for Computational Linguistics. find
  5. Mohab Elkaref, Movina Moses, Shinnosuke Tanaka, James Barry, and Geeth Mel. 2023. NLPeople at NADI 2023 shared task: Arabic Dialect Identification with Augmented Context and Multi-Stage Tuning. In Proceedings of ArabicNLP 2023, pages 642–646, Singapore (Hybrid). Association for Computational Linguistics.
  6. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding Dataset Difficulty with V-Usable Information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988–6008. PMLR. find
  7. Murhaf Fares. 2024. AraT5-MSAizer: Translating Dialectal Arabic to MSA. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 124–129, Torino, Italia. ELRA and ICCL.
  8. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics.
  9. Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazifar, Di Jin, Yang Liu, and Dilek Hakkani-Tur. 2023. Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1463–1476, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  10. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, Ismail Berrada, and Houda Bouamor. 2024. AraFinNlp 2024: The First Arabic Financial NLP Shared Task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  11. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  12. Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874, Dublin, Ireland. Association for Computational Linguistics.
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. ↗
  14. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.
  15. Cecilia Ying and Stephen Thomas. 2022. Label errors in BANKING77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 139– 143, Dublin, Ireland. Association for Computational Linguistics. ↗
  16. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 2308–2313, New York, NY, USA. Association for Computing Machinery. find
  17. Nadhem Zmandar, Mahmoud El-Haj, and Paul Rayson. 2021. Multilingual Financial Word Embeddings for Arabic, English and French. In 2021 IEEE International Conference on Big Data (Big Data), pages 4584–4589. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fares-2024-babelbot-arafinnlp2024-fine-tuning-t5/