Elamur Research⌕
☰
home / papers / 2024 / SMASH at AraFinNLP2024: Benchmarking Arabic BERT Models on the Intent Detection

SMASH at AraFinNLP2024: Benchmarking Arabic BERT Models on the Intent Detection

Paper · 2024 · Youssef Al Hariri and Ibrahim Abu Farha

Summary

The paper reports Team SMASH's entry in Subtask 1 (Multi-dialect Intent Detection) of the AraFinNLP 2024 shared task. The task is to assign one of 77 banking intents to Arabic customer queries from ArBanking77, an Arabic adaptation of Banking77. The authors fine-tuned nine pretrained Arabic and multilingual encoders, plus AraBART, using HuggingFace. They compared three setups:

  • plain 77-way classification;
  • a two-level scheme, where the model first picks one of 11 hand-built intent groups and then a group-specific model picks the exact intent;
  • a variant that adds dialect-to-MSA machine translations (AraT5) to the training data.

The two-level scheme helped almost every model. MARBERTv2 reached F1 0.9470 on the Palestinian validation set, but only 0.7866 macro F1 on the blind test, which placed 9th. The test set contains dialects (Moroccan, Saudi, Tunisian) that were absent from training. The study gives a side-by-side benchmark of Arabic BERT variants under one recipe. It also shows that translating inputs at test time hurts, while adding translations as extra training data can help.

Key points

  • Data: ArBanking77 has 39,315 annotated queries over 77 intents. Training covers only MSA and Palestinian (21,554 queries). The test split of 15,296 queries also includes Moroccan, Saudi and Tunisian.
  • Models compared: AraBART, AraBERTv0.2-base, AraBERTv2-base, mBERT, CAMeLBERT-DA, CAMeLBERT-Mix, MARBERT, MARBERTv2 and QARiB.
  • Direct fine-tuning results: QARiB (F1 0.9273) and MARBERT (0.9270) led on PAL val. Training ran for 7 epochs with AdamW, lr 2e-5 and batch 128.
  • Hierarchical setup:
    • 11 parent groups, of which Cards is the largest with 23 intents.
    • 12 fine-tuned models per backbone (1 group model plus 11 intent models).
    • MARBERTv2 rose by about 5 points to 0.9470.
    • AraBART collapsed to 0.6248.
  • Translation results:
    • Evaluating on translated validation queries lowered all models to roughly 0.78–0.81.
    • Training on original plus translated queries and testing on the originals gave MARBERT 0.9473 and lifted AraBART back to 0.9196.

Interesting ideas

  • I1. Group fine-grained labels by shared topic, then classify coarse-to-fine with a dedicated small model per group, so each decision faces fewer, more distinguishable options.
  • I2. Machine-translated copies work better as extra training material than as a preprocessing step at inference, because translation errors at inference propagate straight into the decision.
  • I3. The gap between a strong in-dialect validation score and the much weaker blind test score shows how misleading validation can be when the test contains unseen language varieties.
  • I4. Running many backbones under one fixed recipe revealed that the same architectural trick can help most models yet badly hurt one (AraBART).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Frame or speech-act selection could first choose a broad family (e.g., a VerbNet top class) and then disambiguate within it, which also yields an explainable two-step trace.
  • I2 (learn): Silver or paraphrased data can enlarge training sets, but evaluation should stay on untouched original text so that augmentation noise is not hidden.
  • I3 (learn): Evaluation suites should include held-out genres or registers that are absent from training, acting as a control against overestimating in-domain scores.
  • I4 (tree): When comparing encoders for gparse, any structural change (e.g., label factoring) should be tested across all backbones rather than assumed to transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Youssef Al Hariri and Ibrahim Abu Farha. 2024. SMASH at AraFinNLP2024: Benchmarking Arabic BERT Models on the Intent Detection. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.35

@inproceedings{hariri2024smash,
  title = {{SMASH at AraFinNLP2024: Benchmarking Arabic BERT Models on the Intent Detection}},
  author = {Youssef {Al Hariri} and Ibrahim Abu Farha},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.35},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.35}
}

References

26 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591, Portland, Oregon, USA. Association for Computational Linguistics.
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  4. Fathi Abid, Slah Bahloul, and Mourad Mroua. 2016. Financial development and economic growth in mena countries. Journal of Policy Modeling, 38(6):1099– 1117. ↗
  5. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  6. Ibrahim Abu Farha and Walid Magdy. 2021. A comparative study of effective approaches for arabic sentiment analysis. Information Processing & Management, 58(2):102438. ↗
  7. Abeer Abuzayed and Hend Al-Khalifa. 2021. Sarcasm and sentiment detection in Arabic tweets using BERT-based models and data augmentation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 312–317, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  8. Ali Saleh Alammary. 2022. Bert models for arabic text classification: a systematic review. Applied Sciences, 12(11):5720. ↗
  9. Meshrif Alruily. 2022. Arrasa: Channel optimization for deep learning-based arabic nlu chatbot framework. Electronics, 11(22). ↗
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  11. Abdallah M. Bashir, Abubakr Hassan, Benjamin Rosman, Daniel Duma, and Mohanad Ahmed. 2018. Implementation of a neural natural language understanding component for arabic dialogue systems. Procedia Computer Science, 142:222–229. Arabic Computational Linguistics. find
  12. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. ↗
  13. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J. Jansen. 2020. Improving Arabic text categorization using transformer training diversification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 226–236, Barcelona, Spain (Online). Association for Computational Linguistics.
  14. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021a. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. ↗
  15. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, Wassim El-Hajj, et al. 2021b. A panoramic survey of natural language processing in the arab world. Communications of the ACM, 64(4):72–81. ↗
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  17. Hani Elgabou and Dimitar Kazakov. 2017. Building dialectal Arabic corpora. In Proceedings of the Workshop Human-Informed Translation and Interpreting Technology, pages 52–57, Varna, Bulgaria. Association for Computational Linguistics, Shoumen, Bulgaria. find
  18. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University - Computer and Information Sciences, 33(5):497–507.
  19. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  20. Farah Ayman Suleiman Jarrar. 2021. The impact of stock market development on economic growth in jordan. Journal of Applied Economic Sciences (JAES), 16(71):57–73. ↗
  21. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics.
  22. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31–42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  24. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, Ismail Berrada, and Houda Bouamor. 2024. AraFinNlp 2024: The first arabic financial nlp shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  25. Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. A survey of joint intent detection and slot filling models in natural language understanding. ACM Comput. Surv., 55(8). ↗
  26. Nadhem Zmandar, Mo El-Haj, and Paul Rayson. 2023. FinAraT5: A text to text model for financial Arabic text understanding and generation. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 262–273, Vienna, Austria. NOVA CLUNL, Portugal.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-arafinnlp2024-benchmarking-arabic-bert/