Elamur Research⌕
☰
home / papers / 2023 / ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic

ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic

Paper · 2023 · Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia et al.

Summary

Arabic has almost no labelled data for intent detection, which leaves Arabic chatbots behind those for other languages. The authors built ArBanking77 from the English Banking77 corpus (13,083 banking queries, 77 intents). They first ran every query through Google Translate. Student annotators then corrected the Modern Standard Arabic (MSA) output and rewrote each query in Palestinian dialect (PAL), sometimes adding extra paraphrases in either variety. A separate review pass checked semantic correctness and label consistency. The final corpus has 31,404 queries. An AraBERT model fine-tuned on this data reaches F1 0.9209 on MSA and 0.8995 on PAL. The authors also tested the model with smaller training sets and with noisy, misspelled inputs of the kind seen in live chat. Performance holds up fairly well, and adding noisy examples to training helps most when data is scarce. Weak zero-shot results led the authors to argue that both MSA and dialect need their own annotation. The dataset and models are publicly released.

Key points

  • The corpus has 15,537 MSA queries and 15,867 PAL queries, 31,404 in total, across 77 intents. Each English source query has at least one MSA and one PAL counterpart.
  • Twenty-six annotators, all 3rd- and 4th-year students, worked over several months. They revised 2,104 machine translations (about 16%), for example replacing a "shipping" sense of transfer with the banking term for a money transfer.
  • The AraBERT fine-tuned model scores F1 0.9209 on MSA and 0.8995 on PAL when trained on the combined data.
  • Training on only 20% of the data lowers scores by 4.51% on MSA and 6.32% on PAL.
  • At 20% data, the noise-augmented model beats the clean one on clean test sets (PAL 0.8421 vs 0.8363; MSA 0.8801 vs 0.8758). On noisy PAL tests, augmentation narrows the gap by roughly 5%.

Interesting ideas

  • I1. Each source item is stored as a fixed tuple: intent, the English original, revised MT, an optional second MSA version, a PAL version, and an optional second PAL version. This yields parallel multi-variety data with paraphrases built in.
  • I2. Writing and reviewing are organised in opposite ways. One intent's items are split among 2–5 writers to increase diversity, while one reviewer gets a whole intent plus closely related intents to keep labels consistent.
  • I3. Spelling errors were deliberately kept during review. The evaluation then crosses training-noise type against test-noise type, at 20/50/100% data sizes.
  • I4. The authors measured lexical overlap between MSA and PAL to check whether dialect writers were simply copying the MSA shown to them.
  • I5. Low-resource curves show that the damage from noise shrinks as training data grows, with the biggest loss between 20% and 50%.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): the same slot layout could hold several reference realisations per meaning graph, so generation and the compressor are scored against a set of acceptable outputs rather than a single string.
  • I2 (learn): when LLM teacher tiers produce silver frames, generation could be spread across teachers, while review is grouped by neighbouring VerbNet classes so boundary confusions become visible.
  • I3 (tokens): build noisy copies of the en test sets and report a train-noise × test-noise grid, to see where the tagger and lemmatizer break and whether adding noise to training is safe for clean text.
  • I4 (back): an overlap check between source and output could flag translations through the meta-world that copy surface words instead of re-expressing the meaning.
  • I5 (learn): plotting robustness against the size of silver data would show where extra training data matters most for the perceptron parser and where it no longer pays off.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birzeit UniversityBirzeit · Palestine38 works · 798 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden and Sana Ghanem. 2023. ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.22

@inproceedings{jarrar2023arbanking77,
  title = {{ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic}},
  author = {Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden and Sana Ghanem},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.22},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.22}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. find
  3. Pre-training Giusepppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor. ↗
  4. Samyadeep Basu, Amr Sharaf, Karine Ip Kiun Chong, Alex Fischer, Vishal Rohra, Michael Amoake, Hazem El-Hammamy, Ehi Nosakhare, Vijay Ramani, and Benjamin Han. 2022. Strategies to improve few-shot learning for intent classification and slot-filling. pages 17–25. Association for Computational Linguistics. find
  5. Chiva Olivia Bilah, Teguh Bharata Adji, and Noor Akhmad Setiawan. 2022. Intent detection on indonesian text using convolutional neural network. In 2022 IEEE International Conference on Cybernetics and Computational Intelligence (CyberneticsCom), pages 174–178. find
  6. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. ↗
  7. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190. ↗
  8. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  11. AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. Arsas: An arabic speech-act and sentiment corpus of tweets. OSACT, 3:20. find
  12. Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple arabic dialects. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4130–4138.
  13. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  14. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. find
  15. Mohammad Hijjawi, Zuhair Bandar, and Keeley Crockett. 2013. User’s utterance classification using machine learning for arabic conversational agents. In 2013 5th International Conference on Computer Science and Information Technology, pages 223–232. find
  16. Mohammad Hijjawi, Zuhair Bandar, Keeley Crockett, and David Mclean. 2014. Arabchat: An arabic conversational agent. In 2014 6th International Conference on Computer Science and Information Technology (CSIT), pages 227–237. find
  17. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  18. Mustafa Jarrar. 2021. The arabic ontology - an arabic wordnet with ontologically clean content. Applied Ontology Journal, 16(1):1–26. ↗
  19. Mustafa Jarrar, Nizar Habash, Diyam Akra, and Nasser Zalmout. 2014. Building a corpus for palestinian arabic: a preliminary study. In Proceedings of the EMNLP 2014, Workshop on Arabic Natural Language, pages 18–27. Association For Computational Linguistics.
  20. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: An annotated corpus for the palestinian arabic dialect. Journal Language Resources and Evaluation, 51(3):745– 775. ↗
  21. Mustafa Jarrar, Eman Karajah, Muhammad Khalifa, and Khaled Shaalan. 2021. Extracting synonyms from bilingual dictionaries. In Proceedings of the 11th International Global Wordnet Conference (GWC2021), pages 215–222. Global Wordnet Association. find
  22. Ahlam Fuad and Maha Al-Yahya. 2022. Recent developments in arabic conversational ai: A literature review. IEEE Access, 10:23842–23859. ↗
  23. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  24. Sana Ghanem, Mustafa Jarrar, Radi Jarrar, and Ibrahim Bounhas. 2023. A benchmark and scoring algorithm for enriching arabic synonyms. In Proceedings of the 12th International Global Wordnet Conference (GWC2023), pages 215–222. Global Wordnet Association.
  25. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammad Khalilia. 2023a. Salma: Arabic sense-annotated corpus and wsd benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  26. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023b. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  27. Alaa Joukhadar, Huda Saghergy, Leen Kweider, and Nada Ghneim. 2019. Arabic dialogue act recognition for textual chatbot systems. In Proceedings of The First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019-Short Papers, pages 43– 49. ↗
  28. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. A lexical distance study of arabic dialects. Procedia Computer Science, 142:2–13. Arabic Computational Linguistics. find
  29. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  30. Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Arabic fine-grained entity recognition. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  31. Ridha Mezzi, Aymen Yahyaoui, Mohamed Wassim Krir, Wadii Boulila, and Anis Koubaa. 2022. Mental health intent recognition for arabic-speaking patients using the mini international neuropsychiatric interview (mini) and bert model. Sensors, 22(3). ↗
  32. Eman Naser-Karajah, Nabil Arman, and Mustafa Jarrar. 2021. Current trends and approaches in synonyms extraction: Potential adaptation to arabic. In Proceedings of the 2021 International Conference on Information Technology (ICIT), pages 428–434, Amman, Jordan. IEEE.
  33. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian arabic dialects with morphological annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  34. Lorenz Nigst, Maxim Romanov, Sarah Bowen Savant, Masoumeh Seydi, and Peter Verkinderen. 2020. Openiti: a machine-readable corpus of islamicate texts. nd http://doi.org/10.5281/zenodo, 4075046. ↗
  35. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprache. find
  36. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic gigaword fifth edition ldc2011t11. Philadelphia: Linguistic Data Consortium. find
  37. Sana Shams and Muhammad Aslam. 2022. Improving user intent detection in urdu web queries with capsule net architectures. Applied Sciences, 12(22). ↗
  38. Sana Shams, Muhammad Aslam, and Ana Maria Martinez-Enriquez. 2019. Lexical intent recognition in urdu queries using deep neural networks. In Advances in Soft Computing, pages 39–50, Cham. Springer International Publishing. find
  39. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. ↗
  40. Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022. Knn-contrastive learning for out-of-domain intent classification. pages 5129–5141. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2023-arbanking77-intent-detection-neural-model/