Elamur Research⌕
☰
home / papers / 2024 / SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers

SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers

Paper · 2024 · Ashraful Paran, Symom Shohan, Md. Hossain et al.

Summary

This shared-task paper addresses intent classification for Arabic customer queries in banking, where data and tools are scarce and dialectal variation is a challenge. The team entered Subtask-1 of AraFinNLP2024 and used the ArBanking77 corpus, an Arabic derivative of the English Banking77 with 77 intent labels covering Modern Standard Arabic (MSA) and Palestinian (PAL) queries. They compared nine classifiers. Three were classical models over TF-IDF or BoW features, three were CNN-based networks over Word2Vec or FastText embeddings, and three were fine-tuned transformers. One transformer was an English Banking77 model that received machine-translated input. Fine-tuned AraBERTv2 performed best, with a micro F1 of 82.08%, and the team placed 5th. Classical and neural baselines fell far behind, and the deep models trailed even the SVM. Error analysis attributes many failures to two causes: test-set dialects not seen during training, and intent labels with very similar meanings. The work shows that pretrained models matched to the language beat translation pipelines and shallow features on dialect-heavy intent detection.

Key points

  • ArBanking77 data: 21,559 training texts (MSA and PAL in roughly equal shares), 2,464 development texts and 11,721 test texts, spread over 77 intents.
  • Transformer scores (F1): AraBERTv2 reached 82.08%, CamelBERT 73.00% and BERT-Banking77 68.56%.
  • BERT-Banking77 classified Arabic queries after zero-shot translation to English with Helsinki-NLP/opus-mt-ar-en and llama-3-8b-instruct.
  • Baseline scores (F1):
    • The best classical model was SVM with TF-IDF at 61.69%.
    • The best neural model was CNN with Word2Vec at 43.33%.
    • CNN+LSTM and CNN+BiLSTM scored lower than the plain CNN.
  • AraBERTv2 classified 9,621 of 11,721 test items correctly and missed 2,100.
    • The test set contained dialects absent from training.
    • The classical and neural models looked good on dev but dropped on test.

Interesting ideas

  • I1. An English-trained intent model fed with machine-translated Arabic did clearly worse than a native Arabic model, which puts a measurable price on routing meaning through a pivot language.
  • I2. The test set included dialects missing from training, which exposed a gap that the dev scores had hidden. This argues for keeping a held-out variety split in any evaluation.
  • I3. The error analysis traced failures to near-synonymous labels, such as two different top-up intents, rather than to random noise. This points to inspecting label confusions to find inventory design problems.
  • I4. The study ran a full ladder of baselines on the same data: shallow features, CNN models and pretrained transformers. This showed that deeper networks without pretraining can lose to a linear SVM.
  • I5. The corpus reuses the English Banking77 label set for Arabic queries, which keeps a shared intent inventory across languages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When testing translation through the meta-world, compare against a pivot-translation baseline so the gain from going through meaning is quantified.
  • I2 (tree): Score the parsers on a treebank genre or dialect left out of training, alongside the in-domain dev set, so overfitting to one style shows up early.
  • I3 (frames): Check which speech-act or frame labels prag confuses most often, and merge or sharpen pairs that differ only in fine detail.
  • I4 (learn): Include bag-of-words and TF-IDF classifiers as negative controls in evaluations, so any structured, explainable method must demonstrably beat them.
  • I5 (answer): Align Mova's question or intent categories with Banking77 so that banking-style QA tests can be imported and compared directly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chittagong University of Engineering & TechnologyChittagong · Bangladesh25 works · 29 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ashraful Paran, Symom Shohan, Md. Hossain, Jawad Hossain, Shawly Ahsan and Mohammed Moshiul Hoque. 2024. SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.38

@inproceedings{paran2024semanticcuetsync,
  title = {{SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers}},
  author = {Ashraful Paran, Symom Shohan, Md. Hossain, Jawad Hossain, Shawly Ahsan and Mohammed Moshiul Hoque},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.38},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.38}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wassen Aldjanabi, Abdelghani Dahou, Mohammed AA Al-qaness, Mohamed Abd Elaziz, Ahmed Mohamed Helmi, and Robertas Damaševičius. 2021. Arabic offensive and hate speech detection using a cross-corpora multi-task learning model. In Informatics, volume 8, page 69. MDPI. ↗
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  3. Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. 1992. A training algorithm for optimal mar-Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023b. Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics. Saurabh Kumar, Suman Deep, and Pourush Kalra. Enhancing customer service in banking with ai: Intent classification using distilbert.
  4. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, and Ismail Berrada. 2024. AraFinNlp 2024: The first arabic financial nlp shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  5. Ridha Mezzi, Aymen Yahyaoui, Mohamed Wassim Krir, Wadii Boulila, and Anis Koubaa. 2022. Mental health intent recognition for arabic-speaking patients using the mini international neuropsychiatric interview (mini) and bert model. Sensors, 22(3):846. ↗
  6. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  7. Sana Shams, Bareera Sadia, and Muhammad Aslam. 2022. Intent detection in urdu queries using fine-tuned bert models. In 2022 16th International Conference on Open Source Systems and Technologies (ICOSST), pages 1–6. IEEE. find
  8. Tokunaga Takenobu. 1994. Text categorization based on weighted inverse document frequency. Information Processing Society of Japan, SIGNL, 94(100):33–40. find
  9. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  10. Lihua Wang, Huiting Yang, Feng Li, Wenzhong Yang, and Zhenwan Zou. 2022. Intent detection model based on dual-channel feature fusion. In 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), volume 6, pages 1862– 1867. IEEE. find
  11. Pei Wang, Keqing He, Yejie Wang, Xiaoshuai Song, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, and Weiran Xu. 2024. Beyond the known: Investigating llms performance on out-of-domain intent detection. arXiv preprint arXiv:2402.17256. ↗
  12. Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth. 2020. Self-attention networks for intent detection. arXiv preprint arXiv:2006.15585. ↗
  13. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020a. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867. ↗
  14. Jianguo Zhang, Kazuma Hashimoto, Yao Wan, Zhiwei Liu, Ye Liu, Caiming Xiong, and Philip S Yu. 2021. Are pretrained transformers robust in intent classification? a missing ingredient in evaluation of out-of-scope intent detection. arXiv preprint arXiv:2106.04564. ↗
  15. Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020b. Intent detection with wikihow. arXiv preprint arXiv:2009.05781. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paran-2024-semanticcuetsync-arafinnlp2024-classification-cross-dialect/