Elamur Research⌕
☰
home / papers / 2024 / AlexuNLP24 at AraFinNLP2024: Multi-Dialect Arabic Intent Detection with Contrastive Learning in Banking Domain

AlexuNLP24 at AraFinNLP2024: Multi-Dialect Arabic Intent Detection with Contrastive Learning in Banking Domain

Paper · 2024 · Hossam Elkordi, Ahmed Sakr, Marwan Torki et al.

Summary

The paper tackles intent detection for Arabic banking queries when the evaluation data contains dialects absent from training. It is a submission to subtask 1 of the AraFinNLP2024 shared task. The provided ArBanking77 data, an Arabic rendering of Banking77 with 77 intents, includes only MSA and Palestinian in its train and validation splits, while the test set adds Moroccan, Tunisian and Saudi. To close this gap, the authors machine-translated the MSA training portion into the three missing dialects with NLLB.

Training uses MARBERT in two stages. First, its [CLS] sentence embedding is shaped with a supervised triplet loss, where examples of the same intent count as positives whatever their dialect. Then a small feed-forward head is trained with class-weighted cross-entropy.

Augmentation brought the largest gain. The final system combines eight variants (L2 vs cosine distance, cleaned vs raw text, MARBERTv1 vs v2) in an ensemble. It placed second with about 0.876 test micro-F1. The work suggests that synthetic dialect data plus contrastive training can partly make up for missing annotated dialect resources.

Key points

  • Data: ArBanking77, with about 10.7k MSA and 10.8k PAL training queries. NLLB added 9,694 training and 1,039 validation queries each for MOR, TUN and SAU. The test split has 11,721 samples.
  • Contrastive stage: MARBERTv1 encoder, triplet loss with margin 1, 200 epochs, learning rate 1e-5. The classifier head uses linear layers of sizes 256 and 77 with LayerNorm and GELU, trained for 20 epochs with class weights.
  • Contrastive pre-training matters most when the encoder is frozen. Validation macro-F1 with only the head trained rose from 0.8272 to 0.9090. With full fine-tuning it rose from 0.9263 to 0.9308.
  • Augmentation lifted per-dialect validation scores sharply: Tunisian from 0.66 to 0.90, Moroccan from 0.80 to 0.95. On test, micro-F1 went from 0.8036 to 0.8482.
  • Neither text cleaning (normalization, punctuation and diacritic removal) nor MARBERTv2 gave a consistent improvement. The full eight-model ensemble reached 0.9487 validation macro-F1 and 0.8763 test micro-F1 (the abstract reports 0.8762).

Interesting ideas

  • I1. The encoder is first trained so that queries with the same label sit close together and queries with different labels sit apart, regardless of dialect; only then is a classifier attached, which makes the representation itself robust to variety.
  • I2. Missing varieties in the training data are synthesized by translating the standard-language portion, and the effect is then measured separately for each variety rather than only on average.
  • I3. Instead of picking one best model, the authors check which variant wins on each class; the spread of winners across classes justifies an ensemble.
  • I4. Error analysis traced many mistakes to the translation step dropping or changing meaning (e.g., losing "debit" or "transfer"), so some apparent model errors were really label-noise errors.
  • I5. Class weights in the loss give rare intents the same influence on training as frequent ones, counteracting bias toward dominant labels.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse's encoder could be pre-shaped with triplets of tokens that share a UD relation across different treebanks or English varieties, so that dialect-specific surface forms map to the same arc decisions.
  • I2 (learn): silver data could be extended with automatically rewritten registers (informal, British/American, child-style text), with scores reported per register to expose which ones the parser still fails on.
  • I3 (learn): when witness parsers vote on silver trees, their votes could be weighted per dependency relation according to which parser is historically best on that relation.
  • I4 (learn): augmented or translated training items could be checked for meaning drift by comparing their frames with the source sentence's frames, and mismatching items discarded as a negative control.
  • I5 (tokens): the tagger and morphology models could weight rare tags or features so that infrequent categories are not systematically absorbed by common ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hossam Elkordi, Ahmed Sakr, Marwan Torki and Nagwa El-Makky. 2024. AlexuNLP24 at AraFinNLP2024: Multi-Dialect Arabic Intent Detection with Contrastive Learning in Banking Domain. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.37

@inproceedings{elkordi2024alexunlp24,
  title = {{AlexuNLP24 at AraFinNLP2024: Multi-Dialect Arabic Intent Detection with Contrastive Learning in Banking Domain}},
  author = {Hossam Elkordi, Ahmed Sakr, Marwan Torki and Nagwa El-Makky},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.37},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.37}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Waheed Ahmed Abro, Guilin Qi, Muhammad Aamir, and Zafar Ali. 2022. Joint intent detection and slot filling using weighted finite state transducer and bert. Applied Intelligence, 52(15):17356–17370. ↗
  3. Moath Al Ali, Bassel Zaity, Pavel Drobintsev, Hazem Wannous, Igor Chernoruckiy, and Andrey Filchenkov. 2020. Joint slot filling and intent detection in spoken language understanding by hybrid cnn-lstm model. In Proceedings of the 2020 1st International Conference on Control, Robotics and Intelligent System, pages 112–117. find
  4. Hala J. Alshahrani, Khaled Tarmissi, Hussain Alshahrani, Mohamed Ahmed Elfaki, Ayman Yafoz, Raed Alsini, Omar Alghushairy, and Manar Ahmed Hamza. 2022. Computational linguistics with deep-learning-based intent detection for natural language understanding. Applied Sciences, 12(17). ↗
  5. Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063. ↗
  6. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. ↗
  7. Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. 2016. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc, volume 1, page 3. find
  8. Naaima Boudad, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb. 2018. Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal, 9(4):2479–2490. ↗
  9. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. ↗
  10. Qinkai Chen. 2021. Stock movement prediction with financial news using contextualized embedding from bert. arXiv preprint arXiv:2107.08721. ↗
  11. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Rehab Duwairi and Ftoon Abushaqra. 2021. Syntactic-and morphology-based text augmentation framework for arabic sentiment analysis. PeerJ Computer Science, 7:e469. ↗
  14. Rashmi Gangadharaiah and Balakrishnan Narayanaswamy. 2019. Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 564–569, Minneapolis, Minnesota. Association for Computational Linguistics. find
  15. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  16. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University - Computer and Information Sciences, 33(5):497–507.
  17. Changai He, Sibao Chen, Shilei Huang, Jian Zhang, and Xiao Song. 2019. Using convolutional neural network with bert for intent determination. In 2019 International Conference on Asian Language Processing (IALP), pages 65–70. IEEE. find
  18. Abdelrahman H. Hefny, Georgios A. Dafoulas, and Manal A. Ismail. 2020. Intent classification for a management conversational assistant. In 2020 15th International Conference on Computer Engineering and Systems (ICCES), pages 1–6. find
  19. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv: Learning. find
  20. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics.
  21. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. ArXiv, abs/1609.01454. ↗
  22. Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2021. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pages 4513–4519. find
  23. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, and Ismail Berrada. 2024. AraFinNlp 2024: The first arabic financial nlp shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  24. Suman Ravuri and Andreas Stolcke. 2016. A comparative study of recurrent neural network models for lexical domain classification. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6075–6079. IEEE. find
  25. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. AlexU-AIC at Arabic hate speech 2022: Contrast to classify. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 200–208, Marseille, France. European Language Resources Association. ↗
  26. Rawan Tahssin, Youssef Kishk, and Marwan Torki. 2020. Identifying nuanced dialect for Arabic tweets with deep learning and reverse translation corpus extension system. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 288–294, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  27. Antonio Uva, Pierluigi Roberti, and Alessandro Moschitti. 2020. Dialog-based help desk through automated question answering and intent detection. find
  28. Computational Linguistics CLiC-it 2020, page 443. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  30. Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5259–5267, Florence, Italy. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elkordi-2024-alexunlp24-arafinnlp2024-multi-dialect-arabic/