Elamur Research⌕
☰
home / papers / 2024 / AraFinNLP 2024: The First Arabic Financial NLP Shared Task

AraFinNLP 2024: The First Arabic Financial NLP Shared Task

Paper · 2024 · Sanad Malaysha, Mo El-Haj, Saad Ezzini et al.

Summary

AraFinNLP 2024 is the first shared task on Arabic financial NLP. It targets banking customer queries, where most existing financial NLP work covers only English. The organizers proposed two subtasks:

  • Multi-dialect Intent Detection: classify a query into one or more of 77 banking intents, without being told which dialect it is written in.
  • Cross-dialect Translation and Intent Preservation: render MSA queries in Gulf (Saudi), Moroccan, Palestinian and Tunisian Arabic while keeping their intent.

The resource is an updated ArBanking77, which is derived from the English Banking77 and now holds about 39k parallel queries. Participants trained only on the MSA and Palestinian portion. The test data also contained Saudi, Moroccan and Tunisian queries that were never seen in training. External data and pretrained models were explicitly allowed because Arabic dialects are low-resource.

Of 45 registered teams, 11 submitted to Subtask 1 and only one to Subtask 2. Intent detection reached a micro F1 near 0.88, but dialect translation scored a BLEU of just 1.667. The task establishes a benchmark for dialect-robust banking chatbots and shows that generating dialects is still largely unsolved.

Key points

  • The updated ArBanking77 contains about 39k parallel queries in MSA plus four dialects, each labeled with at least one of 77 banking intents.
  • The training split held roughly 31k MSA and Palestinian queries. Moroccan, Tunisian and Saudi examples appeared only at test time.
  • Participation: 45 teams registered, 11 were active in Subtask 1, and 1 in Subtask 2.
  • Top Subtask 1 results:
    • The MA team won with F1 0.8773, using fine-tuned BERT-based models, contrastive loss and data augmentation.
    • AlexUNLP24 came second at 0.8762 with QARiB and MARBERTv2.
    • Other teams scored lower, for example SMASH at 0.7866.
  • The only Subtask 2 submission scored 1.667 BLEU, showing how hard MSA-to-dialect generation remains.

Interesting ideas

  • I1. Training on the standard variety plus one dialect and then testing on dialects never seen in training measures true variety generalization rather than memorization.
  • I2. Hiding the dialect label at test time forces systems to cope with mixed input without an oracle telling them which variety they face.
  • I3. AlexUNLP24 found that translating dialect queries into MSA before classification hurt accuracy, because normalization can erase signal.
  • I4. Translation quality was judged partly by whether the original intent survives, not only by surface overlap such as BLEU.
  • I5. The winning system used contrastive loss to pull same-intent paraphrases together across varieties, making class boundaries robust to surface variation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build negative-control evaluation splits from genres or registers (tweets, transcripts, archaic tales) that are absent from silver training data, to show whether the parser generalizes or overfits.
  • I2 (tree): Evaluate gparse and the perceptron parser on unlabeled mixed-register input, rather than routing by known source, to expose dialect-conversion blind spots.
  • I3 (tokens): Be wary of aggressively normalizing nonstandard English spellings before tagging. Compare tagging with and without normalization before adopting it.
  • I4 (back): Score the text → graph → text compressor and translation by re-parsing the output and checking that the same frames and speech act come back.
  • I5 (frames): Use contrastive training over paraphrases that share a frame or speech act, so classification depends on meaning rather than wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations🎓Lancaster UniversityLancaster · United Kingdom35 works · 1,615 citations🎓King Saud UniversityRiyadh · Saudi Arabia37 works · 527 citations🎓Université Mohammed VI PolytechniqueMarrakesh · Morocco18 works · 171 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammed Khalilia, Mustafa Jarrar, Sultan Almujaiwel, Ismail Berrada and Houda Bouamor. 2024. AraFinNLP 2024: The First Arabic Financial NLP Shared Task. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.34

@inproceedings{malaysha2024arafinnlp,
  title = {{AraFinNLP 2024: The First Arabic Financial NLP Shared Task}},
  author = {Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammed Khalilia, Mustafa Jarrar, Sultan Almujaiwel, Ismail Berrada and Houda Bouamor},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.34},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.34}
}

References

48 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics.
  2. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7088–7105. Association for Computational Linguistics. ↗
  3. Carla Abreu, Henrique Cardoso, and Eugénio Oliveira. 2019. FinDSE@FinTOC-2019 shared task. In Proceedings of the Second Financial Narrative Processing Workshop (FNP 2019), pages 69–73, Turku, Finland. Linköping University Electronic Press. find
  4. Hanna Abi Akl, Anubhav Gupta, and Dominique Mariko. 2019. FinTOC-2019 shared task: Finding title in text blocks. In Proceedings of the Second Financial Narrative Processing Workshop (FNP 2019), pages 58–62, Turku, Finland. Linköping University Electronic Press. find
  5. Moustafa Al-Hajj and Mustafa Jarrar. 2021. ArabGloss-BERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. IN-COMA Ltd. find
  6. Alaa Aljabari, Lina Duaibes, Mustafa Jarrar, and Mohammed Khalilia. 2024. Event-Arguments Extraction Corpus and Modeling using BERT for Arabic. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  7. Mohammed Alshahrani, Fuxi Zhu, Mohammed Al-ghaili, Eshrag Refaee, and Mervat Bamiah. 2018. Borsah: An arabic sentiment financial tweets corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  8. Yusuf Arslan, Kevin Allix, Lisa Veiber, Cedric Lothritz, Tegawendé F Bissyandé, Jacques Klein, and Anne Goujon. 2021. A comparison of pre-trained language models for multi-class text classification in the financial domain. In Companion Proceedings of the Web Conference 2021, pages 260–268. find
  9. Sylvio Barbon Junior, Paolo Ceravolo, Sven Groppe, Mustafa Jarrar, Samira Maghool, Florence Sèdes, Soror Sahri, and Maurice Van Keulen. 2024. Are Large Language Models the New Interface for Data Pipelines? In Proceedings of the International Workshop on Big Data in Emergent Distributed Environments, BiDEDE ’24, New York, NY, USA. Association for Computing Machinery. ↗
  10. Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alexandre Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Y. Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. 2023. Seamless: Multilingual expressive and streaming speech translation. CoRR, abs/2312.05187. ↗
  11. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. ↗
  12. Damir Cavar and Matthew Josefy. 2018. Mapping deep nlp to knowledge graphs: An enhanced approach to analyzing corporate filings with regulators. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  13. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A Panoramic survey of Natural Language Processing in the Arab Worlds. Commun. ACM, 64(4):72–81. find
  14. Mahmoud El-Haj, Paulo Alves, Paul Rayson, Martin Walker, and Steven Young. 2020. Retrieving, classifying and analysing narrative commentary in unstructured (glossy) annual reports published as pdf files. Accounting and Business Research, 50(1):6–34. ↗
  15. Mahmoud El-Haj, Paul Rayson, Paulo Alves, and Steven Young. 2018. Towards a multilingual financial narrative processing system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  16. Mahmoud El-Haj, Paul Rayson, Martin Walker, Steven Young, and Vasiliki Simaki. 2019. In search of meaning: Lessons, resources and next steps for computational analysis of financial discourse. Journal of Business Finance & Accounting, 46(3-4):265–306. find
  17. Mahmoud El-Haj, Paul Rayson, Steven Young, and Martin Walker. 2014. Detecting document structure in a very large corpus of uk financial reports. In LREC, volume 14, pages 1335–1338. find
  18. Mahmoud El-Haj, Antonio Moreno Sandoval, and José Antonio Jiménez Millán. 2021. Machine learning models for classifying spanish beaters and non-beaters financial reports. In Financial narrative processing in spanish, pages 179–198. Tirant Humanidades. find
  19. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + Baladi: Towards a Levantine Corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  20. Tymaa Hammouda, Mustafa Jarrar, and Mohammed Khalilia. 2024. SinaTools: Open Source Toolkit for Arabic Natural Language Understanding. In Proceedings of the 2024 AI in Computational Linguistics (ACLING 2024), Procedia Computer Science, Dubai. ELSEVIER. find
  21. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, WANLP 2021, Kyiv, Ukraine (Virtual), April 9, 2021, pages 92–104. Association for Computational Linguistics.
  22. Mustafa Jarrar. 2008. Towards Effectiveness and Transparency in e-Business Transactions, An Ontology for Customer Complaint Management, chapter 7. IGI Global. find
  23. Mustafa Jarrar. 2021. The Arabic Ontology - An Arabic Wordnet with Ontologically Clean Content. Applied Ontology Journal, 16(1):1–26. ↗
  24. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa’ Omar. 2023a. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 748– 758. ACL.
  25. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023b. ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 276–287. ACL.
  26. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023c. ArBanking77: Intent detection neural model and a new dataset in modern and dialectical Arabic. In Proceedings of ArabicNLP 2023, pages 276–287, Singapore (Hybrid). Association for Computational Linguistics.
  27. Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2024. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  28. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  29. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023d. SALMA: Arabic Sense-annotated Corpus and WSD Benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 359–369. ACL.
  30. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023e. Lisan: Yemeni, Irqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  31. Rasmus Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. 2023. Multifin: A dataset for multilingual financial nlp. In Findings of the Association for Computational Linguistics: EACL 2023, pages 894–909. find
  32. Rasmus Kær Jørgensen and Christian Igel. 2021. Machine learning for financial transaction classification across companies using character-level word embeddings of text fields. Intelligent Systems in Accounting, Finance and Management, 28(3):159–172. ↗
  33. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  34. Oi Yee Kwong. 2018. Analysis and annotation of english-chinese financial terms for benchmarking and language processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  35. Matthew Lamm, Arun Chaganty, Dan Jurafsky, Christopher D. Manning, and Percy Liang. 2018. Qsrl: A semantic role-labeling schema for quantitative facts. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  36. Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Arabic Fine-Grained Entity Recognition. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 310–323. ACL.
  37. Tim Loughran and Bill McDonald. 2016. Textual analysis in accounting and finance: A survey. Journal of Accounting Research, 54(4):1187–1230. find
  38. Sanad Malaysha, Mustafa Jarrar, and Mohammed Khalilia. 2023. Context-Gloss Augmentation for Improving Arabic Target Sense Verification. In Proceedings of the 12th International Global Wordnet Conference (GWC2023). Global Wordnet Association.
  39. Sanad Malaysha, Mustafa Jarrar, and Mohammed Khalilia. 2024. NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness. In In Proceedings of the SemEval 2024 Shared Task 1 (Semantic Relatedness). ACL.
  40. Abir Naskar, Tirthankar Dasgupta, Sudeshna Jana, and Lipika Dey. 2022. Atl at fincausal 2022: Transformer based architecture for automatic causal sentence detection and cause-effect extraction. In Proceedings of the 4th Financial Narrative Processing Workshop (FNP 2022), pages 131–134, Marseille, France. European Language Resources Association. of the 4th Financial Narrative Processing Workshop (FNP 2022), pages 27–35, Marseille, France. European Language Resources Association. find
  41. Elias Zavitsanos, Aris Kosmopoulos, George Giannakopoulos, Marina Litvak, Blanca Carbajo-Coronado, Antonio Moreno-Sandoval, and Mahmoud El-Haj. 2023. The financial narrative summarisation shared task (fns 2023). In Proceedings of the 2023 IEEE International Conference on Big Data (BigData), pages 2890–2896, Sorrento, Italy. find
  42. Nadhem Zmandar, Tobias Daudert, Sina Ahmadi, Mahmoud El-Haj, and Paul Rayson. 2022. Cofif plus: A french financial narrative summarisation corpus. In Proceedings of the 4th Financial Narrative Processing Workshop (FNP 2022), pages 1622–1639, Marseille, France. European Language Resources Association. find
  43. Nadhem Zmandar, Mahmoud El-Haj, and Paul Rayson. 2021. Multilingual financial word embeddings for arabic, english and french. In 2021 IEEE International Conference on Big Data (Big Data), pages 4584–4589. find
  44. Nadhem Zmandar, Mo El-Haj, and Paul Rayson. 2023. FinAraT5: A text to text model for financial Arabic text understanding and generation. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 262–273, Vienna, Austria. NOVA CLUNL, Portugal.
  45. Jan Štihec, Senja Pollak, and Martin Žnidaršič. 2021. Preliminary experimentation with combinations and extensions of forward-looking sentence detection wordlists. In Proceedings of the 3rd Financial Narrative Processing Workshop, pages 26–30, Lancaster, United Kingdom. Association for Computational Linguistics. find
  46. Martin Žnidaršič, Jasmina Smailović, Jan Gorše, Miha Grčar, Igor Mozetič, and Senja Pollak. 2018. Trust and doubt terms in financial tweets and periodic reports. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  47. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian Arabic Dialects with Morphological Annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 12–23. ACL.
  48. Gakuto Tsutsumi and Takehito Utsuro. 2022. Detecting causes of stock price rise and decline by machine reading comprehension with bert. In Proceedings find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malaysha-2024-arafinnlp-2024-first-arabic-financial/