Elamur Research⌕
☰
home / papers / 2024 / BFCI at AraFinNLP2024: Support Vector Machines for Arabic Financial Text Classification

BFCI at AraFinNLP2024: Support Vector Machines for Arabic Financial Text Classification

Paper · 2024 · Nsrin Ashraf, Hamada Nayel, Mohammed Aldawsari et al.

Summary

This paper presents the BFCI/BFCAI team's entry to subtask 1 of the AraFinNLP 2024 shared task. The task is intent detection for Arabic banking customer queries written in multiple dialects. The team used ArBanking77, an Arabic rendering of the English Banking77 corpus with 77 intent labels, and trained only on its MSA portion. Their system is a deliberately simple classical pipeline: queries become TF-IDF vectors over unigrams and bigrams, and three scikit-learn classifiers are compared on them: Multi-Layer Perceptron, Stochastic Gradient Descent and a linear Support Vector Machine. On the development set the SVM was slightly ahead of the MLP and well ahead of SGD, so it was submitted. On the blind test set it reached about 49% micro F1 and placed 11th. The large gap between development and test scores is the main lesson. The authors attribute it to overfitting and argue that the SVM's regularization made it the most robust of the three. The work serves as a lightweight baseline that uses no external resources.

Key points

  • Data: ArBanking77 (Jarrar et al., 2023) is derived from Banking77 (Casanueva et al., 2020). It has 13,083 queries across 77 intents in MSA and Palestinian Arabic. The authors used only the MSA split, reported as 10,733 train, 1,230 validation and 11,721 test items.
  • Features: TF-IDF with an n-gram range of 1–2 (unigrams plus bigrams) and max_df = 1.0.
  • Classifiers and settings:
    • MLP: 20 hidden units, adam solver, logistic activation, 500 iterations.
    • SGD: log loss, 1000 iterations.
    • SVM: linear kernel with L1 regularization.
    • Random seeds were fixed for reproducibility.
  • Development F1: SVM 85.12%, MLP 84.46%, SGD 74.65%.
  • Test result: the submitted SVM scored 49.10% F1 on the blind test, ranking 11th in the shared task.

Interesting ideas

  • I1. Comparing several basic classifiers on one shared, transparent feature set isolates the classifier's contribution from the representation's, and it yields a cheap, inspectable baseline before heavier models are tried.
  • I2. A near 36-point drop from development to blind test is a warning sign: in-distribution validation can badly overstate real performance when held-out data differs.
  • I3. Choosing the more strongly regularized model (an L1 linear SVM) for robustness under overfitting trades a little dev accuracy for simpler decision boundaries.
  • I4. Fixing random seeds for every classifier makes reported numbers exactly reproducible, which matters when model differences are under one point, as with SVM versus MLP here.
  • I5. Treating customer queries as a closed set of 77 fine-grained intents is a useful design for mapping a request onto an actionable category.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Any learned component, such as a frame or speech-act classifier, could first be benchmarked against a linear model over explicit UD-derived features, so we can see how much a complex model actually adds.
  • I2 (learn): Evaluation should always include a genuinely held-out or shifted test set, such as other genres or dialects, next to the dev split, and the gap should be reported as a first-class metric.
  • I3 (tree): The perceptron parser could favour sparse, regularized feature weights, which may help it generalize across treebanks and keep its decisions easier to inspect.
  • I4 (learn): Seeding all stochastic steps would let us tell real improvements apart from run-to-run noise in silver-data training and self-play.
  • I5 (frames): Speech-act recognition could include a finite inventory of request or intent types for task-oriented text, attached to the frame rather than guessed from bag-of-words features.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Prince Sattam Bin Abdulaziz UniversityAl Kharj · Saudi Arabia17 works · 489 citations🎓Benha UniversityBanhā · Egypt14 works · 89 citations🎓Mangalore UniversityMangaluru · India2 works · 13 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nsrin Ashraf, Hamada Nayel, Mohammed Aldawsari, Hosahalli Shashirekha and Tarek Elshishtawy. 2024. BFCI at AraFinNLP2024: Support Vector Machines for Arabic Financial Text Classification. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.42

@inproceedings{ashraf2024bfci,
  title = {{BFCI at AraFinNLP2024: Support Vector Machines for Arabic Financial Text Classification}},
  author = {Nsrin Ashraf, Hamada Nayel, Mohammed Aldawsari, Hosahalli Shashirekha and Tarek Elshishtawy},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.42},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.42}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed H. AbuElAtta, Mahmoud Sobhy, Ahmed A. El-Sawy, and Hamada Nayel. 2023. Arabic regional dialect identification (ardi) using pair of continuous bag-of-words and data augmentation. International Journal of Advanced Computer Science and Applications, 14(11). find
  2. Nsrin Ashraf, Hamada Nayel, and Mohamed Taha. 2022. Misinformation detection in arabic tweets: A case study about covid-19 vaccination. Benha Journal of Applied Sciences, 7(5):265–268. ↗
  3. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. ↗
  4. Shadi Diab. 2019. Optimizing stochastic gradient descent in text classification based on fine-tuning hyper-parameters approach. A case study on automatic classification of global terrorist attacks. CoRR, abs/1902.06542. ↗
  5. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics. svm and nlp. International journal of electrical and computer engineering (IJECE), 9(3):2152–2163.
  6. H. L. Shashirekha and Hamada A. Nayel. 2016. A comparative study of segment representation for biomedical named entity recognition. In 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 1046–1052. find
  7. Mahmoud Sobhy, Ahmed H. Abu El-Atta, Ahmed A. El-Sawy, and Hamada Nayel. 2022. Word representation models for Arabic dialect identification. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 474–478, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  8. Nadhem Zmandar, Mahmoud El-Haj, and Paul Rayson. 2021. Multilingual financial word embeddings for arabic, english and french. In 2021 IEEE International Conference on Big Data (Big Data), pages 4584–4589. find
  9. Nadhem Zmandar, Mo El-Haj, and Paul Rayson. 2023. FinAraT5: A text to text model for financial Arabic text understanding and generation. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 262–273, Vienna, Austria. NOVA CLUNL, Portugal.
  10. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, Ismail Berrada, and Houda Bouamor. 2024. AraFinNlp 2024: The first arabic financial nlp shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  11. Hamada Nayel. 2020. NAYEL at SemEval-2020 task 12: TF/IDF-based approach for automatic offensive language detection in Arabic tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2086–2089, Barcelona (online). International Committee for Computational Linguistics.
  12. Krutika Patil, Medha Badamikar, and Sheetal Sonawane. 2023. Nlp based text summarization of fintech rfps. In 2023 International Conference on Sustainable Computing and Data Communication Systems (IC-SCDS), pages 865–869. find
  13. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  14. Chaudhary Jashubhai Rameshbhai and Joy Paulose. 2019. Opinion mining on newspaper headlines using find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ashraf-2024-bfci-arafinnlp2024-support-vector-machines/