Elamur Research⌕
☰
home / papers / 2025 / Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems

Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems

Paper · 2025 · Abdulla Alshabanah and Murali Annavaram

Summary

The paper asks whether recommendation systems serve Arabic speakers equally, given how strongly regional dialects differ. The authors label users by dialect using recent Arabic NLP tools. A BERT-based dialect identification model (Keleg and Magdy, 2023) assigns each user the dialect that dominates their reviews, across 18 country-level classes. Reviews whose ALDi dialectness score falls below 0.1 are relabelled as MSA. Users are then grouped at three levels: MSA versus dialectal, five regions (Nile, Gulf, Levant, Maghreb, MSA), and individual countries.

Two recommenders, TTNN and DeepFM, are trained with a max-margin ranking loss on two Arabic book-review corpora, BRAD and LABR. They are evaluated with leave-one-out top-k metrics (HR@10, NDCG@10). Fairness is measured with UGF*, a multi-group version of User-oriented Group Fairness defined as the largest gap in average metric between any two groups.

Recommendation quality differs by dialect, and the gap widens as groups become finer. Some countries, such as Saudi Arabia, Oman and Palestine, fall well behind. This shows that language-aware NLP can expose hidden biases in other systems.

Key points

  • Datasets: BRAD and LABR, both converted to implicit feedback with a leave-one-out split. Models: TTNN and DeepFM, with Matrix Factorization reported in the appendix. Metrics: HR@10 and NDCG@10.
  • Dialect labels come from a BERT-based ADI model covering 18 country dialects. An ALDi score below 0.1 overrides the label to MSA.
  • In LABR with DeepFM, dialectal users outscore MSA users (HR@10 0.079 vs 0.058). The authors connect this to average test-item degree: dialectal items have 40.0 interactions versus 36.2 for MSA items, while BRAD shows the reverse pattern (95.1 vs 93.2).
  • At country level, some groups exceed 0.1 HR@10, while Saudi Arabia, Oman and Palestine often stay under 0.05. Which region ranks best changes with the dataset and model; for example, Maghreb leads in BRAD with 0.065 and 0.077.
  • UGF* using HR@10 is roughly 0.0004–0.02 for the MSA-vs-dialectal split and rises to about 0.14–0.25 at country level. NDCG@10 follows the same trend, and no mitigation method is proposed.

Interesting ideas

  • I1. Measuring the same disparity at several nested grouping levels (binary, regional, country) shows how a gap grows as groups get smaller, instead of reporting one arbitrary split.
  • I2. The fairness gap for many groups is reduced to one number: the largest difference in mean score between any pair of groups. This makes it easy to track and compare across models.
  • I3. A discrete classifier label is combined with a continuous "how nonstandard is this text" score, and the score overrides the label below a threshold. This cleanly separates formal from informal input.
  • I4. Before blaming a group gap on the grouping itself, the authors check a confounder: how popular the test items are.
  • I5. An NLP signal (dialect) is used to audit a non-NLP system, revealing biases that standard evaluation hides.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Report parser LAS sliced by nested text categories (all text → register → specific genre or source) to see where accuracy collapses rather than averaging it away.
  • I2 (learn): Use a max-pairwise-gap score over treebanks or genres as a regression alarm alongside overall accuracy when retraining gparse or the perceptron parser.
  • I3 (tokens): A continuous informality or nonstandardness score could flag chatty or noisy English before tagging, routing it to more tolerant tokenization and morphology rules.
  • I4 (answer): When QA or word-problem accuracy differs between text types, first test confounders such as sentence length or entity count as a negative control before concluding that the text type is the cause.
  • I5 (answer): Linguistic features extracted from the UD tree and frames could slice the error analysis of the student readers, showing which construction types the answering stage systematically fails on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Abdulla Alshabanah and Murali Annavaram. 2025. Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.860

@inproceedings{alshabanah2025mind,
  title = {{Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems}},
  author = {Abdulla Alshabanah and Murali Annavaram},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.860},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.860}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Abdelrahim Elmadany, Chiyu Zhang, Houda Bouamor, Nizar Habash, et al. 2023. Nadi 2023: The fourth nuanced arabic dialect identification shared task. In Proceedings of Arabic-NLP 2023, pages 600–613.
  2. Muhammad Abdul-Mageed, Amr Keleg, Abdelrahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. Nadi 2024: The fifth nuanced arabic dialect identification shared task. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 709– 728.
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Abdelrahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. Nadi 2021: The second nuanced arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, Abdelrahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. Nadi 2022: The third nuanced arabic dialect identification shared task. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97.
  5. Abdulla Alshabanah and Murali Annavaram. 2025. On using Arabic language dialects in recommendation systems. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2178–2186, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  6. Abdulla Alshabanah, Keshav Balasubramanian, and Murali Annavaram. 2025. Meta-learn to unlearn: Enhanced exact machine unlearning in recommendation systems with meta-learning. Proceedings on Privacy Enhancing Technologies. ↗
  7. Maha Jarallah Althobaiti. 2020. Automatic arabic dialect identification systems for written texts: A survey. International Journal of Computational Linguistics (IJCL), 11(3):61–89. Revised: 31-10-2020; Published: 01-12-2020. ↗
  8. Mohamed Aly and Amir Atiya. 2013. Labr: A large scale arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. ↗
  9. Keshav Balasubramanian, Abdulla Alshabanah, Elan Markowitz, Greg Ver Steeg, and Murali Annavaram. 2024. Biased user history synthesis for personalized long-tail item recommendation. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys ’24, page 189–199, New York, NY, USA. Association for Computing Machinery. find
  10. A. Bergman and Mona Diab. 2022. Towards responsible natural language annotation for the varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022, pages 364–371, Dublin, Ireland. Association for Computational Linguistics.
  11. Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, New York, NY, USA. find
  12. Ashraf Elnagar and Omar Einea. 2016. Brad 1.0: Book reviews in arabic dataset. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pages 1–8. IEEE. find
  13. Ashraf Elnagar, Leena Lulu, and Omar Einea. 2018. An annotated huge dataset for standard and colloquial arabic reviews for subjective sentiment analysis. Procedia computer science, 142:182–189. find
  14. Charles A. Ferguson. 1959. 15(2):325–340. Diglossia. find
  15. WORD, Carlos A. Gomez-Uribe and Neil Hunt. 2016. The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manage. Inf. Syst., 6(4). ↗
  16. Asnat Greenstein-Messica and Lior Rokach. 2018. Personal price aware multi-seller recommender system: Evidence from ebay. Know.-Based Syst., 150(C):14–26. ↗
  17. Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: A factorization-machine based neural network for CTR prediction. CoRR, abs/1703.04247. ↗
  18. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. CoRR, abs/1708.05031. ↗
  19. Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Ajit Kumthekar, Zhe Zhao, Li Wei, and Ed Chi, editors. 2019. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. find
  20. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics.
  21. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Proceedings of ArabicNLP 2023, pages 385–398, Singapore (Hybrid). Association for Computational Linguistics.
  22. Amr Keleg, Walid Magdy, and Sharon Goldwater. 2024. Estimating the level of dialectness predicts inter-annotator agreement in multi-dialect Arabic datasets. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 766–777, Bangkok, Thailand. Association for Computational Linguistics.
  23. Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37. ↗
  24. Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented fairness in recommendation. In Proceedings of the web conference 2021, pages 624–632. find
  25. G. Linden, B. Smith, and J. York. 2003. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing, 7(1):76–80. ↗
  26. Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. 2019. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs/1906.00091. ↗
  27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32. ↗
  28. Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International Conference on Data Mining, pages 995–1000. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshabanah-2025-mind-dialect-nlp-advancements-uncover/