Elamur Research⌕
☰
home / papers / 2025 / Shawarma Chats: A Benchmark Exact Dialogue & Evaluation Platter in Egyptian, Maghrebi & Modern Standard Arabic–-A Triple-Dialect Feast for Hungry Language Models

Shawarma Chats: A Benchmark Exact Dialogue & Evaluation Platter in Egyptian, Maghrebi & Modern Standard Arabic–-A Triple-Dialect Feast for Hungry Language Models

Paper · 2025 · Kamyar Zeinalipour, Mohamed Zaky Saad, Oumaima Attafi et al.

Summary

Arabic has no dialogue benchmark that ties conversations to a source text and also covers Modern Standard Arabic, Egyptian and Maghrebi varieties together. The authors build ShawarmaChats, a set of 30,000 six-turn, two-speaker dialogues derived from 10,000 Arabic Wikipedia paragraphs, with 10,000 dialogues per variety.

They first asked five frontier LLMs to write 1,500 pilot dialogues. Two native-speaker experts graded these from A to D, and the resulting gold set was used to pick two models: the strongest generator (Gemini 2.5 Flash) and the judge whose ratings best matched the humans (GPT-4o). Any dialogue below grade A goes through at most two rounds of repair. In each round the grader writes a rationale and the generator rewrites. Whatever still fails is fixed by hand.

This loop raised the share of grade-A dialogues from 85.94% to 99.48%, so humans had to touch only about 0.52% of items. LoRA fine-tuning of six open models on the corpus improved grader scores and BERTScore, BLEU and ROUGE, most clearly for models above 7B. The work shows a cheap, human-anchored way to produce grounded, dialect-aware data.

Key points

  • Source pool: the THAW collection of over 200,000 Arabic Wikipedia articles, sorted by GPT-4 into 29 categories. Articles were filtered by length (at least 150 words), title shape and popularity, and 10,000 paragraphs were sampled from them.
  • Pilot design: 100 paragraphs × 3 dialect-specific prompts × 5 LLMs gave 1,500 seeds. The LLMs were GPT-4o, Gemini 2.5 Flash, Qwen-Plus, DeepSeek-Chat and Mistral Large, and each returned its dialogue as structured JSON.
  • Human gold: an Egyptian and a Moroccan annotator rated dialogues independently on four criteria: fluency, faithfulness, coherence and dialect accuracy. They reached Cohen's κ = 0.794 and then settled every disagreement into a single consensus grade.
  • Judge quality: GPT-4o agreed with the experts 80% of the time overall and reached 96.3% precision on its grade-A calls for the chosen generator.
  • Downstream test: six open LLMs (1B–24B) were LoRA-tuned on the corpus. Models of 7B and above gained the most, up to +2.1 grader points. The dataset, LoRA weights, prompts and evaluation code are released.

Interesting ideas

  • I1. Choose the automatic judge by checking it against a small expert-adjudicated set, focusing on precision for the "accept" grade rather than overall agreement, since accepted items are what flow into the corpus.
  • I2. A bounded critique-and-revise cycle works well: the grader must explain what is wrong, the generator fixes it, the loop stops after two passes, and the small residue goes to a human.
  • I3. The best generator and the best judge turned out to be different models, so the two roles were deliberately split instead of letting one model grade its own work.
  • I4. Build evaluation items by deriving multi-turn exchanges from a source paragraph and requiring every turn to stay faithful to it, so hallucinations can be checked against the source.
  • I5. Have annotators grade blind and independently first, measure κ, and only then adjudicate into one gold label, which yields both a reliability figure and a clean reference.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Rank LLM teacher tiers and witness parsers by how often the silver trees they approve are actually correct on a hand-checked sample, not by overall accuracy.
  • I2 (learn): Self-induced repair rules could run the same way, with a checker stating why a parse or frame is wrong, a repairer proposing a fix, and items still failing after two attempts sent to manual review.
  • I3 (back): When generating text from the meaning graph, use a separate scorer (e.g. re-parsing the output and comparing graphs) instead of trusting the generator's own confidence.
  • I4 (answer): Generate grounded question-and-answer dialogues from fables and tales, then verify each answer against the world model built from the original text.
  • I5 (learn): Apply this two-stage protocol when building small gold sets for negative-control evaluation of silver trees and frames.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SienaSiena · Italy9 works · 198 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksDialogue systems · 624 works

Cite

Cite as Kamyar Zeinalipour, Mohamed Zaky Saad, Oumaima Attafi, Marco Maggini and Marco Gori. 2025. Shawarma Chats: A Benchmark Exact Dialogue & Evaluation Platter in Egyptian, Maghrebi & Modern Standard Arabic–-A Triple-Dialect Feast for Hungry Language Models. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.39

@inproceedings{zeinalipour2025shawarma,
  title = {{Shawarma Chats: A Benchmark Exact Dialogue \& Evaluation Platter in Egyptian, Maghrebi \& Modern Standard Arabic–-A Triple-Dialect Feast for Hungry Language Models}},
  author = {Kamyar Zeinalipour, Mohamed Zaky Saad, Oumaima Attafi, Marco Maggini and Marco Gori},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.39},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.39}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022b. Evaluating attribution in dialogue systems: The begin benchmark. Transactions of the Association for Computational Linguistics, 10:1066–1083. ↗
  2. Alexander Fabbri, Wojciech Kryściński, Bryan Mc Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Reevaluating summariza tion evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  3. Ahlam Fuad, Maha AlYahya, and Fahad Alruwili. 2022. Araconv: Developing an arabic taskoriented dia logue system using multilingual transformer model mt5. Applied Sciences, 12(4):1881. find
  4. Karthik Gopalakrishnan, Bahar Hedayatnia, Qinlang Hu, Huda Khayrallah, Ryan Meltz, Ashwin Venkatesh, and et al. 2019. Topicalchat: Towards knowledge grounded opendomain conversations. In Proceed ings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 132–142. find
  5. Prakhar Gupta, ChienSheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Dialfact: A benchmark for factchecking in dialogue. In Proceedings of the 60th Annual Meeting of the Association for Compu tational Linguistics (Volume 1: Long Papers), pages 3785–3801, Dublin, Ireland. Association for Compu tational Linguistics. find
  6. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q2: Evaluating factual consistency in knowledge grounded dialogues via question generation and ques tion answering. In Proceedings of the 2021 Confer ence on Empirical Methods in Natural Language Pro cessing, pages 7856–7870, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Aditya Abhay Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Computing Surveys, 57(6). ↗
  8. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can llms actually cor rect their own mistakes? a critical survey of self correction of llms. Transactions of the Association for Computational Linguistics, 12:1417–1440. find
  9. Abdullah Khered, Youcef Benkhedda, and Riza Batista Navarro. 2025. Dial2msaverified: A multidialect arabic social media dataset for neural machine trans lation to modern standard arabic. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics. find
  10. Charles Koutchéme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024. Open source language models can provide feedback: Eval uating llms’ ability to help students using gpt4 as a judge. In Proceedings of the 2024 Conference on Innovation and Technology in Computer Science Ed ucation, pages 52–58, Milan, Italy. ACM. find
  11. Linhao Ye, Zhikai Lei, Jianghao Yin, Qin Chen, Jie Zhou, and Liang He. 2024. Boosting conversa tional question answering with finegrained retrieval augmentation and selfcheck. Proceedings of SIGIR 2024, pages 2301–2305. find
  12. Kamyar Zeinalipour, Achille Fusco, Asya Zanollo, Marco Maggini, and Marco Gori. 2024a. Harnessing llms for educational contentdriven italian crossword generation. arXiv preprint arXiv:2411.16936. ↗
  13. Kamyar Zeinalipour, Neda Jamshidi, Fahimeh Akbari, Marco Maggini, Monica Bianchini, Marco Gori, and 1 others. 2025a. Persianmcqinstruct: A comprehen sive resource for generating multiplechoice ques tions in persian. In Proceedings of the First Workshop on Language Models for LowResource Languages, pages 344–372. Association for Computational Lin guistics. find
  14. Kamyar Zeinalipour, Yusuf Gökberk Keptiğ, Marco Maggini, and Marco Gori. 2024b. Automating turk ish educational quiz generation using large language models. In International Conference on Intelligent Systems and Pattern Recognition, pages 246–260. Springer. find
  15. Kamyar Zeinalipour, Yusuf Gökberk Keptiğ, Marco Maggini, Leonardo Rigutini, and Marco Gori. 2024c. A turkish educational crossword puzzle generator. In International Conference on Artificial Intelligence in Education, pages 226–233. Springer. find
  16. Kamyar Zeinalipour, Mehak Mehak, Fatemeh Parsamo tamed, Marco Maggini, and Marco Gori. 2024d. Ad vancing student writing through automated syntax feedback. In International Workshop on AI in Ed ucation and Educational Research, pages 52–66. Springer. find
  17. Salima Lamsiyah, Kamyar Zeinalipour, Matthias Brust, Marco Maggini, Pascal Bouvry, Christoph Schommer, and 1 others. 2025. Arabicsense: A benchmark for evaluating commonsense reasoning in arabic with large language models. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL4), pages 1–11.
  18. Kamyar Zeinalipour, Moahmmad Saad, Marco Maggini, and Marco Gori. 2025b. From Arabic text to puz zles: LLMdriven development of Arabic educational crosswords. In Proceedings of the First Workshop on Language Models for LowResource Languages, pages 479–495, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  19. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yu jia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llmsasjudges: A comprehensive survey on llmbased evaluation methods. arXiv preprint arXiv:2412.05579. ↗
  20. Kamyar Zeinalipour, Mohamed Zaky Saad, Marco Mag gini, and Marco Gori. 2025c. From arabic text to puzzles: Llmdriven development of arabic educa tional crosswords. arXiv preprint arXiv:2501.11035. ↗
  21. ChinYew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out (ACL Workshop), pages 74–81.
  22. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. Selfcheckgpt: Zeroresource blackbox halluci nation detection for generative large language models. In Proceedings of EMNLP 2023, pages 9004–9017. find
  23. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. Proceedings of ACL 2020, pages 7881–7892.
  24. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Wein berger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of ICLR. find
  25. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, and et al. 2022. Towards a unified multidimensional evaluator for text generation. In Proceedings of EMNLP 2022, pages 1740–1755. find
  26. Andrea Zugarini, Kamyar Zeinalipour, Achille Fusco, and Asya Zanollo. 2024a. Ecwcaeducational cross word clues answering: A calamita challenge. In Pro ceedings of the 10th Italian Conference on Computa tional Linguistics (CLiCit 2024), pages 1239–1244. find
  27. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024b. Clueinstruct: Textbased clue generation for educational crossword puzzles. arXiv preprint arXiv:2404.06186. Batch size: 4 sequences × 2 gradaccumulation steps ⇒ effective batch of 8. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeinalipour-2025-shawarma-chats-benchmark-exact-dialogue/