Elamur Research⌕
☰
home / papers / 2026 / WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

Paper · 2026 · Zien Sheikh Ali, Hamdy Mubarak, Soon-Gyo Jung et al.

Summary

Voice assistants built as a speech-recognition stage followed by an LLM are hard to diagnose. When a user dislikes an answer, the cause could be a misrecognized utterance, a request that was unclear or impossible from the start, or a weak LLM. WASIL is an Arabic resource designed to pull these causes apart.

About 93 speakers from Algeria, Egypt, Sudan and Syria talked freely with a Fanar-based assistant over nine days, guided by daily topics. They produced 9,304 spoken prompts and marked responses with like/dislike, plus finer complaint categories. The release pairs each prompt with its audio, ASR output, the assistant's reply, a dialect label and an answerability label.

Gold transcripts for a subset were made cheaply: annotators from the speaker's own country corrected automatic transcripts, using agreement between several ASR systems as a guide. With these, the authors relate answerability to dislikes and benchmark several LLMs. Each model answers both the ASR and the gold transcript, and multiple LLM judges score the replies without reference answers.

This appears to be the first in-the-wild Arabic spoken-LLM dataset with real user feedback.

Key points

  • Scale and split: 9,304 turns from 93 users. Of these, 2,573 have gold transcripts, split into a 1,777-turn test set and a 988-turn feedback set; 6,602 turns form a training set. The test set is public as QCRI/WASIL on Hugging Face.
  • Systems: Initial transcripts come from Fanar-Aura-STT-1. Recordings were also run through a parallel pipeline using Gemini for both ASR and LLM, and ALLaM was used as another LLM in the cascade.
  • Labels:
    • Dialect: MSA, English, mixed, Egyptian, Sudanese, Algerian, Syrian.
    • Answerability: answerable/clear, ambiguous (needs clarification), out-of-domain/unsupported, not-a-request/noise.
  • Dislike taxonomy: a 10-way multi-label scheme covering:
    • instruction failure and factual errors;
    • style/format and evasion;
    • cultural/Islamic misalignment, disturbing content and religious inaccuracy;
    • grammar/odd characters;
    • responses that are too brief or too long.
  • Transcript quality control: annotators were matched to the speaker's country, followed by an expert's random second-pass review.

Interesting ideas

  • I1. Agreement between independent recognizers marks which transcripts are probably right, so human effort goes only where systems disagree; this makes gold data far cheaper than transcribing from scratch.
  • I2. Labeling whether an input was answerable at all keeps inherently bad requests from being blamed on upstream errors, which makes error attribution honest.
  • I3. Feeding the downstream model both a noisy and a clean version of the same input, then comparing outputs, shows which upstream errors actually matter rather than counting all of them equally.
  • I4. Because a multi-label failure taxonomy tracks each kind of error on its own rather than merging everything into a single score, specific weaknesses can be targeted.
  • I5. Several LLM judges scoring answers without gold references allow comparison of systems on open-ended outputs where no reference exists.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use agreement among the witness parsers to decide which sentences become silver data automatically and which go to an LLM teacher or human review, so the review budget is spent only on contested arcs.
  • I2 (answer): Readers on tests and fable QA could first classify a question as answerable, ambiguous or unsupported by the text, and abstain or ask for clarification instead of guessing; evaluation would then score these cases separately.
  • I3 (tree): Run frames and world construction on gold versus predicted UD trees and keep the parse errors that change the world or the answer; this gives a meaning-aware parser metric beyond LAS.
  • I4 (learn): Tag each wrong answer with an explainable failure type (bad tree, wrong role, missed coreference, missing world knowledge, arithmetic slip) so self-induced repair rules can be aimed at the dominant category.
  • I5 (back): Score text regenerated by the text → graph → text compressor with a panel of judges on meaning preservation against the source, flagging items where the judges disagree for closer inspection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 1,484 worksSpeech & spoken language · 233 works

Cite

Cite as Zien Sheikh Ali, Hamdy Mubarak, Soon-Gyo Jung, Hunzalah Hassan Bhatti, Firoj Alam and Shammur Absar Chowdhury. 2026. WASIL: In-the-Wild Arabic Spoken Interactions with LLMs. arXiv:2605.16364.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.16364 · arXiv 2605.16364

@misc{ali2026wasil,
  title = {{WASIL: In-the-Wild Arabic Spoken Interactions with LLMs}},
  author = {Zien Sheikh Ali, Hamdy Mubarak, Soon-Gyo Jung, Hunzalah Hassan Bhatti, Firoj Alam and Shammur Absar Chowdhury},
  year = {2026},
  doi = {10.48550/arxiv.2605.16364},
  eprint = {2605.16364},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.16364}
}

References

56 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Wang, H. Ren, Z. Lu, M. Zhan, and H. Li, “Voiceassistant-eval: Benchmarking ai assistants across listening, speaking, and viewing,” arXiv preprint arXiv:2509.22651, 2025. ↗
  2. Y. Hou, H. Liu, Y. Wang, Z. Cheng, R. Wu, Q. Gu, Y. Wang, and Y. Wang, “SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant,” in Interspeech 2025, 2025, pp. 5713–5717. find
  3. J. Billa, “The cascade equivalence hypothesis: When do speech llms behave like asr→llm pipelines?” arXiv preprint arXiv:2602.17598, 2026. ↗
  4. M. Kubis, P. Skórzewski, M. Sowański, and T. Zietkiewicz, “Back transcription as a method for evaluating robustness of natural language understanding models to speech recognition errors,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 11 824–11 835. find
  5. M. Galbraith, “An analysis of dialogue repair in voice assistants,” arXiv preprint arXiv:2311.03952, 2024. ↗
  6. H. Men, Y. Hu, Y. He, Y. Gao, X. Mou, and Y. Xu, “Reject or not?: A benchmark for voice assistant query rejection in smart home scenario and an improved method based on llms,” arXiv preprint arXiv:2512.10257, 2025. ↗
  7. Y. Chen, X. Yue, C. Zhang, X. Gao, R. T. Tan, and H. Li, “VoiceBench: Benchmarking llm-based voice assistants,” Transactions of the Association for Computational Linguistics, vol. 14, pp. 378–398, 2026. ↗
  8. S. Kim, A. Arora, D. Le, C.-F. Yeh, C. Fuegen, O. Kalinli, and M. L. Seltzer, “Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding,” in Interspeech 2021, 2021, pp. 1977–1981. find
  9. J. Harvill, R. Khaziev, S. Li, R. Cogill, L. Wang, G. Chennupati, and H. Thadakamalla, “Significant asr error detection for conversational voice assistants,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 606–11 610. find
  10. Z. Liu, S. Kim, and O. Kalinli, “Evaluating Speech Recognition Performance Towards Large Language Model Based Voice Assistants,” in Interspeech 2024, 2024, pp. 4099–4103. find
  11. B. Talafha, K. Kadaoui, S. M. Magdy, M. Habiboullah, C. M. Chafei, A. O. El-Shangiti, H. Zayed, M. C. Tourad, R. Alhamouri, R. Assi, A. Alraeesi, H. Mohamed, F. Alwajih, A. Mohamed, A. El Mekki, E. M. B. Nagoudi, B. D. M. Saadia, H. A. Alsayadi, W. Al-Dhabyani, S. Shatnawi, Y. Ech-chammakhy, A. Makouar, Y. Berrachedi, M. Jarrar, S. Shehata, I. Berrada, and M. Abdul-Mageed, “Casablanca: Data and models for multidialectal Arabic speech recognition,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 21 745–21 758. ↗
  12. A. Ali, S. Chowdhury, M. Afify, W. El-Hajj, H. Hajj, M. Abbas, A. Hussein, N. Ghneim, M. Abushariah, and A. Alqudah, “Connecting arabs: Bridging the gap in dialectal speech recognition,” Communications of the ACM, vol. 64, no. 4, pp. 124–129, 2021. ↗
  13. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, “WildChat: 1m chatgpt interaction logs in the wild,” in The Twelfth International Conference on Learning Representations, 2024. ↗
  14. F. Alam, A. E. Shahroor, M. A. Hasan, Z. S. Ali, H. H. Bhatti, M. B. Kmainasi, S. A. Chowdhury, B. Mousi, F. Dalvi, N. Durrani, and N. Milic-Frayling, “OASIS: A multilingual and multimodal dataset for culturally grounded spoken visual qa,” arXiv preprint arXiv:2510.06371, 2025. ↗
  15. Z. S. Ali, H. H. Bhatti, R. N. Nandi, S. A. Chowdhury, and F. Alam, “MENASpeechBank: A reference voice bank with persona-conditioned multi-turn conversations for audiollms,” arXiv preprint arXiv:2602.07036, 2026. [Online]. Available: https://arxiv.org/abs/2602.07036 ↗
  16. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez et al., “Chatbot arena: An open platform for evaluating llms by human preference,” in International Conference on Machine Learning. PMLR, 2024, pp. 8359–8388. ↗
  17. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023, pp. 46 595–46 623. find
  18. D. Bohus and A. I. Rudnicky, “Sorry, i didn’t catch that! – an investigation of non-understanding errors and recovery strategies,” in Proceedings of SIGDIAL 2005, 2005. find
  19. G. Tur, A. Deoras, and D. Hakkani-Tur, “Detecting out-of-domain utterances addressed to a virtual personal assistant,” in Proceedings of Interspeech 2014, 2014. find
  20. H. A. Rahmani, X. Wang, Y. Feng, Q. Zhang, E. Yilmaz, and A. Lipani, “A survey on asking clarification questions datasets in conversational systems,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 2698–2716. find
  21. J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER),” in Proceedings of the 1997 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 1997. find
  22. A. Ali, W. Magdy, and S. Renals, “Multi-reference evaluation for dialectal speech recognition system: A study for Egyptian ASR,” in Proceedings of the Second Workshop on Arabic Natural Language Processing, N. Habash, S. Vogel, and K. Darwish, Eds. Beijing, China: Association for Computational Linguistics, Jul. 2015, pp. 118–126. [Online]. Available: https://aclanthology.org/W15-3213/ ↗
  23. S. Wray, H. Mubarak, and A. Ali, “Best practices for crowdsourcing dialectal Arabic speech transcription,” in Proceedings of the 4th Workshop on Arabic Natural Language Processing, 2015. find
  24. J. Prakash, B. Kumar, K. Hacioglu, B. Sharma, S. Gopalan, M. Chetlur, S. Venkatesan, and A. Stolcke, “Better pseudo-labeling with multi-asr fusion and error correction by speechllm,” in Interspeech 2025, 2025. find
  25. F. Team, U. Abbas, M. S. Ahmad, F. Alam, E. Altinisik, E. Asgari, Y. Boshmaf, S. Boughorbel, S. Chawla, S. Chowdhury et al., “Fanar: An arabic-centric multimodal generative ai platform,” arXiv:2501.13944, 2025. ↗
  26. G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. ↗
  27. M. S. Bari, Y. Alnumay, N. A. Alzahrani, N. M. Alotaibi, H. A. Alyahya, S. AlRashed, F. A. Mirza, S. Z. Alsubaie, H. A. Alahmed, G. Alabduljabbar, R. Alkhathran, Y. Almushayqih, R. Alnajim, S. Alsubaihi, M. A. Mansour, S. A. Hassan, D. M. Alrubaian, A. Alammari, Z. Alawami, A. Al-Thubaity, A. Abdelali, J. Kuriakose, A. Abujabal, N. Al-Twairesh, A. Alowisheq, and H. Khan, “ALLam: Large language models for arabic and english,” in The Proc. of 30th ICLR, 2025. ↗
  28. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, Aug. 2023, accepted by TMLR (OpenReview). ↗
  29. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022.
  30. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” arXiv preprint arXiv:2311.07911, 2023. ↗
  31. S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring how models mimic human falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 3214–3252. [Online]. Available: https://aclanthology.org/2022.acl-long.229/ ↗
  32. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, Dec. 2022. ↗
  33. J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “OR-bench: An over-refusal benchmark for large language models,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. PMLR, 13–19 Jul 2025, pp. 11 515–11 542. ↗
  34. A. R. Fabbri, W. Kryściński, B. McCann, C. Xiong, R. Socher, and D. Radev, “SummEval: Re-evaluating summarization evaluation,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 391–409, 2021. ↗
  35. Y. Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec, “Cultural bias and cultural alignment of large language models,” PNAS Nexus, vol. 3, no. 9, p. pgae346, Sep. 2024. ↗
  36. K. Jones, K. Walker, C. Caruso, and S. Strassel, “MAGLIC the maghrebi language identification corpus,” in Proceedings of the Speaker and Language Recognition Workshop Odyssey 2024, 2024, pp. 86–90. find
  37. I. Hamed, F. Eryani, D. Palfreyman, and N. Habash, “ZAEBUC-Spoken a multilingual multidialectal arabic-english speech corpus,” in Proceedings of LREC-COLING 2024, 2024, pp. 17 770– 17 782. find
  38. R. Artstein and M. Poesio, “Survey article: Inter-coder agreement for computational linguistics,” Computational linguistics, vol. 34, no. 4, pp. 555–596, 2008.
  39. S. Akasaki and M. Sassano, “Detecting ambiguous utterances in an intelligent assistant,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Industry Track. Association for Computational Linguistics, 2024, pp. 386–394. find
  40. L.-M. Zhan, H. Liang, B. Liu, L. Fan, X.-M. Wu, and A. Y. S. Lam, “Out-of-scope intent detection with self-supervision and discriminative training,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2021, pp. 3521–3532. ↗
  41. H. Lang, Y. Zheng, B. Hui, F. Huang, and Y. Li, “Out-of-domain intent detection considering multi-turn dialogue contexts,” in Proceedings of LREC-COLING 2024. ELRA Language Resource Association, 2024, pp. 12 539–12 552. find
  42. H. Bunt, V. Petukhova, E. Gilmartin, C. Pelachaud, A. Fang, S. Keizer, and L. Prevot, “The iso standard for dialogue act annotation, second edition,” in Proceedings of the 12th LREC, 2020, pp. 549–558.
  43. K. L. Gwet, “Computing inter-rater reliability and its variance in the presence of high agreement,” British Journal of Mathematical and Statistical Psychology, vol. 61, no. 1, pp. 29–48, 2008. find
  44. M. Elmahdy, R. Gruhn, W. Minker, and S. Abdennadher, “Cross-lingual acoustic modeling for dialectal arabic speech recognition.” in INTERSPEECH, 2010, pp. 873–876. find
  45. S. A. Chowdhury, A. Hussein, A. Abdelali, and A. Ali, “Towards one model to rule all: Multilingual strategy for dialectal code-switching arabic asr,” in Proc. Interspeech 2021, 2021, pp. 2466– 2470.
  46. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 27 730– 27 744.
  47. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealToxicityPrompts: Evaluating neural toxic degeneration in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 3356–3369. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.301/ ↗
  48. A. Gatt and E. J. Krahmer, “Survey of the state of the art in natural language generation: Core tasks, applications and evaluation,” Journal of Artificial Intelligence Research, vol. 61, no. 1, pp. 65– 170, 2018. ↗
  49. T. Naous, M. J. Ryan, A. Ritter, and W. Xu, “Having beer after prayer? measuring cultural bias in large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 16 366– 16 393.
  50. F. Alwajih, A. El Mekki, H. Mubarak, M. Hawasly, A. Mohamed, and M. Abdul-Mageed, “PalmX 2025: The first shared task on benchmarking LLMs on Arabic and islamic culture,” in Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, K. Darwish, A. Ali, I. Abu Farha, S. Touileb, I. Zitouni, A. Abdelali, S. Al-Ghamdi, S. Alkhereyf, W. Zaghouani, S. Khalifa, B. AlKhamissi, R. Almatham, I. Hamed, Z. Alyafeai, A. Alowisheq, G. Inoue, K. Mrini, and W. Alshammari, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 774–789. find
  51. F. Alam, M. A. Hasan, and S. A. Chowdhury, “SpokenNativQA: Multilingual everyday spoken queries for llms,” in Proceedings of the 26th Interspeech Conference (Interspeech 2025). Rotterdam, The Netherlands: ISCA, Aug. 2025. find
  52. J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin, “Qwen2.5-omni technical report,” 2025. ↗
  53. A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram et al., “Openai gpt-5 system card,” arXiv preprint arXiv:2601.03267, 2025. ↗
  54. OpenAI, “GPT-4 technical report,” OpenAI, Tech. Rep., 2023. ↗
  55. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson et al., “Paperbench: Evaluating ai’s ability to replicate ai research,” in ICML. PMLR, 2025, pp. 56 843–56 873. find
  56. X. Guo, U. Tyagi, A. Gosai, P. Vergara, J. Park, E. G. H. Montoya, C. B. C. Zhang, B. Hu, Y. He, B. Liu et al., “Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning,” arXiv preprint arXiv:2510.12712, 2025. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2026-wasil-wild-arabic-spoken-interactions/