Elamur Research⌕
☰
home / papers / 2024 / A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domain

A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domain

Summary

Arabic speech recognition is especially hard on phone calls. Speakers mix many dialects, switch into English and talk informally, and the audio is often noisy or degraded. Benchmarks for this setting were missing, so the authors built one from real recordings of agents talking with customers. The calls came from a MENA cloud communication provider, with user consent and with sensitive business details removed. The resource contains 132 hours of manually transcribed speech, covering six regional dialect groups and speakers from 13 countries. Noise conditions range from quiet rooms and offices to cars and are distributed evenly. Calls were cut at silences into clips shorter than 30 seconds, transcribed under shared guidelines, and passed through automatic quality checks. Five systems were then scored with WER and CER to give baselines. Even the best one, Google's Chirp, gets about half of all words wrong, and Whisper performs worst. The benchmark shows that call-centre Arabic remains far from solved and gives the field a realistic common test bed.

Key points

  • The data is 132 hours of agent–client call audio at 16kHz, mono, 16-bit. It spans six dialect regions, speakers from 13 countries, and domains such as education, entertainment and e-commerce, mostly in support and sales roles.

  • Transcription was done by 41 annotators, and 13 reviewers then checked their work.

  • After annotation, automatic checks were run:

    • every transcript must use only the allowed character set;
    • very short or empty audio files were removed;
    • language ID confirmed that each clip is Arabic;
    • every clip must have a transcript.

    The data was then split into tiers by cleanliness and domain relevance.

  • Baseline results:

    SystemWERCER
    Chirp48.9%22.4%
    Google API67.1%40.60%
    Meta M4T V167.8%34.3%
    Azure API71.88%39.04%
    Whisper Large V183.8%52.3%

  • Google API has a slightly better WER than Meta M4T but a clearly worse CER. The two metrics therefore rank these systems in opposite orders.

Interesting ideas

  • I1. Scoring at two levels of granularity (words and characters) exposes cases where systems swap places, so a single headline number can mislead about which system is better. — For Mova (learn): Evaluate tokenizer and lemmatizer output with both whole-token accuracy and character- or morpheme-level error, so near-miss errors are not hidden by an exact-match score.
  • I2. The test set deliberately balances acoustic conditions, from clean to noisy, so that robustness is measured across realistic difficulty rather than on the easy cases only. — For Mova (tree): Build parser evaluation sets with equal shares of clean edited prose, informal text and messy web text, and report LAS per stratum so a gain on clean news cannot mask a loss on noisy input.
  • I3. A cheap automatic gate runs after human annotation: it checks the character set, drops empty items, verifies the language and confirms every input is paired with an output. This catches pipeline defects before they contaminate evaluation. — For Mova (learn): Put the same kind of gate in front of silver treebanks agreed by witness parsers, rejecting sentences in the wrong language, odd scripts or empty trees, and any sentence lacking a full tree.
  • I4. The annotation guidelines fix surface conventions up front, such as whether numbers are written as digits or words. Otherwise, scores partly reflect formatting choices instead of real errors. — For Mova (answer): For maths word problems, normalise numerals and number words to one canonical form before comparing answers, so a correct quantity is not marked wrong for its spelling.
  • I5. Long recordings are cut at natural pauses under a hard length cap, which makes units easier to annotate and to process. — For Mova (tokens): When the sentence splitter faces very long run-on text, fall back to splitting at natural boundaries such as semicolons or discourse markers under a maximum length, keeping downstream parsing tractable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.04280 · arXiv 2403.04280

@misc{obaidah2024new,
  title = {{A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domain}},
  author = {Obaidah, Qusai Abo and Za'ter, Muhy Eddin and Jaljuli, Adnan and Mahboub, Ali and Hakouz, Asma and Al-Rfooh, Bashar and Estaitia, Yazan},
  year = {2024},
  doi = {10.48550/arxiv.2403.04280},
  eprint = {2403.04280},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.04280}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Farghaly and Khaled Shaalan. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22, 2009. find
  2. María Pilar Fernández-Gallego and Doroteo T Toledano. A study of data augmentation for asr robustness in low bit rate contact center recordings including packet losses. Applied Sciences, 12(3):1580, 2022. find
  3. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Abdel Monem. Challenges in arabic natural language processing. In Computational linguistics, speech and image processing for arabic language, pages 59–83. World Scientific, 2019. find
  4. Ramzi A Haraty and Omar El Ariss. Casra+: A colloquial arabic speech recognition application. American Journal of Applied Sciences, 4(1):23–32, 2007. find
  5. Ashraf Elnagar, Sane M Yagi, Ali Bou Nassif, Ismail Shahin, and Said A Salloum. Systematic literature review of dialectal arabic: identification and detection. IEEE Access, 9:31010–31042, 2021. find
  6. Hamzah A Alsayadi, Abdelaziz A Abdelhamid, Islam Hegazy, and Zaki T Fayed. Arabic speech recognition using end-to-end deep learning. IET Signal Processing, 15(8):521–534, 2021. find
  7. Seham Nasr, Rehab Duwairi, and Muhannad Quwaider. End-to-end speech recognition for arabic dialects. Arabian Journal for Science and Engineering, pages 1–17, 2023. find
  8. Rostislav Kolobov, Olga Okhapkina, Olga Omelchishina, Andrey Platunov, Roman Bedyakin, Vyacheslav Moshkin, Dmitry Menshikov, and Nikolay Mikhaylovskiy. Mediaspeech: Multilanguage asr benchmark and dataset. arXiv preprint arXiv:2103.16193, 2021. ↗
  9. Tuka AlHanai, Wei-Ning Hsu, and James Glass. Development of the mit asr system for the 2016 arabic multi-genre broadcast challenge. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 299–304. IEEE, 2016. find
  10. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. Qasr: Qcri aljazeera speech resource–a large scale annotated arabic speech corpus. arXiv preprint arXiv:2106.13000, 2021. ↗
  11. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596, 2023. ↗ our card
  12. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: sentence-level multimodal and language-agnostic representations. arXiv e-prints, pages arXiv–2308, 2023. find
  13. Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610, 2019. find our card
  14. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022. ↗
  15. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR, 2023. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/obaidah-2024-new-benchmark-evaluating-automatic-speech/