Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German

Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German

Summary

Commercial speech recognition is usually advertised with error rates measured on clean, single-language test sets. Those numbers say little about speakers who mix languages within one sentence. The authors, from Perle AI, built a benchmark of 1,200 code-switched utterances, 300 for each of four pairs: Egyptian Arabic–English, Saudi (Najdi/Hijazi) Arabic–English, Persian–English and German–English. Native speakers recorded the utterances conversationally on ordinary headset or USB microphones in quiet rooms, which mimics remote-work audio. Hard samples were chosen in two stages. First, a cheap rule-based difficulty score filtered candidates. Then a GPT-4o plus Gemini 1.5 Pro ensemble ranked only the survivors, cutting LLM cost by roughly 91%. Five commercial systems were scored with both WER and multilingual BERTScore. ElevenLabs Scribe v2 led everywhere. The two metrics ranked systems identically on the Arabic and Persian pairs, but WER made the gaps look about three times larger. It counts valid alternative transliterations as errors. The work shows that metric choice and stratified reporting change how providers should be compared. The data is released on Hugging Face.

Key points

  • The dataset is Perle-ai/ASR_Code_Switch: 4 language pairs × 300 utterances, with topics labelled by GPT-4o using a taxonomy induced from 240 sampled transcripts.

  • Stage 1 computes H_Score (0–10) from five weighted signals:

    • script mix ratio (0.30), with its peak set at the empirical median of 0.35;
    • token script alternation (0.30);
    • regex-detected morphological blends such as Arabic article + English word (0.20);
    • length (0.10);
    • type–token ratio (0.10).

    The script-based signals cannot work for German–English, since both languages use Latin script.

  • Scribe v2 reached 13.2% WER overall and a BERTScore of 0.936. On Egyptian Arabic its WER was 13.1%, against 45.9% for the next-best system.

  • Azure AI Speech averaged 43.6% WER with continuous language-ID mode, 2.0 points better than the default at-start mode. German–English improved from 20.5% to 17.1%.

  • WER and BERTScore (bert-base-multilingual-cased) agreed on rankings with Kendall τ = 1.0 for the Arabic and Persian pairs. WER exaggerated the size of the differences roughly threefold.

Interesting ideas

  • I1. Run a cheap, interpretable rule-based scorer over the whole pool and send only its top candidates to costly LLM judges. This keeps most of the selection quality at a fraction of the price. — For Mova (learn): Before asking LLM teacher tiers to adjudicate silver parses, rank sentences with cheap signals (witness-parser disagreement, length, rare constructions), so teacher budget goes only to the hardest cases.
  • I2. Anchor each heuristic's saturation point to a corpus statistic, such as the median mix ratio or the 75th/90th percentile, rather than a theoretical maximum. — For Mova (tree): Difficulty scores for picking parser training or evaluation sentences could cap features like clause depth or non-projectivity at corpus percentiles, keeping the score meaningful and explainable.
  • I3. Check whether a surface metric and a meaning-aware metric agree on ranking (τ) but disagree on magnitude. Agreement on order does not mean the reported gap sizes are trustworthy. — For Mova (learn): Compare exact-match LAS-style scores with graph-level semantic equivalence when evaluating systems, and report both, so cosmetic variants are not inflated into large regressions.
  • I4. Accept several valid surface forms as correct, through multi-reference or normalisation-aware scoring, so that legitimate spelling or script variants are not punished. — For Mova (back): Score text → graph → text round trips against a set of acceptable paraphrases, or compare at the graph level, rather than penalising every wording difference.
  • I5. Report results split by difficulty and subgroup, and record test material under realistic deployment conditions, because averages hide where systems break. — For Mova (answer): Break test and word-problem accuracy down by question difficulty and text type, and use naturally messy inputs, so weak spots in the readers stay visible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.19069 · arXiv 2605.19069

@misc{abdoli2026benchmarking,
  title = {{Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German}},
  author = {Abdoli, Sajjad and Al-Sumaidaee, Ghassan and Taylor, Clayton W. and ElShiekh, Ahmad and Rashad, Ahmed},
  year = {2026},
  doi = {10.48550/arxiv.2605.19069},
  eprint = {2605.19069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.19069}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Auer, Peter, ed. (2013). Code-Switching in Conversation: Language, Interaction and Identity. London: Routledge. find
  2. Baevski, Alexei et al. (2020). “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”. In: Advances in Neural Information Processing Systems. Vol. 33, pp. 12449–12460. find our card
  3. Deepgram (2025a). Introducing Nova-3: Setting a New Standard for AI-Driven Speech-to-Text. Deepgram Blog. Accessed: May 2026. url: https://deepgram.com/learn/introducing-nova-3-speech-to-text-api. — (2025b). Nova-3 Multilingual Goes GA: Real-Time Code-Switching Across 10 Languages. Deepgram Developer Changelog. Accessed: May 2026. url: https://developers.deepgram.com/changelog/2025/3/31. ↗
  4. Deuchar, Margaret et al. (2014). Building Bilingual Corpora. Miami Bangor Corpus. find
  5. Devlin, Jacob et al. (2019). “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”. In: Proceedings of NAACL-HLT, pp. 4171–4186. find
  6. Diwan, Anuj et al. (2021). “Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages”. In: Proceedings of Interspeech, pp. 2446–2450. find
  7. ElevenLabs (2025). Models: Scribe v2. ElevenLabs Documentation. Accessed: May 2026. url: https://elevenlabs.io/docs/overview/models#scribe-v2. ↗
  8. Google Cloud (2025). Chirp 3 Transcription: Enhanced Multilingual Accuracy. Google Cloud Speech-to-Text Documentation. Accessed: May 2026. url: https://docs.cloud.google.com/speech-to-text/docs/ models/chirp-3. ↗
  9. Gumperz, John J. (1982). Discourse Strategies. Cambridge: Cambridge University Press. find
  10. Hamed, Injy, Mohamed Elmahdy, and Slim Abdennadher (2017). “Collection and Annotation of Egyptian Arabic-English Code-Switched Data”. In: Proceedings of the Arabic Natural Language Processing Workshop, pp. 124–130. find
  11. Hamed, Injy, Ngoc Thang Vu, and Slim Abdennadher (May 2020). “ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English”. eng. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. Ed. by Nicoletta Calzolari et al. Marseille, France: European Language Resources Association, pp. 4237–4246. isbn: 979-10-95546-34-4. url: https://aclanthology.org/2020.lrec-1.523/. ↗
  12. Levenshtein, Vladimir I. (1966). “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals”. In: Soviet Physics Doklady 10.8, pp. 707–710. find
  13. Lyu, Dau-Cheng et al. (2010). “SEAME: A Mandarin-English Code-Switching Speech Corpus in South-East Asia”. In: Proceedings of Interspeech, pp. 1986–1989. find
  14. Makhoul, John et al. (1999). “Performance Measures for Information Extraction”. In: Proceedings of DARPA Broadcast News Workshop, pp. 249–252. find
  15. McInnes, Leland, John Healy, and James Melville (2018). “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction”. In: arXiv preprint arXiv:1802.03426. ↗ our card
  16. Microsoft Corporation (2024). Implement language identification — Azure AI Speech. Microsoft Azure Documentation. Accessed: May 2026. url: https://learn.microsoft.com/en-us/azure/ai-services/ speech-service/language-identification. ↗
  17. Myers-Scotton, Carol (1997). Duelling Languages: Grammatical Structure in Codeswitching. Oxford: Oxford University Press. find
  18. OpenAI (2025). Introducing Next-Generation Audio Models in the API. OpenAI Blog. Accessed: May 2026. url: https://openai.com/index/introducing-our-next-generation-audio-models/. ↗
  19. Poplack, Shana (1980). “Sometimes I’ll Start a Sentence in Spanish Y TERMINO EN ESPAÑOL: Toward a Typology of Code-Switching”. In: Linguistics 18.7–8, pp. 581–618. find
  20. Radford, Alec et al. (2023). “Robust Speech Recognition via Large-Scale Weak Supervision”. In: Proceedings of ICML. find our card
  21. Reimers, Nils and Iryna Gurevych (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”. In: Proceedings of EMNLP-IJCNLP, pp. 3982–3992. find
  22. Shia, Chi-Jiun et al. (2004). “Language boundary detection and identification of mixed-language speech based on map estimation”. In: 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing. Vol. 1. IEEE, pp. I–381. find
  23. Toshniwal, Shubham et al. (2018). “Multilingual Speech Recognition with a Single End-to-End Model”. In: Proceedings of ICASSP, pp. 4904–4908. find
  24. Watanabe, Shinji et al. (2017). “Hybrid CTC/Attention Architecture for End-to-End Speech Recognition”. In: Proceedings of Interspeech, pp. 3408–3412. find
  25. Xie, Peng et al. (2026). “SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset”. In: Advances in Neural Information Processing Systems 38. find
  26. Zhang, Tianyi et al. (2019). “Bertscore: Evaluating text generation with bert”. In: arXiv preprint arXiv:1904.09675. A. ASR Systems and API Parameters Tables 8 and 9 document the systems evaluated and the exact API parameters used for the benchmark. B. H_Score Formula and Signal Weights Full H_Score composite (Equation 7): ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-commercial-asr-systems-code/