ArFake: A Robust Framework for Multi-Dialect Arabic Speech Spoofing Detection Benchmark

Summary

Neural text-to-speech systems now produce voices that are hard to tell apart from real ones. Anti-spoofing benchmarks such as ASVspoof mostly cover English, so Arabic dialects have had almost no dedicated test resources. ArFake addresses this with a pipeline for both creating and detecting fake Arabic speech across eight dialect regions.

The authors take real utterances from the Casablanca corpus and resynthesize them with four TTS/voice-cloning systems. They judge the fakes three ways: how easily classifiers separate them from real audio, Whisper ASR word error rate, and human MOS ratings. From the bona fide audio and three of the generators they build a mixed corpus. On it they train detectors of two kinds: small heads on frozen speech-model embeddings, and classical classifiers on MFCC features.

Robustness is tested in three settings: in-domain, against a generator never seen in training, and against a dialect never seen in training. The reported in-domain and unseen-generator results reach about 96% and 97%. The released dataset and protocols form a first reference point for securing Arabic speech against deepfakes.

Key points

Source data is Casablanca: eight Arabic dialect regions, roughly six hours each. The spoofs come from XTTS-v2, FishSpeech, ArTST and VITS, and each generator is treated as a separate attack family.
The final corpus has 54,413 utterances, with 31,302 for train+validation and 23,111 for test. Training is about 30% bona fide and 70% spoofed, with per-generator sampling rates that differ. VITS is fully held out for the cross-generator (LOGO) test.
When a detector is trained per generator, FishSpeech fakes are the hardest to catch. Whisper-large reaches 6.92% EER on them, against 14.92% for the RawNet2 baseline. HuBERT-base gets 0.07% EER on XTTS-v2. ArTST and VITS fakes are separated almost perfectly, even by MFCC-based SVMs.
Detector heads are two feed-forward layers with ReLU and dropout 0.5, trained with Adam at lr 1e-3. They sit on top of HuBERT-base, wav2vec2, Whisper-small or Whisper-large embeddings.
The human evaluation used 12 native Arabic speakers rating realism on a 1–5 scale. Each generator contributed eight clips, one per dialect.

Interesting ideas

I1. The paper uses how easily a classifier tells synthetic output from genuine data as a measure of distance from the real distribution: the easier the separation, the worse the generator. — For Mova (back): Train a simple discriminator between human-written sentences and sentences regenerated from the meaning graph, and track its accuracy as a naturalness score for generation.
I2. Under Leave-One-Generator-Out, an entire source of data is withheld from training and used only at test time, which mimics meeting a brand-new system in deployment. — For Mova (learn): Drop one witness parser or one LLM teacher tier when building silver data, then check whether the trained parser still holds up on that source's style of output.
I3. Under Leave-One-Dialect-Out, training covers all varieties but one, and testing uses the excluded one, so robustness to linguistic shift is measured directly. — For Mova (tree): Train gparse with one UD treebank or genre held out, to see how well the parser and the UD dialect conversion transfer to unseen text varieties.
I4. The authors note that near-perfect scores on weak generators probably come from obvious artifacts rather than real detection skill, so suspiciously high numbers deserve suspicion. — For Mova (learn): Treat sudden perfect scores as a signal to run negative controls, checking for shortcut cues before trusting the result.
I5. ASR word error rate against the original transcript is used as a diagnostic and gating signal for generated audio, not as a measure of realism. — For Mova (back): In the text → graph → text compressor, reparse each regenerated sentence and compare its graph to the source graph, rejecting outputs that drift.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.22808 · arXiv 2509.22808

@misc{elsetohy2025arfake,
  title = {{ArFake: A Robust Framework for Multi-Dialect Arabic Speech Spoofing Detection Benchmark}},
  author = {Elsetohy, Mohamed and Ehab, Alhassan and Mekky, Ali and Hassan, Besher and Shehata, Shady},
  year = {2025},
  doi = {10.48550/arxiv.2509.22808},
  eprint = {2509.22808},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.22808}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

B. Talafha, K. Kadaoui, S. M. Magdy, M. Habiboullah, C. M. Chafei, A. O. El-Shangiti, H. Zayed, R. Alhamouri, R. Assi, A. Alraeesi et al., “Casablanca: Data and models for multidialectal arabic speech recognition,” arXiv preprint arXiv:2410.04527, 2024. ↗
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” 2016. [Online]. Available: https://arxiv.org/abs/1609.03499 ↗ our card
Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” 2017. [Online]. Available: https://arxiv.org/abs/1703.10135 ↗
C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,” 2023. [Online]. Available: https://arxiv.org/abs/2301.02111 ↗ our card
M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, and W.-N. Hsu, “Voicebox: Text-guided multilingual universal speech generation at scale,” 2023. [Online]. Available: https://arxiv.org/abs/2306.15687 ↗
E. Casanova, K. Davis, E. Gölge, G. Göknar, I. Gulea, L. Hart, A. Aljafari, J. Meyer, R. Morais, S. Olayemi, and J. Weber, “Xtts: a massively multilingual zero-shot text-to-speech model,” 2024. [Online]. Available: https://arxiv.org/abs/2406.04904 ↗
S. Liao, Y. Wang, T. Li, Y. Cheng, R. Zhang, R. Zhou, and Y. Xing, “Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis,” 2024. [Online]. Available: https://arxiv.org/abs/2411.01156 ↗
X. Wang, H. Delgado, H. Tak, J. weon Jung, H. jin Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen, N. Evans, K. A. Lee, and J. Yamagishi, “Asvspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,” 2024. [Online]. Available: https://arxiv.org/abs/2408.08739 ↗
J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans, and H. Delgado, “Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,” 2021. [Online]. Available: https://arxiv.org/abs/2109.00537 ↗
A. Nautsch, X. Wang, N. Evans, T. H. Kinnunen, V. Vestman, M. Todisco, H. Delgado, M. Sahidullah, J. Yamagishi, and K. A. Lee, “Asvspoof 2019: Spoofing countermeasures for the detection of synthesized, converted and replayed speech,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 3, no. 2, p. 252–265, Apr. 2021. [Online]. Available: http://dx.doi.org/10.1109/TBIOM.2021.3059479 ↗
N. M. Müller, P. Czempin, F. Dieckmann, A. Froghyar, and K. Böttinger, “Does audio deepfake detection generalize?” 2024. [Online]. Available: https://arxiv.org/abs/2203.16263 ↗
X. Liu, X. Wang, M. Sahidullah, J. Patino, H. Delgado, T. Kinnunen, M. Todisco, J. Yamagishi, N. Evans, A. Nautsch, and K. A. Lee, “Asvspoof 2021: Towards spoofed and deepfake speech detection in the wild,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, p. 2507–2522, 2023. [Online]. Available: http://dx.doi.org/10.1109/TASLP.2023.3285283 ↗
L. Zhang, X. Wang, E. Cooper, N. Evans, and J. Yamagishi, “The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, p. 813–825, 2023. [Online]. Available: http://dx.doi.org/10.1109/TASLP.2022.3233236 ↗
Z. M. Almutairi and H. Elgibreen, “Detecting fake audio of arabic speakers using self-supervised deep learning,” IEEE Access, vol. 11, pp. 72 134–72 147, 2023. find
R. Awaad, A. Hedya, M. Fadel, and A. B. Zaky, “Real-time deepfake detection for ai-generated arabic speech,” in 2024 6th Novel Intelligent and Leading Emerging Sciences Conference (NILES), 2024, pp. 198–201. find
N. M. Müller, P. Kawa, W. H. Choong, E. Casanova, E. Gölge, T. Müller, P. Syga, P. Sperl, and K. Böttinger, “Mlaad: The multi-language audio anti-spoofing dataset,” 2025. [Online]. Available: https://arxiv.org/abs/2401.09512 ↗
K. Kuckreja, P. Gupta, I. Hamed, T. Solorio, M. H. Khan, and A. Dhall, “Tell me habibi, is it real or fake?” 2025. [Online]. Available: https://arxiv.org/abs/2505.22581 ↗
A. Khan, K. M. Malik, and S. Nawaz, “Frame-to-utterance convergence: A spectra-temporal approach for unified spoofing detection,” in ICASSP 2024 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 761–10 765. [Online]. Available: https://arxiv.org/pdf/2309.09837 ↗
H. Toyin, A. Djanibekov, A. Kulkarni, and H. Aldarmaki, “Artst: Arabic text and speech transformer,” in Proceedings of Arabic-NLP 2023, 2023, pp. 41–51. find
J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” 2021. [Online]. Available: https://arxiv.org/abs/2106.06103 ↗
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elsetohy-2025-arfake-robust-framework-multi-dialect/