Elamur Research⌕
☰
home / papers / 2026 / How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages

How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages

Paper · 2026 · Anatoly Marchenko

Summary

Many voice-deepfake detectors rely on jitter, shimmer and harmonics-to-noise ratio (HNR). The assumption is that synthetic voices lack the small biological irregularities of human vocal fold vibration. This paper tests whether that still holds for free, scriptable TTS services of the kind used in mass vishing campaigns.

The author built a natural-speech reference from 29 public emotion corpora (24 languages, 11 families, 35,091 utterances). Emotional speech was chosen because phone fraud exploits urgency and distress. The comparison set is 1,850 clips from Google TTS, Microsoft Edge TTS and the macOS system voice. All three features were extracted with Parselmouth. Separability was assessed in two ways: Mann-Whitney U tests with rank-biserial effect sizes, and three classifiers trained on repeated balanced subsamples with cross-validation.

gTTS remains easy to catch because its output is unnaturally clean. Edge TTS, however, approaches human perturbation profiles, and no classifier reaches F1 above 0.78 against it. For anti-fraud and telephony teams, these features are still useful as one signal among several, but they are not a sufficient defense on their own, especially across languages.

Key points

  • Data: gTTS covers 12 languages (159 samples), macOS 9 languages (635), and Edge TTS 21 languages (1,056). The classifiers are logistic regression, SVM-RBF and Random Forest, evaluated with 10 subsamples × 5-fold CV.
  • Classifier results: Against Edge TTS, the strongest model is SVM-RBF (F1 = 0.78, AUROC = 0.85). This is well short of the roughly 0.90 the author considers necessary for a standalone detector.
  • Effect sizes: Moving from gTTS to Edge TTS, effect sizes fall by 2.1× (shimmer), 4.3× (jitter) and 7.4× (HNR). The largest |r| drops from 0.75 (gTTS) to 0.56 (macOS) to 0.36 (Edge TTS).
  • Ablation: Against Edge TTS, single features score F1 = 0.52 (jitter), 0.57 (shimmer) and 0.60 (HNR). Combining all three reaches only 0.75 with Random Forest.
  • Cross-language results: In 14 of the 21 languages, Edge TTS and natural speech cannot be told apart by jitter once Bonferroni correction is applied. A neutral-only baseline check confirms that the emotional corpora do not explain the result.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSpeech & spoken language · 480 works

Cite

Cite as Anatoly Marchenko. 2026. How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages. Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marchenko2026how,
  title = {{How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages}},
  author = {Anatoly Marchenko},
  booktitle = {Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security},
  year = {2026},
  url = {https://aclanthology.org/2026.nlpaics-1.18/}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We tested three commodity TTS engines against perturbation-based detection across 21 languages (1,850 synthetic vs. 35,091 natural samples from 29 corpora). Three classifiers and ablation analysis all point the same way: effect sizes shrink 2–7× from older to newer TTS, Edge TTS reaches only small e… find
  2. Yannick Jadoul, Bill Thompson, and Bart de Boer. 2018. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics, 71:1–15. find
  3. Aurosweta Mahapatra, Ismail Rasim Ulgen, Abinay Reddy Naini, Carlos Busso, and Berrak Sisman. 2025. Can emotion fool anti-spoofing? In Proc. Interspeech. find
  4. Nicolas M. Müller, Pavel Czempin, Franziska Diekmann, Adam Froghyar, and Konstantin Böttinger. 2022. Does audio deepfake detection generalize? In Proc. Interspeech, pages 2783–2787. find
  5. Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. 2021. End-to-end anti-spoofing with RawNet2. In Proc. ICASSP, pages 6369–6373. find
  6. Ingo R. Titze. 1994. Principles of Voice Production. Prentice Hall. find
  7. Xin Wang, Héctor Delgado, Hemlata Tak, Jee-weon Jung, Hye-jin Shim, Massimiliano Todisco, Ivan Kukanov, Xuechen Liu, Md Sahidullah, Tomi H. Kinnunen, Nicholas Evans, Kong Aik Lee, and Junichi Yamagishi. 2024. ASVspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale. In Proc. find
  8. ASVspoof 2024 Workshop, pages 1–8. find
  9. Kevin Warren, Daniel Olszewski, Seth Layton, Kevin Butler, Carrie Gates, and Patrick Traynor. 2025. Pitch imperfect: Detecting audio deepfakes through acoustic prosodic analysis. arXiv preprint arXiv:2502.14726. — [arXiv:2502.14726](https://arxiv.org/abs/2502.14726) find
  10. Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and Héctor Delgado. 2021. find
  11. ASVspoof 2021: Accelerating progress in spoofed and deepfake speech detection. In Proc. find
  12. ASVspoof 2021 Workshop. find
  13. Tianle Yang, Chengzhe Sun, Siwei Lyu, and Phil Rose. 2026. Forensic deepfake audio detection using segmental speech features. Forensic Science International, 379:112768. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marchenko-2026-how-well-do-commodity-text/