Elamur Research⌕
☰
home / papers / 2026 / Tie-Calibrated COMETKiwi for Speech Translation Quality Estimation: IWSLT2026 Metrics Track

Tie-Calibrated COMETKiwi for Speech Translation Quality Estimation: IWSLT2026 Metrics Track

Summary

This IWSLT 2026 Speech Translation Metrics shared task submission tackles reference-free quality estimation (QE) for English→German and English→Chinese speech translation. The system uses no audio. The frozen COMETKiwi-22 model scores whisper-large-v3 ASR transcripts against MT hypotheses, and a post-processing step called tie calibration follows. Tie calibration rounds scores into buckets of width ϵ, so hypotheses with almost identical scores become exact ties. Kendall τb ignores ties, so these near-ties no longer count as arbitrary misorderings. ϵ is chosen separately for each target language by grid search over multiples of the training-score standard deviation, maximizing per-document τb on the training split only.

On the development set (IWSLT 2025 ACL Talks), average segment-level τb rises from 34.6% to 39.4%, ahead of the SpeechQE and BLASER 2.0 QE baselines. System-level SPA stays roughly unchanged. Chinese benefits more because COMETKiwi's within-document scores cluster more tightly there. The method is cheap and model-agnostic and needs no retraining. However, ASR errors pass undetected, and ϵ must be re-tuned for each new domain, language or QE model.

Key points

  • Dev-set results for segment-level τb: the primary system averages 39.4%, outperforming the baselines COMETKiwi (34.6%), SpeechQE (29.2%) and BLASER 2.0 QE (24.4%). System-level SPA is 88.0%, versus 89.4% for COMETKiwi and 86.0% for SpeechQE.
  • Gains by language pair: +6.7 τb for en-zh and about +3.0 for en-de (+2.7 relative to the authors' reproduced COMETKiwi).
  • Selected bucket widths: ϵ* = 3.497 for en-de and 2.756 for en-zh, both at k = 0.20, which is the upper edge of the search grid. τb grew monotonically with k across the sweep.
  • Mean within-document COMETKiwi σ is 5.49 for en-zh versus 8.31 for en-de, which the authors give as the reason for the larger Chinese gain.
  • Data: training labels come from IWSLT 2023, WMT 2024 and WMT 2025. The test set has 48,044 EN→{DE, ZH} examples across five domains, 64.5% of them ACL talks. Ablations found round bucketing better than floor or ceil, and per-language ϵ better than a shared ϵ for τb.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.iwslt-1.36

@inproceedings{shah2026tie,
  title = {{Tie-Calibrated COMETKiwi for Speech Translation Quality Estimation: IWSLT2026 Metrics Track}},
  author = {Shah, Mubashir Hussain and Fatima, Aymen and Choi, Kiho and Jang, Daehee},
  booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.iwslt-1.36},
  url = {https://doi.org/10.18653/v1/2026.iwslt-1.36}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Sebastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kasztelnik, Antoine Laurent, and 33 others. 2026. Speech translation and metrics in 2026: Findings of the iwslt campaign. In Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026), San Diego, California, US. Association for Computational Linguistics. find
  2. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69–99, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  3. Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, Mrinmaya Sachan, and Jan Niehues. 2026. Early-exit and instant confidence translation quality estimation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 55–76, Rabat, Morocco. Association for Computational Linguistics. find our card

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2026-tie-calibrated-cometkiwi-speech-translation/