Elamur Research⌕
☰
home / papers / 2026 / Scoring the Translation: On Target Automatic Keyword-Based Evaluation of Machine Translation in the Sports Domain

Scoring the Translation: On Target Automatic Keyword-Based Evaluation of Machine Translation in the Sports Domain

Summary

This paper revisits SportsEval, an English→Icelandic test suite submitted to the WMT 2025 General Translation Shared Task. The suite tests how well MT systems handle sports terminology. In the original evaluation, automatic keyword matching scored the top systems far lower than human judges did. The authors set out to find out why and to fix it.

They re-examined all 300 segments, which cover football, basketball, chess, golf and gymnastics. They dropped keywords that are not truly domain terms, capped repetitions, and added alternative Icelandic translations. They also expanded the word-form registry, drawing on the DIM inflection database plus manually listed forms, which matters because Icelandic is highly inflected and compounds freely.

The revised keyword method was run in two variants, whole-word matching and substring matching. Both were compared with manual evaluation of six systems and with the reference-free metrics CometKiwi and GEMBA-MQM.

After revision, automatic scores closely track human judgments. This suggests that a carefully curated keyword approach can reliably evaluate terminology translation into a morphologically rich language. The authors also release English–Icelandic sports term lists.

Key points

  • The test suite has 300 segments: 100 football, 100 basketball, 50 chess, 25 golf and 25 gymnastics. Evaluated term instances were cut from 971 to 717, with 365 keywords appearing once and 145 recurring, at most three times per sport.
  • In the original evaluation, the top four systems scored 47%–55% automatically versus 71%–78% manually. After revision, the gap largely disappears; for example, Gemini-2.5-Pro gets 90.0% with string matching, 88.1% with word matching and 90.7% from human evaluation.
  • Substring matching accounts for compounds containing the term but can produce false positives. Word-token matching is stricter. Both variants are reported.
  • One human evaluator assessed six systems: the top four plus two lower-scoring systems. The two weaker systems served as a check that adding accepted translations based on the top systems' output did not bias the results.
  • The authors make available bilingual English–Icelandic sports terminology lists together with a revised version of the SportsEval suite, and they provide an error analysis of terminology mistakes made by modern MT systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ctpkpj58fwf

@inproceedings{steingrimsson2026scoring,
  title = {{Scoring the Translation: On Target Automatic Keyword-Based Evaluation of Machine Translation in the Sports Domain}},
  author = {Steingrímsson, Steinthór and Sigurdhsson, Einar Freyr},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3ctpkpj58fwf},
  url = {https://doi.org/10.63317/3ctpkpj58fwf}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bjarki Ármannsson, Hinrik Hafsteinsson, Atli Jasonarson, and Steinþór Steingrímsson. 2024. Killing Two Flies with One Stone: An Attempt to Break LLMs Using English→Icelandic Idioms and Proper Names. In Proceedings of the Ninth Conference on Machine Translation (WMT), pages 451–458. Association for Computational Linguistics. ↗
  2. Kristín Bjarnadóttir, Kristín Ingibjörg Hlynsdóttir, and Steinþór Steingrímsson. 2019. DIM: The Database of Icelandic Morphology. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 146–154. Linköping University Electronic Press. find
  3. Lise Lotte Weilgaard Christensen, Hanne Erdman Thomsen, Bodil Nistrup Madsen, Anna-Lena Bucher, Henrik Nilsson, Claudia Dobrina, Håvard Hjulstad, Åsa Holmér, Johan Myking, Anita Nuopponen, Sirpa Suhonen, Anu Ylisalmi, and Ágústa Þorbergsdóttir. 2025. The Nordic Terminology Community. Research and practice. In Terminology throughout History. A discipline in the making, pages 327–364. John Benjamins. ↗
  4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  5. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68. Association for Computational Linguistics. find our card
  6. Steinunn Rut Friðriksdóttir. 2024. The Gender-Queer Test Suite. In Proceedings of the Ninth Conference on Machine Translation (WMT), pages 327–340. Association for Computational Linguistics. find
  7. Selma Dís Hauksdóttir and Steinþór Steingrímsson. 2025. Automated Evaluation for Terminology Translation Related to the EEA Agreement. In Proceedings of the Tenth Conference on Machine Translation (WMT), pages 850–855. Association for Computational Linguistics. find
  8. Marcin Junczys-Dowmunt. 2025. GEMBA-MQM V2: Ten Judgments Are Better Than One. In Proceedings of the Tenth Conference on Machine Translation (WMT), pages 926–933. Association for Computational Linguistics. find
  9. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica M. Lundin, Christof Monz, Kenton Murray, Masaaki Nagata, Stefano Perrella, Lorenzo Proietti, Martin Popel, Maja Popović, Parker Riley, Mariya Shmatova, Steinþór Steingrímsson, Lisa Yankovskaya, and Vilém Zouhar. 2025. Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets. In Proceedings of the Tenth Conference on Machine Translation (WMT), pages 355–413. Association for Computational Linguistics. find our card
  10. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation (WMT), pages 768–775. Association for Computational Linguistics. find our card
  11. Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276– 282. ↗
  12. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. ↗
  13. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. Association for Computational Linguistics. find
  14. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. Association for Computational Linguistics. find our card
  15. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645. Association for Computational Linguistics. ↗ our card
  16. Kirill Semenov, Vilém Zouhar, Tom Kocmi, Dongdong Zhang, Wangchunshu Zhou, and Yuchen Eleanor Jiang. 2023. Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies. In Proceedings of the Eighth Conference on Machine Translation (WMT), pages 663–671. Association for Computational Linguistics. find
  17. Einar Freyr Sigurðsson, Magnús Már Magnússon, Atli Jasonarson, and Steinþór Steingrímsson. 2025. Up to Par? MT Systems Take a Shot at Sports Terminology. In Proceedings of the Tenth Conference on Machine Translation (WMT), pages 856–865. Association for Computational Linguistics. find
  18. Steinþór Steingrímsson, Hrafn Loftsson, and Andy Way. 2023. SentAlign: Accurate and Scalable Sentence Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 256–263. Association for Computational Linguistics. find
  19. Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, and Weining Shen. 2024. SportQA: A Benchmark for Sports Understanding in Large Language Models. ↗
  20. Zhengbang Yang, Haotian Xia, Jingxi Li, Zezhi Chen, Zhuangdi Zhu, and Weining Shen. 2025. Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models Through Question Answering from Text to Video. Electronics, 14(3):461. ↗
  21. F. Yates. 1934. Contingency Tables Involving Small Numbers and the χ 2 Test. Supplement to the Journal of the Royal Statistical Society, 1(2):217– 235. find
  22. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 Shared Task on Quality Estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69–99. Association for Computational Linguistics. find our card
  23. Zeng Zhiliang, Wang Lei, and Liu Qiang. 2024. A method for real-time translation of online video subtitles in sports events. Signal, Image and Video Processing, 19:146. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/steingrimsson-2026-scoring-translation-target-automatic-keyword/