Elamur Research⌕
☰
home / papers / 2026 / A Fine-Grained Linguistic Evaluation of Low-Resource Luxembourgish–English MT

A Fine-Grained Linguistic Evaluation of Low-Resource Luxembourgish–English MT

Summary

Standard MT evaluation scores systems with metrics averaged over random sentences. This ranks models but hides which errors they make. The problem is acute for Luxembourgish (LB), a low-resource language with little parallel data, where evaluation has relied only on reference-based metrics. The author builds the Lux-MT-Test-Suite, the first fine-grained challenge set for LB-EN translation. Most items are adapted from Macketanz et al.'s (2022a) DE-EN test suite. The rest come from the Luxembourgish Online Dictionary (LOD), Döhmer (2020) and the native-speaker author. Outputs are scored semi-automatically: token and regex rules first, then manual review of unmatched outputs. The suite compares three systems: Gemma 3, a fine-tuned version called LuxMT, and GPT-5. Fine-tuning improved many categories, mainly morphology and syntax, with smaller lexical gains. Some items that the base model handled correctly were mistranslated afterwards. GPT-5 stayed ahead overall. All systems still translate too literally (translationese), struggle with idioms and show lexical gaps. The resource helps developers find specific weaknesses rather than relying on opaque scores.

Key points

  • Lux-MT-Test-Suite contains 896 test items in 12 linguistic categories and 36 phenomena, with at least five items per phenomenon. Categories include Ambiguity, False friends, MWE, Verb valency and Subordination. The suite is released on GitHub.
  • The base model is instruction-tuned Gemma 3 with 27B parameters, 8-bit quantised. It was fine-tuned with Unsloth for one epoch at a learning rate of 2e-5.
  • Training data came from LuxAlign (RTL News), which has 89k LB-FR and 28k LB-EN pairs. Filtering with LuxEmbedder at cosine 0.99 left 14k LB-FR and 2.5k LB-EN pairs. Parliamentary transcripts translated with Google Translate and filtered at 0.98 added 20k LB-EN and 18k LB-FR pairs.
  • GPT-5 outperformed the best local model with statistical significance in 5 of the 12 categories. The models also showed different error strategies, such as treating unknown nouns as named entities or leaning on a related language.
  • Limitations:
    • Annotation relied on a single annotator, so no agreement score is available.
    • Correctness is judged as a simple pass/fail.
    • Public examples such as the LOD sentences may have leaked into training data.
    • Evaluation is sentence-level only.
    • Planned extensions are an EN-LB direction and more test items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loresmt-1.12

@inproceedings{rehlinger2026fine,
  title = {{A Fine-Grained Linguistic Evaluation of Low-Resource Luxembourgish–English MT}},
  author = {Rehlinger, Nils},
  booktitle = {Proceedings for the Ninth Workshop on Technologies for Machine Translation of Low Resource Languages (LoResMT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loresmt-1.12},
  url = {https://doi.org/10.18653/v1/2026.loresmt-1.12}
}

References

47 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dimitra Anastasiou. 2022. Enrich4all: A first luxembourgish bert model for a multilingual chatbot. In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages, pages 207–212. find
  2. Mariana Avelino, Vivien Macketanz, Eleftherios Avramidis, and Sebastian Möller. 2022. A test suite for the evaluation of portuguese-english machine translation. In International Conference on Computational Processing of the Portuguese Language, pages 15–25. Springer. find
  3. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, Aljoscha Burchardt, and Sebastian Möller. 2020. Fine-grained linguistic evaluation for state-of-the-art machine translation. arXiv preprint arXiv:2010.06359. ↗
  4. Laura Bernardy. 2022. A Luxembourgish GPT-2 Approach Based on Transfer Learning. Ph.D. thesis, Master’s thesis, University of Trier. find
  5. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  6. Caroline Döhmer. 2018. A new perspective on the luxembourgish genitive. In Germanic genitives, pages 15–36. John Benjamins Publishing Company. ↗
  7. Caroline Döhmer. 2020. Aspekte der luxemburgischen Syntax. BoD–Books on Demand. ↗
  8. Nathalie Entringer, Peter Gilles, Sara Martin, and Christoph Purschke. 2021. Schnëssen. surveying language dynamics in luxembourgish with a mobile research app. Linguistics Vanguard, 7(s1):20190031. ↗
  9. Fernand Fehlen and Andreas Heinz. 2016. Die Luxemburger Mehrsprachigkeit: Ergebnisse einer Volkszäh-lung. transcript Verlag. find
  10. Martin Gellerstam. 1986. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1:88–95. find
  11. Daniela Gierschek. 2022. Detection of sentiment in luxembourgish user comments. find
  12. Peter Gilles, Léopold Edem Ayité Hillah, and Nina HOSSEINI KIVANANI. 2023a. Asrlux: Automatic speech recognition for the low-resource language luxembourgish. In 20. International Conference of Phonetic Sciences (ICPhS). Guarant International, Prague, Unknown/unspecified. find
  13. Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  14. Ulrich Heid and Elke Hildenbrand. 1991. Some practical experience with the use of test suites for the evaluation of systran. In the Proceedings of the Evaluators’ Forum, Les Rasses. Citeseer. find
  15. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. arXiv preprint arXiv:1704.07431. ↗ our card
  16. Juraj Juraska, Tobias Domhan, Mara Finkelstein, Tetsuji Nakagawa, Geza Kovacs, Daniel Deutsch, Pidong Wang, and Markus Freitag. 2025. MetricX-25 and GemSpanEval: Google Translate submissions to the WMT25 evaluation shared task. In Proceedings of the Tenth Conference on Machine Translation, pages 957–968, Suzhou, China. Association for Computational Linguistics. ↗
  17. Margaret King and Kirsten Falkedal. 1990. Using test suites in evaluation of machine translation systems. In COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics. find our card
  18. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. ↗ our card
  19. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, and 1 others. 2025. Findings of the wmt25 general machine translation shared task: Time to stop evaluating on easy test sets. In Proceedings of the Tenth Conference on Machine Translation, pages 355–413. find our card
  20. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch. 2025. Findings of the WMT25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help. In Proceedings of the Tenth Conference on Machine Translation, pages 436– 483, Suzhou, China. Association for Computational Linguistics. find
  21. Danni Liu and Jan Niehues. 2025. Conditions for catastrophic forgetting in multilingual translation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 347–359. find
  22. Arle Lommel, Serge Gladkoff, Alan K Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, and 1 others. 2024. The multi-range theory of translation quality measurement: Mqm scoring models and statistical quality control. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 75–94. find
  23. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica, (12):0455–463. ↗
  24. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. Luxembert: Simple and practical data augmentation in language model pre-training for luxembourgish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5080–5089. find
  25. Anne-Marie Lutgen, Alistair Plum, Christoph Purschke, and Barbara Plank. 2025. Neural text normalization for luxembourgish using real-life variation data. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 115–127. ↗
  26. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, He Wang, Renlong Ai, Shushen Manakhimova, Ursula Strohriegel, Sebastian Möller, and Hans Uszkoreit. 2022a. A linguistically motivated test suite to semi-automatically evaluate german– english machine translation output. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 936–947. find
  27. Vivien Macketanz, Shushen Manakhimova, Eleftherios Avramidis, Ekaterina Lapshinova-koltunski, Sergei Bagdasarov, and Sebastian Möller. 2022b. Linguistically motivated evaluation of the 2022 state-of-the-art machine translation systems for three language directions. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 432–449, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  28. Shushen Manakhimova, Maria Kunilovskaya, Ekaterina Lapshinova-Koltunski, and Eleftherios Avramidis. 2025. Fine-grained evaluation of english-russian mt in 2025: Linguistic challenges mirroring human translator training. In Proceedings of the Tenth Conference on Machine Translation, pages 866–877. find
  29. Andrianos Michail, Corina Raclé, Juri Opitz, and Simon Clematide. 2025. Adapting multilingual embedding models to historical luxembourgish. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025), pages 291–298. ↗
  30. Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sennrich, and Liane Guillou. 2025. Machine translation meta evaluation through translation accuracy challenge sets. Computational Linguistics, 51(1):73–137. ↗ our card
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  32. Fred Philippy, Laura Bernardy, Siwen Guo, Jacques Klein, and Tegawendé F Bissyandé. 2025a. Luxin-struct: A cross-lingual instruction tuning dataset for luxembourgish. arXiv preprint arXiv:2510.07074. ↗
  33. Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Towards a common understanding of contributing factors for cross-lingual transfer in multilingual language models: A review. arXiv preprint arXiv:2305.16768. ↗
  34. Fred Philippy, Siwen Guo, Jacques Klein, and Tegawende Bissyande. 2025b. Luxembedder: A cross-lingual approach to enhanced luxembourgish sentence embeddings. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11369–11379. ↗
  35. Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, and Christoph Purschke. 2024. Luxbank: The first universal dependency treebank for luxembourgish. arXiv preprint arXiv:2411.04813. ↗
  36. Alistair Plum, Tharindu Ranasinghe, and Christoph Purschke. 2025. Text generation models for luxembourgish with limited data: A balanced multilingual strategy. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 93–104. ↗
  37. Lorenzo Proietti, Stefano Perrella, and Roberto Navigli. 2025. Has machine translation evaluation achieved human parity? the human reference and the limits of progress. arXiv preprint arXiv:2506.19571. ↗
  38. Tharindu Ranasinghe, Alistair Plum, Christoph Purschke, and Marcos Zampieri. 2023. Publish or hold? automatic comment moderation in luxembourgish news articles. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 968–978. find
  39. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787. find our card
  40. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7881– 7892. find
  41. Yewei Song, Saad Ezzini, Jacques Klein, Tegawende Bissyande, Clément Lefebvre, and Anne Goujon. 2023. Letz translate: Low-resource machine translation for luxembourgish. In 2023 5th International Conference on Natural Language Processing (IC-NLP), pages 165–170. IEEE. find
  42. Yewei Song, Lujun Li, Cedric Lothritz, Saad Ezzini, Lama Sleem, Niccolo Gentile, Radu State, Tegawendé F Bissyandé, and Jacques Klein. 2025. Is llm the silver bullet to low-resource languages machine translation? arXiv preprint arXiv:2503.24102. ↗
  43. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
  44. Jörg Tiedemann and Santhosh Thottingal. 2020. Opus-mt–building open translation services for the world. In Annual Conference of the European Association for Machine Translation, pages 479–480. European Association for Machine Translation. find
  45. Julian Valline, Cedric Lothritz, and Jordi Cabot. 2025. Luxit: A luxembourgish instruction tuning dataset from monolingual seed data. arXiv preprint arXiv:2510.24434. ↗
  46. Andrew Way. 1991. Developer-oriented evaluation of mt systems. In Proceedings of the evaluators’ forum, pages 237–244. find
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/rehlinger-2026-fine-grained-linguistic-evaluation-low/