Elamur Research⌕
☰
home / papers / 2026 / Mimicking Neural Machine Translation History for Pedagogic Reasons

Mimicking Neural Machine Translation History for Pedagogic Reasons

Summary

The paper tackles a teaching problem: translation students meeting machine translation for the first time struggle to understand today's complex pretrained systems. The author proposes rebuilding the evolution of NMT step by step. The setting is the KU Leuven course Machine Translation Advanced Topics, which has 27 contact hours and 3 ECTS. Because OpenNMT is no longer developed, he wrote his own Python scripts and public Kaggle notebooks, shared on GitHub (VincentCCL/MTAT).

Every system uses the same small Tatoeba English–Dutch corpus from OPUS and is scored with SacreBLEU on the development set. After each epoch, the scripts also print example translations, so students see progress both in numbers and in actual output. The progression covers:

  • RNN models trained from scratch
  • gated cells, bidirectionality, attention and SentencePiece
  • Transformers
  • a multilingual zero-shot system
  • fine-tuned pretrained encoder-decoders
  • prompting of decoder-only LLMs

Quality rises fairly steadily along this sequence. The approach matters because it gives non-technical translators hands-on insight into MT methodology and the role of data. It relies only on free compute, though it needs close individual support and therefore suits small groups.

Key points

  • Data: Tatoeba v2023-04-12 EN–NL has 79,541 sentence pairs, reduced to 79,381 after cleaning (duplicates, length and ratio filters). Development and test sets hold 1,000 pairs each, and the test set is reserved for students' final evaluation.
  • From-scratch models: BLEU climbs from vanilla RNN through LSTM/GRU, biLSTM and attention.
    • LSTM and GRU perform about the same.
    • SentencePiece gives no extra gain, though a joint model beats separate models.
    • The large Transformer makes only one error on the five tracked examples.
  • Multilingual model: a target-language-tagged Transformer trained on EN–NL plus FR–NL data demonstrates zero-shot EN→FR translation. The output is partly correct.
  • Pretrained encoder-decoders:
    • Fine-tuned mBART scores best among the fine-tuned systems.
    • Without fine-tuning, M2M reaches about 41 BLEU, NLLB about 53, and MADLAD-400 56.37, the highest overall.
    • The author warns that Tatoeba data may have leaked into these models' pretraining.
  • Decoder-only models:
    • GPT-2, LLaMA and Mistral are tested with zero-shot, formatted, few-shot and chat prompts, partly through Hugging Face, Blablador and Groq endpoints.
    • These models tend to overgenerate text beyond the translation itself.
    • Everything is designed to fit within Kaggle's free limit of 30 GPU hours per week.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vandeghinste2026mimicking,
  title = {{Mimicking Neural Machine Translation History for Pedagogic Reasons}},
  author = {Vandeghinste, Vincent},
  booktitle = {Proceedings of the 1st International Workshop on Teaching AI-Based Translation and Technologies (TAITT 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.taitt-1.3/}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cho, Kyunghyun, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734. Association for Computational Linguistics. ↗
  2. Fan, Angela, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(1):4839–4886. find
  3. Groq Inc. 2024. Groq API documentation. https://console.groq.com/. Accessed: 2026-04-03. ↗
  4. Hackenbuchner, Janiça and Ralph Krüger. 2023. DataLitMT – Teaching data literacy in the context of machine translation literacy. In Nurminen, Mary, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartı́n, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz, editors, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 285–293, Tampere. European Association for Machine Translation. find
  5. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  6. Jiang, Albert Q., Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv. find
  7. Klein, Guillaume, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Bansal, Mohit and Heng Ji, editors, Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver. Association for Computational Linguistics. find our card
  8. Kudo, Taku and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 66–71, Brussels. Association for Computational Linguistics. find
  9. Kudugunta, Sneha, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A multilingual and document-level large audited dataset. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook. Curran Associates Inc. find our card
  10. Liu, Yinhan, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. In Johnson, Mark, Brian Roark, and Ani Nenkova, editors, Transactions of the Association for Computational Linguistics, Volume 8, pages 726–742, Cambridge. MIT Press. ↗
  11. Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon. Association for Computational Linguistics. ↗
  12. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  13. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Isabelle, Pierre, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311– 318, Philadelphia. Association for Computational Linguistics. find our card
  14. Post, Matt. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels. Association for Computational Linguistics. find our card
  15. Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  16. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485–5551. ↗
  17. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Webber, Bonnie, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  18. Sellam, Thibault, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Jurafsky, Dan, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  19. Strube, Alexandre. 2024. Helmholtz Blablador and the LLM models’ ecosystem. EuroSciPy 2024, Szczecin. find
  20. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS’14: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, pages 3104–3112, Cambridge, MA. Curran Associates, Inc. ↗
  21. Tiedemann, Jörg. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul. European Language Resources Association (ELRA). find
  22. Tiedemann, Jörg. 2020. The Tatoeba translation challenge – Realistic data sets for low resource and multilingual MT. In Barrault, Loı̈c, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors, Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online. Association for Computational Linguistics. find our card
  23. Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  24. Vanroy, Bram, Arda Tezcan, and Lieve Macken. 2023. MATEO: MAchine Translation Evaluation Online. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 389–393, Tampere. European Association for Machine Translation. find
  25. Xue, Linting, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  26. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations (ICLR 2020). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vandeghinste-2026-mimicking-neural-machine-translation-history/