Elamur Research⌕
☰
home / papers / 2026 / Terminology-Aware Retrieval-Augmented Knowledge Distillation for Biomedical Neural Machine Translation

Terminology-Aware Retrieval-Augmented Knowledge Distillation for Biomedical Neural Machine Translation

Summary

Compact student models trained with standard knowledge distillation (KD) keep all their knowledge in fixed parameters. This limits them in specialised domains such as biomedical translation, where parallel data is scarce and specialised terminology must be translated correctly. The authors address this for French-to-English biomedical NMT by combining few-shot KD with retrieval-augmented generation (RAG).

At inference time, a Small Transformer student draws on an external non-parametric memory. The memory holds training sentence pairs and translations produced by the larger teacher. For each input, the student fetches related examples and briefly fine-tunes itself on them, a form of test-time adaptation. The paper compares plain semantic retrieval with a hybrid strategy that ranks candidates by terminology overlap.

The terminology-aware variant beats semantic-only retrieval and the distilled baseline. It reaches or exceeds teacher quality while keeping the speed of a small model. This matters for deploying efficient MT in domains where terminology is critical and data is limited.

Key points

  • The best configuration uses terminology-overlap retrieval over DistilledMemory153K and reaches a mean score of 80.07 at k = 15. The mean combines SacreBLEU, chrF++, COMET, BERTScore and BLEURT.
  • At its peak, this configuration scores chrF++ 84.80 and COMET 90.45.
  • It improves on the strongest semantic retrieval setup by 0.73 points. It improves on the distilled student baseline by 3.24 points, a relative gain of 4.22%.
  • Semantic retrieval (DistilledMemory760K / DistilledMemory153K) tends to keep improving up to k = 31. The terminology-aware method peaks earlier, with k between 12 and 19, and shows diminishing returns beyond that.
  • Bootstrap resampling with compare-mt confirms that the terminology-based students significantly outperform the quantized teacher on SacreBLEU (p < 0.05). Test-time adaptation uses only 10 gradient steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zafar2026terminology,
  title = {{Terminology-Aware Retrieval-Augmented Knowledge Distillation for Biomedical Neural Machine Translation}},
  author = {Zafar, Maria and Bakkali, Souhail and Haque, Rejwanul},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.26/}
}

References

56 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen, Meiqi, Fandong Meng, Yingxue Zhang, Yan Zhang, and Jie Zhou. 2024. Crat: A multi-agent framework for causality-enhanced reflective and retrieval-augmented translation with large language models. find
  2. Conia, Simone, Daniel Lee, Min Li, Umar Farooq Minhas, Saloni Potdar, and Yunyao Li. 2024. Towards cross-cultural machine translation with retrieval-augmented generation from multilingual knowledge graphs. find
  3. Costa-Jussà, Marta R, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  4. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies… find
  5. Fernandez, Jared, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. 2025. Energy considerations of large language model inference and efficiency optimizations. find
  6. Fomicheva, Marina and Lucia Specia. 2019. Taking MT evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515–558, September. find our card
  7. Jin, Xiao, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. 2019. Knowledge distillation via route constrained optimization. find
  8. Junczys-Dowmunt, Marcin, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Liu, Fei and Thamar Solorio,… find our card
  9. Kasai, Jungo, Nikolaos Pappas, Hao Peng, James Cross, and Noah A Smith. 2020. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. arXiv preprint arXiv:2006.10369. find our card
  10. Kim, Yoon and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Su, Jian, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, November. Association for Computational Linguist… find our card
  11. Kim, Sejoon, Mingi Sung, Jeonghwan Lee, Hyunkuk Lim, and Jorge Gimenez Perez. 2024. Efficient terminology integration for LLM-based translation in specialized domains. In Haddow, Barry, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation,… find
  12. Freitag, Markus, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, et al. 2024. Are llms breaking mt metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, p… find our card
  13. Koehn, Philipp. 2004. Statistical significance tests for machine translation evaluation. In Lin, Dekang and Dekai Wu, editors, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain, July. Association for Computational Linguistics. find
  14. Gou, Jianping, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. find
  15. Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. find
  16. Gu, Jiatao, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. find our card
  17. Haque, Rejwanul, Mohammed Hasanuzzaman, and Andy Way. 2020. Analysing terminology translation errors in statistical and neural machine translation. Machine Translation, 34(2):149–195. find
  18. Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. find our card
  19. Li, Juanhui, Sreyashi Nag, Hui Liu, Xianfeng Tang, Sheikh Sarwar, Limeng Cui, Hansu Gu, Suhang Wang, Qi He, and Jiliang Tang. 2025. Learning with less: Knowledge distillation from large language models via unlabeled data. find
  20. Loshchilov, Ilya and Frank Hutter. 2019. Decoupled weight decay regularization. find
  21. Meta. ing ing. 2025. Introducing llama 4: Advanc-multimodal intelligence and reason-https://ai.meta.com/blog/llama-4-multimodal-intelligence/, April. Accessed: 2026-03-26. find
  22. Mirzadeh, Seyed Iman, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191–5198. find our card
  23. Moslem, Yasmin, Rejwanul Haque, John D Kelleher, and Andy Way. 2023. Domain terminology integration into machine translation: Leveraging large language models. arXiv preprint arXiv:2306.01452. find
  24. Moslem, Yasmin. 2025. Efficient speech translation through model compression and knowledge distillation. In Salesky, Elizabeth, Marcello Federico, and Antonis Anastasopoulos, editors, Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 379–388, Vienna,… find
  25. Pang, Jianhui, Fanghua Ye, Derek Fai Wong, Dian Yu, Shuming Shi, Zhaopeng Tu, and Longyue Wang. 2025. Salute the classic: Revisiting challenges of machine translation in the age of large language models. Transactions of the Association for Computational Linguistics, 13:73–95. find
  26. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Isabelle, Pierre, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, P… find our card
  27. Popović, Maja. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Bojar, Ondřej, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, p… find our card
  28. Post, Matt. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. find our card
  29. Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. find
  30. Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. find
  31. Ramos, Rita, Everlyn Asiko Chimoto, Maartje ter Hoeve, and Natalie Schluter. 2025. Grammamt: Improving machine translation with grammar-informed in-context learning. find our card
  32. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Webber, Bonnie, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online, Nov… find our card
  33. Reimers, Nils and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11. find our card
  34. Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  35. Saunders, Danielle. 2022. Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation. Ph.D. thesis, University of Cambridge. find
  36. Sellam, Thibault, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. find
  37. Sun, Siqi, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression. find our card
  38. Sun, Zhiqing, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices. find our card
  39. Sun, Haoxiang, Ruize Gao, Pei Zhang, Baosong Yang, and Rui Wang. 2025. Enhancing machine translation with self-supervised preference data. In Che, Wanxiang, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computati… find
  40. Team, Gemma, Aishwarya Kamath, Johan Ferret, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  41. Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  42. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find
  43. Wang, Fusheng, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. In Zong, Chengqing, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In… find
  44. Wang, Jiaan, Fandong Meng, Yingxue Zhang, and Jie Zhou. 2025. Retrieval-augmented machine translation with unstructured knowledge. find
  45. Xu, Yuhui, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. 2020. Trained rank pruning for efficient deep neural networks. find
  46. Xue, Linting, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin… find
  47. Yang, An, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. 2026. Qwen3.5 technical report: Towards native multimodal agents and specialized reasoning. arXiv preprint arXiv:2602.15000. — [arXiv:2602.15000](https://arxiv.org/abs/2602.15000) find
  48. Zafar, Maria, Souhail Bakkali, and Rejwanul Haque. 2025a. Investigating model compression for neural machine translation in the biomedical domain. In Proceedings of the 33rd International Conference on Artificial Intelligence and Cognitive Science (AICS2025), Dublin, Ireland. find
  49. Zafar, Maria, Patrick J. Wall, Souhail Bakkali, and Rejwanul Haque. 2025b. Confidence-based knowledge distillation to reduce training costs and carbon footprint for low-resource neural machine translation. Applied Sciences, 15(14). find
  50. Zhan, Zaifu, Jun Wang, Shuang Zhou, Jiawen Deng, and Rui Zhang. 2025. Mmrag: Multi-mode retrieval-augmented generation with large language models for biomedical in-context learning. find
  51. Zhang, Ying, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2017. Deep mutual learning. find
  52. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. find
  53. Zhang, Jianyi, Aashiq Muhamed, Aditya Anantharaman, et al. 2023. ReAugKD: Retrieval-augmented knowledge distillation for pre-trained language models. In Rogers, Anna, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguist… find
  54. Zheng, J, H Hong, F Liu, X Wang, J Su, Y Liang, and S Wu. 2024. Dragft: Adapting large language models with dictionary and retrieval augmented finetuning for domain-specific machine translation. arXiv preprint arXiv:2402.15061v2. — [arXiv:2402.15061](https://arxiv.org/abs/2402.15061) find
  55. Zhou, Chunting, Graham Neubig, and Jiatao Gu. 2021. Understanding knowledge distillation in non-autoregressive machine translation. find
  56. Zhou, Wangchunshu, Canwen Xu, and Julian McAuley. 2022. Bert learns to teach: Knowledge distillation with meta learning. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zafar-2026-terminology-aware-retrieval-augmented-knowledge/