Elamur Research⌕
☰
home / papers / 2025 / GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning

GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning

Summary

GrammaMT tackles machine translation for low-resource and endangered languages, where large LLMs lack training data and fine-tuning is impractical. The authors use Interlinear Glossed Text (IGT), the linguist's standard triplet of source sentence, morpheme-by-morpheme gloss, and translation, as extra context in few-shot prompts. They propose three training-free prompting strategies:

  • gloss-shot adds glosses to the in-context examples.
  • chain-gloss has the LLM write a gloss for the input before translating.
  • model-gloss supplies a gloss produced by an external glossing model, GlossLM.

The main model is Llama-3-70B-Instruct, with additional runs on smaller and other models. Evaluation covers three benchmarks: the GlossLM IGT corpus, the 2023 SIGMORPHON Shared Task data on endangered languages, and FLORES as an out-of-domain test. Translation is mostly into English. The grammar-aware prompts improve over standard prompting across low- and high-resource languages. An oracle study with gold glosses shows much larger potential gains. The method matters because it needs only a couple dozen annotated examples, which linguists often already have.

Key points

  • Models and setup: Llama-3-70B-Instruct, Llama-3-8B-Instruct and Mixtral-8x22B-Instruct-v0.1 were run with 4-bit quantization on a single A100 80GB GPU. GPT-4o is reported in an appendix.
  • Number of examples: About N = 21 glossed examples was optimal. Chain-gloss in particular benefits from more demonstrations.
  • Gloss quality: Llama-3 70B reaches under 21% word accuracy when glossing Tsez (Ddo), versus up to 88% for GlossLM. This gap explains why model-gloss performs better.
  • Oracle study: Gold glosses add about 17.46 BLEU on average over few-shot prompting. Zero-shot with a gold gloss gains about 16.02 BLEU over plain zero-shot and even beats few-shot.
  • Role of grammatical labels: Removing the grammatical labels and keeping only the lemmata lowers BLEU. The approach also showed gains when translating from English on SIGMORPHON languages (Gitksan, Lezgi, Natügu, Tsez).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1447

@inproceedings{ramos2025grammamt,
  title = {{GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning}},
  author = {Ramos, Rita and Chimoto, Everlyn Asiko and Hoeve, Maartje Ter and Schluter, Natalie},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1447},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1447}
}

References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021. find
  3. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. Preprint, arXiv:1710.04087. ↗
  4. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023a. Self-alignment with instruction back-translation. arXiv preprint arXiv:2308.06259. ↗ our card
  5. Yiming Cui, Wanxiang Che, Shijin Wang, and Ting Liu. 2022. Lert: A linguistically-motivated pre-trained language model. Preprint, arXiv:2211.05344. ↗
  6. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  7. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, pages 10867–10878. PMLR. ↗
  8. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation. Preprint, arXiv:2302.07856. ↗
  9. Michael Ginn, Sarah Moeller, Alexis Palmer, Anna Stacey, Garrett Nicolai, Mans Hulden, and Miikka Silfverberg. 2023. Findings of the SIGMORPHON 2023 shared task on interlinear glossing. pages 186– 201, Toronto, Canada. find
  10. Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, and Lori Levin. 2024. Glosslm: Multilingual pretraining for low-resource interlinear glossing. arXiv preprint arXiv:2403.06399. ↗
  11. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. ↗
  12. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗ our card
  14. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. Preprint, arXiv:2302.09210. ↗
  15. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  16. Meta. 2024. Llama 3 model card. find
  17. Mistral. 2024. Mixtral-8x22b. find
  18. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276. ↗ our card
  19. Preprint, Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. find our card
  20. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of ChatGPT for machine translation. Singapore. ↗ our card
  21. Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612–618. find our card
  22. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. ↗ our card
  23. Nooshin Pourkamali and Shler Ebrahim Sharifi. 2024. Machine translation with large language models: Prompt engineering for persian, english, and russian directions. Preprint, arXiv:2401.08429. ↗
  24. Ratish Puduppully, Anoop Kunchukuttan, Raj Dabre, Ai Ti Aw, and Nancy F Chen. 2023. Decomposed prompting for machine translation between related languages using large language models. arXiv preprint arXiv:2305.13085. ↗
  25. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Unbabel’s participation in the WMT20 metrics shared task. pages 911–920, Online. find our card
  26. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. ChatGPT MT: Competitive for high- (but not low-) resource languages. pages 392–418, Singapore. find our card
  27. Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically guided neural machine translation. pages 299–305, Berlin, Germany. find
  28. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. pages 5027–5038, Brussels, Belgium. find our card
  29. Zewei Sun, Qingnan Jiang, Shujian Huang, Jun Cao, Shanbo Cheng, and Mingxuan Wang. 2022. Zero-shot domain adaptation for neural machine translation with retrieved phrase-level prompts. arXiv preprint arXiv:2209.11409. ↗
  30. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations. ↗
  31. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. ↗ our card
  32. Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. 2023b. Machine translation with large language models: Prompting, few-shot learning, and fine-tuning with QLoRA. Singapore. find
  33. Zhong Zhou, Lori Levin, David R. Mortensen, and Alex Waibel. 2020. Using interlinear glosses as pivot in low-resource multilingual machine translation. Preprint, arXiv:1911.02709. ↗
  34. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. Preprint, arXiv:2304.04675. ↗ our card
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022a. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903. ↗ our card
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. find our card
  37. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations. find our card
  38. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Preprint, arXiv:2105.13626. ↗
  39. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020. ↗
  40. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. ↗
  41. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, pages 41092–41110. PMLR. ↗ our card
  42. Kexun Zhang, Yee Man Choi, Zhenqiao Song, Taiqi He, William Yang Wang, and Lei Li. 2024. Hire a linguist!: Learning endangered languages with in-context linguistic descriptions. Preprint, arXiv:2402.18025. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ramos-2025-grammamt-improving-machine-translation-grammar/