Elamur Research⌕
☰
home / papers / 2024 / Do Multilingual Language Models Think Better in English?

Do Multilingual Language Models Think Better in English?

Paper · 2024 · Julen Etxaniz, Gorka Azkune, Aitor Soroa et al.

Summary

Translate-test (machine-translating non-English inputs into English before querying a model) often helps multilingual language models. However, the external MT system is trained on large amounts of parallel data the model never saw, so the source of the gain is unclear. The authors isolate the effect of translation with self-translate: the same model first translates the input into English using a 4-shot prompt built from FLORES-200 examples, then solves the task on its own translation. They test XGLM (564M–7.5B) and LLaMA (7B–30B) on XCOPA, XStoryCloze, XNLI, PAWS-X and MGSM. They also compare against the external NLLB-200 3.3B translator. Self-translate beats direct inference in the original language on average for every model. The gains are larger for bigger models and high-resource languages. NLLB still does better, but the difference shrinks with scale. The results show that multilingual models have capabilities they cannot fully use when prompted in non-English languages, even though substantial cross-lingual transfer takes place.

Key points

  • Setup: 7 base models from two families, with extra results for BLOOM, LLaMA 2, OpenLLaMA, RedPajama and PolyLM in the appendix.
    • Translation uses greedy decoding, and each input field (e.g., the XNLI premise and hypothesis) is translated separately.
    • MGSM uses 8-shot chain-of-thought. The other tasks are scored zero-shot by picking the most probable candidate.
  • Main gains: LLaMA 30B rises from 43.1 to 46.6 average accuracy (+3.5), with +6.5 on XNLI. XGLM 7.5B improves from 43.4 to 45.7. The smallest XGLM gains only +0.3 and drops on some tasks, which the authors attribute to its weak translation ability.
  • Cross-lingual transfer: LLaMA outperforms XGLM on MGSM despite trailing it on the other tasks. This indicates that math skills carry over across languages, although the transfer is incomplete.
  • Comparison with NLLB: NLLB-200 gives higher downstream accuracy and COMET scores than self-translate. The gap narrows as the models grow larger.
  • Limitations:
    • Self-translate is slower because it adds a translation step.
    • Only base models were studied.
    • All benchmarks are human-translated, which may introduce evaluation artifacts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain40 works · 16,234 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,343 worksEnglish · 1,373 worksMultilingual & cross-lingual NLP · 467 works

Cite

Cite as Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle and Mikel Artetxe. 2024. Do Multilingual Language Models Think Better in English?. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-short.46

@inproceedings{etxaniz2024do,
  title = {{Do Multilingual Language Models Think Better in English?}},
  author = {Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez {de Lacalle} and Mikel Artetxe},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.46},
  url = {https://doi.org/10.18653/v1/2024.naacl-short.46}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv. find
  2. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. find
  3. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv.
  4. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano…
  5. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  6. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv.
  7. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingual transfer. arXiv. find
  8. Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv. find
  9. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15406– 154… find
  10. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Ma…
  11. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv. find
  12. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv.
  13. Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–58…
  14. Machel Reid and Mikel Artetxe. 2023. On the role of parallel data in cross-lingual transfer learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5999–6006, Toronto, Canada. Association for Computational Linguistics. find
  15. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. arXiv. find
  16. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv.
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv.
  18. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study.
  19. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023b. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/