Elamur Research⌕
☰
home / papers / 2023 / TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

Paper · 2023 · Bibek Upadhayay and Vahid Behzadan

Summary

Making LLMs work in low-resource languages usually means expensive pretraining or full fine-tuning, and there are few multilingual benchmarks to measure progress. The authors propose a cheaper route. They machine-translated Alpaca-52K (GPT-4 version) and Dolly-15K into 132 languages with Google Cloud Translation, releasing the result as the Multilingual Instruction-Tuning Dataset (named MITS in the abstract and MITDS in the body). From these data they built TaCo (Translation-Assisted Cross-Linguality). Each training target walks through a chain of steps: render the non-English instruction in English, answer in English, then translate the answer back into the target language. Guanaco-33B, already tuned on OASST1, was further trained this way with LoRA adapters as a curriculum-learning step. Models for Nepali, Sanskrit, Maithili and Persian were judged by GPT-4 on a translated Vicuna Benchmark. On a low-resource language, TaCo reached a GPT-4 score of 82%, about twice that of plain instruction tuning. This suggests that strong English models can be extended to underserved languages at modest cost.

Key points

  • Resources released: translated Alpaca-52K, Dolly-15K and Vicuna Benchmark sets covering 132 languages, plus LoRA adapters for all four languages (github.com/UNHSAILLab/TaCo).
  • Training data: the combined set has about 67K TaCo-formatted prompts. Translation quality was checked on 1,000 sentences per language using round-trip (en→xx→en) BLEU (SacreBLEU), ChrF and TER.
  • Training setup: LoRA on q_proj, k_proj, v_proj and o_proj, with r=32, alpha=64 and dropout 0.05. The learning rate was 3e-4 and the cutoff length 2000.
  • Evaluation: GPT-4 graded each answer individually on the multilingual Vicuna Benchmark. The headline result is 82% for a low-resource language, roughly double the instruction-tuning-only baseline.
  • Qualitative observation: in the paper's Nepali examples, the model tuned without TaCo tends to fall into repeated sentence loops. The TaCo model gives a coherent answer in English and then in Nepali.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of New HavenWest Haven · United States2 works · 17 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksMultilingual & cross-lingual NLP · 882 worksLow-resource languages · 287 works

Cite

Cite as Bibek Upadhayay and Vahid Behzadan. 2023. TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes. arXiv:2311.10797.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.10797 · arXiv 2311.10797

@misc{upadhayay2023taco,
  title = {{TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes}},
  author = {Bibek Upadhayay and Vahid Behzadan},
  year = {2023},
  doi = {10.48550/arxiv.2311.10797},
  eprint = {2311.10797},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.10797}
}

References

31 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.
  3. Angana Borah. Translationese: A brief introduction. https://anganaborah.medium.com/ translationese-a-brief-introduction-db4575c0ee69, 2021. Accessed: 16-Sep-2022. ↗
  4. Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume…
  5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2023. ↗
  6. Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019. find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
  8. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/ 12/dolly-first-open-commercially-viable-… ↗
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
  10. Philipp Dufter and Hinrich Schütze. Identifying elements essential for bert’s multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4423–4437, 2020.
  11. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. Larger-scale transformers for multilingual masked language modeling. arXiv preprint arXiv:2105.00572, 2021.
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  14. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André FT Martins, François Yvon, et al. Glot500: Scaling multilingual corpora and language models to 500 languages. arXiv preprint arXiv:2305.12182, 2023. find
  15. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2, 2019.
  16. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
  17. Moshe Koppel and Noam Ordan. Translationese and its dialects. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 1318–1326, 2011.
  18. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al. Madlad-400: A multilingual and document-level large audited dataset. arXiv preprint arXiv:2309.04662, 2023.
  19. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation. arXiv preprint arXiv:2305.15011, 2023.
  20. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022.
  21. OpenAI. Gpt-4 technical report, 2023.
  22. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
  23. Maja Popović. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pp. 612–618, 2017.
  24. Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771, 2018.
  25. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  26. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022.
  27. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223–231, 2006. tloen. alpaca-lora…
  28. UNESCO (IESALC). A decade to prevent the disappearance of 3,000 languages. https://www.iesalc.unesco.org/en/2022/02/21/ a-decade-to-prevent-the-disappearance-of-3000-languages/, 2022. Accessed: 2023-04-01. ↗
  29. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b.
  31. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018, 2023. *Table 2: Translation Evaluation Metrics for Low Resources Languages* *Figure 2: Eval…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/upadhayay-2023-taco-enhancing-cross-lingual-transfer/