Elamur Research⌕
☰
home / papers / 2023 / Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic

Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic

Paper · 2023 · Sabri Boughorbel and Majd Hawasly

Summary

Most LLM benchmarks are in English or use closed-form tasks, so we know little about how open models handle open-ended, multi-turn instructions in languages such as Arabic. The authors built ARABIC MT-BENCH by machine-translating MT-Bench's 80 two-turn questions with Google Translate and then curating them by hand. During curation they replaced names with Arabic ones, rewrote English-grammar items, translated the English reference answers instead of regenerating them, and added a request to reply in Arabic. GPT-4 judged both language versions with the unchanged MT-Bench prompt, which allowed a direct English-versus-Arabic comparison per category and per turn for open models such as Jais-13B-chat, Vicuna, LLaMa-2-chat, Phoenix, Noon and BLOOMZ.

Results differed across task types, for example logic versus literacy tasks, and between languages. Base models fine-tuned on multilingual, multi-turn data could rival models trained from scratch on multilingual corpora. Models built on the same BLOOM backbone varied widely, which shows that the fine-tuning recipe matters a lot. An oracle ensemble of small open models would clearly beat any single one. The benchmark is publicly released.

Key points

  • ARABIC MT-BENCH covers 8 categories with 10 questions each and 2 turns per question, 160 prompts in total. It is published as the Hugging Face space QCRI/mt-bench-ar.
  • GPT-4 is the judge and scores answers from 1 to 10 with an explanation. Earlier work found GPT-4 judgments agree with humans 85% of the time, above the 81% agreement between humans.
  • Without any change to its prompt, the GPT-4 judge penalized answers given in English despite the Arabic-reply instruction.
  • Jais-13B-chat was the best single open model in Arabic, with a score of 5.08. A per-question oracle ensemble reached 6.70, about 32% higher; the English ensemble reached 8.2.
  • Jais led five literacy-type categories. Math, coding and reasoning were split among LLaMa-2-13B-chat, Guanaco-33B and Phoenix-inst-chat-7b.

Interesting ideas

  • I1. Localize test items rather than merely translate them: swap personal names and rewrite items that only make sense in the source language, so the benchmark measures skill and not translation artifacts.
  • I2. Translate the gold reference answers from the source instead of regenerating them in the target language, which keeps the two language versions scored against equivalent references.
  • I3. Run the identical judging setup on paired versions of the same question and plot one score against the other per turn; the distance from the diagonal shows the gap, and the shift between turns shows conversational decay.
  • I4. Compute an oracle ensemble by picking the best model per question; this gives an upper bound on how complementary several systems are and shows which system wins which category.
  • I5. Have the judge explain each score, then summarize those explanations per rating band to get a readable profile of typical failures.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When importing school tests or fable QA from other languages or cultures, adapt names and language-bound items so the readers are tested on comprehension rather than odd surface forms.
  • I2 (learn): When building paraphrased or translated evaluation variants, carry the original gold annotations across so score differences reflect the system and not shifted references.
  • I3 (learn): Score paired inputs (original vs. paraphrase, first vs. follow-up question) and plot them against each other to spot where robustness breaks down.
  • I4 (learn): Measure the oracle gain of the witness parsers before relying on their agreement for silver data, which shows whether combining them adds real coverage and in which constructions.
  • I5 (answer): Group the explanation traces of wrong answers by score or error type to produce concise failure profiles that guide which pipeline stage to fix.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Sabri Boughorbel and Majd Hawasly. 2023. Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.11

@inproceedings{boughorbel2023analyzing,
  title = {{Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic}},
  author = {Sabri Boughorbel and Majd Hawasly},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.11},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.11}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2023. Benchmarking Arabic AI with large language models. find
  2. Zaid Alyafeai, Maged S Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating Arabic NLP tasks using ChatGPT models. arXiv preprint arXiv:2306.16322. ↗
  3. Zhengda Bian, Hongxin Liu, Boxiang Wang, Haichen Huang, Yongbin Li, Chuanrui Wang, Fan Cui, and Yang You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110.14883. ↗
  4. Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2023. Phoenix: Democratizing ChatGPT across languages. arXiv preprint arXiv:2304.10453. ↗
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. ↗
  6. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023. OpenAssistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327. ↗
  7. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. ↗
  8. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLora: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314. ↗
  10. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The Flan Collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688. ↗
  11. Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. 2022. MMDialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.05719. ↗
  12. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747. ↗
  13. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. find
  14. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. ↗
  15. Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021. Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1121–1133. find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. ↗
  17. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP.
  18. MosaicML. 2023. Introducing MPT-30B: Raising the bar for open-source foundation models. Accessed: 2023-09-09. find
  19. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. ↗
  20. OpenAI. 2023. GPT-4 technical report.
  21. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. ↗
  22. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ↗
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models.
  24. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boughorbel-2023-analyzing-multilingual-competency-llms-multi/