Elamur Research⌕
☰
home / papers / 2024 / Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings

Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings

Paper · 2024 · Krishno Dey, Prerona Tarannum, Md. Arid Hasan et al.

Summary

The paper asks how well large language models handle widely spoken but under-resourced South Asian languages compared with English. The authors tested three models: GPT-4, Llama-2-70b-chat and Gemini Pro. They covered Bangla, Hindi and Urdu alongside English. Two cross-lingual benchmarks were used: XNLI for natural language inference and SIB-200 for topic classification. All experiments used zero-shot prompting with a fixed instruction template. They built five prompt settings by mixing the language of the instruction and the input, partly through cross-lingual translation of the original English prompts. Malformed model outputs were cleaned in post-processing, and remaining invalid labels were counted as errors. GPT-4 scored highest in almost every setting and language. Prompts written in English produced better results than prompts in the low-resource languages, even when the alternative was a translated prompt. Llama 2 performed worst because it largely failed to understand non-English instructions. The study matters because these three languages together have over a billion speakers. It shows that multilingual LLMs still lag in these languages, which points to a need for more language-specific data and resources.

Key points

  • Models and data: The study evaluates GPT-4, Llama-2-70b-chat and Gemini Pro on XNLI (entailment/contradiction/neutral) and SIB-200 (topics such as Entertainment, Geography, Health, Politics, Science/Technology, Sports, Travel). Languages are Bangla (BN), Hindi (HI), Urdu (UR) and English (EN).
  • Why these languages: The authors cite roughly 610M Hindi, 273M Bangla and 232M Urdu speakers, ranking 3rd, 7th and 10th worldwide. They argue these languages are poorly covered by existing resources: IndicXNLI and IndicLLMSuite lack Urdu, and BUFFET lacks Bangla.
  • Ranking: GPT-4 led across all five prompt settings (P1–P5) on both datasets, with one exception. In setting P3 on SIB-200, Gemini Pro gave the best Urdu result.
  • Gemini Pro limitations: Gemini Pro blocked prompts it judged harmful, even after its safety settings were relaxed, though only about 1–4 samples went unpredicted. It also returned many stray tokens and invalid labels.
  • Llama 2 limitations: The authors attribute Llama 2's weak results to training data that is about 90% English. It also showed a strong bias toward predicting the "Neutral" class on XNLI.
  • Other findings: The authors report that prompt translation quality had little effect, and that translating prompts occasionally helped performance.
  • Stated limitations: The study uses zero-shot prompting only, with no few-shot experiments, and covers a narrow task range because cost restricted experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Macquarie UniversitySydney · Australia42 works · 5,776 citations🎓UNSW SydneySydney · Australia34 works · 1,509 citations🎓University of New BrunswickFredericton · Canada5 works · 97 citations🎓Daffodil International UniversityDhaka · Bangladesh2 works · 24 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksMultilingual & cross-lingual NLP · 828 worksLow-resource languages · 267 works

Cite

Cite as Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak and Usman Naseem. 2024. Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings. arXiv:2410.13153.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.13153 · arXiv 2410.13153

@misc{dey2024better,
  title = {{Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings}},
  author = {Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak and Usman Naseem},
  year = {2024},
  doi = {10.48550/arxiv.2410.13153},
  eprint = {2410.13153},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.13153}
}

References

61 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2024. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. Preprint, arXiv:2309.07445.
  2. Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. Indicxnli: Evaluating multilingual inference for indian languages. arXiv preprint arXiv:2204.08776. find
  3. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528.
  4. Syeda Nahida Akter, Zichun Yu, Aashiq Muhamed, Tianyue Ou, Alex Bäuerle, Ángel Alexander Cabrera, Krish Dholakia, Chenyan Xiong, and Graham Neubig. 2023. An in-depth look at gemini’s language abilities. arXiv preprint arXiv:2312.11444. find
  5. Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305.14857.
  6. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th international conference on World Wide Web companion, pages 759–760. find
  7. Emily Bender. 2019. The\# benderrule: On naming the languages we study and why it matters. The Gradient, 14:34. find
  8. Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin, Md Saiful Islam, M. Sohel Rahman, Anindya Iqbal, and Rifat Shahriyar. 2021. Banglabert: Combating embedding barrier in multilingual models for low-resource language understanding. Preprint, arXiv:2101.00204. — [arXiv:2101.00204](https://arxiv.org/abs/2101.00204) find
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  10. Sharanya Chakravarthy, Anjana Umapathy, and Alan W Black. 2020. Detecting entailment in code-mixed hindi-english conversations. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 165–170. find
  11. Anusha Chhabra and Dinesh Kumar Vishwakarma. 2023. A literature survey on multimodal and multilingual automatic hate speech identification. Multimedia Systems, pages 1–28. find
  12. Willy Chung, Samuel Cahyawijaya, Bryan Wilie, Holy Lovenia, and Pascale Fung. 2023. Instructtods: Large language models for end-to-end task-oriented dialogue systems. arXiv preprint arXiv:2310.08885. find
  13. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… find
  14. Tahmid Hasan, Abhik Bhattacharjee, Kazi Samin, Masum Hasan, Madhusudan Basak, M Sohel Rahman, and Rifat Shahriyar. 2020b. Not low-resource anymore: Aligner ensembling, batch filtering, and new datasets for bengali-english machine translation. arXiv preprint arXiv:2009.09359. find
  15. Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2020. A survey on recent approaches for natural language processing in low-resource scenarios. arXiv preprint arXiv:2010.12309.
  16. Ming Shan Hee, Shivam Sharma, Rui Cao, Palash Nandi, Preslav Nakov, Tanmoy Chakraborty, and Roy Ka-Wei Lee. 2024. Recent advances in hate speech moderation: Multimodality and the role of large models. arXiv preprint arXiv:2401.16727. — [arXiv:2401.16727](https://arxiv.org/abs/2401.16727) find
  17. Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. arXiv preprint arXiv:1908.10423.
  18. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004.
  19. José Antonio García-Díaz, Ronghao Pan, and Rafael Valencia-García. 2023. Leveraging zero and few-shot learning for enhanced model generality in hate speech detection in spanish and english. Mathematics, 11(24):5004. find
  20. Yi Huang, Junlan Feng, Shuo Ma, Xiaoyu Du, and Xiaoting Wu. 2020. Towards low-resource semi-supervised dialogue generation with meta-learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4123–4128. find
  21. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.
  22. Khondoker Ittehadul Islam, Sudipta Kar, Md Saiful Islam, and Mohammad Ruhul Amin. 2021. Sentnob: A dataset for analysing sentiment on noisy bangla texts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3265–3271. find
  23. Reto Gubelmann, Aikaterini-Lida Kalouli, Christina Niklaus, and Siegfried Handschuh. 2023. When truth matters-addressing pragmatic categories in natural language inference (nli) by large language models (llms). In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (* SEM…
  24. Md Arid Hasan, Firoj Alam, Anika Anjum, Shudipta Das, and Afiyat Anjum. 2023a. Blp-2023 task 2: Sentiment analysis. In Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), pages 354–364. find
  25. Md Arid Hasan, Shudipta Das, Afiyat Anjum, Firoj Alam, Anika Anjum, Avijit Sarker, and Sheak Rashed Haider Noori. 2023b. Zero-and few-shot prompting with llms: A comparative study with fine-tuned models for bangla sentiment analysis. arXiv preprint arXiv:2308.10783. find
  26. Md Arid Hasan, Jannatul Tajrin, Shammur Absar Chowdhury, and Firoj Alam. 2020a. Sentiment classification in bangla textual content: A comparative study. In 2020 23rd international conference on computer and information technology (ICCIT), pages 1–6. IEEE. find
  27. Md Saroar Jahan and Mourad Oussalah. 2023. A systematic review of hate speech automatic detection using natural language processing. Neurocomputing, page 126232. find
  28. Mohsinul Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, M Saiful Bari, and Enamul Hoque. 2023. Benllmeval: A comprehensive evaluation into the potentials and pitfalls of large language models on bengali nlp. arXiv preprint arXiv:2309.13173. — [arXiv:2309.13173](https://arxiv.org/abs/2309.13173) find
  29. Tannon Kew, Florian Schottmann, and Rico Sennrich. 2023. Turning english-centric llms into polyglots: How much multilinguality is needed? arXiv preprint arXiv:2312.12683. find
  30. Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh M Khapra, et al. 2024. Indicllmsuite: A blueprint for creating pre-training and fine-tuning datasets for indian languages. arXiv…
  31. Fajri Koto, Tilman Beck, Zeerak Talat, Iryna Gurevych, and Timothy Baldwin. 2024. Zero-shot sentiment analysis in low-resource languages using a multilingual sentiment lexicon. arXiv:2402.02113. arXiv preprint Akshi Kumar and Victor Hugo C Albuquerque. 2021. Sentiment analysis using xlm-r transforme… find
  32. Xiaoyi Liu, Mao Teng, Shuangtao Yang, and Bo Fu. 2023. Knowdee at blp-2023 task 2: Improving bangla sentiment analysis using ensembled models with pseudo-labeling. In Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), pages 273–278. find
  33. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  34. Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023. New trends in machine translation using large language models: Case examples with chatgpt. arXiv preprint arXiv:2305.01181. — [arXiv:2305.01181](https://arxiv.org/abs/2305.01181) find
  35. Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00436. find
  36. Khalid Bin Muhammad and SM Aqil Burney. 2023. Innovations in urdu sentiment analysis using machine and deep learning techniques for two-class classification of symmetric datasets. Symmetry, 15(5):1027. find
  37. Bhavish Pahwa and Bhavika Pahwa. 2023. Bphigh at semeval-2023 task 7: Can fine-tuned cross-encoders outperform gpt-3.5 in nli tasks on clinical trial data? In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1936– 1944. find
  38. Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian languages: An overview of sail_code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) find
  39. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) find
  40. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. arXiv preprint arXiv:2310.14799.
  41. Yeasin Ar Rahman, Mahtabul Alam Sohan, Khalid Ibn Zinnah, and Mohammed Moshiul Hoque. 2017. A framework for building a natural language interface for bangla. In 2017 international conference on electrical, computer and communication engineering (ecce), pages 935–940. IEEE. find
  42. Nitin Rane. 2023. Enhancing mathematical capabilities through chatgpt and similar generative artificial intelligence: Roles and challenges in solving mathematical problems. Available at SSRN 4603237. find
  43. Nathaniel R Robinson, Perez Ogayo, David R Mortensen, and Graham Neubig. 2023. Chatgpt mt: Competitive for high-(but not low-) resource languages. arXiv preprint arXiv:2309.07423.
  44. Sunayana Sitaram, Monojit Choudhury, Barun Patra, Vishrav Chaudhary, Kabir Ahuja, and Kalika Bali. 2023. Everything you need to know about multilingual llms: Towards fair, performant and reliable models for languages of the world. In Proceedings of the 61st Annual Meeting of the Association for Comp… find
  45. Xiaofei Sun, Xiaoya Li, Shengyu Zhang, Shuhe Wang, Fei Wu, Jiwei Li, Tianwei Zhang, and Guoyin Wang. 2023. Sentiment analysis through llm negotiations. arXiv preprint arXiv:2311.01876.
  46. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  47. Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family. In International Semantic Web Conference, pages 348–367. Springer. find
  48. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  50. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382.
  51. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
  52. Frank Xing. 2024. Designing heterogeneous llm agents for financial sentiment analysis. arXiv preprint arXiv:2401.05799. — [arXiv:2401.05799](https://arxiv.org/abs/2401.05799) find
  53. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674. find
  54. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2023. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data.
  55. Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. arXiv preprint arXiv:1805.07513.
  56. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Summit: Iterative text summarization via chatgpt. arXiv preprint arXiv:2305.14835. find
  57. Min Zhang, Jianfeng He, Taoran Ji, and Chang-Tien Lu. 2024. Don’t go to extremes: Revealing the excessive sensitivity and calibration limitations of llms in implicit hate speech detection. arXiv preprint arXiv:2402.11406. find
  58. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023b. Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005.
  59. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  60. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. arXiv preprint arXiv:2306.13304.
  61. Steven Zimmerman, Udo Kruschwitz, and Chris Fox. 2018. Improving hate speech detection with deep learning ensembles. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dey-2024-better-ask-english-evaluation-large/