Elamur Research⌕
☰
home / papers / 2024 / Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages

Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages

Paper · 2024 · Zihao Li, Yucheng Shi, Zirui Liu et al.

Summary

LLMs are trained on corpora dominated by English. For example, about 92.65% of GPT-3 tokens and 89.7% of LlaMa2 pre-training data are English. As a result, low-resource languages get weaker support, yet there are few quantitative ways to measure this gap. The authors propose Language Ranker, an intrinsic metric that ranks languages using a model's hidden states instead of task outputs. They take parallel sentence pairs from OPUS-100 and feed each English sentence and its non-English counterpart into the model. They then compare the last-token representations with cosine similarity, averaged over several intermediate layers. Languages are ordered by this score. Agreement between models' rankings is measured with a longest-common-ordered-sublist ratio. Across LlaMa2, LlaMa3, Qwen, Mistral-v0.1 and Gemma, high-resource languages consistently land closer to English than low-resource ones. The scores also track each language's share of pre-training data. This gives a cheap, task-independent way to diagnose multilingual bias. It could also guide the design of more balanced training corpora.

Key points

  • Probing data: OPUS-100 (English-centric), filtered to 94 language pairs with 2000 samples each. The pairs span high-resource (German, French, Chinese) and low-resource languages (Oriya, Kannada, Kazakh, Igbo, Turkmen).
  • Score: cosine similarity of last-token representations, averaged over layers 5, 10, 15, 20 and 25. Major European languages typically score above 0.6, while low-resource ones often fall below 0.4.
  • Link to pre-training share (LlaMa2):
    • German (0.17% of the corpus) scores 0.723 and French scores 0.737.
    • Welsh, Persian, Urdu and Kannada each make up ≤0.01% and score between 0.236 and 0.396.
  • Consistency across models: 7B models produce largely consistent rankings, and the pattern holds for 13B LlaMa2 and Qwen. Qwen's extra Chinese training lifts Chinese to the level of other high-resource languages.
  • "Double variance" check: this measures the spread of the top covariance eigenvalues of a language's embeddings. It shows that low-resource languages occupy narrow, unbalanced subspaces, with high double variance paired with low similarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MinnesotaMinneapolis · United States9 works · 1,833 citations🎓University of GeorgiaAthens · United States9 works · 26,161 citations🎓New Jersey Institute of TechnologyNewark · United States4 works · 1,234 citations🎓Wake Forest UniversityWinston-Salem · United States3 works · 884 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Language models & pretraining · 1,343 works Low-resource languages · 166 works

Cite

Cite as Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu and Mengnan Du. 2024. Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages. arXiv:2404.11553.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.11553 · arXiv 2404.11553

@misc{li2024language,
  title = {{Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages}},
  author = {Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu and Mengnan Du},
  year = {2024},
  doi = {10.48550/arxiv.2404.11553},
  eprint = {2404.11553},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.11553}
}

References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. our card
  2. Ahuja, S.; Aggarwal, D.; Gumma, V.; Watts, I.; Sathe, A.; Ochieng, M.; Hada, R.; Jain, P.; Axmed, M.; Bali, K.; and Sitaram, S. 2024. MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks. arXiv:2311.07463. our card
  3. Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. our card
  4. Blasi, D.; Anastasopoulos, A.; and Neubig, G. 2021. Systematic Inequalities in Language Technology Performance across the World’s Languages. arXiv preprint arXiv:2110.06733. our card
  5. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.… find
  6. Gurnee, W.; and Tegmark, M. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207. find
  7. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. our card
  8. Huang, H.; Tang, T.; Zhang, D.; Zhao, W. X.; Song, T.; Xia, Y.; and Wei, F. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S… our card
  9. Ju, T.; Sun, W.; Du, W.; Yuan, X.; Ren, Z.; and Liu, G. 2024. How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study. arXiv preprint arXiv:2402.16061. find
  10. Kalyan, K. S.; Rajasekharan, A.; and Sangeetha, S. 2021. Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542. — [arXiv:2108.05542](https://arxiv.org/abs/2108.05542) find
  11. Lankford, S.; Afli, H.; and Way, A. 2024. Transformers for Low-Resource Languages: Is F\\’eidir Linn! arXiv preprint arXiv:2403.01985. Li, K.; Patel, O.; Viégas, F.; Pfister, H.; and Wattenberg, M. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neura… our card
  12. Liu, C.; Zhang, W.; Zhao, Y.; Luu, A. T.; and Bing, L. 2024. Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models. arXiv preprint arXiv:2403.10258. find
  13. Marks, S.; and Tegmark, M. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. our card
  14. Meta-AI. 2024. LlaMa3. https://github.com/meta-llama/ llama3. Accessed: 2024-06-14. ↗
  15. OpenAI. 2023. GPT-3 Dataset Statistics. https://github.com/ openai/gpt-3/tree/master/dataset statistics. ↗
  16. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744. our card
  17. Qin, L.; Chen, Q.; Wei, F.; Huang, S.; and Che, W. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. arXiv preprint arXiv:2310.14799. our card
  18. Qin, L.; Chen, Q.; Zhou, Y.; Chen, Z.; Li, Y.; Liao, L.; Li, M.; Che, W.; and Yu, P. S. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925. Schäfer, A.; Ravfogel, S.; Hofmann, T.; Pimentel, T.; and Schlag, I. 2024. Language Imbala… find
  19. Shen, L.; Tan, W.; Chen, S.; Chen, Y.; Zhang, J.; Xu, H.; Zheng, B.; Koehn, P.; and Khashabi, D. 2024. The language barrier: Dissecting safety challenges of llms in multilingual contexts. arXiv preprint arXiv:2401.13136. our card
  20. Steck, H.; Ekanadham, C.; and Kallus, N. 2024. Is Cosine-Similarity of Embeddings Really About Similarity? In Companion Proceedings of the ACM on Web Conference 2024, WWW ’24. ACM. Team, G.; Mesnard, T.; Hardin, C.; Dadashi, R.; Bhupatiraju, S.; Pathak, S.; Sifre, L.; Rivière, M.; Kale, M. S.; Love… find
  21. Tiedemann, J. 2020. The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, 1174–1182. Online: Association for Computational Linguistics. our card
  22. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  23. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30. our card
  24. Wendler, C.; Veselovsky, V.; Monea, G.; and West, R. 2024. Do Llamas Work in English? On the Latent Language of Multilingual Transformers. arXiv:2402.10588. our card
  25. Xie, Y.; Chen, X.; Zhan, H.; Shivakumara, P.; Yin, B.; Liu, C.; and Lu, Y. 2024. Weakly supervised scene text generation for low-resource languages. Expert Systems with Applications, 237: 121622. find
  26. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  27. Zhang, B.; Williams, P.; Titov, I.; and Sennrich, R. 2020. Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1628–1639. Online: Association for Computational Linguistics. our card
  28. Zhang, X.; Li, S.; Hauer, B.; Shi, N.; and Kondrak, G. 2023. Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7915–7927. find
  29. Zou, A.; Phan, L.; Chen, S.; Campbell, J.; Guo, P.; Ren, R.; Pan, A.; Yin, X.; Mazeika, M.; Dombrowski, A.-K.; et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/