Elamur Research⌕
☰
home / papers / 2026 / A Principled Framework for Evaluating on Typologically Diverse Languages

A Principled Framework for Evaluating on Typologically Diverse Languages

Paper · 2026 · Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen et al.

Summary

Multilingual NLP models are often described as working "across languages," but they can only be tested on a small subset of the world's languages. Studies usually claim their evaluation sets are typologically diverse by picking languages from different families. The authors show that this is unreliable. Family membership does not guarantee structural difference, and random picks within families give inconsistent samples.

They borrow the concepts of sampling universe, frame and sample from linguistic typology. From these they build a framework that measures pairwise typological distance using Grambank features and then selects maximally distant languages with two algorithms established in typology. Against baselines such as convenience and family-based selection, their samples score higher on four diversity metrics.

In zero-shot downstream experiments on Taxi1500, diverse samples give a more realistic picture of how well models generalize. The framework also works with other distance types, such as geography, and can guide dataset expansion or find similar languages for transfer learning.

Key points

  • The typological distances are computed from Grambank, which covers grammatical features only and has incomplete coverage.
  • Sample diversity is scored with four metrics: MPD, FVO, FVI and H. Non-deterministic baselines are averaged over 10 runs.
  • Downstream tests use 20-language samples (k = 20) on Taxi1500, with FloRes-200 also examined as a sampling frame.
  • The framework extends to geographic diversity: the MaxMin objective on coordinate distances selects twenty widely spread languages.
  • The released Python package, typdiv-sampling, is available at github.com/esther2000/typdiv-sampling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations🎓Aalborg UniversityAalborg · Denmark12 works · 743 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, Anders Schlichtkrull, Miryam de Lhoneux and Johannes Bjerva. 2026. A Principled Framework for Evaluating on Typologically Diverse Languages. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.577

@article{ploeger2026principled,
  title = {{A Principled Framework for Evaluating on Typologically Diverse Languages}},
  author = {Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, Anders Schlichtkrull, Miryam {de Lhoneux} and Johannes Bjerva},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.577},
  url = {https://doi.org/10.1162/coli.a.577}
}

References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, Marah, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. https://arxiv.org/abs/2412.08905 Ács, Judit, Ákos Kádár, and Andras Kornai. 2021. Subword pool… ↗
  2. Aikhenvald, Alexandra Y. 2003b. Teaching Tariana, an endangered language from northwest Amazonia. International Journal of the Sociology of Language, 2003(161):125–139. https://doi.org/10.1515/ijsl.2003.025 Bakker, Dik. 2010. Language sampling. Oxford handbooks in linguistics. https://doi.org/10.10… — [doi:10.1515/ijsl.2003.025](https://doi.org/10.1515/ijsl.2003.025) ↗
  3. Bender, Emily M. 2009. Linguistically naı̈ve != language independent: Why NLP needs linguistic typology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous? pages 26–32. https://doi.org/10.3115 /1642038.1642044…
  4. SIGTYP 2020 shared task: Prediction of typological features. In Proceedings of the Second Workshop on Computational Research in Linguistic Typology, pages 1–11. https://doi.org/10.18653/v1/2020.sigtyp-1.1 Bugliarello, Emanuele, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki… ↗
  5. Conneau, Alexis, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association… find
  6. Dryer, Matthew S. and Martin Haspelmath. 2013. WALS Online. Leipzig: Max Planck Institute for Evolutionary Anthropology. find
  7. Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Georgi, Ryan, Fei Xia, and William Lewis. 2010. Comparing language similarity across genetic and typologically-based groupings. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 385–393. find
  9. Greenberg, Joseph Harold. 1963. Some universals of grammar with particular reference to the order of meaningful elements. In Universals of Language. MIT Press, pages 40–70. Gutierrez-Vasques, Ximena, Christian Bentz, and Tanja Samardžić. 2023. Languages through the looking glass of BPE compression… find
  10. Hershcovich, Daniel, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural…
  11. John Wiley & Sons. Kuo, Ching Chung, Fred Glover, and Krishna S. Dhir. 1993. Analyzing and modeling the maximum diversity problem by zero-one programming. Decision Sciences, 24(6):1171–1185. https://doi.org/10.1111/j.1540 -5915.1993.tb00509.x Littell, Patrick, David R. Mortensen, Ke Lin, Katherine K… ↗
  12. Miestamo, Matti, Dik Bakker, and Antti Arppe. 2016. Sampling for variety. Linguistic Typology, 20(2):233–296. https://doi.org/10.1515/lingty -2016-0006 Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositio… — [doi:10.1515/lingty](https://doi.org/10.1515/lingty) ↗
  13. Nichols, Johanna and Balthasar Bickel. 2009. The AUTOTYP genealogy and geography database: 2009 release. URL: https://github.com/autotyp/autotyp-data Parreño, Francisco, Ramón Álvarez-Valdés, and Rafael Martı́. 2021. Measuring diversity. A review and an empirical analysis. European Journal of O… — [doi:10.1016/j.ejor.2020](https://doi.org/10.1016/j.ejor.2020) ↗
  14. Perkins, Revere D. 1988. The covariation of culture and grammar. Studies in Syntactic Typology, pages 359–378. https://doi.org/10.1075/tsl.17.22per Petrov, Aleksandar, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances… ↗
  15. Wichmann, Søren and Eric W. Holman. 2010. Pairwise comparisons of typological profiles. In Jan Wohlgemuth and Michael Cysouw, editors, Rethinking Universals: How Rarities Affect Linguistic Theory. De Gruyter Mouton, pages 241–254. https://doi.org/10.1515/9783110220933.241 Xu, Hongzhi, Jordan Kodner… ↗
  16. Zhang, Xiang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927. http… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ploeger-2026-principled-framework-evaluating-typologically-diverse/