Elamur Research⌕
☰
home / papers / 2026 / Form and Meaning in Intrinsic Multilingual Evaluations

Form and Meaning in Intrinsic Multilingual Evaluations

Paper · 2026 · Wessel Poelman and Miryam de Lhoneux

Summary

Multilingual research often ranks models or languages by intrinsic scores such as perplexity, computed on multi-parallel test sets. The premise is that translations with identical meaning make the comparison fair. Poelman and de Lhoneux challenge this premise. The cross-entropy objective behind these scores measures information content in Shannon's sense, meaning how predictable a specific string is, not what it says. Two paraphrases with the same meaning can therefore get noticeably different scores.

The authors review six intrinsic metrics and show how they derive from NLL. They formalize which factors enter a cross-language comparison: tokenizers, training corpora, and mono- vs. multilingual models. They then analyze metric behavior on two multi-parallel corpora with both model types and run paraphrase experiments to test consistency. Their conclusion is that the metrics are not universally comparable. Depending on which valid paraphrase ends up in the test set, judgments about which model or language is "better" can reverse. This matters for any project, including Ukrainian NLP work, that uses perplexity on FLORES-style data to judge model quality or how hard a language is to model.

Key points

  • Metrics examined: NLL, PPL, BPC, BPEC (Cotterell et al., 2018), Information Parity/IP (Tsvetkov and Kipnis, 2024) and MRR (Limisiewicz et al., 2023).
    • All except MRR are transformations of NLL.
    • BPEC and IP express a target language's BPC relative to English.
    • MRR depends only on the rank of the correct token, not its probability.
  • Two setups are distinguished: a single multilingual model, or separate monolingual models with their own tokenizers and corpora. Scores from different segmentations are only potentially comparable after rescaling to a shared unit such as characters.
  • Core failure mode: if a sentence's score in one language falls inside the spread of scores for several paraphrases in another language, the outcome depends on which paraphrase is chosen. The paper illustrates this with four German paraphrases vs. one English sentence, and with a hypothetical Spanish/English example where PPLs of 20, 22 and 18 flip the model ranking.
  • Data mentioned: FLORES-200 is cited as the usual benchmark for such comparisons. The authors credit the WMT paraphrase dataset for the paraphrase work. The excerpt does not explicitly name the two multi-parallel corpora used.
  • Proposed directions: the authors link the problem to the form-meaning debate (e.g., Bender and Koller's 2020 "octopus" argument). As alternatives they suggest:
    • semantically annotated parallel resources, following Sproat et al. (2014);
    • meaning representations such as UMR or DRS, which currently lack FLORES-200-level coverage;
    • a still-unformulated statistical law that accounts for paraphrase.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium46 works · 7,833 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Wessel Poelman and Miryam de Lhoneux. 2026. Form and Meaning in Intrinsic Multilingual Evaluations. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.113

@inproceedings{poelman2026form,
  title = {{Form and Meaning in Intrinsic Multilingual Evaluations}},
  author = {Wessel Poelman and Miryam {de Lhoneux}},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.113},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.113}
}

References

62 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations. In Proceedings of the 15th Conference of…
  2. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623. Association for Computational Linguistics.
  3. Thomas Bauwens. 2024. Bits-per-character and its relation to perplexity. find
  4. Emily M. Bender. 2011. On Achieving and Evaluating Language-Independence in NLP. Linguistic Issues in Language Technology, 6. find
  5. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198. Association for Computational Linguistics.
  6. Christian Bentz. 2023. The Zipfian Challenge: Learning the statistical fingerprint of natural languages. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 27–37. Association for Computational Linguistics. find
  7. Terra Blevins and Luke Zettlemoyer. 2022. Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574. Association for Computational Linguistics. find
  8. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024a. When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096. Association for Computational…
  9. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024b. Goldfish: Monolingual Language Models for 350 Languages. find
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  11. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are All Languages Equally Hard to Language-Model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
  12. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012–1031. find
  13. Gottlob Frege. 1892. Über Sinn Und Bedeutung [On Sense and Meaning]. Zeitschrift für Philosophie Und Philosophische Kritik, 100(1):25–50. find
  14. Markus Freitag, George Foster, David Grangier, Colin Cherry, et al. 2020a. Human-Paraphrased References Improve Neural Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 1183–1192. Association for Computational Linguistics. find
  15. Markus Freitag, David Grangier, and Isaac Caswell. 2020b. BLEU might be Guilty but References are not Innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71. Association for Computational Linguistics.
  16. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  17. Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction. Transactions of the Association for Computational Linguistics, 6:451–465.
  18. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, et al. 2024. The Llama 3 Herd of Models.
  19. Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1):335–346. find
  20. Gustav Herdan. 1960. Type-Token Mathematics. Mouton. find
  21. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1). find
  22. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. Association for Computationa…
  23. Jaap Jumelet and Willem Zuidema. 2023. Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4354–4369. Association for Computational Linguistics. find
  24. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86. find
  25. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower Perplexity is Not Always Human-Like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  26. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, et al. 2021. Datasets: A Community Library for Natura…
  27. Tomasz Limisiewicz, Jiří Balhar, and David Mareček. 2023. Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681. Association for Computational Lingui… find
  28. Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. 2024. MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  29. Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In Proceedings of the… find
  30. Clara Meister and Ryan Cotterell. 2021. Language Model Evaluation Beyond Perplexity. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5328–5339. Associa… find
  31. Sabrina J. Mielke. 2019. Can you compare perplexity across different segmentations? find
  32. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What Kind of Language Is Hard to Language-Model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989. Association for Computational Linguistics.
  33. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translatio… find
  34. Xenia Ohmer, Elia Bruni, and Dieuwke Hupke. 2024. From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency. Computational Linguistics, 50(4):1507– 1556.
  35. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology Matters: A Multilingual Language Modeling Analysis. Transactions of the Association for Computational Linguistics, 9:261–276.
  36. Steven T. Piantadosi. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112– 1130. find
  37. Esther Ploeger, Wessel Poelman, Miryam de Lhoneux, and Johannes Bjerva. 2024. What is ”Typological Diversity” in NLP? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5681–5700. Association for Computational Linguistics. find
  38. Wessel Poelman, Thomas Bauwens, and Miryam de Lhoneux. 2025. Confounding Factors in Relating Model Performance to Morphology. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7273–7298. Association for Computational Linguistics.
  39. Wessel Poelman and Miryam de Lhoneux. 2025. The Roles of English in Evaluating Multilingual Language Models. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 492– 498. Universit… find
  40. Willard Van Orman Quine. 1960. find
  41. Word and Object [2013 Reissue]. The MIT Press. find
  42. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, et al. 2022. BLOOM: A 176B-Parameter Open-A… find
  43. John R. Searle. 1980. Minds, brains, and programs. Behavioral and Brain Sciences, 3(3):417–424. find
  44. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics.
  45. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. find
  46. Claude Elwood Shannon. 1951. Prediction and entropy of printed English. The Bell System Technical Journal, 30(1):50–64. find
  47. Richard Sproat. 2023. Symbols: An Evolutionary History from the Stone Age to the Future. Springer Nature Switzerland. find
  48. Richard Sproat, Bruno Cartoni, HyunJeong Choe, David Huynh, Linne Ha, Ravindran Rajakumar, Evelyn Wenzel-Grondie, et al. 2014. A Database for Measuring Linguistic Information Content. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 967–974.… find
  49. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2017. Do neural nets learn statistical laws behind natural language? PLOS ONE, 12(12):e0189326. find
  50. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2019. Evaluating Computational Language Models with Scaling Properties of Natural Language. Computational Linguistics, 45(3):481–513. find
  51. Kushal Tatariya, Artur Kulmizev, Wessel Poelman, Esther Ploeger, Marcel Bollmann, Johannes Bjerva, Jiaming Luo, Heather Lent, and Miryam de Lhoneux. 2025. How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP.
  52. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC‘12), pages 2214–2218. European Language Resources Association (ELRA). find
  53. Alexander Tsvetkov and Alon Kipnis. 2024. Information Parity: Measuring and Predicting the Multilingual Capabilities of Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7971–7989. Association for Computational Linguistics. find
  54. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, et al. 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual…
  55. Jens E. L. Van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Tim O’Gorman, Andrew Cowell, William Croft, Chu-Ren Huang, Jan Hajič, James H. Martin, Stephan Oepen, Martha Palmer, James Pustejovsky, et al. 2021. Designing a Uniform Meaning Representation for Natural Langua… find
  56. Ada Wan. 2022. Fairness in Representation for Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling. In International Conference on Learning Representations. find
  57. Ethan Gotlieb Wilcox, Michael Y. Hu, Aaron Mueller, Alex Warstadt, Leshem Choshen, Chengxu Zhuang, Adina Williams, Ryan Cotterell, and Tal Linzen. 2025. Bigger is not always better: The importance of human-scale language modeling for psycholinguistics. Journal of Memory and Language, 144:104650. find
  58. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, et al. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proc…
  59. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  60. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual Universal Sentence Encoder for Semantic Retrieval. In Proceedings of the 58th Annual Meeting of the Association…
  61. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534. European Language Resources Association (ELRA).
  62. George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poelman-2026-form-meaning-intrinsic-multilingual-evaluations/