Elamur Research⌕
☰
home / papers / 2026 / From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks

From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks

Summary

Olympiad-style linguistic puzzles are increasingly used to test LLM reasoning, but new puzzles are scarce because writing them takes expert effort. The authors ask whether two dominant formats, Rosetta Stone (mostly given translations, fill in the rest) and Match-Up (pair shuffled items across languages), are two views of the same underlying task. If so, one generation procedure could produce both. They designed a systematic way to turn UKLO Rosetta Stone puzzles, together with their questions and answer keys, into Match-Up versions. This produced a corpus of 96 puzzle pairs. Thirty pairs were evaluated: two experienced high-school solvers attempted 16, and LLMs attempted all 30. The human solvers performed no worse on the synthetic Match-Up items than on the originals, which suggests the conversion yields usable puzzles. Both humans and models showed an all-or-nothing pattern on Match-Up tasks, especially for morphology. The released dataset and the documented solver strategies offer a starting point for automatic puzzle generation and for comparing human and machine reasoning.

Key points

  • The corpus contains 96 Rosetta Stone/Match-Up pairs built from UKLO puzzles. It is available at github.com/ef2020/lrec2026-data, includes UKLO-derived metadata, and contains no personal data.
  • In the evaluation, 16 pairs were solved by two expert high-school solvers and all 30 by LLMs. Rosetta answers were scored by exact match only.
  • On Match-Up, solvers and models tended either to fully succeed or to fail completely. This was most pronounced for morphology puzzles, which suggests the format targets a distinct kind of reasoning.
  • A survey of 235 UKLO problems from 2010–2025 found 206 distinct languages, of which 186 appear only once. This confirms that the competition deliberately avoids familiar languages.
  • Interviews identified two useful solver strategies: counting recurring substrings and using proper names as anchors (e.g., Mary/Meeri, Alice/Alicja). Stated limitations include the tiny human sample (N=2), uneven coverage of topics and languages, and some puzzles that cannot be converted, such as multi-template verb systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3yq3khhygtya

@inproceedings{majmudar2026rosetta,
  title = {{From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks}},
  author = {Majmudar, Neh and Huang, Anne and Hu, Jinfan Frank and Filatova, Elena},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3yq3khhygtya},
  url = {https://doi.org/10.63317/3yq3khhygtya}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrew M. Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A. Chi, Ryan Chi, Scott A. Hale, and Hannah Rose Kirk. 2024. LingOly: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. In Proceedings of the Thirty-Eighth Annual Conference on Neural In… find
  2. Bozhidar Bozhanov and Ivan Derzhanski. 2013. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL. find
  3. Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. 2024. ModeLing: A novel dataset for testing linguistic reasoning in language models. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages… find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep… find our card
  5. Henry Allan Gleason. 1955. Workbook in descriptive linguistics. Publisher Holt, Rinehartand Winston. find
  6. Jude Khouja, Karolina Korgul, Simi Hellsten, Lingyi Yang, Vlad Neacsu, Harry Mayne, Ryan Kearns, Andrew Bean, and Adam Mahdi. 2025. Lingoly-too: Disentangling reasoning from knowledge with templatised orthographic obfuscation. find
  7. Neh Majmudar and Elena Filatova. 2025. Can LLMs Solve and Generate Linguistic Olympiad Puzzles? In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Suzhou, China. find
  8. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196. find our card
  9. OpenAI. 2025. GPT-5 system card. https://cdn.openai.com/gpt-5-system-card. pdf. [Accessed 13-10-2025]. ↗
  10. Gözde Gül Şahin, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych. 2020. PuzzLing Machines: A Challenge on Learning From Small Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1241–1254. find
  11. Andrey A. Zaliznyak. 1963. Linguistics puzzles (in Russian). In Tatyana N. Moloshnaya, editor, Structural Typology Research. find
  12. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. 2024. A comprehensive survey on pretrained foundation models: A history from BERT to Chat-GPT. International Journal of Machine Learning and Cybernetics, abs/2302.09419:1–65. — [arXiv:2302.09419](https://arxiv.org/abs/2302.09419) find
  13. Alfred N. Zhurinsky. 1993. Word, Letter, Number: A discussion of self-sufficient linguistic problems with an analysis of a hundred samples of the genre (in Russian). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/majmudar-2026-rosetta-match-up-paired-corpus/