Elamur Research⌕
☰
home / papers / 2026 / Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages

Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages

Summary

Semantic similarity search can show scholars conceptual links between ancient texts that keyword or rule-based tools miss. For Latin and Ancient Greek, however, there has been no standard way to compare retrieval models. The authors fill this gap with Gretino, which they present as the first benchmark for semantic search in Latin, Ancient Greek, and across the two languages. It has two parts. The synthetic Silver portion was produced with ChatGPT and then checked by experts. Its queries deliberately express ideas no ancient author would plausibly have written, so that they cannot accidentally match real passages. The smaller Gold portion is built entirely from genuine classical passages. The team benchmarks four existing encoders. They then fine-tune SPhilBERTa with SimCSE-style contrastive learning on aligned Greek–Latin texts. Monolingual scores barely move, but cross-lingual retrieval improves noticeably. This suggests that curated parallel data is a useful lever for classical-language search. The work belongs to the DaMSym retrieval suite developed within the Italian ITSERR project.

Key points

  • Dataset size: 240 queries, each linked to five relevant Latin/Greek passages.
    • Gretino Silver: 200 queries and 1,000 targets, split equally between the two languages.
    • Gretino Gold: 40 queries and 200 targets taken from authentic texts.
  • Models evaluated: GreBERTa, LaBERTa, PhilBERTa and SPhilBERTa.
  • Fine-tuned variants: two SimCSE models built on SPhilBERTa.
    • SimCSE1 stopped after 10 epochs and SimCSE2 after 6, both through early stopping.
  • Main result: in cross-lingual settings (Latin→Greek and Greek→Latin), SimCSE2 consistently beats the SPhilBERTa baseline, with P@5 gains reaching about 6%. Single-language scores stay roughly the same.
  • Availability: the code is released at github.com/gpucce/index_encoders_eval_itserr. Planned future work is to add more unrelated distractor documents so the benchmark becomes harder.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ipryhrqmwvi

@inproceedings{toyin2026gretino,
  title = {{Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages}},
  author = {Toyin, Hawau Olamide and Iezzi, Federico and Scapini, Elia and Federico, Giulio and Puccetti, Giovanni},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3ipryhrqmwvi},
  url = {https://doi.org/10.63317/3ipryhrqmwvi}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Maria Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando Freitas. 2022. Restoring and attributing ancient texts using deep neural networks. Nature, 603:280–283. ↗
  2. David Bamman and Patrick J. Burns. 2020. Latin BERT: A contextual language model for classical philology. CoRR, abs/2009.10053. ↗
  3. Rufus Behr. 2024. Behr at EvaLatin 2024: Latin dependency parsing using historical sentence embeddings. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 198–202, Torino, Italia. ELRA and ICCL. find
  4. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034. Association for Computational Linguistics. ↗ our card
  5. Charlie Cowen-Breen, Creston Brooks, Barbara Graziosi, and Johannes Haubold. 2023. Logion: Machine-learning based detection and correction of textual errors in greek philology. In Proceedings of the Ancient Language Processing Workshop, pages 170–178. INCOMA Ltd., Shoumen, Bulgaria. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  8. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. GREEK-BERT: The greeks visiting sesame street. arXiv:2008.12014. arXiv. ↗
  9. Kevin Krahn, Derrick Tate, and Andrew C. Lamicela. 2023. Sentence embedding models for Ancient Greek using multilingual knowledge distillation. In Proceedings of the Ancient Language Processing Workshop, pages 13–22, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. ↗
  10. Wouter Mercelis. 2024. KU leuven / brepols-CTLO at EvaLatin 2024: Span extraction approaches for Latin dependency parsing. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 203–206, Torino, Italia. ELRA and ICCL. find
  11. Wouter Mercelis and Alek Keersmaekers. 2022. An ELECTRA model for Latin token tagging tasks. In Proceedings of the Second Workshop on Language Technologies for Historical and Ancient Languages, pages 189–192, Marseille, France. European Language Resources Association. find
  12. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  13. Nils Reimers and Iryna Gurevych. 2020. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525. Association for Computational Linguistics. find our card
  14. Frederick Riemenschneider and Anette Frank. 2023a. Exploring large language models for classical philology. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15181– 15199, Toronto, Canada. Association for Computational Linguistics. find
  15. Frederick Riemenschneider and Anette Frank. 2023b. Graecia capta ferum victorem cepit. detecting latin allusions to ancient greek literature. In Proceedings of the Ancient Language Processing Workshop, pages 30–38. INCOMA Ltd., Shoumen, Bulgaria. ↗
  16. Frederick Riemenschneider and Kevin Krahn. 2024. Heidelberg-boston @ SIGTYP 2024 shared task: Enhancing low-resource language analysis with character-aware hierarchical transformers. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 131–141. Association for Computational Linguistics. find
  17. Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021. A pilot study for BERT language modelling and morphological analysis for ancient and medieval greek. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 128–137. Association for Computational Linguistics. find
  18. Andreas Spanopoulos. 2022. Language models for ancient greek. find
  19. Rachele Sprugnoli, Federica Iurescia, and Marco Passarotti. 2024. Overview of the EvaLatin 2024 evaluation campaign. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 190–197. ELRA and ICCL. find
  20. Rachele Sprugnoli, Francesco Mambrini, Marco Passarotti, and Giovanni Moretti. 2023. The sentiment of latin poetry. annotation and automatic analysis of the odes of horace. IJCoL. Italian Journal of Computational Linguistics, 9(1). Number: 1 Publisher: Accademia University Press. ↗
  21. Milan Straka, Jana Straková, and Federica Gamba. 2024. ÚFAL LatinPipe at EvaLatin 2024: Morphosyntactic analysis of Latin. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 207–214, Torino, Italia. ELRA and ICCL. find
  22. Li Sun, Florian Luisier, Kayhan Batmanghelich, Dinei Florencio, and Cha Zhang. 2023. From characters to words: Hierarchical pre-trained language model for open-vocabulary language understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3605– 3620, Toronto, Canada. Association for Computational Linguistics. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc. find
  24. Ivan P. Yamshchikov, Alexey Tikhonov, Yorgos Pantis, Charlotte Schubert, and Jürgen Jost. 2022. BERT in plutarch’s shadows. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6071– 6080, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  25. Tariq Yousef, Chiara Palladino, Farnoosh Shamsian, Anise d’Orange Ferreira, and Michel Ferreira dos Reis. 2022. An automatic model and gold standard for translation alignment of ancient greek. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5894–5905. European Language Resources Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toyin-2026-gretino-greek-latin-dataset-benchmark/