SimLex-999 for Modern Greek
Summary
Benchmarks of human word-similarity judgements remain a standard way to evaluate word representations intrinsically. Until now, Greek had no such resource. There was only a translated WordSim353, which measures association rather than similarity and reused the English scores, plus a relatedness set for Ancient Greek. The authors fill this gap by building a Modern Greek SimLex-999.
They translated the 999 English pairs using the Multi-SimLex guidelines, replacing items where Greek would merge two words into one. They then collected ratings through a Qualtrics questionnaire. Participants were split into 11 tranches, and each rater also scored a shared consistency set used to detect and correct systematic bias. After quality filtering, 90 native-speaker responses remained.
The resulting ratings correlate well with other language editions, most strongly with English. The benchmark was then used to evaluate static and transformer models. FastText scored highest, slightly ahead of GREEK-BERT, while most other models correlated weakly. This is the first similarity-based intrinsic evaluation for Greek that includes transformer models.
Key points
-
Data collection and quality control: 119 survey responses were reduced to 90 valid ones. Exclusions covered incomplete surveys, failed synonym checkpoint questions, and all-zero answers. Seventeen raters' scores were shifted by 1–2 points based on the 20-pair consistency set.
-
Rating coverage and agreement: each pair received about 5 ratings on average, ranging from 1 to 15. Inter-rater agreement on the consistency set was low, with Krippendorff's α = 0.10.
-
Cross-language sanity check: Greek ratings correlate at 0.70 with English SimLex-999 and 0.65 with Italian.
-
Rating patterns:
- Synonyms received the highest ratings.
- Merely associated pairs received the lowest, below antonyms, which suggests raters grasped the similarity/association distinction.
- Verbs were rated slightly lower than nouns, the reverse of the English pattern. The authors suggest this may be because Greek lacks an infinitive, so verbs appear in first-person present forms.
-
Model results (Spearman, best layer, context-free embeddings):
Model Correlation FastText 0.443 GREEK-BERT 0.404 GreekBART (encoder) 0.387 XLM-RoBERTa 0.129 mBERT 0.088 PaloBERT 0.036 GreekDeBERTa-v3-base 0.024 GPT2-Greek −0.045 The dataset is released at github.com/bloemj/Simlex999-Greek.
Cited by
Cite
@inproceedings{mylonadis2026simlex,
title = {{SimLex-999 for Modern Greek}},
author = {Mylonadis, Leonidas and Bloem, Jelke},
booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
year = {2026},
doi = {10.63317/2ynm43iouxft},
url = {https://doi.org/10.63317/2ynm43iouxft}
}References
- Georgios Alexandridis, Iraklis Varlamis, Konstantinos Korovesis, George Caridakis, and Panagiotis Tsantilas. 2021. A survey on sentiment analysis and opinion mining in Greek social media. Information, 12(8):331. *Figure 1: Spearman Correlation Across Transformer Layers in GREEK-BERT.* a correlation… find
This page in Markdown → https://md.research.elamur.ai/free/papers/mylonadis-2026-simlex-999-modern-greek/