Elamur Research⌕
☰
home / papers / 2026 / SimLex-999 for Modern Greek

SimLex-999 for Modern Greek

Summary

Benchmarks of human word-similarity judgements remain a standard way to evaluate word representations intrinsically. Until now, Greek had no such resource. There was only a translated WordSim353, which measures association rather than similarity and reused the English scores, plus a relatedness set for Ancient Greek. The authors fill this gap by building a Modern Greek SimLex-999.

They translated the 999 English pairs using the Multi-SimLex guidelines, replacing items where Greek would merge two words into one. They then collected ratings through a Qualtrics questionnaire. Participants were split into 11 tranches, and each rater also scored a shared consistency set used to detect and correct systematic bias. After quality filtering, 90 native-speaker responses remained.

The resulting ratings correlate well with other language editions, most strongly with English. The benchmark was then used to evaluate static and transformer models. FastText scored highest, slightly ahead of GREEK-BERT, while most other models correlated weakly. This is the first similarity-based intrinsic evaluation for Greek that includes transformer models.

Key points

  • Data collection and quality control: 119 survey responses were reduced to 90 valid ones. Exclusions covered incomplete surveys, failed synonym checkpoint questions, and all-zero answers. Seventeen raters' scores were shifted by 1–2 points based on the 20-pair consistency set.

  • Rating coverage and agreement: each pair received about 5 ratings on average, ranging from 1 to 15. Inter-rater agreement on the consistency set was low, with Krippendorff's α = 0.10.

  • Cross-language sanity check: Greek ratings correlate at 0.70 with English SimLex-999 and 0.65 with Italian.

  • Rating patterns:

    • Synonyms received the highest ratings.
    • Merely associated pairs received the lowest, below antonyms, which suggests raters grasped the similarity/association distinction.
    • Verbs were rated slightly lower than nouns, the reverse of the English pattern. The authors suggest this may be because Greek lacks an infinitive, so verbs appear in first-person present forms.

  • Model results (Spearman, best layer, context-free embeddings):

    ModelCorrelation
    FastText0.443
    GREEK-BERT0.404
    GreekBART (encoder)0.387
    XLM-RoBERTa0.129
    mBERT0.088
    PaloBERT0.036
    GreekDeBERTa-v3-base0.024
    GPT2-Greek−0.045

    The dataset is released at github.com/bloemj/Simlex999-Greek.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2ynm43iouxft

@inproceedings{mylonadis2026simlex,
  title = {{SimLex-999 for Modern Greek}},
  author = {Mylonadis, Leonidas and Bloem, Jelke},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/2ynm43iouxft},
  url = {https://doi.org/10.63317/2ynm43iouxft}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Georgios Alexandridis, Iraklis Varlamis, Konstantinos Korovesis, George Caridakis, and Panagiotis Tsantilas. 2021. A survey on sentiment analysis and opinion mining in Greek social media. Information, 12(8):331. *Figure 1: Spearman Correlation Across Transformer Layers in GREEK-BERT.* a correlation… find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mylonadis-2026-simlex-999-modern-greek/