📄 GreekCommonGen: A Benchmark for Evaluating Generative Commonsense Reasoning in Greek

Summary

Greek has few evaluation resources for large language models. In particular, it had no benchmark for generative commonsense reasoning (GCSR), in which a model must write a plausible everyday sentence that uses a given set of concepts. The authors create GreekCommonGen by adapting the English CommonGen in two stages. First, reference sentences were machine-translated with MADLAD-3B. Then a native speaker post-edited them for grammar, fluency and meaning, for example fixing "bow" rendered as the weapon instead of the decorative knot. Only after that were the concept sets translated, following fixed conventions: verbs in first-person singular present and nouns in nominative singular. Curation took roughly three months. The authors then test Greek LLMs from the Meltemi and Llama-Krikri families under zero-shot and few-shot prompting and several decoding strategies. They score outputs with BLEU-4 and CIDEr adapted to Greek, plus human ratings. The results section is missing from the provided text, so specific scores are not reported here. The freely released dataset fills a gap for evaluating Greek text generation.

Key points

• Dataset size:
  • Test split: 1,004 reference sentences over 342 concept sets.
  • Training split: 3,868 references over 1,843 concept sets.
  • Each concept set has 1–3 references.
  • Released on GitHub (aristotelisStam/CommonGen_Greek).
• Adapted metrics:
  • BLEU-4 is computed over lemmas produced by spaCy's el_core_news_sm, after Unicode normalization, accent removal and lowercasing.
  • CIDEr uses log-smoothed document frequencies.
  • SPICE was not ported to Greek.
• Human evaluation: 50 concept sets were rated on 5-point Likert scales for grammaticality, commonsense plausibility and concept coverage.
• Models evaluated:
  • Meltemi-7B-Instruct-v1.5 (5-bit GGUF).
  • Llama-Krikri-8B-Instruct, both full and 5-bit GGUF.
  • A 4-bit Llama-Krikri-8B-Instruct fine-tuned with QLoRA on the training split.
• Related resources: Comparable GCSR benchmarks exist for other languages: COCOTEROS (Spanish) and MULTICOM (English, Spanish, Dutch, Valencian).