📄 An Icelandic Linguistic Benchmark for Large Language Models

Summary

The paper tackles the lack of a carefully built test of how well large language models handle Icelandic grammar, a lower-resource language. The authors, native speakers with backgrounds in linguistics and language technology, wrote 1160 benchmark items by hand. The items cover 19 subcategories of syntax, morphology and semantics. Five prompting formats are used:

• yes/no grammaticality judgments
• yes/no well-formedness checks on compound nouns
• fill-in-the-blank items
• one-word fragment answers
• direct question answering

A separate set of 102 translation items, scored by hand, tests whether models produce correct Icelandic. All instructions are written in English. Answers are scored by exact match against a reference answer.

Thirteen commercial and open models were evaluated. Claude-3-5-Sonnet scored highest, and only three models exceeded 70%, so even strong systems have clear gaps. The authors argue that a small, expert-curated and fully inspectable set is a useful complement to large auto-generated suites like BLiMP. They also openly discuss the drawbacks of this choice and of prompt-based evaluation.

Key points

• The main set has 1160 manually written items across 19 subcategories and five methods, plus 102 translation items. The largest subcategories are word formation (280 items), word sense disambiguation (150) and gapping (120).
• Main-set scores:
  • Claude-3-5-Sonnet: 77.24%
  • gpt-4o-2024-08-06: 72.59%
  • Claude-3-Opus: 71.90%
  • All other models: 43.02% to 66.47%. The 4-bit GPTQ version of gpt-sw3-20b-instruct scored lowest.
• On the translation subset, GPT-4o scored 65.7% and Claude-3-5-Sonnet 64.7%.
  • Garden-path sentences and gender agreement were difficult for both.
  • Pronoun anaphora in the translations was close to perfect.
• Log-probability comparison, as advocated by Hu and Levy (2023), was not possible because the closed APIs exposed little or no probability data. The authors used prompting instead.
• An ongoing native-speaker study has so far looked only at gender agreement. It shows near-unanimous agreement with the benchmark's intended answers.

Interesting ideas

• I1. Each yes/no item is also asked in negated form ("is this incorrect?"), and word-sense pairs are presented in both orders, so a model's built-in tendency toward one answer cannot inflate its score.
• I2. When an answer names a noun, the scorer accepts either the inflected form found in the sentence or its nominative citation form, which respects rich case morphology without loosening the check.
• I3. A Wug test asks for past tenses of invented weak verbs, which tests productive morphology rather than memorized forms.
• I4. Translation items are built so that the gender of a pronoun or adjective in the output reveals which antecedent the model chose, turning coreference into something visible in the generated text.
• I5. The authors reject edit-distance scoring because a grammatical and an ungrammatical answer may differ by only one or two letters, so they insist on strict matching.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation harness could pair every binary probe with its inverse and with swapped presentation order as a standing negative control. This would separate real competence from answer bias in student components.
• I2 (answer): QA readers could be scored as correct if they return either the surface token or its lemma, with the accepted forms listed explicitly so the scoring stays auditable.
• I3 (tokens): Nonce-word probes, such as made-up verbs and nouns, could check whether the morphology and lemmatizer in crate en apply rules to unseen stems rather than relying on lookup.
• I4 (back): The generator could be tested with sentences where the target-language agreement must reflect the resolved referent in the meaning graph, exposing coreference errors that would otherwise stay hidden.
• I5 (learn): Morphology and generation outputs should be scored by exact form match, since fuzzy similarity metrics would credit precisely the inflection errors being measured.