📄 BLiMP-IT: Harnessing Automatic Minimal Pair Generation for Italian Language Model Evaluation

Summary

Italian lacked a benchmark that would test the grammatical competence of language models on minimal pairs, as BLiMP does for English. The authors create such a resource, BLiMP-IT, and describe the automatic part of its construction in detail. A minimal pair consists of two nearly identical sentences that differ in only one morphosyntactic feature. One of them is grammatical, while the other violates a particular rule. A correct model should rate the former as more probable. BLiMP-IT combines and adapts the existing datasets COnVERSA, AcCompl-it and BLiMP and supplements them with new pairs. For automatic generation, the authors used a corpus of Italian texts for a child audience. From it they extracted a lexicon, tagged it for POS/UPOS using spaCy, and manually annotated the nouns for animacy. Words from the lexicon are then inserted into manually designed tag-based templates. In this way, pairs can be generated in large numbers, and each is guaranteed to differ only in the feature under investigation. The authors show that this pipeline works: it provides broad coverage of grammatical phenomena and reduces the need for manual annotation.

Key points

• BLiMP-IT covers 78 morphosyntactic phenomena grouped into four categories: Agreement and Inflection, Verb Class and Argument Structure, Pronouns and Non-local Dependencies. Among them are, in particular, determiner–noun agreement, placement of reflexive clitics, long-distance wh-dependencies and island effects.
• Data sources: the Italian COnVERSA and AcCompl-it, as well as the adapted English BLiMP with 67 minimal-pair paradigms.
• The corpus for the lexicon was originally prepared for the BabyLM challenge. After cleaning, it contains 2,431,038 tokens, with an overall TTR of 0.03. The corpus has five parts:
  • CHILDES — 346,155 tokens;
  • subtitles of children's films from OpenSubtitles — 700,729;
  • Zecchino D'Oro songs — 222,572;
  • VoLIP telephone conversations — 58,039;
  • fairy tales — 1,287,826.
• The benchmark can be used in two ways: by comparing the log-likelihood of the two sentences in a pair, or by giving the model a binary acceptability classification task.
• The resources are available at nets-lab.github.io/blimpit/, and the generated data at nets-lab.github.io/blimpit-generation/. The authors note that the generated data are preliminary and are still being refined.