📄 Evaluating CxG Generalisation in LLMs via Construction-Based NLI Fine Tuning

Summary

Earlier work by Scivetti et al. (2025) showed that LLMs handle a frequent construction like the Resultative well, even with odd verbs. However, they misread a rarer construction with the same NP V NP ADJ shape, the Depictive, as if it were a Resultative. This paper tests whether that weakness comes from too little exposure to the task, which targeted training could fix.

The authors build ConTest-NLI, a synthetic NLI resource of about 80k sentences covering eight English constructions. These range from lexically fixed patterns (Let-alone) to abstract ones (Resultative, Caused-motion). Premise–hypothesis–label triples come from hand-written syntactic templates filled with mid-frequency vocabulary. A model-in-the-loop filter with partial human validation then screens them. LLaMA 3.1 8B Instruct and Mistral 8B Instruct are fine-tuned on this data and evaluated on the hand-built CxN-NLI and CxN-NLI-Distinction benchmarks.

Accuracy rises on CxN-NLI but barely moves on the Distinction set, where look-alike constructions must be separated by meaning. The conclusion is that fine-tuning mostly teaches surface regularities rather than transferable constructional semantics. This is a caution against assuming that more targeted data alone closes the abstraction gap.

Key points

• ConTest-NLI has about 80k sentences across the eight constructions used by Scivetti et al. (2025). Each construction gets at least 10,000 examples built from 8–12 templates. The templates vary word order, clause type, voice, adjunct position and optional modifiers.
• Lexical slots use lemmas from the 20–60th frequency percentile of BookCorpus, expanded with WordNet synonyms, hyponyms and antonyms, to reduce overlap with pre-training data.
• Fine-tuning results (macro-accuracy, %):
  • On CxN-NLI, LLaMA-3.1-8B went from 57 to 66 and Mistral-8B from 49 to 63.
  • On CxN-NLI-Distinction, LLaMA went only from 33 to 39 and Mistral from 36 to 37.
• GPT-4o reaches 88 on CxN-NLI and 64 on the Distinction set zero-shot, a drop of 24 points. Three-shot ICL with ConTest-NLI examples adds little (91 and 65).
• The hypothesis was fixed in advance: a gain of more than 5 points over baseline counts as success. Error analysis found surface-cue errors in all eight constructions, such as missed location change for way-manner and ignored scalar or negation cues.

Interesting ideas

• I1. Pair a standard test set with a distractor set where items share syntax but differ in meaning; a gain that appears only on the first reveals pattern-matching rather than understanding.
• I2. Fill template slots with mid-frequency lemmas, broadened through WordNet relations, so that memorised collocations cannot solve the items.
• I3. Fix a numeric acceptance threshold before the experiment and judge it on an independent hand-built benchmark, not only on held-out data from the same generator.
• I4. Treat each construction as a licence for specific inferences and test meaning through entailed, contradicted and neutral hypotheses, e.g., "filled the gallery with paintings" entails the gallery contained them.
• I5. Use a model in the loop to keep valid items that look deceptively easy but trip the model, so the benchmark stays challenging as systems improve.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): build a Resultative-vs-Depictive set ("hammered the metal flat" / "buried them alive") where the UD tree is identical. This checks whether prag assigns a caused end state versus a co-occurring state from lexical cues rather than from structure alone.
• I2 (learn): when generating synthetic sentences for silver data or frame tests, sample vocabulary from a middle frequency band so evaluation is not dominated by familiar word combinations.
• I3 (learn): every repair rule or retraining round should have a pre-set improvement bar plus an out-of-distribution set as a negative control against overfitting to the generator.
• I4 (world): compile construction-level rules that create states in the text world (fill-with gives containment, a resultative gives an end state of the object) and check them with NLI-style probe hypotheses.
• I5 (answer): when generating school-style or fable QA tests, keep the verified items that Mova currently gets wrong to form a growing hard set.