📄 Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations

Summary

An earlier study by the same author reported that NLI hypotheses with non-upward monotonicity operators (downward-entailing or non-monotone triggers) drew lower annotator agreement in ChaosNLI, with Cliff's δ = −0.284. ChaosNLI, however, contains only items whose majority label originally got exactly three of five votes. This paper asks whether the pattern also appears in the unfiltered source populations. The author ran a preregistered test on the SNLI and MNLI development sets. The same rule-based tagger was used, frozen with no rule changes, and agreement was coded as a four-level ordinal derived from the five original labels. The registered prediction failed. All seven contrasts came out positive, so non-upward items agreed slightly more, and none came near the preregistered smallest effect of interest. Simulated tagger errors and a manual audit both support the measurement. The broader lesson is that structure found in a disagreement-filtered resource may be an artifact of how it was selected. Claims about human label variation (HLV) should therefore say explicitly which selection they depend on.

Key points

• Data: SNLI dev (9,986 items), MNLI matched dev (10,000), and MNLI mismatched dev (9,946, secondary). MED (5,382 pairs) was used only to estimate tagger error rates.
• Effect sizes:
  • Every δ was positive, from +0.030 to +0.059, all below the SESOI of |δ| = 0.10.
  • The SNLI primary contrast was non-significant (+0.032).
  • MNLI matched was significant but in the opposite direction to the registration (+0.045, Holm p = 7.1×10⁻⁵).
  • Separating high- from low-agreement items gave an AUC near chance (0.492 SNLI, 0.484 matched).
• The planned bridge to ChaosNLI entropy could not be computed. On the overlap, every item has a 3/5 majority, so the ordinal has zero variance.
• Tagger-noise simulation:
  • Flip rates came from MED, e.g. a symmetric error of 0.101.
  • It covered 36 grid cells with 1,000 replications each.
  • Noise only pulled δ toward zero, and the 97.5th percentile never went above 0.064.
• Manual audit (200 fresh items, written codebook):
  • Four-class agreement with the tagger was 0.875 (κ 0.607).
  • An earlier round without a codebook reached only 0.135.

Interesting ideas

• I1. A pattern found on a filtered subset should be re-tested on the full population it came from, because the filter can create or flip the effect.
• I2. To check that instrument error cannot fake a result, take misclassification rates measured on an anchor set, inject them as random flips, and confirm the effect shrinks rather than appears.
• I3. Freeze the tool before validating it, and do not patch it in response to the audit sample, since that makes the validation circular.
• I4. Rater agreement jumped from 0.135 to 0.875 once a codebook fixed what was being judged. Low agreement can mean the raters were measuring different constructs, not that they were noisy.
• I5. Tool-versus-human disagreements were split into three layers: lexicon gaps, polysemy or construction failures, and operators whose semantics is genuinely contested (e.g. "only", "many"). Only the first two can be fixed by engineering.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver data kept only where several witness parsers agree is exactly such a filtered subset. Error patterns or repair rules mined from it should be re-checked on all sentences, including the ones that were dropped.
• I2 (learn): Measured tagger or parser error rates could be injected into evaluation runs as a negative control. This would show whether a reported gain survives realistic upstream noise.
• I3 (learn): Self-induced repair rules should be learned on one sample and scored on a separate held-out sample that never feeds back into the rules.
• I4 (learn): Prompts for LLM teacher tiers should state the exact judgment target, such as trigger presence versus entailment direction, before their labels are trusted or compared.
• I5 (world): Monotonicity entries in the compiled knowledge layer could carry a "contested" flag for operators like "only" and "many". The world builder would then keep both readings explicitly rather than committing to one.