📄 More Labels or Cases? Assessing Label Variation in Natural Language Inference

Summary

The paper argues that summarizing many crowd votes in NLI, either as a majority label or as raw class frequencies, gives an unreliable picture of how uncertain an instance's label really is. The authors analyze ChaosSNLI: 1,514 SNLI development pairs, each relabeled by 100 quality-controlled annotators. They fit a Bayesian multinomial mixture model, estimated with an EM procedure. The model treats the votes as generated by hidden "true" classes and returns a posterior probability over those classes for every instance. They plot each instance in a ternary diagram and compare the borders of the latent classes with the borders implied by majority voting. The latent borders move in ways that make sense semantically. A bootstrap study then shrinks either the number of instances or the number of votes per instance. Class borders stay fairly stable with few instances, as long as each has many votes. With many instances but few votes, estimation breaks down. The practical lesson is that, when label variation is expected, annotation budgets should go to more labels per item.

Key points

• The data is ChaosSNLI, the SNLI part of ChaosNLI, with N = 1,514 instances and J = 100 votes each. The instances are cases where the original five SNLI annotators split 3–2.
• By majority vote, 53.7% of instances are neutral, 27.8% entailment and 18.5% contradiction. This suggests neutral is the class people find hardest to agree on.
• The model is a pooled one, meaning all annotators are assumed equally skilled. Pooling is unavoidable here because ChaosSNLI has no annotator IDs.
• The stability tests used J = 100 with N ∈ {50, 100, 250, 500}, and N = 1000 with J ∈ {5, 10, 25, 50}. Class borders were bootstrapped and compared with the borders from the full data.
• Code and data are released at github.com/corneliagru/label-variation-nli.

Interesting ideas

• I1. Instead of keeping a winner-takes-all label, treat each item's vote counts as a draw from a mixture over hidden classes, and let the whole dataset decide where class boundaries lie.
• I2. For a three-way decision, place every item in a ternary plot by its vote shares and overlay both the majority borders and the learned borders. This makes it visible where the two kinds of borders disagree.
• I3. Test how stable an estimate is by bootstrapping subsamples along two separate axes: fewer items versus fewer votes per item. This shows which resource actually limits reliability.
• I4. Spend annotation effort on many judgments for contested cases, not on broad coverage with thin labeling, since the uncertainty lives mainly in the votes.
• I5. When annotator identities are known, measure their effect by removing one annotator's votes and refitting the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Agreement counts from the witness parsers on each arc or deprel could feed such a mixture, so silver labels become posteriors with calibrated confidence rather than plain majority picks.
• I2 (answer): The entailed / unknown / contradicted verdicts of the readers could be audited this way across several runs or teachers, exposing zones where the readers are systematically confused.
• I3 (learn): Before growing the silver corpus, rerun this kind of experiment to decide whether adding witness parsers or LLM teacher calls beats adding more sentences.
• I4 (learn): Route sentences where witnesses disagree to extra teacher tiers or repeated queries, and accept a single cheap label where witnesses already agree.
• I5 (tree): Leave-one-witness-out refits could reveal which parser or UD dialect conversion biases the agreed silver trees, and by how much.