📄 Embracing Ambiguity: Shifting the Training Target of NLI Models

Summary

NLI benchmarks contain many pairs where annotators disagree, yet models are usually trained on a single gold label. The authors ask whether training directly on the annotator label distribution helps. They build AmbiNLI, a trial resource of 91,717 premise–hypothesis pairs. Each pair carries a probability distribution derived from the 5-annotator labels in SNLI and MNLI, or from UNLI's scalar scores via a piecewise-linear mapping. Items overlapping with ChaosNLI are removed. A BERT-base model, first trained on SNLI+MNLI gold labels, is then finetuned with cross-entropy against these soft targets. It is evaluated on ChaosNLI, which has 100 annotations per item, using Jensen-Shannon Divergence (JSD) and accuracy.

Soft-label finetuning on the SNLI/MNLI part (AmbiSM) cuts JSD by about a fifth. It also beats an otherwise identical model trained on gold labels in accuracy, and gives better frozen-encoder features for transfer tasks. UNLI-derived data hurts unless near-certain items are filtered out. The authors present this as evidence that collecting and releasing multi-annotator labels is worthwhile, not as a final dataset.

Key points

• AmbiNLI contains 18,152 SNLI, 18,048 MNLI and 55,517 UNLI examples. The authors explicitly discourage reusing it and call for new multi-annotation collection instead.
• UNLI conversion maps a value p to a 3-way distribution:
  • p < 0.5 becomes (0, 2p, 1−2p).
  • Otherwise it becomes (2p−1, 2−2p, 0).
• Setup: BERT-base with a softmax head, batch size 128 and learning rate 1e-5, after 3 epochs of gold-label pretraining on SNLI+MNLI.
• With AmbiSM, ChaosSNLI JSD falls from 0.2379 to 0.1893 and ChaosMNLI JSD from 0.3349 to 0.2619 (about 20.5% and 21.7% decreases). Accuracy rises slightly over the same-data gold-label model.
• UNLI items with p < 0.05 or p > 0.97 produce extreme distributions. Removing them lowers AmbiU JSD by roughly 20%.
• Three-fold cross-validation on ChaosMNLI shows average accuracy of 0.5210 with soft labels versus 0.5009 with gold labels.
• JSD stays stable across three entropy bins, while accuracy drops as entropy grows.

Interesting ideas

• I1. Instead of collapsing annotator votes into one label, train on the vote proportions as a soft target; the model then learns how contested an item is, not just its majority answer
• I2. Compare two models trained on exactly the same examples, differing only in hard versus soft targets, so any gain is attributable to the target itself
• I3. A graded scalar judgment can be turned into a categorical distribution with a simple piecewise-linear map, but saturated values near the extremes skew training and are worth filtering
• I4. Bin the evaluation set by annotator-disagreement entropy and report metrics per bin, to check that the model's own uncertainty tracks how ambiguous items really are
• I5. Judge representation quality by freezing the encoder and training only a small head on downstream tasks

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When several witness parsers disagree on silver data, the fraction agreeing on each head or label could serve as a soft training target, rather than discarding or majority-voting the disputed arcs.
• I2 (learn): Any change to how silver labels are aggregated should be tested with this same-data control, alongside existing negative controls, to isolate the effect of the labeling scheme.
• I3 (frames): Graded confidence scores for implicature or irony could be mapped into label distributions the same way, with near-certain extremes capped or dropped to avoid overconfident training.
• I4 (answer): Test questions for the readers could be grouped by how much humans or teachers disagree, verifying that answer confidence falls on genuinely ambiguous fable or word-problem items.
• I5 (tree): An encoder in gparse trained with soft attachment targets could be probed with frozen layers on frame or tagging tasks, to see whether it carries more transferable structure than one trained on hard labels.