Elamur Research⌕
☰
home / papers / 2020 / Natural Language Inference with Mixed Effects

Natural Language Inference with Mixed Effects

Paper · 2020 · William Gantt, Benjamin Kane and Aaron Steven White

Summary

NLI datasets usually collapse several crowd judgments per text-hypothesis pair into one gold label. This hides real disagreement and mixes it with annotators' personal response habits. Gantt, Kane and White add annotator identity to the task and borrow the mixed effects idea from statistics. In one version, each annotator gets a bias term on the output of the classification or regression head (random intercepts). In the other, each annotator gets their own head (random slopes). Both kinds of annotator parameters are tied to a shared prior. They fit these heads on a RoBERTa encoder and train directly on the raw annotations of two datasets that record who gave each label: MegaVeridicality and MegaNegRaising. When annotators are seen in training, random intercepts clearly beat the annotator-blind model. They even beat the best score any annotator-blind model could reach. Random slopes give no further gain. For unseen annotators, both random effects models degrade sharply. The authors conclude that annotators differ mostly in how they use the response scale, not in how they interpret sentences. They argue that dataset builders should release annotator IDs.

Key points

  • Data: MegaVeridicality (MV) has 3,938 pairs, 39,380 three-way categorical labels and 507 annotators. MegaNegRaising (MN) has 7,936 pairs, 79,360 bounded continuous [0,1] labels and 1,108 annotators. Each pair has 10 annotators.
  • Output models:
    • Categorical labels use softmax over the head's potentials plus an annotator offset drawn from a zero-mean Gaussian.
    • Continuous labels use a Beta likelihood. The annotator shifts both its mean and its precision.
  • Setup: pretrained RoBERTa-base (no NLI fine-tuning), a head with 128 hidden units and ReLU, Adam, negative log-likelihood loss, and 5-fold cross-validation. Folds are split four ways: RANDOM, PREDICATE, STRUCTURE and ANNOTATOR.
  • Scoring: results are rescaled so that 0 equals a trivial baseline and 1 equals the best possible fixed model, which predicts the per-pair majority label or mean.
    • Random intercepts score about 1.05–1.15 on MV and about 1.53 on MN when annotators are known.
    • The fixed model scores only 0.23–0.35 on MN.
    • Under ANNOTATOR folds, random intercepts fall to 0.98 on MV and 0.20 on MN; random slopes fall to 0.42 and 0.05.
  • Bias analysis: annotator biases vary most for the neutral label. Neutral is anticorrelated with both contradiction and entailment, which suggests the three-way NLI scale behaves as nominal rather than ordinal.

Interesting ideas

  • I1. Give every label source a small additive offset on the output logits, tied to a shared prior, so that sources with few labels are pulled toward the population average.
  • I2. Report scores rescaled between a trivial baseline and the ceiling of the best model that ignores who labelled the data, so different metrics and datasets become comparable.
  • I3. Build cross-validation folds so that a whole predicate or syntactic structure appears in only one fold, testing lexical and structural generalisation instead of memorisation.
  • I4. Inspect learned annotator offsets with the item-specific signal set to zero, which shows how a judge answers when there is no evidence.
  • I5. Keep the "who judged" information in parameters separate from the linguistic encoder instead of feeding it in as input tokens, so sentence representations stay free of judge identity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Each witness parser or LLM teacher tier could get its own learned bias when silver labels are merged. Systematic quirks of one witness would then be absorbed instead of being taken as disagreement about the sentence.
  • I2 (learn): Evaluation reports could place each model between a majority-label control and an oracle-aggregate ceiling. This makes it obvious when a gain is real and when it beats what aggregation alone allows.
  • I3 (learn): Held-out-verb and held-out-construction splits would make strong negative-control tests for frame or role learners. They would reveal models that only memorise frequent verbs.
  • I4 (learn): Zeroing the input-driven part would expose each teacher's or witness's default preferences. Those defaults could then be audited, for example a habit of overusing "neutral".
  • I5 (frames): Veridicality and implicature judgments could be stored with source-specific adjustments outside the frame representation. The frames themselves would then stay clean and explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as William Gantt, Benjamin Kane and Aaron Steven White. 2020. Natural Language Inference with Mixed Effects. Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gantt2020natural,
  title = {{Natural Language Inference with Mixed Effects}},
  author = {William Gantt, Benjamin Kane and Aaron Steven White},
  booktitle = {Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics},
  year = {2020},
  url = {https://aclanthology.org/2020.starsem-1.9/}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Agresti. 2014. Categorical Data Analysis. John Wiley & Sons. find
  2. Hannah An and Aaron White. 2020. The lexical and grammatical sources of neg-raising inferences. Proceedings of the Society for Computation in Linguistics, 3(1):220–233. ↗
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  4. Dallas Card and Noah A. Smith. 2018. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1636– 1646, New Orleans, Louisiana. Association for Computational Linguistics. find
  5. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. ↗ our card
  6. Paul Felt, Eric Ringger, Kevin Seppi, and Jordan Boyd-Graber. 2018. Learning from measurements in crowdsourcing models: Inferring ground truth from diverse annotation types. In International Conference on Computational Linguistics. find
  7. Andrew Gelman and Jennifer Hill. 2014. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, New York City. find
  8. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166, Hong Kong, China. Association for Computational Linguistics. find
  9. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics. our card
  10. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. Experiments with crowdsourced re-annotation of a POS tagging data set. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 377–382, Baltimore, Maryland. Association for Computational Linguistics. find
  11. Aikaterini-Lida Kalouli, Annebeth Buis, Livy Real, Martha Palmer, and Valeria de Paiva. 2019. Explaining simple natural language inference. In Proceedings of the 13th Linguistic Annotation Workshop, pages 132–143, Florence, Italy. Association for Computational Linguistics. find
  12. John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. 2018. Understanding deep learning performance through an examination of test set difficulty: A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4711–4716, Brussels, Belgium. Association for Computational Linguistics. find
  13. John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648–657, Austin, Texas. Association for Computational Linguistics. find
  14. John P. Lalor, Hao Wu, and Hong Yu. 2019. Learning latent parameters without human response patterns: Item response theory with artificial crowds. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4249– 4259, Hong Kong, China. Association for Computational Linguistics. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. ↗
  16. Richard McElreath. 2020. Statistical Rethinking: A Bayesian course with examples in R and Stan. CRC Press. find
  17. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. ↗ our card
  18. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018a. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585. ↗ our card
  19. Silviu Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. 2018b. A probabilistic annotation model for crowdsourcing coreference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1926–1937, Brussels, Belgium. Association for Computational Linguistics. our card
  20. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗ our card
  21. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card
  22. Ines Rehbein and Josef Ruppenhofer. 2017. Detecting annotation noise in automatically labelled data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1160–1170, Vancouver, Canada. Association for Computational Linguistics. find
  23. Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne, Australia. Association for Computational Linguistics. find
  24. Aaron Steven White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, pages 221–234, Amherst, MA. GLSA Publications. find
  25. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. ↗
  26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  27. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. ↗ our card
  28. Yi Zhang, Zachary Ives, and Dan Roth. 2019. Evidence-based trustworthiness. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 413–423, Florence, Italy. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gantt-2020-natural-language-inference-mixed-effects/