Elamur Research⌕
☰
home / papers / 2023 / Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging

Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging

Paper · 2023 · Aarne Talman, Hande Celikkanat, Sami Virpioja et al.

Summary

Human readers often disagree on whether one sentence follows from another, yet NLI benchmarks usually reduce each example to a single majority label. Talman et al. test whether Bayesian-style uncertainty modelling can make an NLI classifier both more accurate and closer to the spread of human judgements. They fine-tune RoBERTa-base on SNLI and MNLI and compare three setups:

  • a plain baseline;
  • Stochastic Weight Averaging (SWA), which averages parameters collected along the training trajectory;
  • SWA-Gaussian (SWAG), which adds a diagonal plus low-rank covariance and averages predictions from several sampled models at test time.

They evaluate in-domain and across datasets, including SICK. They also use cross entropy to compare predicted label distributions with the five-annotator label distributions. Both averaging methods raise accuracy by roughly 0.7–1.2 points in-domain and usually help in cross-dataset tests. SWAG gives the best match to annotator disagreement in every setting. The study shows that cheap posterior approximations can make NLU models better calibrated to genuine interpretive variation.

Key points

  • Model and tools: RoBERTa-base from Hugging Face Transformers, trained on majority-vote labels. SWA comes from PyTorch 1.12 and SWAG from the Maddox et al. implementation. One weight sample is taken per epoch, and SWAG predictions average 20 sampled models.

  • In-domain accuracy (5 seeds):

    Test setBaseSWASWAG
    SNLI90.8091.4791.59
    MNLI-m86.5387.6087.76
    MNLI-mm86.3187.3487.51

    MNLI is evaluated on its development sets.

  • Cross-dataset accuracy: SNLI→MNLI-m rises from 77.31 to about 79.3–79.7. Results on SICK stay near 57% for all methods, and for MNLI→SICK the averaged models are slightly worse than the baseline.

  • Fit to annotator distributions (cross entropy): SNLI goes from 0.83 (base) to 0.75 (SWA) and 0.69 (SWAG). SNLI→MNLI-m goes from 1.13 to 0.90 and 0.80. SWAG beats SWA in every case.

  • Code: released at github.com/Helsinki-NLP/uncertainty-aware-nli.

Interesting ideas

  • I1. Score a model not only on majority-label accuracy but also on cross entropy against the full multi-annotator label distribution, which rewards honest uncertainty on genuinely ambiguous items.
  • I2. Get an almost free ensemble by averaging checkpoints from the optimizer's own path (one per epoch), which tends to land in wider optima that generalize better.
  • I3. Where a calibrated model and human disagreement diverge sharply, the cause is often a questionable annotation, so the gap can steer active learning toward noisy labels.
  • I4. Use the cheaper method (SWA) as the baseline the costlier one (SWAG) must beat, so extra computation is justified only by measured gains.
  • I5. Train on one corpus and test on another (SNLI→MNLI-mm, →SICK) to reveal how much a gain is domain-specific; SICK exposed that NLI training data transfers poorly to rule-built logic items.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treat the vote split among witness parsers as a soft target and check whether gparse's arc/label probabilities match that split, not just the agreed tree.
  • I2 (tree): Apply epoch-wise SWA when training gparse, and sample SWAG weights to obtain per-arc confidence that later stages can use to flag shaky attachments.
  • I3 (learn): Send silver sentences where witnesses disagree but the model is confident (or the reverse) to the LLM teacher tier, or use them to propose repair rules.
  • I4 (learn): Any added machinery, such as a heavier parser or an extra teacher tier, should be compared against its simplest averaged or ensembled counterpart before adoption.
  • I5 (answer): Evaluate readers trained on fables against school-course texts and maths word problems to see whether improvements in frames and world-building actually transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations🎓Aalto UniversityEspoo · Finland13 works · 648 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Aarne Talman, Hande Celikkanat, Sami Virpioja, Markus Heinonen and Jörg Tiedemann. 2023. Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging. Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{talman2023uncertainty,
  title = {{Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging}},
  author = {Aarne Talman, Hande Celikkanat, Sami Virpioja, Markus Heinonen and Jörg Tiedemann},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  year = {2023},
  url = {https://aclanthology.org/2023.nodalida-1.37/}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? does it matter? Structural Safety, 31(2):105–112. ↗
  2. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385– 393, Montréal, Canada. Association for Computational Linguistics.
  3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. ↗
  4. Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, Ali Ghodsi, and Phillippe Langlais. 2022. Improving generalization of pre-trained language models via stochastic weight averaging. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4948–4954, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  6. Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. 2019. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗
  7. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. ↗
  8. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  9. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA. PMLR. ↗
  10. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
  11. Yingbo Gao, Christian Herold, Zijian Yang, and Hermann Ney. 2022. Revisiting checkpoint averaging for neural machine translation. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 188–196, Online only. Association for Computational Linguistics. find
  12. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  13. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876– 885. Association For Uncertainty in Artificial Intelligence (AUAI). ↗
  14. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616–9625. find
  15. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  16. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. 2022. When do flat minima optimizers work? In Advances in Neural Information Processing Systems. find
  17. Urja Khurana, Eric Nalisnick, and Antske Fokkens. 2021. How emotionally stable is ALBERT? testing robustness with stochastic weight averaging on a sentiment analysis task. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 16–31, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  18. Aarne Talman and Stergios Chatzikyriakidis. 2019. Testing the generalization power of neural network models across NLI benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talman-2023-uncertainty-aware-natural-language-inference/