Elamur Research⌕
☰
home / papers / 2022 / Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression

Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression

Paper · 2022 · Yuxia Wang, Daniel Beck, Timothy Baldwin et al.

Summary

Pre-trained language models used for text regression tend to be overconfident when they are wrong. This limits their use in high-stakes areas such as clinical decision support. Calibration work in NLP has mostly targeted classifiers, so Wang et al. study how well regression models built on pre-trained LMs can express uncertainty. They compare four estimation techniques: sparse Gaussian processes, Bayesian linear regression, Bayes by backprop, and MC dropout. Each is tested in pipeline setups (fixed SBERT embeddings) and end-to-end fine-tuning setups. Evaluation covers three tasks and ten datasets, using intrinsic calibration metrics and downstream use. On top of these estimates, the authors propose a simple way to grow training data in low-resource domains: keep only instances whose predictions carry low variance. This drops noisy gold or pseudo labels and keeps clean ones. In both self-training and active-learning scenarios, the selection strategy consistently improved accuracy and generalization. The work shows that uncertainty estimates are useful both for interpreting predictions and for curating data.

Key points

  • Tasks and data:
    • STS: STS-B, MedSTS, N2C2-STS, BIOSSES, EBMSASS
    • Sentiment rating: Yelp, PeerRead
    • MT quality estimation (direct assessment): WMT 2020 en-zh, ru-en, si-en, covering high-, medium- and low-resource language pairs
  • Methods excluded: Deep ensembles were left out because training several models on split data worsens overfitting when data is scarce. MC dropout was used as the ensemble-style method instead.
  • Fixed features vs. fine-tuning: Bayesian LR and sparse GP on fixed features had lower calibration error and NLPD than end-to-end fine-tuned networks, but weaker correlation. With expressive enough embeddings, they matched deep uncertainty models.
  • Bayes by backprop: Its variance on STS-B was near zero (about 0.005 on average), producing infinite NLPD. It was also overconfident on wrong predictions.
  • Detecting annotator disagreement: The large-variance criterion flagged high-disagreement labels in EBMSASS and BIOSSES with Acc ≈ 0.44 and F1 ≈ 0.39 on average. This is weak as a standalone classifier but effective for filtering out-of-domain training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓RMIT UniversityMelbourne · Australia17 works · 703 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yuxia Wang, Daniel Beck, Timothy Baldwin and Karin Verspoor. 2022. Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00483

@article{wang2022uncertainty,
  title = {{Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression}},
  author = {Yuxia Wang, Daniel Beck, Timothy Baldwin and Karin Verspoor},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00483},
  url = {https://doi.org/10.1162/tacl_a_00483}
}

References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, and Vladimir Makarenkov, and Saeid Nahav. 2020. A review of uncertainty quantification in deep learning: Techniques, applications and challe… — [doi:10.1016/j.inffus](https://doi.org/10.1016/j.inffus) find
  2. Daniel Beck, Trevor Cohn, and Lucia Specia. 2014. Joint emotion analysis via multi-task Gaussian processes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1798–1803. https://doi.org/10 .3115/v1/D14-1190 find
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, volume 382, pages 41–48. ACM. https://doi.org/10.1145/1553374 .1553380 \#\# Conclusion We comprehensively investig… ↗
  4. Zsolt Bitvai and Trevor Cohn. 2015. Predicting peer-to-peer loan rates using Bayesian non-linear regression. In Proceedings of the AAAI Conference on Artificial Intelligence. arXiv:2003.07892. https://doi.org/10 .18653/v1/2020.emnlp-main.21 find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural networks. In International Conference on Machine Learning, pages 1613–1622. http://proceedings.mlr.press/v37/blundell15.pdf. ↗
  7. Enrico Camporeale and Algo Carè. 2020. Estimation of accurate and calibrated uncertainties in deterministic models. CoRR, abs/2003.05103. — [arXiv:2003.05103](https://arxiv.org/abs/2003.05103) find
  8. Piero Esposito. 2020. BLiTZ – Bayesian Layers in Torch Zoo (a Bayesian deep learing library for Torch). https://github.com/piEsposito/blitz-bayesian-deep -learning/. ↗
  9. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. Vancouver, Canada.…
  10. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050–1059. http://proceedings.mlr.press/v48/gal16.pdf. ↗
  11. Chacha Chen, Junjie Liang, Fenglong Ma, Lucas M. Glass, Jimeng Sun, and Cao Xiao. 2020. Unite: Uncertainty-based health risk prediction leveraging multi-sourced data. arXiv preprint arXiv:2010.11389. https://doi.org/10.1145/3442381.3450087 — [doi:10.1145/3442381.3450087](https://doi.org/10.1145/3442381.3450087) ↗
  12. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
  13. Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. arXiv preprint arXiv: 2010.12821.
  14. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. 2021. Uncertainty-aware machine translation evaluation. CoRR, abs/2109.06352. https://doi.org/10.18653/v1/2021.findings-emnlp.330 ↗
  15. Courtney Corley and Rada Mihalcea. 2005. Measuring the semantic similarity of texts. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 13–18. https://doi.org/10 .3115/1631862.1631865
  16. Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. 2007. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243–268. https://doi.org/10.1111/j.1467-9868.2007.00587.x Sharon E. Davis, Thomas A. Lasko, Guanh… — [doi:10.1111/j.1467-9868.2007.00587.x](https://doi.org/10.1111/j.1467-9868.2007.00587.x) ↗
  17. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. https://doi.org/10.1017/S1351324915 000339 ↗
  18. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321–1330. find
  19. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. arXiv preprint Hamed Hassanzadeh, Anthony Nguyen, and Karin Verspoor. 2019. Quantifying semantic similarity of clinical evidence in the biomedical literature to facilitate related evidence synthesis. Journal of Biomedical I… find
  20. José Miguel Hernández-Lobato and Ryan Adams. 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning, pages 1861–1869. http://proceedings.mlr.press/v37/hernandez -lobatoc15.pdf. .mlr.press/v80/kuleshov18a/kuleshov 18a.… ↗
  21. M. Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, volume 1. https://papers.nips.cc/paper/2010/file/e57c6b956a6521b28495f2886c a0977a-Paper.pdf. ↗
  22. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems. https://arxiv.org/pdf/1612.01474.pdf Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2… ↗
  23. Max-Heinrich Laves, Sontje Ihler, Jacob F. Fast, Lüder A. Kahrs, and Tobias Ortmaier. 2020. Well-calibrated regression uncertainty in medical imaging with deep learning. In Medical Imaging with Deep Learning, pages 393–412. http://proceedings.mlr.press/v121/laves20a/laves20a.pdf. ↗
  24. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
  25. Specia Lucia, Fomicheva Marina, Blain Frédéric, Guzmán Paco, Chaudhary Vishrav, Fonseca Erick, and Martins André. 2020. find
  26. WMT 2020 quality estimation dataset. https://www.statmt.org/wmt20/qualityestimation -task.html. ↗
  27. Brian Keith, Exequiel Fuentes, and Claudio Meneses. 2017. A hybrid approach for sentiment analysis applied to paper. In Proceedings of ACM SIGKDD Conference. find
  28. Kristian Miok, Gregor Pirs, and Marko Robnik-Sikonja. 2020. Bayesian methods for semi-supervised text annotation. arXiv preprint arXiv:2010.14872. find
  29. Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pages 5574–5584. https://proceedings.neurips.cc/paper/2017/hash/2650d6 089a6d640c5e85b2b88265dc2b-Abstrac t.html. ↗
  30. Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. 2019. Bayesian batch active learning as sparse subset approximation. In Advances in Neural Information Processing Systems, volume 32, pages 6359–6370. https://proceedings.neurips.cc/paper/2019/file/84c2d48 60a0fc27… ↗
  31. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. 2018. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pages 2796–2804. http://proceedings find
  32. Daniel Preoţiuc-Pietro and Trevor Cohn. 2013. A temporal model of text periodicities using Gaussian processes. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 977–988. find
  33. Felix Leibfried, Vincent Dutordoir, S. T. John, and Nicolas Durrande. 2020. A tutorial on sparse gaussian processes and variational inference. arXiv preprint arXiv:2012.13962. — [arXiv:2012.13962](https://arxiv.org/abs/2012.13962) find
  34. Puria Radmard, Yassir Fathullah, and Aldo Lipani. 2021. Subsequence based deep active learning for named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCN… find
  35. Joachim Sicking, Maram Akila, Maximilian Pintz, Tim Wirtz, Asja Fischer, and Stefan Wrobel. 2021. A novel regression loss for non-parametric uncertainty optimization. arXiv preprint arXiv:2101.02726. — [arXiv:2101.02726](https://arxiv.org/abs/2101.02726) find
  36. Gizem Soğancıoğlu, Hakime Öztürk, and Arzucan Özgür. 2017. BIOSSES: A semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14):i49–i58. https://doi.org/10.1093/bioinforma tics/btx238, PubMed: 28881973 — [doi:10.1093/bioinforma](https://doi.org/10.1093/bioinforma) ↗
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  38. Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. 2019. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897–5906. http://proceedings.mlr.press/v97/song19a/song19a.pdf C. Rusmassen and C. Williams. 2005. Gaussian process for machine learning. https:… — [doi:10.7551/mitpress/3206.001.0001](https://doi.org/10.7551/mitpress/3206.001.0001) ↗
  39. Omkar Sabnis. 2018. Yelp review dataset. https://www.kaggle.com/omkarsabnis/yelp-reviews-dataset Burr Settles. 2009. Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences. http://burrsettles.com/pub/settles.active learning.pdf. ↗
  40. Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25–27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Inte… find
  41. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020. Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. arXiv preprint arXiv:2010.08240. https://doi.org/10.18653/v1/2021 .naacl-main.28 ↗
  42. Michalis Titsias. 2009. Variational learning of inducing variables in sparse Gaussian processes. In Artificial intelligence and statistics, pages 567–574. http://proceedings.mlr.press/v5/titsias09a/titsias 09a.pdf. ↗
  43. Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. 2019. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3459–3467. http://proceedings.mlr.press/v89/vaicenavici u… ↗
  44. Yu Wan, Baosong Yang, Derek F. Wong, Yikai Zhou, Lidia S. Chao, Haibo Zhang, and Boxing Chen. 2020. Self-paced learning for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16–20, 2020, pages 1074–1080… find
  45. Aili Shen, Daniel Beck, Bahar Salehi, Jianzhong Qi, and Timothy Baldwin. 2019. Modelling uncertainty in collaborative document quality assessment. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 191–201, Hong Kong, China. Association for Computational Linguistics. find
  46. Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2018. MedSTS: A resource for clinical semantic textual similarity. Language Resources and Evaluation, pages 1–16. https://doi.org/10.1007/s10579-018 -9431-1 ↗
  47. Yanshan Wang, Sunyang Fu, Feichen Shen, Sam Henry, Ozlem Uzuner, and Hongfang Liu. 2020a. find
  48. The 2019 n2c2/OHNLP track on clinical semantic textual similarity: Overview. JMIR Medical Informatics, 8(11). https://doi.org/10.2196/23375 putational Linguistics. https://doi.org/10.18653/v1/2020.bionlp-1.11 ↗
  49. Yuxia Wang, Karin Verspoor, and Timothy Baldwin. 2020c. Learning from unlabeled data for clinical semantic textual similarity. In Proceedings of the 3rd Clinical NLP Workshop, Online. EMNLP. https://doi.org/10 .18653/v1/2020.clinicalnlp-1.25 find
  50. Boyang Xue, Jianwei Yu, Junhao Xu, Shansong Liu, Shoukang Hu, Zi Ye, Mengzhe Geng, Xunying Liu, and Helen Meng. 2021. Bayesian transformer language models for speech recognition. arXiv preprint arXiv:2102.04754. https://doi.org/10.1109/ICASSP39728 .2021.9414046 — [doi:10.1109/icassp39728](https://doi.org/10.1109/icassp39728) ↗
  51. Yuxia Wang, Fei Liu, Karin Verspoor, and Timothy Baldwin. 2020b. Evaluating the utility of model configurations and data augmentation on clinical semantic textual similarity. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 105–111, Online. Association for Com-E… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2022-uncertainty-estimation-reduction-pre-trained/