Elamur Research⌕
☰
home / papers / 2015 / Posterior calibration and exploratory analysis for natural language processing models

Posterior calibration and exploratory analysis for natural language processing models

Paper · 2015 · Khanh Nguyen and Brendan O'Connor

Summary

Probabilistic NLP models produce full distributions over outputs, yet they are usually judged only by their single best prediction. Nguyen and O'Connor argue that these posteriors should be checked directly for calibration, meaning that events predicted with, say, 70% confidence should actually occur about 70% of the time. Overconfident models weaken any downstream method that relies on lower-ranked alternatives, such as K-best lists, sampling or minimum Bayes risk decoding.

The authors propose a practical way to measure miscalibration based on the calibration term of the Brier score decomposition. Because NLP confidence scores are heavily skewed toward 0 and 1, they replace fixed-width bins with equal-count adaptive bins and add a simulation-based confidence interval. They use this to compare common generative and discriminative tagging and classification models.

They also introduce a coreference sampling algorithm that carries uncertainty into a political event extraction task on New York Times articles, yielding credible intervals for event trends. The work matters because calibrated uncertainty tells analysts when an NLP-based conclusion can be trusted.

Key points

  • Calibration metric: CalibErr is defined as the RMS gap between predicted probability and empirical label frequency. It comes from the calibration–refinement decomposition of the Brier score (DeGroot and Fienberg, 1983).
  • Adaptive binning (Algorithm 1): Each bin holds an equal number of points (bin size β), which keeps standard errors bounded by 0.5/√β across the probability range.
  • Confidence interval (Algorithm 2): The 95% interval for calibration error is obtained by sampling per-bin label frequencies from a binomial-normal approximation.
  • Exploratory analysis: Coreference posterior samples produce 95% credible intervals for counts of "attack" events per 3-month period. For example, the Yugoslavia and NATO counts in mid-1999 overlap heavily, so a 1-best system would suggest a difference the model cannot actually support.
  • Error analysis: Event predictions near 50% probability expose difficult coreference cases. One example is "Mr. Hussein" being linked to Iraq in only about half of the samples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations🎓University of Massachusetts AmherstAmherst Center · United States67 works · 64,591 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1182 · arXiv 1508.05154

@inproceedings{nguyen2015posterior,
  title = {{Posterior calibration and exploratory analysis for natural language processing models}},
  author = {Khanh Nguyen and Brendan O'Connor},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1182},
  eprint = {1508.05154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1182}
}

References

48 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman, Brendan O’Connor, and Noah A. Smith. Learning latent personas of film characters. In Proceedings of ACL, 2013.
  2. Paul N. Bennett. Assessing the calibration of naive Bayes’ posterior estimates. Technical report, Carnegie Mellon University, 2000. find
  3. David M. Blei and Peter I. Frazier. Distance dependent Chinese restaurant processes. The Journal of Machine Learning Research, 12: 2461–2488, 2011. find
  4. Elizabeth Boschee, Premkumar Natarajan, and Ralph Weischedel. Automatic extraction of events from open source text for predictive forecasting. Handbook of Computational Approaches to Counterterrorism, page 51, 2013. find
  5. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. find
  6. Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643):1512–1519, 2009. find
  7. Niko Brümmer and George Doddington. Likelihood-ratio calibration using prior-weighted proper scoring rules. arXiv preprint arXiv:1307.7981, 2013. — [arXiv:1307.7981](https://arxiv.org/abs/1307.7981) find
  8. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of LREC, 2014.
  9. Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. The statistician, pages 12–22, 1983. find
  10. Pedro Domingos and Michael Pazzani. On the optimality of the simple Bayesian classifier under zero-one loss. Machine learning, 29(2-3):103– 130, 1997. find
  11. Greg Durrett and Dan Klein. Easy victories and uphill battles in coreference resolution. In EMNLP, pages 1971–1982, 2013.
  12. Greg Durrett and Dan Klein. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics, 2:477–490, 2014.
  13. Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 618– 626. Association for Computa… find
  14. Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian data analysis. Chapman and Hall/CRC, 3rd edition, 2013. find
  15. Kevin Gimpel and Noah A. Smith. Rich source-side context for statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 9–17, 2008. find
  16. Kevin Gimpel and Noah A. Smith. Softmax-margin CRFs: Training log-linear models with cost functions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 733–736. Association for Computational Linguistics, 20… find
  17. Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1100–1111, Seattle, Washington, USA, October 2013. Association for Computat… find
  18. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. find
  19. Joshua Goodman. Parsing algorithms and metrics. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 177–183, Santa Cruz, California, USA, June 1996. Association for Computational Linguistics. doi: 10.3115/981863.981887. URL http://www.aclweb.org/antholog… ↗
  20. Aria Haghighi and Dan Klein. Unsupervised coreference resolution in a nonparametric Bayesian model. In Annual Meeting, Association for Computational Linguistics, volume 45, page 848, 2007.
  21. Robert E. Kass, Bradley P. Carlin, Andrew Gelman, and Radford M. Neal. Markov chain Monte Carlo in practice: a roundtable discussion. The American Statistician, 52(2):93–100, 1998. find
  22. Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 169–176, Boston, Massachusetts, USA, May 2 - May 7 2004. Association for Computational Linguistics.
  23. Kalev Leetaru and Philip A. Schrodt. GDELT: Global data on events, location, and tone, 1979– 2012. In ISA Annual Convention, volume 2, page 4, 2013. find
  24. Jimmy Lin and Alek Kolcz. Large-scale machine learning at Twitter. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 793–804. find
  25. Michael C. McCord, J. William Murdock, and Branimir K. Boguraev. Deep parsing in Watson. IBM Journal of Research and Development, 56 (3.4):3–1, 2012. find
  26. David Mimno and David Blei. Bayesian checking for topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 227–237, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/D11-1021. ↗
  27. Makoto Miwa, Sampo Pyysalo, Tadayoshi Hara, and Jun’ichi Tsujii. Evaluating dependency representations for event extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 779–787, Beijing, China, August 2010. find
  28. Coling 2010 Organizing Committee. URL http://www.aclweb.org/anthology/C10-1088. ↗
  29. Allan H. Murphy and Robert L. Winkler. A general framework for forecast verification. Monthly Weather Review, 115(7):1330–1338, 1987. find
  30. Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. Advances in neural information processing systems, 14: 841, 2002. find
  31. Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, pages 625–632, 2005. find
  32. Brendan O’Connor, Brandon Stewart, and Noah A. Smith. Learning to extract international relations from political context. In Proceedings of ACL, 2013. find
  33. Naoaki Okazaki. Crfsuite: a fast implementation of conditional random fields (CRFs), 2007. URL http://www.chokkan.org/software/crfsuite/. ↗
  34. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830… find
  35. John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in large margin classifiers. MIT Press (2000), 1999. URL http://research.microsoft.com/pubs/69187/svmprob.ps.gz. ↗
  36. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. CoNLL-2011 shared task: Modeling unrestricted coreference in Ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27. Association for Co…
  37. Jonathon Read. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In Proceedings of the ACL Student Research Workshop, pages 43–48. Association for Computational Linguistics, 2005. find
  38. Evan Sandhaus. The New York Times Annotated Corpus. Linguistic Data Consortium, LDC2008T19, 2008. find
  39. Philip A. Schrodt. Precedents, progress, and prospects in political event data. International Interactions, 38(4):546–569, 2012. find
  40. Philip A. Schrodt, Shannon G. Davis, and Judith L. Weddle. KEDS – a program for the machine coding of event data. Social Science Computer Review, 12(4):561 –587, December 1994. doi: 10.1177/089443939401200408. URL http://ssc.sagepub.com/content/12/4/561.abstract. — [doi:10.1177/089443939401200408](https://doi.org/10.1177/089443939401200408) ↗
  41. Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. Joint inference of entities, relations, and coreference. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, pages 1– 6. find
  42. David A. Smith and Jason Eisner. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787–794, Sydney, Australia, July 2006. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P06-2101. ↗
  43. Veselin Stoyanov, Alexander Ropson, and Jason Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In International Conference on Artificial Intelligence and Statistics, pages 725–733, 2011. find
  44. Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. A global joint model for semantic role labeling. Computational Linguistics, 34(2):161–191, 2008.
  45. John W. Tukey. Curves as parameters, and touch estimation. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 681–694, Berkeley, Calif., 1961. University of California Press. URL http://projecteuclid.… ↗
  46. Ashish Venugopal, Andreas Zollmann, Noah A. Smith, and Stephan Vogel. Wider pipelines: Nbest alignments and parses in MT training. In Proceedings of AMTA, 2008. find
  47. Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006. find
  48. Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of KDD, pages 694–699. ACM, 2002. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2015-posterior-calibration-exploratory-analysis-natural/