Elamur Research⌕
☰
home / papers / 2025 / Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph

Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph

Paper · 2025 · Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev et al.

Summary

Uncertainty quantification (UQ) can help flag hallucinated or low-quality outputs of large language models. Earlier work on UQ for LLMs used incompatible methods and evaluation setups, so findings were hard to compare.

The authors build a unified benchmark on top of the LM-Polygraph framework. It implements many UQ techniques under one evaluation protocol, grouped into information-based, sample-diversity, density-based and reflexive families, with both white-box and black-box variants. The benchmark covers three task types:

  • selective question answering;
  • selective generation (machine translation and text summarization);
  • claim-level fact-checking, supported by a new automatic pipeline for English, Chinese, Arabic and Russian.

The authors also add a calibration-oriented metric. It measures how well confidence normalization methods turn raw uncertainty into bounded, interpretable scores, and they propose a normalization baseline to test against it.

Experiments span eleven datasets. No single method wins everywhere, because task type and output length strongly affect which family performs best. The open, MIT-licensed benchmark makes future UQ research more comparable and easier to begin.

Key points

  • Selective QA/generation results (PRR with AlignScore and COMET):
    • Sample-diversity methods, especially SAR, DegMat and Semantic Entropy, achieve the best mean results.
    • The MSP baseline lags behind but remains reasonable.
  • Output length matters:
    • For very short answers (under 7 symbols), information-based scores such as MSP, Perplexity, CCP and TokenSAR perform best.
    • For longer outputs, NLI-based black-box methods lead.
    • SAR is the most stable across all lengths.
  • Fact-checking setup: biographies of 100 people were generated with Mistral 7B, Jais 13B, Vikhr 7B and Yi 6B. Extracted claims per setting ranged from 1,031 (Arabic) to 3,104 (Russian).
  • Fact-checking results: CCP gives the best ROC-AUC/PR-AUC in every setting except Chinese, where it ties with P(True).
  • Normalization: binned and isotonic PCC give clearly better calibration (lower MSE against AlignScore) than linear or quantile normalization. Monotonic normalizers keep the PRR of the raw scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia60 works · 21,199 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates47 works · 4,456 citations🎓National Research University Higher School of EconomicsMoscow · Russia16 works · 39,548 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov and Artem Shelmanov. 2025. Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00737

@article{vashurin2025benchmarking,
  title = {{Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph}},
  author = {Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov and Artem Shelmanov},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl\_a\_00737},
  url = {https://doi.org/10.1162/tacl_a_00737}
}

References

2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. 2019. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations. find
  2. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translatio…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/