Elamur Research
☰
home / papers / 2024 / Measuring short-form factuality in large language models

Measuring short-form factuality in large language models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2411.04368

Summary

Large language models often state false facts with confidence, and factuality is hard to measure when answers are long and full of claims. The OpenAI authors narrow the problem to short questions that each have exactly one verifiable answer, and they release SimpleQA, a benchmark of 4,326 such questions. Human AI trainers wrote the questions adversarially: a question was kept only if at least one of four GPT-4-era model answers got it wrong. A second trainer then had to reach the same answer on their own, and the answer had to be supported by sources from two different web domains. A prompted ChatGPT grader labels each model response as correct, incorrect or not attempted. Frontier models score below 50%; the best result in the paper is 42.7% for o1-preview. The authors also use SimpleQA to test calibration. Models are more accurate when they report higher confidence or give the same answer more often, but they consistently claim more confidence than their accuracy supports. The benchmark gives an inexpensive, low-noise way to track hallucinations and whether models know the limits of their own knowledge.

Key points

  • Dataset: 4,326 questions across many topics. The largest groups are Science & Technology (858), Politics (709) and Art (550). About a third of the answers are dates and about a quarter are people. Wikipedia is a source for roughly 3.5k of the questions. The data is released at github.com/openai/simple-evals.
  • Quality check: a third trainer re-answered 1,000 sampled questions. From the disagreements, the authors estimate that about 3% of the benchmark's reference answers are wrong. The main causes are ambiguous questions and sources that contradict each other.
  • Metrics: the paper reports overall correct, correct given attempted, and an F-score (the harmonic mean of the two). The authors note that the F-score can still reward guessing.
  • Results (correct %): o1-preview 42.7, GPT-4o 38.2, Claude-3.5-sonnet 28.9, GPT-4o-mini 8.6, o1-mini 8.1. Claude models decline to answer much more often than the others (Claude-3.5-sonnet does not attempt 35% of the questions).
  • Calibration: the larger models (o1-preview, GPT-4o) are better calibrated than their mini versions. This holds both for confidence the model states and for how often it repeats an answer across 100 samples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 127 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.04368 · arXiv 2411.04368

@misc{wei2024measuring,
  title = {{Measuring short-form factuality in large language models}},
  author = {Wei, Jason and Karina, Nguyen and Chung, Hyung Won and Jiao, Yunxin Joy and Papay, Spencer and Glaese, Amelia and Schulman, John and Fedus, William},
  year = {2024},
  doi = {10.48550/arxiv.2411.04368},
  eprint = {2411.04368},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.04368}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Agrawal, M. Suzgun, L. Mackey, and A. T. Kalai. Do language models know when they’re hallucinating references? Findings of EACL, 2024. URL https://arxiv.org/abs/2305.18248. ↗
  2. P. Anthropic. Claude 3 model card, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model\_Card\_Claude\_3.pdf. ↗
  3. Q. Cheng, T. Sun, W. Zhang, S. Wang, X. Liu, M. Zhang, J. He, M. Huang, Z. Yin, K. Chen, et al. Evaluating hallucinations in chinese large language models. arXiv preprint arXiv:2310.03368, 2023. URL https://openreview.net/forum?id=1AXvGjfF0V. — [arXiv:2310.03368](https://arxiv.org/abs/2310.03368) ↗
  4. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc of ACL, 2017. URL https://arxiv. org/abs/1705.03551. find our card
  5. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K.… find
  6. A. T. Kalai. Personal communication, July 2024. Communication on July 9, 2024. find
  7. S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint, 2024. URL https://arxiv.org/abs/2409.12941. ↗
  8. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. TACL, 2019. URL ht… find our card
  9. J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In EMNLP, 2023. URL https://arxiv. org/abs/2305.11747. find our card
  10. S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, 2022a. URL https://aclanthology.org/2022.acl-long.229. ↗ our card
  11. S. Lin, J. Hilton, and O. Evans. Teaching models to express their uncertainty in words. In TMLR, 2022b. URL https://arxiv.org/abs/2205.14334. ↗
  12. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In EMNLP, 2023. URL https://aclanthology.org/2023. emnlp-main.741. ↗ our card
  13. OpenAI. Hello gpt-4o, 2024a. URL https://openai.com/index/hello-gpt-4o/. ↗
  14. OpenAI. Openai o1-mini, 2024b. URL https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/. ↗
  15. OpenAI. Learning to reason with llms, 2024c. learning-to-reason-with-llms/. find
  16. T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le, and T. Luong. Freshllms: Refreshing large language models with search engine augmentation, 2023. URL https://arxiv.org/abs/2310.03214. ↗ our card
  17. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In ICML, 2023. URL https://arxiv.org/abs/2203.11171. ↗ our card
  18. J. Wei, C. Yang, X. Song, Y. Lu, N. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, and Q. V. Le. Long-form factuality in large language models. In NeurIPS, 2024. URL https://arxiv.org/abs/2403.18802. ↗ our card
  19. Y. Zhao, J. Zhang, I. Chern, S. Gao, P. Liu, J. He, et al. FELM: Benchmarking factuality evaluation of large language models. NeurIPS, 2024. URL https://arxiv.org/abs/2310.00741. ↗
  20. URL https://openai.com/index/ ↗