Elamur Research⌕
☰
home / papers / 2026 / Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit

Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit

Paper · 2026 · Julien Sylvestre

Summary

SciFact is a widely cited benchmark for scientific claim verification, also part of BEIR. Over six years its expert-assigned gold labels were never checked, and no errata were published. This paper audits the 209 dev and 564 training claim–document pairs that have gold evidence. The audit has two stages. First, GPT-5.4-mini screens every pair cheaply. Then one human annotator reviews all pairs, using GPT-5.4 as a conversational second opinion on the flagged ones, and checks each correction against the source PubMed paper.

The audit finds label errors in both splits, and most dev errors wrongly mark a claim as SUPPORT. Correcting the dev labels improves zero-shot macro-F1 for GPT-5.4 and Claude Haiku 4.5 models. These gains are about as large as the margins separating top leaderboard systems, so existing rankings may be partly driven by annotation noise. The authors publish corrected labels and a blind re-annotation packet. They ask for an independent multi-annotator re-audit and acknowledge that their review relied on a single human annotator.

Key points

  • Error rates: 11 dev-split pairs turned out to be mislabeled, a rate of 5.3% with a 95% confidence interval of 2.7–9.2%, while the training split contained 13 mislabeled pairs (2.3%, interval 1.2–3.9%).
  • Screening: the automated stage cost $0.11 in API fees and flagged 57 dev pairs, most of which were false alarms. The full manual review found 3 further errors that the screen had missed.
  • Error types and direction: the dev errors fall into four types: label reversal (4), outcome mismatch (3), entity mismatch (3) and overgeneralization (1). Nine of the 11 wrongly assign SUPPORT (one-sided binomial p=0.033, two-sided p=0.065).
  • Binary F1 impact: after correction, binary macro-F1 rises by +3.8 for GPT-5.4-mini (84.9→88.7), +1.7 for GPT-5.4-nano and +2.98 for Claude Haiku 4.5. For comparison, MultiVerS and ARSJoint are separated by about 1.25 F1 on the test leaderboard.
  • 3-way F1 and contamination: Haiku's 3-way macro-F1 rises by +9.24 (61.85→71.09) when six pairs are relabeled as NEI. A claim-only probe does not suggest memorization explains the gains. Corrections, the blind packet and scripts are on GitHub (Kefez/scifact-audit-bionlp2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Julien Sylvestre. 2026. Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.9

@inproceedings{sylvestre2026gold,
  title = {{Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit}},
  author = {Julien Sylvestre},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.9},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.9}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saso Cemerski, Jayajit Das, Emanuele Giurisato, Mary A. Markiewicz, Paul M. Allen, Arup K. Chakraborty, and Andrey S. Shaw. 2008. The balance between T cell receptor signaling and degradation at the center of the immunological synapse is determined by antigen quality. Immunity, 29(3):414– 422. find
  2. Daria Esterházy, Ina Stützer, Haiyan Wang, Markus P. Rechsteiner, Jeremy Beauchamp, Heinz Döbeli, Hans Hilpert, Hugues Matile, Michael Prummer, Alexander Schmidt, and 1 others. 2011. Bace2 is a β cell-enriched protease that regulates pancreatic β cell function and mass. Cell Metabolism, 14(3):365–37… find
  3. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL-HLT, pages 107–112.
  4. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 49(1):157–198.
  5. Miloš Košprdić, Adela Ljajić, Darija Medvecki, Bojana Bašaragin, and Nikola Milošević. 2024. Scientific claim verification with fine-tuned NLI models. In Proceedings of IC3K/KMIS. find
  6. Fan-Tao Meng, Yan-Li Wang, Juan Liu, Jing Zhao, Rui-Yuan Liu, and Jiang-Ning Zhou. 2012. ApoE genotypes are associated with age at natural menopause in Chinese females. Age, 34(3):653–660. find
  7. Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor, and Roi Reichart. 2025. Are LLMs better than reported? Detecting label errors and mitigating their effect on model performance. In Proceedings of EMNLP.
  8. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of EMNLP.
  9. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of NeurIPS. find
  10. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the ACL, 7:677–694.
  11. Susanna Rucker and Alan Akbik. 2023. CleanCoNLL: A nearly noise-free named entity recognition dataset. In Proceedings of EMNLP. find
  12. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of EMNLP. find
  13. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of NeurIPS Datasets and Benchmarks. find
  14. Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. 2025. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461. find
  15. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of EMNLP, pages 7534–7550.
  16. David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022a. MultiVerS: Improving scientific claim verification with weak supervision and full-document context. In Findings of NAACL. find
  17. David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022b. SciFact-Open: Towards open-domain scientific claim verification. In Findings of EMNLP. and contamination-probe scripts; (5) the statistical-analysis script that reproduces every CI and binomial test in this…
  18. Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of ACL.
  19. Zhiwei Zhang, Jihua Fang, and Barbara Plank. 2021. Abstract, rationale, stance: A joint model for scientific claim verification. In Proceedings of EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sylvestre-2026-gold-label-errors-scifact-benchmark/