Elamur Research⌕
☰
home / papers / 2026 / NSF-SciFy: Mining the NSF Awards Database for Scientific Claims

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims

Paper · 2026 · Delip Rao, Weiqiu You, Eric Wong et al.

Summary

Existing datasets for checking scientific claims are small, number in the thousands of claims, and mostly cover biomedicine or health, drawing on published papers or fact-checking sources. This work uses a new source: abstracts of funded National Science Foundation grants. The authors parsed the full NSF Awards database (1970 to September 2024) and used zero-shot prompting of Claude-3.5-Sonnet to pull out two kinds of statements from each abstract. The first is claims, meaning assertions or assumptions presented as true. The second is investigation proposals, meaning planned research activities. Extracting both together kept the model from mislabeling planned work as established fact. The resulting resource, NSF-SciFy, has about 2.8M claims and is at least an order of magnitude larger than earlier claim corpora. The authors tested it on three tasks: claim extraction, proposal extraction and non-technical abstract generation. Fine-tuning 7B models on the data greatly improved both extraction tasks, while abstract generation gained only slightly. The dataset supports large-scale claim verification and meta-science research.

Key points

  • Scale. NSF-SciFy covers 412,155 parseable award abstracts with about 2.8M claims. It has two subsets:
    • NSF-SciFy-MatSci: 16,031 Division of Materials Research awards and about 114K claims, described as the first claim dataset for materials science.
    • NSF-SciFy-20K: 20K awards from five directorates (MPS, GEO, ENG, CSE, BIO) and about 135K claims.
  • Record contents. Each record holds the award metadata, technical and non-technical abstracts, extracted claims and extracted proposals. Non-technical abstracts appear in roughly 81% of awards. Linked publications, resolved via DOIs, appear mainly for awards from 2014 onward.
  • Extraction method. Claude-3.5-Sonnet ran at temperature 0 and returned structured JSON output.
  • Fine-tuning results. The authors fine-tuned Mistral-7B-v0.3 and Qwen2.5-7B and scored them with new LLM-judged metrics. Relative gains often exceeded 100%, and F1 for claim and proposal extraction roughly doubled.
  • Limitations and release. The zero-shot extraction favors precision over recall by design. All data and models are released for research and commercial use at github.com/darpa-scify/NSFSciFy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Delip Rao, Weiqiu You, Eric Wong and Chris Callison-Burch. 2026. NSF-SciFy: Mining the NSF Awards Database for Scientific Claims. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2118

@inproceedings{rao2026nsf,
  title = {{NSF-SciFy: Mining the NSF Awards Database for Scientific Claims}},
  author = {Delip Rao, Weiqiu You, Eric Wong and Chris Callison-Burch},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2118},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2118}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1). find
  2. Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282, Online. Association for Comp…
  3. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hend… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  4. Thomas Diggelmann, Jordan L. Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. CoRR, abs/2012.00614.
  5. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  6. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  7. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  8. Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, Online. Association for Computational Linguistics.
  9. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  10. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  11. Isabelle Mohr, Amelie Wührl, and Roman Klinger. 2022. CoVERT: A corpus of fact-checked biomedical COVID-19 tweets. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 244–257, Marseille, France. European Language Resources Association. find
  12. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/. Accessed: 2025-10-01. ↗
  13. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276.
  14. Preprint, Minsu Park, Suman Kalyan Maity, Stefan Wuchty, and Dashun Wang. 2024. Interdisciplinary papers supported by disciplinary grants garner deep and broad scientific impact. Preprint, arXiv:2303.14732. — [arXiv:2303.14732](https://arxiv.org/abs/2303.14732) find
  15. Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2025. StyleDistance: Stronger content-independent style embeddings with synthetic parallel examples. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of… find
  16. Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. COVID-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural La…
  17. Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021. Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499–3512, Punta Cana, Dominican Republic. Association for Computational Linguistics…
  18. Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818–8847, Miami, Florida, USA. Association for Computational Linguistics.
  19. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association…
  20. David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. SciFact-open: Towards open-domain scientific claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4719–4734, Abu Dhabi, United Arab Emirates. Asso…
  21. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rao-2026-nsf-scify-mining-nsf-awards/