Elamur Research⌕
☰
home / papers / 2025 / Sparks of Science: Hypothesis Generation Using Structured Paper Data

Sparks of Science: Hypothesis Generation Using Structured Paper Data

Paper · 2025 · Charles O'Neill, Tirthankar Ghosal, Roberta R\uaileanu et al.

Summary

LLMs asked to propose research ideas often produce output that is either unoriginal or impractical. Part of the problem is that no dataset has treated scientific hypothesis generation as a standard conditional text generation task. The authors build HypoGen, about 5,500 problem–hypothesis records drawn from papers at leading computer science conferences. OpenAI's o1 rewrites each paper into four parts:

  • a Bit, the prevailing assumption;
  • a Flip, the proposed alternative;
  • a Spark, a short phrase naming the core insight;
  • a first-person Chain-of-Reasoning, taken from the paper body, that traces the move from Bit to Flip.

LLaMA 3.1-8B and its R1-distilled variant are fine-tuned on these records and receive only the Bit at inference. Claude 3.7 Sonnet, acting as judge, rates the fine-tuned models clearly better overall and more feasible than one-shot prompting, though somewhat less novel. Human-written hypotheses still win. The resource gives AI-assisted ideation a reproducible benchmark and a source of training data.

Key points

  • Dataset: HypoGen has roughly 5,500 Bit-Flip-Spark + Chain-of-Reasoning entries. It is released under the MIT license on Hugging Face as UniverseTBD/hypogen-dr1.
  • Fine-tuned vs. one-shot: According to the LLM judge, fine-tuned models beat their one-shot counterparts on overall quality in 86–92% of comparisons and on feasibility in 74–86%. The non-fine-tuned versions won on novelty (54–86%).
  • Human hypotheses: Human-written hypotheses were preferred overall in 82–90% of comparisons. The fine-tuned models still won 62–64% of feasibility comparisons against them.
  • Diversity trade-off: Fine-tuning lowered output diversity. For R1-distilled LLaMA, the Idea Distinctness Index fell from 0.7146 to 0.6288.
  • Robustness checks and limitations: Re-judging with o3-mini largely confirmed the trends. A small human evaluation favoured the fine-tuned R1 model (70% preferred, 25% tie). The authors acknowledge that relying on LLM judges and covering only computer science are limitations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓University College LondonLondon · United Kingdom88 works · 19,432 citations🎓Australian National UniversityCanberra · Australia23 works · 6,522 citations🔬Oak Ridge National LaboratoryOak Ridge · United States3 works · 44 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Charles O'Neill, Tirthankar Ghosal, Roberta R\uaileanu, Mike Walmsley, Thang Bui, Kevin Schawinski and Ioana Ciuc\ua. 2025. Sparks of Science: Hypothesis Generation Using Structured Paper Data. arXiv:2504.12976.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.12976 · arXiv 2504.12976

@misc{oneill2025sparks,
  title = {{Sparks of Science: Hypothesis Generation Using Structured Paper Data}},
  author = {Charles O'Neill, Tirthankar Ghosal, Roberta Ruaileanu, Mike Walmsley, Thang Bui, Kevin Schawinski and Ioana Ciucua},
  year = {2025},
  doi = {10.48550/arxiv.2504.12976},
  eprint = {2504.12976},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.12976}
}

References

61 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. Ahmed M. Alaa, Boris van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:231942787. ↗
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR), 2015. URL https://arxiv.org/abs/1409.0473. ↗
  4. Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, 5:277 – 280, 2023. URL https://api.semanticscholar.org/CorpusID:258361324. ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  8. Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. Evaluation metrics for language models. 1998. find
  9. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080, 2024. find
  10. Kai-Hendrik Cohrs, Emiliano Diaz, Vasileios Sitokonstantinou, Gherardo Varando, and Gustau Camps-Valls. Large language models for causal hypothesis generation in science. Machine Learning: Science and Technology, 6(1):013001, 2025. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  12. Simona Doboli, Fanshu Zhao, and Alex Doboli. New measures for evaluating creativity in scientific publications. arXiv preprint arXiv:1406.7582, 2014. — [arXiv:1406.7582](https://arxiv.org/abs/1406.7582) find
  13. Katherine Elkins and Jon Chun. Can gpt-3 pass a writers turing test? Journal of Cultural Analytics, 5(2), 2020. find
  14. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025.
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  16. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. find
  17. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ArXiv, abs/1904.09751, 2019. URL https://api.semanticscholar.org/CorpusID:127986954. ↗
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  19. Peter Alexander Jansen, Marc-Alexandre Cot’e, Tushar Khot, Erin Bransom, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. ArXiv, abs/2406.06769, 2024. URL https://ap… find
  20. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
  21. Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. Can large language models unlock novel scientific research ideas? arXiv preprint arXiv:2409.06185, 2024.
  22. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. ArXiv, abs/2305.11747, 2023. URL https://api.semanticscholar.org/CorpusID:258832847. ↗
  23. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://api.semanticscholar.org… ↗
  24. James R Lindes and Wray Peter. Improving knowledge extraction from llms for robotic task learning through agent analysis. arXiv preprint arXiv:2306.06770, 2023. — [arXiv:2306.06770](https://arxiv.org/abs/2306.06770) find
  25. Ling Liu and Mans Hulden. Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models. ArXiv, abs/2104.06483, 2021. URL https://api.semanticscholar.org/CorpusID:233231232. ↗
  26. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. find
  27. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024a.
  28. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408.06292, 2024b. URL https://api.semanticscholar.org/CorpusID:271854887. ↗
  29. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey on large language models for scientific research. 2025. URL https://api.semanticscholar.org/ CorpusID:275358054. ↗
  30. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. ArXiv, abs/2407.01725, 2024. URL https://api.semanticsc… ↗
  31. Potsawee Manakul, Adian Liusie, and Mark John Francis Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. ArXiv, abs/2303.08896, 2023. URL https://api.semanticscholar.org/CorpusID:257557820. ↗
  32. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11: 652–670, 2021. URL https://api.… ↗
  33. Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. ArXiv, abs/2305.14552, 2023. URL https://api.semanticscholar.org/CorpusID: 258865517. ↗
  34. Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. ArXiv, abs/2202.00666, 2022. URL https://api.semanticscholar.org/CorpusID:246442062. — [arXiv:2202.00666](https://arxiv.org/abs/2202.00666) ↗
  35. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502.14499, 2025. find
  36. David Noever and Forrest McKee. Numeracy from literacy: Data science as an emergent skill from large language models. ArXiv, abs/2301.13382, 2023. URL https://api.semanticscholar.org/CorpusID:256416333. — [arXiv:2301.13382](https://arxiv.org/abs/2301.13382) ↗
  37. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhangren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. ArXiv, abs/2311.05965, 2023. URL https://api.semanticscholar.org/CorpusID:265128781. ↗
  38. Mark A Runco and Garrett J Jaeger. The standard definition of creativity. Creativity research journal, 24(1):92–96, 2012. find
  39. Samuel Schmidgall and Michael Moor. Agentrxiv: Towards collaborative autonomous research. arXiv preprint arXiv:2503.18102, 2025. find
  40. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025.
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. ArXiv, abs/1508.07909, 2015. URL https://api.semanticscholar.org/CorpusID:1114678. ↗
  42. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration. ArXiv, abs/2412.15701, 2024. URL https://api.semanticscholar.org/CorpusID:274964990. ↗
  43. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023.
  44. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.
  45. Dean Keith Simonton. Creativity in science: Chance, logic, genius, and zeitgeist. Cambridge University Press, 2004. find
  46. Heidi Angell Strøm. Creativity in science-scientific essay. 2018. find
  47. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. ArXiv, abs/2202.06417, 2022. URL https://api.semanticscholar.org/CorpusID:246823043. — [arXiv:2202.06417](https://arxiv.org/abs/2202.06417) ↗
  48. Kyle Swanson, Wesley Wu, Nash L. Bulaong, John E. Pak, and James Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, 2024. doi: 10.1101/2024.11.11.623004. find
  49. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  50. S. M. Towhidul Islam Tonmoy, S. M. Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024.
  51. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:258841365. ↗
  52. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  53. Tong Xie, Yuwei Wan, Wei Huang, Yufei Zhou, Yixuan Liu, Qingyuan, Linghu, Shaozhou Wang, Chunyu Kit, Clara Grazian, W. Zhang, Bram, and Hoex. Large language models as master key: Unlocking the secrets of materials science with gpt. ArXiv, abs/2304.02213, 2023. URL https://api.semanticscholar.org/Cor… — [arXiv:2304.02213](https://arxiv.org/abs/2304.02213) ↗
  54. Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models. ArXiv, abs/2411.02382, 2024a. URL https://api.semanticscholar.org/CorpusID:273821167. ↗
  55. Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models. arXiv preprint arXiv:2411.02382, 2024b.
  56. Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv:2010.02903, 2020.
  57. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. find
  58. Chen Zhang. User-controlled knowledge fusion in large language models: Balancing creativity and hallucination. ArXiv, abs/2307.16139, 2023. URL https://api.semanticscholar.org/CorpusID:260334043. — [arXiv:2307.16139](https://arxiv.org/abs/2307.16139) ↗
  59. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypothesis generation with large language models. ArXiv, abs/2404.04326, 2024a. URL https://api.semanticscholar.org/CorpusID:269005868. ↗
  60. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326, 2024b. find
  61. Caleb Ziems, William B. Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? ArXiv, abs/2305.03514, 2023. URL https://api.semanticscholar.org/CorpusID:258547324. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/o-neill-2025-sparks-science-hypothesis-generation-using/