Elamur Research⌕
☰
home / papers / 2025 / Literature Meets Data: A Synergistic Approach to Hypothesis Generation

Literature Meets Data: A Synergistic Approach to Hypothesis Generation

Paper · 2025 · Haokun Liu, Yangqiaoyu Zhou, Mingxuan Li et al.

Summary

LLM-based hypothesis generation has mostly followed one of two paths. Theory-driven methods mine published papers but adapt poorly to new data. Data-driven methods induce patterns from labeled examples but may overfit a single dataset. The authors ask whether combining the two helps, and they build what they call the first system that does so. Paper summaries are produced from PDFs converted with S2ORC-doc2json. These summaries are then injected into the HypoGeniC pipeline in two ways:

  • HypoRefine, where literature and data agents alternately refine newly proposed hypotheses.
  • A union strategy, which merges a literature-only bank with a data-driven bank after removing redundant hypotheses.

The approach is tested on five classification datasets that span identifying deception, AI-generated content and signs of mental stress, as well as predicting how persuasive an argument is. It generalizes better out of distribution than few-shot prompting, literature-only and data-only baselines. In a human study, the generated hypotheses helped people make better judgments. The work suggests that grounding data-driven discovery in prior knowledge yields hypotheses that are both more useful and more transferable.

Key points

  • On out-of-distribution test sets, accuracy gains over the baselines were:
    • +8.97% vs. few-shot inference
    • +15.75% vs. literature-only generation
    • +3.37% vs. data-only generation (HypoGeniC)
  • Human participants given the hypotheses improved accuracy by 7.44% on deception detection and 14.19% on AIGC detection. All participants rated the hypotheses as helpful.
  • Annotators judged literature-based and data-driven hypotheses as giving meaningfully different information in 84% of pairs for deception detection and 80% for AIGC detection, which indicates the two sources complement each other.
  • HypoRefine hypotheses scored significantly higher in clarity and plausibility than the commercial baselines NotebookLM and HyperWrite. NotebookLM's slight edge in novelty was not significant.
  • Code and data are released at github.com/ChicagoHAI/hypothesis-generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ChicagoChicago · United States45 works · 50,617 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Haokun Liu, Yangqiaoyu Zhou, Mingxuan Li, Chenfei Yuan and Chenhao Tan. 2025. Literature Meets Data: A Synergistic Approach to Hypothesis Generation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.12

@inproceedings{liu2025literature,
  title = {{Literature Meets Data: A Synergistic Approach to Hypothesis Generation}},
  author = {Haokun Liu, Yangqiaoyu Zhou, Mingxuan Li, Chenfei Yuan and Chenhao Tan},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.12},
  url = {https://doi.org/10.18653/v1/2025.acl-long.12}
}

References

57 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Lit-Search: A retrieval benchmark for scientific literature search. Preprint, arXiv:2407.18940. — [arXiv:2407.18940](https://arxiv.org/abs/2407.18940) find
  2. Peter Auer. 2003. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3:397–422. find
  3. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. ResearchAgent: Iterative research idea generation over scientific literature with large language models. Preprint, arXiv:2404.07738.
  4. Rafael M. Batista and James Ross. 2024. Words that work: Using language to generate hypotheses. find
  5. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2024. ScienceAgentBench: Toward rigorous assessme… find
  6. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of EMNLP-IJCNLP. find
  7. Son Doan, Amanda Ritchart, Nicholas Perry, Juan D Chaparro, and Mike Conway. 2017. How do you \#relax when you’re \#stressed? a content analysis and infodemiology study of stress-related tweets. JMIR Public Health and Surveillance, 3(2):e35. find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of ACL.
  10. Cristina Garbacea and Chenhao Tan. 2025. Hyperalign: Hypotheses-driven personalized alignment. Preprint, arXiv:2505.00038. — [arXiv:2505.00038](https://arxiv.org/abs/2505.00038) find
  11. Alireza Ghafarollahi and Markus J. Buehler. 2024. Sci-Agents: Automating scientific discovery through multi-agent intelligent graph reasoning. Preprint, arXiv:2409.05556. find
  12. Pierpaolo Goffredo, Mariana Chaves, Serena Villata, and Elena Cabrio. 2023. Argument-based detection and classification of fallacies in political debates. In Proceedings of EMNLP. find
  13. Google. 2024. NotebookLM. find
  14. Pär Anders Granhag and Aldert Vrij. 2005. Deception detection. Psychology and law: An empirical perspective, pages 43–92. find
  15. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. 2024. BLADE: Benchmarking language model agents for data-driven science. Preprint, arXiv:2… find
  16. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. DS-Agent: Automated data science by empowering large language models with case-based reasoning. In Proceedings of ICML. find
  17. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. 2024. InfiAgent-DABench: Evaluating agents on data analysis tasks. In Proceedings of ICML, Proceedings of… find
  18. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. MLAgentBench: Evaluating language agents on machine learning experimentation. In Proceedings of ICML.
  19. Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. 2024. DISCOVERYWORLD: A virtual environment for developing and evaluating automated scientific discovery agents. Preprint, arXiv:2406.06769. find
  20. Hao Kang and Chenyan Xiong. 2024. ResearchArena: Benchmarking llms’ ability to collect and organize information as research agents. Preprint, arXiv:2406.10291. — [arXiv:2406.10291](https://arxiv.org/abs/2406.10291) find
  21. Jiwei Li, Myle Ott, and Claire Cardie. 2013. Identifying manipulated offerings on review portals. In Proceedings of EMNLP. find
  22. Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. Towards a general rule for identifying deceptive opinion spam. In Proceedings of ACL. find
  23. Mingxuan Li, Hanchen Li, and Chenhao Tan. 2025. Hypoeval: Hypothesis-guided evaluation for natural language generation. Preprint, arXiv:2504.07174. find
  24. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024. MLR-Copilot: Autonomous machine learning research based on large language models agents. Preprint, arXiv:2408.14033. find
  25. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of ACL, Online.
  26. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI scientist: Towards fully automated open-ended scientific discovery. Preprint, arXiv:2408.06292.
  27. Jens Ludwig and Sendhil Mullainathan. 2024. Machine learning as a tool for hypothesis generation*. The Quarterly Journal of Economics, page qjad055. find
  28. Sonia J Lupien, Bruce S McEwen, Megan R Gunnar, and Christine Heim. 2009. Effects of stress throughout the lifespan on the brain, behaviour and cognition. Nature Reviews Neuroscience, pages 434–445. find
  29. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2024. DiscoveryBench: Towards data-driven discovery with large language models. Preprint, arXiv:2407.01725. find
  30. Sushil Kumar Maurya, Dinesh Singh, and Ashish Kumar Maurya. 2022. Deceptive opinion spam detection approaches: a literature survey. Applied Intelligence, 53(2):2189–2234. find
  31. OpenAI. 2023. GPT-4 technical report.
  32. OthersideAI. 2024. HyperWrite. find
  33. Myle Ott, Claire Cardie, and Jeffrey T. Hancock. 2013. Negative deceptive opinion spam. In Proceedings of NAACL. find
  34. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. find
  35. Amalie Brogaard Pauli, Isabelle Augenstein, and Ira Assent. 2024. Measuring and benchmarking large language models’ capabilities to generate persuasive language. Preprint, arXiv:2406.17753. find
  36. Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. 2024. Topicgpt: A prompt-based topic modeling framework. In Proceedings of NAACL.
  37. Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. 2018. Crowdsourcing a large corpus of clickbait on Twitter. In Proceedings of COLING. find
  38. Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. 2024. CiteME: Can language models accurately cite scientific claims? Preprint, arXiv:2407.12861. find
  39. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  40. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In Proceedings of ICLR. find
  41. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475.
  42. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024a. Language models as inductive reasoners. In Proceedings of EACL.
  43. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. Preprint, arXiv:2409.04109.
  44. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of WWW. find
  45. Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2023. The science of detecting llm-generated texts. Preprint, arXiv:2303.07205. — [arXiv:2303.07205](https://arxiv.org/abs/2303.07205) find
  46. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… find
  47. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024b. Large language models for automated open-domain scientific hypotheses discovery. In Proceedings of ACL. find
  48. Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, and Qiaozhu Mei. 2024. MASSW: A new dataset and benchmark tasks for ai-assisted scientific workflows. Preprint, arXiv:2406.06357. find
  49. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven discovery of distributional differences via language descriptions. In Proceedings of NeurIPS.
  50. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326. find
  51. Elsbeth Turcan and Kathleen McKeown. 2019. Dreaddit: A reddit dataset for stress analysis in social media. Preprint, arXiv:1911.00133. find
  52. Jean H.M. Wagemans. 2023. How to identify an argument type? on the hermeneutics of persuasive discourse. Journal of Pragmatics, 203:117–129. find
  53. Xiangxuan Wan and Li Tian. 2024. User stress detection using social media text: A novel machine learning approach. International Journal of Computers Communications & Control, 19. find
  54. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of ACL. find
  55. Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2020. Persuasion for good: Towards a personalized persuasive dialogue system for social good. Preprint, arXiv:1906.06725.
  56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS.
  57. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F. Wong, and Lidia S. Chao. 2024. A survey on llm-generated text detection: Necessity, methods, and future directions. Preprint, arXiv:2310.14724.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-literature-meets-data-synergistic-approach/