Elamur Research⌕
☰
home / papers / 2024 / Hypothesis Generation with Large Language Models

Hypothesis Generation with Large Language Models

Paper · 2024 · Yangqiaoyu Zhou; Haokun Liu; Tejes Srivastava; Hongyuan Mei; and Chenhao Tan

Summary

The paper asks whether large language models can propose scientific hypotheses directly from labeled data, a step usually done informally by researchers. The authors introduce HypoGeniC (Hypothesis Generation in Context). An LLM first drafts hypotheses from a handful of examples. It then refines them over the training set, treating each hypothesis as an arm in a multi-armed bandit. A UCB-style reward combines a hypothesis's observed accuracy with an exploration bonus. Training examples that many top hypotheses misclassify are collected in a "wrong example bank," which prompts the model to write new hypotheses that fill the gaps. The final hypotheses drive an interpretable classifier with several inference strategies, such as using the single best-accuracy hypothesis or filtering hypotheses and then voting with weights.

The method was tested on one synthetic task and three social-science tasks: deception detection and headline or tweet popularity. It clearly beats few-shot prompting and often rivals or surpasses fine-tuned models. The hypotheses also transfer across LLMs and to out-of-distribution data. Some of them confirm findings from the literature, while others suggest new ideas, which points to LLMs as practical aids for data-driven discovery.

Key points

  • Accuracy over few-shot prompting: gains of 31.7% on SHOE SALES (synthetic), 13.9% on DECEPTIVE REVIEWS, 3.3% on HEADLINE POPULARITY and 24.9% on TWEET POPULARITY.
  • Comparison with supervised oracles: On a pair of challenging real-world datasets, HypoGeniC outperforms the supervised learning baseline, with margins of 12.1% and 11.6%. It matches or exceeds RoBERTa and Llama-2-7B everywhere except DECEPTIVE REVIEWS.
  • Cross-model and out-of-distribution transfer: hypotheses produced by GPT-3.5-turbo work for inference with Mixtral. On an out-of-distribution deceptive-review set, they outperform fine-tuned RoBERTa.
  • Settings and cost: the wrong example bank holds 10 items, and hypothesis bank sizes of 3 and 20 were tested. Running GPT-3.5-turbo averaged about $2.05 and 1.5 hours per experiment.
  • Release: code and data are public at github.com/ChicagoHAI/hypothesis-generation. The authors flag the risk of the method reinforcing biases and recommend human oversight.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Toyota Technological Institute at ChicagoChicago · United States37 works · 13,529 citations🎓University of ChicagoChicago · United States54 works · 50,989 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Yangqiaoyu Zhou; Haokun Liu; Tejes Srivastava; Hongyuan Mei; and Chenhao Tan. 2024. Hypothesis Generation with Large Language Models. arXiv:2404.04326.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.nlp4science-1.10 · arXiv 2404.04326

@misc{mei2024hypothesis,
  title = {{Hypothesis Generation with Large Language Models}},
  author = {Yangqiaoyu Zhou; Haokun Liu; Tejes Srivastava; Hongyuan Mei; and Chenhao Tan},
  year = {2024},
  doi = {10.18653/v1/2024.nlp4science-1.10},
  eprint = {2404.04326},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.nlp4science-1.10}
}

References

40 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric T Anderson and Duncan I Simester. 2014. Reviews without a purchase: Low ratings, loyal customers, and deception. Journal of Marketing Research, 51(3):249–269. find
  2. Anthropic. 2023. Claude 2. find
  3. Peter Auer. 2002. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422. find
  4. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. ResearchAgent: Iterative research idea generation over scientific literature with large language models. Preprint, arXiv:2404.07738.
  5. Akshina Banerjee and Oleg Urminsky. 2021. The language that drives engagement: A systematic large-scale analysis of headline experiments. Social Science Research Network. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  7. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2020. DreamCoder: growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning. Philosophical Transactions of the Royal Soci…
  8. Kristina Gligorić, Ashton Anderson, and Robert West. 2019. Causal effects of brevity on style and success in social media. In Proceedings of ACM HCI. find
  9. Pär Anders Granhag and Aldert Vrij. 2005. Deception detection. Psychology and law: An empirical perspective, pages 43–92. find
  10. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. In Proceedings of ACL.
  11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361.
  12. Junsol Kim and Byungkyu Lee. 2023. AI-augmented surveys: Leveraging large language models and surveys for opinion prediction. Preprint, arXiv:2305.09620. — [arXiv:2305.09620](https://arxiv.org/abs/2305.09620) find
  13. Vivian Lai, Han Liu, and Chenhao Tan. 2020. "Why is ’Chicago’ deceptive?" Towards building model-driven tutorials for humans. In Proceedings of CHI. find
  14. Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of FAccT. find
  15. Jiwei Li, Myle Ott, and Claire Cardie. 2013. Identifying manipulated offerings on review portals. In Proceedings of EMNLP. find
  16. Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. Towards a general rule for identifying deceptive opinion spam. In Proceedings of ACL, pages 1566–1576, Baltimore, Maryland. Association for Computational Linguistics. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. ArXiv.
  18. Jens Ludwig and Sendhil Mullainathan. 2024. Machine learning as a tool for hypothesis generation*. The Quarterly Journal of Economics, page qjad055. find
  19. Jorge Nathan Matias, Kevin Munger, Marianne Aubin Le Quere, and Charles R. Ebersole. 2021. The upworthy research archive, a time series of 32,487 experiments in U.S. media. Scientific Data, 8. find
  20. Mistral. 2023. Mixtral of experts. find
  21. OpenAI. 2023b. Gpt-4 technical report.
  22. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of ACL.
  23. Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. 2024. Topicgpt: A prompt-based topic modeling framework. In Proceedings of NAACL.
  24. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
  25. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In Proceedings of ICLR. find
  26. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475.
  27. Albert Rothenberg. 1995. Creative cognitive processes in kekule’s discovery of the structure of the benzene molecule. The American journal of psychology, pages 419–438. find
  28. Mark Sadoski, Ernest T Goetz, and Maximo Rodriguez. 2000. Engaging texts: Effects of concreteness on comprehensibility, interest, and recall in four text types. Journal of Educational Psychology, 92(1):85. find
  29. Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762):854–856. find
  30. Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The effect of wording on message propagation: Topic-and author-controlled natural experiments on twitter. In Proceedings of ACL. find
  31. Joshua B. Tenenbaum, Charles Kemp, Thomas L. Griffiths, and Noah D. Goodman. 2011. How to grow a mind: Statistics, structure, and abstraction. Science, 331:1279 – 1285. find
  32. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  33. Trieu Trinh, Yuhuai Tony Wu, Quoc Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625:476– 482.
  34. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of ACL.
  35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In Proceedings of ICLR.
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS.
  37. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024a. Language models as inductive reasoners. In Proceedings of EACL.
  38. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024b. Large language models for automated open-domain scientific hypotheses discovery. In Proceedings of ACL.
  39. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven discovery of distributional differences via language descriptions. In Proceedings of NeurIPS.
  40. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, pages 1–55.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2024-hypothesis-generation-large-language/