Elamur Research⌕
☰
home / papers / 2023 / Large Language Models for Automated Open-domain Scientific Hypotheses Discovery

Large Language Models for Automated Open-domain Scientific Hypotheses Discovery

Paper · 2023 · Zonglin Yang, Xinya Du, Junxian Li et al.

Summary

The paper addresses hypothetical induction, the reasoning scientists use to move from observations to explanatory hypotheses. Earlier benchmarks were limited because their observations were hand-picked sentences, which made them closed-domain, and their target hypotheses were mostly commonsense facts that LLMs may have memorized. The authors define a harder task, TOMATO (automated open-domain hypothetical induction). In TOMATO, a system receives only raw web text and must produce research hypotheses that are new, valid and useful. They build a benchmark from 50 social science papers published in top journals after January 2023. Experts annotated each paper's main hypothesis, background and inspirations, and linked them to matching web passages. To solve the task, they propose MOOSE, a multi-module LLM prompting pipeline with three feedback loops: present, past and future. Both GPT-4 judging and expert assessment rate MOOSE above an LLM baseline, and each feedback type adds further gains. Experts judged many outputs to be both previously unpublished and plausible. This suggests LLMs could act as research assistants that help scientists form hypotheses.

Key points

  • Dataset: 50 social science publications from 2023 onward, covering 7 disciplines such as Psychology, Management and Marketing. The raw corpus comes mainly from news, Wikipedia and business reviews, so it can scale cheaply.
  • Annotations: Each paper has its main hypothesis, background, inspirations, matched web passages, reasoning process and complexity level. The manual selections serve only as a human-performance reference, not as system input.
  • Framework: MOOSE uses three feedback mechanisms:
    • present-feedback: a module refines its own output right away using critiques from another LLM;
    • past-feedback: an earlier module is revised based on feedback given to a later module;
    • future-feedback: a module passes justifications or draft suggestions forward to later modules.
  • Evaluation: GPT-4 and PhD-level experts rated outputs on validity, novelty and helpfulness. MOOSE-base beat the baseline, and the feedback loops improved results step by step.
  • Resources: Data, code and generated hypotheses are released at github.com/ZonglinY/MOOSE (arXiv:2309.02726).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓The University of Texas at DallasRichardson · United States63 works · 5,710 citations🎓Singapore University of Technology and DesignSingapore · Singapore54 works · 11,534 citations🎓Huazhong University of Science and TechnologyWuhan · China59 works · 8,310 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria and Erik Cambria. 2023. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. arXiv:2309.02726.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.02726 · arXiv 2309.02726

@misc{yang2023large,
  title = {{Large Language Models for Automated Open-domain Scientific Hypotheses Discovery}},
  author = {Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria and Erik Cambria},
  year = {2023},
  doi = {10.48550/arxiv.2309.02726},
  eprint = {2309.02726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.02726}
}

References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
  3. Daniil A. Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research capabilities of large language models. CoRR, abs/2304.05332.
  4. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence…
  5. Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376.
  6. Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process… find
  7. Jia Gao, Ying Rong, Xin Tian, and Yuliang Yao. 2023. Improving convenience or saving face? an empirical analysis of the use of facial recognition payment technology in retail. Information Systems Research. find
  8. Vinod Goel, Gorka Navarrete, Ira A Noveck, and Jérôme Prado. 2017. The reasoning brain: The interplay between cognitive neuroscience and theories of reasoning. find
  9. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  10. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural L…
  11. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoR… find
  12. John D Norton. 2003. A little survey of induction. find
  13. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  14. CoRR, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.… — Training language models to follow instructions with human feedback
  15. Feng Pan, Rutu Mulkar-Mehta, and Jerry R. Hobbs. 2011. Annotating and learning event durations in text. Comput. Linguistics, 37(4):727–752.
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  17. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. CoRR, abs/2302.12813.
  18. Zonglin Yang, Xinya Du, Alexander Rush, and Claire Cardie. 2020. Improving event duration prediction via time-aware pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3370–3378, Online. Association for Computational Linguistics. find
  19. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350. find
  20. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven discovery of distributional differences via language descriptions. CoRR, abs/2302.14233.
  21. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. CoRR, abs/2311.05965. find
  22. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  23. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR, abs/2303.11366.
  24. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023a. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60. find
  25. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023b. Learning to generate novel scientific directions with contextualized literature-based discovery. CoRR, abs/2305.14259.
  26. Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 4393… find
  27. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024. Language models as inductive reasoners. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 209–225, St.…
  28. Zonglin Yang, Xinya Du, Erik Cambria, and Claire Cardie. 2023a. End-to-end case-based reasoning for commonsense knowledge base completion. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3509–3522, Dubrovnik, Croatia. Association… find
  29. Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023b. Logical reasoning over natural language as knowledge representation: A survey. CoRR, abs/2303.12023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-automated/