Elamur Research⌕
☰
home / papers / 2025 / Automated Hypothesis Validation with Agentic Sequential Falsifications

Automated Hypothesis Validation with Agentic Sequential Falsifications

Paper · 2025 · Kexin Huang, Ying Jin, Ryan Li et al.

Summary

Large language models can now propose scientific hypotheses faster than people can check them. Many of these claims are also abstract statements that cannot be tested directly. The authors present POPPER, an agent-based system that judges free-form natural-language hypotheses by trying to refute them, following Karl Popper's idea of falsification. One LLM agent derives a measurable consequence of the hypothesis and frames it as a sub-test with an explicit null and alternative. A second agent writes and runs Python analyses on large existing datasets and returns a p-value. POPPER converts these p-values into e-values and combines them sequentially. This lets it decide whether to continue gathering evidence or stop, while keeping the Type-I error under a chosen level even when tests are dependent. Across six domains, including biology, economics and sociology, POPPER limited false validations and gained power over existing methods. In an expert study, it matched human accuracy in about one tenth of the time. The result is a scalable, statistically sound filter for machine-generated hypotheses.

Key points

  • Two agents, the Experiment Design Agent and the Experiment Execution Agent, share the work. A typical execution round involves up to 14 steps and favours non-parametric tests. Code is available at github.com/snap-stanford/POPPER.
  • Benchmarks include DiscoveryBench and TargetVal-IL2. Claude Haiku 3.5 showed inflated Type-I error, while Llama 3.3 70B, GPT-4o, Claude Sonnet 3.5 and o1 kept error reasonably controlled.
  • A user study with nine PhD-level biostatisticians and computational biologists found comparable validation quality with about 10× less time.
  • Human ratings of 90 falsification proposals reached Kendall's W = 0.91. The ReleCheck agent's ratings correlated with human ratings (Spearman's ρ = 0.55) but were somewhat lenient: it accepted 85% of proposals versus 77% for humans.
  • The main failure sources were misread p-values (35.9%) and weak experiment design (28.1%). Hallucination accounted for only 0.8% of failures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candes and Jure Leskovec. 2025. Automated Hypothesis Validation with Agentic Sequential Falsifications. arXiv:2502.09858.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.09858 · arXiv 2502.09858

@misc{huang2025automated,
  title = {{Automated Hypothesis Validation with Agentic Sequential Falsifications}},
  author = {Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candes and Jure Leskovec},
  year = {2025},
  doi = {10.48550/arxiv.2502.09858},
  eprint = {2502.09858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.09858}
}

References

65 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agassi, J. Popper and his popular critics: Thomas kuhn, paul feyerabend and imre lakatos. In SpringerBriefs in Philosophy. Springer, 2014. doi: 10.1007/978-3-319-06587-8. — [doi:10.1007/978-3-319-06587-8](https://doi.org/10.1007/978-3-319-06587-8) find
  2. Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., and Gao, T. Litsearch: A retrieval benchmark for scientific literature search, 2024. URL https://arxiv.org/ abs/2407.18940. ↗
  3. Alet, F., Lopez-Contreras, J., Koppel, J., Nye, M., Solar-Lezama, A., Lozano-Perez, T., Kaelbling, L., and Tenenbaum, J. A large-scale benchmark for few-shot program induction and synthesis. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volu… find
  4. Impact Statement This work introduces P OPPER , a statistically rigorous agentic framework for hypothesis validation using Large Language Model (LLM) agents. By combining advanced natural language processing capabilities with robust statistical methodologies, P OPPER addresses the critical challenge… — ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
  5. From an ethical perspective, P OPPER ’s emphasis on rigorous statistical validation and Type-I error control mitigates the risks associated with hallucinated or unsupported hypotheses. This ensures that research resources are directed toward meaningful and plausible hypotheses, reducing the potentia… find
  6. Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300, 1995. find
  7. Brown, M. B. 400: A method for combining non-independent, one-sided tests of significance. Biometrics, pp. 987–992, 1975. \#\# Acknowledgement Collaboration, O. S. Estimating the reproducibility of psychological science. Science, 349(6251):aac4716, 2015. find
  8. We thank Tatsunori Hashimoto and members of the Jure Leskovec lab for discussions and for providing feedback on our manuscript. We thank the expert user study participants: Consortium, G. The gtex consortium atlas of genetic regulatory effects across human tissues. Science, 369(6509): 1318–1330, 202… find
  9. D’Arcy, M., Hope, T., Birnbaum, L., and Downey, D. Marg: Multi-agent review generation for scientific papers, 2024. URL https://arxiv.org/abs/2401.04259. ↗
  10. Ioannidis, J. P. Why most published research findings are false. PLoS medicine, 2(8):e124, 2005. find
  11. Jun, E., Birchfield, M., De Moura, N., Heer, J., and Just, R. Hypothesis formalization: Empirical findings, software limitations, and design implications. ACM Transactions on Computer-Human Interaction (TOCHI), 29(1):1–28, 2022. find
  12. Fisher, R. A. Design of experiments. British Medical Journal, 1(3923):554, 1936. find
  13. Fisher, R. A. Statistical methods for research workers. In Breakthroughs in statistics: Methodology and distribution, pp. 66–70. Springer, 1970. find
  14. Kuhn, T. S. The Structure of Scientific Revolutions. University of Chicago Press, Chicago, 1st edition, 1962. find
  15. Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. Large language models are not strong abstract reasoners, 2024. URL https://arxiv.org/abs/2305.19555. ↗
  16. Lakatos, I. The Methodology of Scientific Research Programmes. Cambridge University Press, Cambridge, 1978. find
  17. Godfrey-Smith, P. Theory and reality: An introduction to the philosophy of science. University of Chicago Press, 2009. find
  18. Lehr, S. A., Caliskan, A., Liyanage, S., and Banaji, M. R. Chatgpt as research scientist: Probing gpt’s capabilities as a research librarian, research ethicist, data generator and data predictor, 2024. URL https://arxiv.org/ abs/2406.14765. — [arXiv:2406.14765](https://arxiv.org/abs/2406.14765) ↗
  19. Goodman, N. Fact, Fiction, and Forecast. Harvard University Press, Cambridge, MA, 1983. find
  20. Li, M. Y., Vajipey, V., Goodman, N. D., and Fox, E. B. Critical: Critic automation with language models. arXiv preprint arXiv:2411.06590, 2024a. — [arXiv:2411.06590](https://arxiv.org/abs/2411.06590) find
  21. Grünwald, P., de Heide, R., and Koolen, W. M. Safe testing. In 2020 Information Theory and Applications Workshop (ITA), pp. 1–54. find
  22. Li, R., Patel, T., Wang, Q., and Du, X. Mlr-copilot: Autonomous machine learning research based on large language models agents, 2024b. URL https://arxiv.org/abs/2408.14033. ↗
  23. Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R.-J., Lyu, D., Mao, Y., Pan, Y., Wu, T., Yu, J., Zhang, Y., Zhang, T. M., Zhu, L., Merrill, M. A., Heer, J., and Althoff, T. Blade: Benchmarking language model agents for data-driven science, 2024. URL https://arxiv.org/ abs/2408.09667. ↗
  24. Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., McFarland, D., and Zou, J. Can large language models provide useful feedback on research papers? a large-scale empirical analysis, 2023. URL https://arxiv.org/abs/ 2310.01783. ↗
  25. Han, S. J., Ransom, K., Perfors, A., and Kemp, C. Inductive reasoning in humans and large language models, 2023. URL https://arxiv.org/abs/2306.06548. — [arXiv:2306.06548](https://arxiv.org/abs/2306.06548) ↗
  26. Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
  27. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https://arxiv.org/abs/2408.06292. ↗
  28. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
  29. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. NeurIPS, 36, 2024. find
  30. MacArthur, J., Bowler, E., Cerezo, M., Gil, L., Hall, P., Hastings, E., Junkins, H., McMahon, A., Milano, A., Morales, J., et al. The new nhgri-ebi catalog of published genome-wide association studies (gwas catalog). Nucleic acids research, 45(D1):D896–D901, 2017. find
  31. Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725, 2024.
  32. Ifargan, T., Hafner, L., Kern, M., Alcalay, O., and Kishony, R. Autonomous llm-driven research from data to human-verifiable research papers, 2024. URL https://arxiv.org/abs/2404.17605. ↗
  33. Manning, B. S., Zhu, K., and Horton, J. J. Automated social science: Language models as scientist and subjects, 2024. URL https://arxiv.org/abs/2404.11794. refinement, 2024. URL https://arxiv.org/abs/ 2310.08559. — [arXiv:2404.11794](https://arxiv.org/abs/2404.11794) ↗
  34. Maxwell, N. Popper, kuhn, lakatos and aim-oriented empiricism. arXiv preprint, 2012. find
  35. Ridnik, T., Kredo, D., and Friedman, I. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024.
  36. Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines, 2023. URL https://arxiv.org/abs/2307.04721. ↗
  37. Rubin, M. The replication crisis is less of a ”crisis” in lakatos’ philosophy of science. European Journal for Philosophy of Science, 15(5), 2025. doi: 10.1007/ s13194-024-00629-x. find
  38. Moskvichev, A., Odouard, V. V., and Mitchell, M. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain, 2023. URL https://arxiv.org/abs/2305.07141. ↗
  39. Schmidt, R., Steinhart, Z., Layeghi, M., Freimer, J. W., Bueno, R., Nguyen, V. Q., Blaeschke, F., Ye, C. J., and Marson, A. Crispr activation and interference screens decode stimulation responses in primary human t cells. Science, 375(6580):eabj4008, 2022. find
  40. Neyman, J. and Pearson, E. S. On the use and interpretation of certain test criteria for purposes of statistical inference part i. Biometrika, 20(1-2):175–240, 1928. find
  41. Shafer, G. The language of betting as a strategy for statistical and scientific communication. arXiv preprint arXiv:1903.06991, 2019. — [arXiv:1903.06991](https://arxiv.org/abs/1903.06991) find
  42. Neyman, J. and Pearson, E. S. The testing of statistical hypotheses in relation to probabilities a priori. In Mathematical proceedings of the Cambridge philosophical society, volume 29, pp. 492–510. Cambridge University Press, 1933. find
  43. Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv.org/abs/2409.04109. ↗
  44. Tang, X., Zheng, Z., Li, J., Meng, F., Zhu, S.-C., Liang, Y., and Zhang, M. Large language models are in-context semantic reasoners rather than symbolic reasoners, 2023. URL https://arxiv.org/abs/2305.14825. ↗
  45. Oughtred, R., Stark, C., Breitkreutz, B.-J., Rust, J., Boucher, L., Chang, C., Kolas, N., ODonnell, L., Leung, G., McAdam, R., et al. The biogrid interaction database: 2019 update. Nucleic acids research, 47(D1):D529– D541, 2019. find
  46. Thompson, W. H. and Skau, S. On the scope of scientific hypotheses. Royal Society Open Science, 10(8):230607, 2023. find
  47. Philosophy Institute. Imre lakatos approach: Bridging popper and kuhn in philosophy of science, 2023. URL https://philosophy.institute/ Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., philosophy-of-science-and-cosmology/ Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., imre-la… ↗
  48. Routledge, 2005. van Fraassen, B. C. The Scientific Image. find
  49. Clarendon Press, Oxford, 1980. find
  50. Press, C. U. Normal science and dogmatism, paradigms and progress: Kuhn versus popper and lakatos. 2009. find
  51. Vovk, V. and Wang, R. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736– 1754, 2021. find
  52. Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., and Bethge, M. Citeme: Can language models accurately cite scientific claims?, 2024. URL https://arxiv.org/abs/2407.12861. ↗
  53. Wang, Q., Downey, D., Ji, H., and Hope, T. Scimon: Scientific inspiration machines optimized for novelty, 2024a. URL https://arxiv.org/abs/2305.14259. ↗
  54. Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., and Ren, X. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis Wang, R. and Ramdas, A. False discovery rate control with e-values. Journal of… find
  55. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. ICLR, 2024b. find
  56. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models, 2024c. URL https://arxiv.org/abs/2309.05660. ↗
  57. Webb, T., Holyoak, K. J., and Lu, H. Emergent analogical reasoning in large language models, 2023. URL https://arxiv.org/abs/2212.09196. ↗
  58. Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., and Cambria, E. Are large language models really good logical reasoners? a comprehensive evaluation and beyond, 2024a. URL https://arxiv.org/abs/2306.09841. ↗
  59. Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations, 2024b. URL https://arxiv.org/ abs/2305.18354. ↗
  60. Yang, Z., Dong, L., Du, X., Cheng, H., Cambria, E., Liu, X., Gao, J., and Wei, F. Language models as inductive reasoners, 2024a. URL https://arxiv.org/abs/ 2212.10923. ↗
  61. Yang, Z., Du, X., Li, J., Zheng, J., Poria, S., and Cambria, E. Large language models for automated open-domain scientific hypotheses discovery, 2024b. URL https://arxiv.org/abs/2309.02726. ↗
  62. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. ICLR, 2023.
  63. Zhang, X., Xie, Y., Huang, J., Ma, J., Pan, Z., Liu, Q., Xiong, Z., Ergen, T., Shim, D., Lee, H., and Mei, Q. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows, 2024. URL https://arxiv.org/abs/ 2406.06357. ↗
  64. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595–46623, 2023.
  65. Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2025-automated-hypothesis-validation-agentic-sequential/