Elamur Research⌕
☰
home / papers / 2025 / Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems

Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems

Paper · 2025 · Jiayi Geng, Howard Chen, Dilip Arumugam et al.

Summary

The paper asks whether LLMs can act as dependable "AI scientists" by testing a basic research skill: inferring the hidden mechanism of a black-box system from its input–output behavior. The authors build three controlled reverse-engineering tasks, drawn from cognitive science: list-mapping programs, formal languages, and math equations. They compare two conditions. In one, the model only receives passively observed data. In the other, it can also intervene by choosing its own queries and seeing the black box's responses. The resulting hypotheses are scored against a Bayesian inference baseline.

With passive data alone, LLMs plateau well below the Bayesian ideal. Active querying improves results but does not close the gap. When one model's intervention data is handed to another model, much of the benefit disappears. This suggests the gain comes from a model testing its own beliefs, which parallels findings in human learning.

The work offers a principled benchmark for scientific reasoning. It also warns that simple, controlled systems already expose weaknesses, and that experimental knowledge may not transfer cleanly between AI agents.

Key points

  • The three black-box domains are list-mapping programs, formal languages, and math equations. Performance is reported with a descriptive score for programs and formal languages, and with 1 − RMSE for equations.
  • Intervention helps models escape two failure modes:
    • Overcomplication: inventing overly complex hypotheses or assuming prior knowledge about the system.
    • Overlooking: ignoring observations or drawing generic conclusions without checking them.
  • In the experiments, models were allowed to reason once every five queries, and this reasoning generally improved intervention. Structured "functional" reasoning gave no additional gain, which hints that reverse-engineering ability differs from formal reasoning skill.
  • In the transfer experiment, Llama-3.3-70B-Instruct was given intervention data collected by GPT-4o. It matched or slightly beat its observation-only baseline but consistently fell short of using its own interventions.
  • Stated limitations: only noise-free black boxes were studied, and the task set is narrow. Code is available at github.com/JiayiGeng/reverse-engineering.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Jiayi Geng, Howard Chen, Dilip Arumugam and Thomas L. Griffiths. 2025. Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems. arXiv:2505.17968.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.17968 · arXiv 2505.17968

@misc{geng2025are,
  title = {{Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems}},
  author = {Jiayi Geng, Howard Chen, Dilip Arumugam and Thomas L. Griffiths},
  year = {2025},
  doi = {10.48550/arxiv.2505.17968},
  eprint = {2505.17968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.17968}
}

References

83 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Abernethy, Kareem Amin, Michael Kearns, and Moez Draief. Large-Scale Bandit Problems and KWIK Learning. In International Conference on Machine Learning, pp. 588–596, 2013. find
  2. Dana Angluin. Queries and Concept Learning. Machine Learning, 2:319–342, 1988. find
  3. Dana Angluin and Carl H Smith. Inductive Inference: Theory and Methods. ACM Computing Surveys (CSUR), 15(3):237–269, 1983. find
  4. Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet, 2024. ↗
  5. Dilip Arumugam and Thomas L Griffiths. Toward Efficient Exploration by Large Language Model Agents. arXiv preprint arXiv:2504.20997, 2025. — [arXiv:2504.20997](https://arxiv.org/abs/2504.20997) find
  6. Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, pp. 679–684, 1957. find
  7. Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023. find
  8. David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017. find
  9. Ronen I Brafman and Moshe Tennenholtz. R-MAX – A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3:213–231, 2002. find
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  11. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025.
  12. Kathryn Chaloner and Isabella Verdinelli. Bayesian Experimental Design: A Review. Statistical Science, pp. 273–304, 1995. find
  13. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do Not Think that Much for 2+3=? On the Overthinking of o1-like LLMs. arXiv preprint arXiv:2412.21187, 2024.
  14. Sahil Chopra, Michael Henry Tessler, and Noah D Goodman. The first crank of the cultural ratchet: Learning and transmitting concepts through language. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 41, 2019. find
  15. Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: A Large Language Model Walks into A Psychology Lab. arXiv preprint arXiv:2402.18225, 2024. — [arXiv:2402.18225](https://arxiv.org/abs/2402.18225) find
  16. Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025.
  17. MH DeGroot. Uncertainty, Information, and Sequential Experiments. The Annals of Mathematical Statistics, 33(2):404–419, 1962. find
  18. Marc Deisenroth and Carl E Rasmussen. PILCO: A Model-Based and Data-Efficient Approach to Policy Search. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 465–472, 2011. find
  19. Armen Der Kiureghian and Ove Ditlevsen. Aleatory or Epistemic? Does it Matter? Structural Safety, 31(2):105–112, 2009. find
  20. Fabian Falck, Ziyu Wang, and Chris Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. arXiv preprint arXiv:2406.00793, 2024.
  21. Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational Bayesian optimal experimental design. Advances in Neural Information Processing Systems, 32, 2019. find
  22. Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452, 2023. find
  23. Kanishk Gandhi, Michael Y Li, Lyle Goodyear, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D Goodman. BoxingGym: Benchmarking progress in automated experimental design and model discovery. arXiv preprint arXiv:2501.01540, 2025.
  24. Yolanda Gil, Mark Greaves, James Hendler, and Haym Hirsh. Amplify scientific discovery with artificial intelligence. Science, 346(6206):171–172, 2014. find
  25. Ellen R Girden. ANOVA: Repeated measures. Number 84. Sage, 1992. find
  26. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andr… — Accelerating scientific discovery with Co-Scientist
  27. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  28. Thomas L Griffiths, Jian-Qiao Zhu, Erin Grant, and R Thomas McCoy. Bayes in the Age of Intelligent Machines. Current Directions in Psychological Science, 33(5):283–291, 2024. find
  29. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  30. Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361, 2025.
  31. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024.
  32. Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. What do learning dynamics reveal about generalization in llm reasoning? arXiv preprint arXiv:2411.07681, 2024. find
  33. Alexander Ku, Declan Campbell, Xuechunzi Bai, Jiayi Geng, Ryan Liu, Raja Marjieh, R Thomas McCoy, Andrew Nam, Ilia Sucholutsky, Veniamin Veselovsky, et al. Using the tools of cognitive science to understand large language models at different levels of analysis. arXiv preprint arXiv:2503.13401, 2025. — [arXiv:2503.13401](https://arxiv.org/abs/2503.13401) find
  34. Salvatore La Torre, Parthasarathy Madhusudan, and Gennaro Parlato. A robust class of context-sensitive languages. In 22nd Annual IEEE Symposium on Logic in Computer Science (LICS 2007), pp. 161–170, 2007. find
  35. Andrew Kyle Lampinen, Stephanie C Y Chan, Ishita Dasgupta, Andrew J Nam, and Jane X Wang. Passive learning of active causal strategies in agents and language models. Advances in Neural Information Processing Systems, 2023. find
  36. Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1–23, 2025. find
  37. Lihong Li and Michael L Littman. Reducing Reinforcement Learning to KWIK Online Regression. Annals of Mathematics and Artificial Intelligence, 58:217–237, 2010. find
  38. Lihong Li, Michael L Littman, and Thomas J Walsh. Knows What It Knows: A Framework for Self-Aware Learning. In Proceedings of the 25th International Conference on Machine Learning, pp. 568–575, 2008. find
  39. Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4):986–1005, 1956. find
  40. Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2:285–318, 1988. find
  41. Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling Exploration and Exploitation for Meta-Reinforcement Learning Without Sacrifices. In International Conference on Machine Learning, pp. 6925–6935, 2021. find
  42. Ryan Liu, Jiayi Geng, Joshua C Peterson, Ilia Sucholutsky, and Thomas L Griffiths. Large language models assume people are more rational than we really are. arXiv preprint arXiv:2406.17055, 2024. — [arXiv:2406.17055](https://arxiv.org/abs/2406.17055) find
  43. Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333, 2024. — [arXiv:2410.21333](https://arxiv.org/abs/2410.21333) find
  44. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  45. Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, and Bo Zheng. Deconstructing long chain-of-thought: A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503.16385, 2025.
  46. Doug Markant and Todd Gureckis. Category learning through active sampling. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32, 2010. find
  47. Douglas B Markant and Todd M Gureckis. Is it better to select or to receive? learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1):94, 2014. find
  48. R Thomas McCoy and Thomas L Griffiths. Modeling rapid language learning by distilling bayesian priors into artificial neural networks. arXiv preprint arXiv:2305.14701, 2023. — [arXiv:2305.14701](https://arxiv.org/abs/2305.14701) find
  49. R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121, 2024. find
  50. Ziqi Ni, Yahao Li, Kaijia Hu, Kunyuan Han, Ming Xu, Xingyu Chen, Fengqi Liu, Yicong Ye, and Shuxin Bai. Matpilot: an llm-enabled ai materials scientist under the framework of human-machine collaboration. arXiv preprint arXiv:2411.08063, 2024. — [arXiv:2411.08063](https://arxiv.org/abs/2411.08063) find
  51. Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. Sparks of science: Hypothesis generation using structured paper data. arXiv preprint arXiv:2504.12976, 2025.
  52. Ian Osband, Daniel Russo, and Benjamin Van Roy. (More) Efficient Reinforcement Learning via Posterior Sampling. Advances in Neural Information Processing Systems, 26:3003–3011, 2013. find
  53. Martin L. Puterman. Markov Decision Processes—Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994. find
  54. Chandan K Reddy and Parshin Shojaee. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28601–28609, 2025. find
  55. Ronald L Rivest and Robert E Schapire. Diversity-Based Inference of Finite Automata. In 28th Annual Symposium on Foundations of Computer Science (SFCS 1987), pp. 78–87, 1987. find
  56. Ronald L Rivest and Robert E Schapire. Inference of Finite Automata Using Homing Sequences. In Proceedings of the Twenty-First Annual ACM Symposium on Theory of Computing, pp. 411– 420, 1989. find
  57. Milena Rmus, Akshay K. Jagadish, Marvin Mathony, Tobias Ludwig, and Eric Schulz. Towards automation of cognitive modeling using large language models. arXiv preprint arXiv:2502.00879, 2025. — [arXiv:2502.00879](https://arxiv.org/abs/2502.00879) find
  58. Joshua S Rule, Steven T Piantadosi, Andrew Cropper, Kevin Ellis, Maxwell Nye, and Joshua B Tenenbaum. Symbolic metaprogram search improves learning efficiency and explains rule learning in humans. Nature Communications, 15(1):6847, 2024. find
  59. Amin Sayedi, Morteza Zadimoghaddam, and Avrim Blum. Trading off Mistakes and Don’t-Know Predictions. Advances in Neural Information Processing Systems, 23, 2010. find
  60. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025.
  61. Burr Settles. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.
  62. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Llm-sr: Scientific equation discovery via programming with large language models. International Conference on Learning Representations, 2025.
  63. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. arXiv preprint arXiv:2409.04109, 2024.
  64. Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To CoT or not to CoT? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183, 2024.
  65. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025.
  66. Alexander L Strehl and Michael L Littman. An Analysis of Model-based interval estimation for Markov Decision Processes. Journal of Computer and System Sciences, 74(8):1309–1331, 2008. find
  67. Malcolm JA Strens. A Bayesian framework for reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 943–950, 2000. find
  68. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025.
  69. Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh International Conference on Machine Learning, pp. 216–224, 1990. find
  70. Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160–163, 1991. find
  71. Richard S Sutton and Andrew G Barto. Introduction to Reinforcement Learning. MIT Press, 1998. find
  72. István Szita and Csaba Szepesvári. Agnostic KWIK learning and Efficient Approximate Reinforcement Learning. In Proceedings of the 24th Annual Conference on Learning Theory, pp. 739–772, 2011. find
  73. Sebastian B Thrun and Knut Möller. Active exploration in dynamic environments. Advances in Neural Information Processing Systems, 4, 1991. find
  74. Thomas J Walsh, István Szita, Carlos Diuk, and Michael L Littman. Exploring Compact Reinforcement-Learning Representations with Linear Regression. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 591–598, 2009. find
  75. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific Discovery in the Age of Artificial Intelligence. Nature, 620(7972):47–60, 2023. find
  76. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585, 2025.
  77. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  78. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations, 2021.
  79. Yuan Yang and Steven T. Piantadosi. One model for the learning of language. Proceedings of the National Academy of Sciences, 119(5):e2021865119, 2022. find
  80. Lance Ying, Katherine M Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L Griffiths, and Joshua B Tenenbaum. On benchmarking human-like intelligence in machines. arXiv preprint arXiv:2502.20502, 2025.
  81. Cedegao E Zhang, Katherine M Collins, Lionel Wong, Mauricio Barba, Adrian Weller, and Joshua B Tenenbaum. People use fast, goal-directed simulation to reason about novel games. arXiv preprint arXiv:2407.14095, 2024. — [arXiv:2407.14095](https://arxiv.org/abs/2407.14095) find
  82. Jian-Qiao Zhu and Thomas L Griffiths. Eliciting the priors of large language models using iterated in-context learning. arXiv preprint arXiv:2406.01860, 2024. — [arXiv:2406.01860](https://arxiv.org/abs/2406.01860) find
  83. Jian-Qiao Zhu and Thomas L Griffiths. Incoherent probability judgments in large language models. arXiv preprint arXiv:2401.16646, 2024. — [arXiv:2401.16646](https://arxiv.org/abs/2401.16646) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geng-2025-large-language-models-reliable/