Elamur Research⌕
☰
home / papers / 2026 / Sound Agentic Science Requires Adversarial Experiments

Sound Agentic Science Requires Adversarial Experiments

Paper · 2026 · Dionizije Fa and Marko Culjak

Summary

This position paper argues that LLM-based agents for scientific data analysis face a verification problem that coding agents do not. A software agent can run its code against tests, specifications and user feedback, so each iteration narrows the set of acceptable programs. In empirical fields such as biomedicine, only new experiments, controls and replications can rule out explanations, and analysis agents cannot perform them. Cheap re-analysis therefore multiplies defensible-looking narratives rather than eliminating them. Drawing on Popper, Fisher and Pearl, the authors conclude that agent outputs without experimental confirmation should count as hypotheses, not publishable findings.

To illustrate the risk, they give two coding agents the same NHANES 2017–2018 data with opposite instructions about vitamin D and depression. Using reasonable specification choices, each agent obtains the result it was asked for. The proposed remedy is a falsification-first norm: observational claims produced with agent help should be accompanied by evidence of adversarial testing. Agents make such refutation attempts nearly as cheap as the original analyses, so skipping them becomes harder to excuse.

Key points

  • Toy experiment setup:
    • Data: NHANES 2017–2018 files DEMO_J, DPQ_J and VID_J, merged on SEQN.
    • Variables: serum 25(OH)D (LBXVIDMS) against the PHQ-9 total score.
    • Agents: run in Codex CLI with GPT-5.2-Codex; the prompts were written by GPT-5.2.
    • Code and results: github.com/dionizijefa/verification.
  • Agent A was told to find a protective link. It used survey weights, broad confounder adjustment and a seasonality control. Result: −0.045 PHQ-9 points per 10 nmol/L (95% CI −0.068 to −0.023, p = 0.0006).
  • Agent B was told to find no link. It used unweighted OLS, adjustment for age and sex only, and a restricted sample. Result: +0.0005 points per 1 nmol/L (95% CI −0.0050 to +0.0060, p = 0.855).
  • Conceptual framing (Figure 1): in software, the code-generation loop itself acts as the experiment. In data analysis, flexible modeling choices widen the space of hypotheses instead of shrinking it.
  • Recommendation: use the same agent as a critic. It should propose alternative explanations and run targeted checks for confounding, selection effects, measurement error and arbitrary analytic choices.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ZagrebZagreb · Croatia11 works · 1,179 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Dionizije Fa and Marko Culjak. 2026. Sound Agentic Science Requires Adversarial Experiments. arXiv:2604.22080.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.22080 · arXiv 2604.22080

@misc{fa2026sound,
  title = {{Sound Agentic Science Requires Adversarial Experiments}},
  author = {Dionizije Fa and Marko Culjak},
  year = {2026},
  doi = {10.48550/arxiv.2604.22080},
  eprint = {2604.22080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.22080}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dhruv Ahlawat, Vaibhav Mishra, Somaditya Singh, Mohd Zaki, Vaibhav Bihani, Hargun Singh Grover, Biswajit Mishra, Santiago Miret, Mausam, and N. M. Anoop Krishnan. A family of large language models for materials research with insights into model adaptability in continued pretraining. Nature Machine I… — [doi:10.1038/s42256-026-01199-8](https://doi.org/10.1038/s42256-026-01199-8) find
  2. Christopher A. Bail. Can generative ai improve social science? Proceedings of the National Academy of Sciences, 121(21), May 2024. ISSN 1091-6490. doi: 10.1073/pnas.2314021121. URL http://dx.doi.org/10.1073/pnas.2314021121. — [doi:10.1073/pnas.2314021121](https://doi.org/10.1073/pnas.2314021121) ↗
  3. C. Glenn Begley and Lee M. Ellis. Raise standards for preclinical cancer research. Nature, 483 (7391):531–533, March 2012. ISSN 1476-4687. doi: 10.1038/483531a. URL http://dx.doi.org/10.1038/483531a. — [doi:10.1038/483531a](https://doi.org/10.1038/483531a) ↗
  4. CDC and NCHS. Nhanes 2017–2018: Questionnaires, datasets, and related documentation, 2018. URL https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/ default.aspx?BeginYear=2017. ↗
  5. CDC and NCHS. National health and nutrition examination survey, 2017–2018: Demographic variables and sample weights (demo j) — data documentation, codebook, and frequencies, 2020. URL https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/ DataFiles/DEMO_J.htm. Data file: DEMO J.xpt. ↗
  6. CDC and NCHS. National health and nutrition examination survey, 2017–2018: Mental health — depression screener (dpq j) — data documentation, codebook, and frequencies, 2020. URL https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/ DataFiles/DPQ_J.htm. Data file: DPQ J.xpt. ↗
  7. CDC and NCHS. National health and nutrition examination survey, 2017–2018: Vitamin d (vid j) — data documentation, codebook, and frequencies, 2022. URL https://wwwn.cdc.gov/ Nchs/Data/Nhanes/Public/2017/DataFiles/VID_J.htm. Data file: VID J.xpt. ↗
  8. Dionizije Fa, Marko Culjak, Bruno Pandza, and Mateo Cupic. Bioagent bench: An ai agent evaluation suite for bioinformatics, 2026. URL https://arxiv.org/abs/2601.21800. — [arXiv:2601.21800](https://arxiv.org/abs/2601.21800) ↗
  9. R. A. Fisher. The Design of Experiments. Oliver and Boyd, Edinburgh, 1935. find
  10. GitHub Staff. Octoverse, October 2025. URL https://github.blog/news-insights/ octoverse/octoverse-a-new-developer-joins-github-every-second-as\\ -ai-leads-typescript-to-1/. Updated January 30, 2026. ↗
  11. Marta Guasch-Ferré, Anpan Satija, Stacey A. Blondin, et al. Meta-analysis of randomized controlled trials of red meat consumption in comparison with various comparison diets on cardiovascular risk factors. Circulation, 139(15):1828–1845, 2019. doi: 10.1161/CIRCULATIONAHA.118.035225. — [doi:10.1161/circulationaha.118.035225](https://doi.org/10.1161/circulationaha.118.035225) find
  12. Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candes, and Jure Leskovec. Automated hypothesis validation with agentic sequential falsifications. In Forty-second International Conference on Machine Learning, 2025.
  13. John P. A. Ioannidis. Why most published research findings are false. PLoS Medicine, 2(8):e124, August 2005. doi: 10.1371/journal.pmed.0020124. Epub 2005-08-30. Erratum in: PLoS Med. 2022 Aug 25;19(8):e1004085. doi:10.1371/journal.pmed.1004085. — [doi:10.1371/journal.pmed.0020124](https://doi.org/10.1371/journal.pmed.0020124) find
  14. Nikita Mehandru, Amanda K. Hall, Olesya Melnichenko, Yulia Dubinina, Daniel Tsirulnikov, David Bamman, Ahmed Alaa, Scott Saponas, and Venkat S. Malladi. Bioagents: Bridging the gap in bioinformatics analysis with multi-agent systems. Scientific Reports, 15(1), November 2025. ISSN 2045-2322. doi: 10.… — [doi:10.1038/s41598-025-25919-z](https://doi.org/10.1038/s41598-025-25919-z) find
  15. Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition, 2009. find
  16. Karl Popper. The Logic of Scientific Discovery. Hutchinson, 1959. English translation/revision of Logik der Forschung (1934). find
  17. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex N… — OpenAI GPT-5 System Card
  18. Alexus A. Smith, Edmund L. Wong, Ronan C. Donovan, Brad A. Chapman, Ryan Harry, Pooyan Tirandazi, Paulina Kanigowska, Elizabeth A. Gendreau, Robert H. Dahl, Michal Jastrzebski, Jose E. Cortez, Christopher J. Bremner, José C. Morales Hemuda, James Dooner, Ian Graves, Rahul Karandikar, Christopher Li… find
  19. Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5):702–712, 2016. doi: 10.1177/1745691616658637. — [doi:10.1177/1745691616658637](https://doi.org/10.1177/1745691616658637) find
  20. Erick H Turner, Annette M Matthews, Eftihia Linardatos, Robert A Tell, and Robert Rosenthal. Selective publication of antidepressant trials and its influence on apparent efficacy. The New England journal of medicine, 358(3):252—260, January 2008. ISSN 0028-4793. doi: 10. 1056/nejmsa065779. URL http:… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fa-2026-sound-agentic-science-requires/