Elamur Research⌕
☰
home / papers / 2026 / Position: Correct Answer, Wrong Mechanism — When AI Scientists Defend General Claims Their Own Data Contradicts

Position: Correct Answer, Wrong Mechanism — When AI Scientists Defend General Claims Their Own Data Contradicts

Paper · 2026 · Steven Young Eulig

Summary

The paper argues that judging AI scientist systems only by whether their final answer is right hides a serious failure. An agent can reach a correct-looking result while justifying it with reasoning that is false or that overreaches beyond what it tested. The author calls this pattern Correct Answer, Wrong Mechanism (CAWM). The study ran 28 episodes in which coding agents tried to rediscover a known particle-identification observable, the leading-edge photon fraction f_early, in a Geant4 Cherenkov detector simulation. Twenty episodes used Claude Opus 4.6 under four prompt conditions, ranging from fully open-ended to a deliberately biased prior. Eight more episodes probed two other frontier models.

CAWM occurred in 4/20 primary and 3/8 cross-model episodes. Honesty and mechanism correctness turned out to be separable: an agent could revise a false belief and still defend an observable with physics that its own data refuted. The author proposes evaluating systems on three separate axes: outcome, mechanism fidelity and epistemic honesty. The paper also offers cheap detection checks. The conclusion is that agents are dependable tools but not yet trustworthy co-authors.

Key points

  • Setup: Four prompt variants were used, N=5 each. A was open-ended, B added a five-step scaffold, C was a positive control that supplied f_early, and D gave a partially misleading prior. Impact parameters were d = 10–25 m at E = 40 GeV.
  • Why the main alternative misleads: The competing observable σ(hitTime) fails to discriminate at d = 10 m but separates muons from electrons at 20–25 m. An agent that tests it in only one regime therefore draws conclusions that do not transfer.
  • Discovery versus execution: No open-ended Prompt A agent chose f_early as its primary observable (0/5), and three defended their choice with incorrect physics. With the formula given (Prompt C), all five agents reproduced it.
  • Scaffolding and honesty: The scaffold removed CAWM for Claude (0/5) but not in the cross-model probe (2/2). Under Prompt D, all five agents rejected the false claim, yet episode 19 still argued for σ using reasoning its own data contradicted.
  • Detection: A one-step regime-shift check combined with a recomputation step flags every CAWM case. The simulation data (ROOT files), prompts and verification script are released at github.com/seulig/g4 agent icml 2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Steven Young Eulig. 2026. Position: Correct Answer, Wrong Mechanism — When AI Scientists Defend General Claims Their Own Data Contradicts. arXiv:2606.23175.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.23175 · arXiv 2606.23175

@misc{eulig2026position,
  title = {{Position: Correct Answer, Wrong Mechanism — When AI Scientists Defend General Claims Their Own Data Contradicts}},
  author = {Steven Young Eulig},
  year = {2026},
  doi = {10.48550/arxiv.2606.23175},
  eprint = {2606.23175},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.23175}
}

References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Labs and deployers. (i) Role boundaries matter. LLM coding agents are reliable at tool-level tasks and unreliable at co-author-level observable selection in the simulation-based discovery setting studied here (Table 3). (ii) Scaffolding Agostinelli, S., Allison, J., Amako, K., Apostolakis, J., Arauj… find
  2. Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., and Clune, J. Towards end-to-end automation of AI research. Nature, 651:914–919, 2026. find
  3. Beel, J., Kan, M.-Y., and Baumgart, M. Evaluating Sakana’s AI scientist: Bold claims, mixed results, and a promising future? ACM SIGIR Forum, 2025. arXiv preprint arXiv:2502.14297.
  4. Luo, Z., Kasirzadeh, A., and Shah, N. B. The more you automate, the less you see: Hidden pitfalls of AI scientist systems. arXiv preprint arXiv:2509.08713, 2025.
  5. NeurIPS 2025 AI4Science Workshop (Spotlight). find
  6. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.
  7. McCoy, R. T., Pavlick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
  8. Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., Dey, V., Xue, M., Baker, F. N., Burns, B., Adu-Ampratwum, D., Huang, X., Ning, X., Gao, S., Su, Y., and Sun, H. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific di… — ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  9. Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadolski, T., Sulovari, A., Landsness, E. C., Barabasi, D. L., Narayanan, S., Evans, N., Reddy, S., Foiani, M., Kamal, A., Shriver, L. P., Cao, F., Wassie, A. T., Laurent, J. M., Melville-Green, E., Caldas, M., Bou, A., Roberts, K. F., Zagorac, S., Or… — Kosmos: An AI Scientist for Autonomous Discovery
  10. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. find
  11. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., Saab, K., Popovici, D., Blum, J., Zhang, F., Chou, K., Hassidim, A., Gokturk, B., Vahdat, A., Kohli, P., Matias, Y., Carroll, A., Kulkarni, K., Tomasev, N., Guan, Y., Dhil… — Accelerating scientific discovery with Co-Scientist
  12. Niven, T. and Kao, H.-Y. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019. find
  13. Ren, R., Agarwal, A., Mazeika, M., Menghini, C., Vacareanu, R., Kenstler, B., Yang, M., Barrass, I., Gatti, A., Yin, X., Trevino, E., Geralnik, M., Khoja, A., Lee, D., Yue, S., and Hendrycks, D. The MASK benchmark: Disentangling honesty from accuracy in AI systems. arXiv preprint arXiv:2503.03750, 2… — [arXiv:2503.03750](https://arxiv.org/abs/2503.03750) find
  14. Jansen, P., Côté, M.-A., Khot, T., Bransom, E., Dalvi Mishra, B., Majumder, B. P., Tafjord, O., and Clark, P. DiscoveryWorld: A virtual environment for developing and evaluating automated scientific discovery agents. Advances in Neural Information Processing Systems, 37, 2024.
  15. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. Language models don’t always say what they think: Unfaithful explanations in Chain-of-Thought prompting. Advances in Neural Information Processing Systems, 36, 2023. — Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  16. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Te… — Measuring Faithfulness in Chain-of-Thought Reasoning
  17. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
  18. Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., and Ha, D. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025.
  19. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations, 2024. — Let's Verify Step by Step

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eulig-2026-position-correct-answer-wrong/