Elamur Research⌕
☰
home / papers / 2026 / Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

Paper · 2026 · Seyedali Mohammadi, Manas Gaur and Francis Ferraro

Summary

LLMs are increasingly used in research pipelines, but it is unclear how well they judge whether a scientific claim is feasible, meaning whether it fits existing knowledge and can be tested experimentally. The authors treat this as a binary decision (FEASIBLE/INFEASIBLE) accompanied by a short justification. They build a controlled-knowledge protocol in which each hypothesis is shown in one of four ways: alone, with extracted experiment descriptions, with extracted outcomes, or with both. A stability analysis then randomly removes parts of the experiment and outcome lists to see how predictions degrade. Five proprietary models are tested on MATTER-OF-FACT and REASONS, with post-cutoff filtering to reduce pretraining leakage.

Outcome evidence generally improves accuracy beyond what the models know internally. Experiment descriptions are fragile: incomplete context can push performance below the hypothesis-only baseline, often in a non-monotonic way. The authors trace this to three behaviours: matching on surface topic, failing to check whether the evidence is relevant, and anchoring on partial fragments. The framework matters because it reveals weaknesses that standard claim-verification and RAG evaluations miss.

Key points

  • Protocol: There are four conditions (H, H+E, H+O, H+E+O). Reveal fractions k1 (experiments) and k2 (outcomes) take values in {0, 0.5, 1.0}, with R = 5 random subsamples per configuration, reported as mean and standard deviation.
  • Metrics: Accuracy, macro-F1 and MCC are reported, plus a below-baseline rate. A lexical-overlap score (ROUGE) against gold explanations is used on MATTER-OF-FACT only as a diagnostic.
  • Models: gpt-5.1, gpt-4o, Gemini-2.5-Pro, Gemini-2.5-Flash and Grok-4.1-fast, all prompted with identical instructions.
  • Data:
    • MATTER-OF-FACT (Jansen et al., 2025) contains claims that a source paper may support, contradict or only partly address.
    • REASONS (Saxena et al., 2025) is a positive-feasibility set.
    • Leakage control relies on model knowledge cutoffs: GPT-4o (Oct 2023) uses the full test split, while GPT-5.1 (Sep 2024) and Gemini-2.5 (Jan 2025) use narrower 2024–2025 subsets.
  • Evidence and resources: Experiments and outcomes are extracted once per instance and the full paper text is never shown; a manual audit checked extraction coverage. Code and prompts are available at github.com/mohammadi-ali/scify (DARPA SciFy program).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, Baltimore CountyBaltimore · United States14 works · 915 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInterpretability · 130 works

Cite

Cite as Seyedali Mohammadi, Manas Gaur and Francis Ferraro. 2026. Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.50

@inproceedings{mohammadi2026experiments,
  title = {{Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models}},
  author = {Seyedali Mohammadi, Manas Gaur and Francis Ferraro},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.50},
  url = {https://doi.org/10.18653/v1/2026.acl-short.50}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, and Tristan Miller. 2025. Transforming science with large language models: A survey on ai-assisted scien…
  2. Peter Jansen, Samiah Hassan, and Ruoyao Wang. 2025. Matter-of-fact: A benchmark for verifying the feasibility of literature-supported claims in materials science. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4090–4102, Suzhou, China. Association fo… find
  3. Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh. 2025. Beyond designer’s knowledge: Generating materials design hypotheses via a large language model. Acta Materialia, 297:121307. find
  4. Seyedali Mohammadi, Bhaskara Hanuma Vedula, He-mank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, and Manas Gaur. 2025. Do LLMs adhere to label definitions? examining their receptivity to external label definitions. In Proceedings of the 2025 Conference on Empirical Methods in Natural… find
  5. Yumi Nakamura, Hiroshi Watanabe, Aiko Tanaka, Masato Yasui, Jun Nishihira, and Norihito Murayama. 2020. Effect of increased daily water intake and hydration on health in japanese adults. Nutrients, 12(4). find
  6. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  7. Delip Rao, Weiqiu You, Eric Wong, and Chris Callison-Burch. 2025. NSF-SciFy: Mining the NSF awards database for scientific claims. In Proceedings of The 5th New Frontiers in Summarization Workshop, pages 183–198, Hybrid. Association for Computational Linguistics.
  8. Yash Saxena, Deepa Tilwani, Ali Mohammadi, Edward Raff, Amit Sheth, Srinivasan Parthasarathy, and Manas Gaur. 2025. Attribution in scientific literature: New benchmark and methods. Preprint, arXiv:2405.02228. find
  9. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13545–13565, Bangkok, Thailand. Association for Comput… find
  10. Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. 2025. From automation to autonomy: A survey on large language models in scientific discovery. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17733–17750… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammadi-2026-experiments-or-outcomes-probing-scientific/