Elamur Research⌕
☰
home / papers / 2026 / Can Large Language Models Infer Causal Relationships from Real-World Text?

Can Large Language Models Infer Causal Relationships from Real-World Text?

Paper · 2026 · Ryan Saklad, Aman Chadha, Oleg V. Pavlov et al.

Summary

Most tests of LLM causal reasoning use short synthetic passages where cause and effect are stated outright. This work asks whether models can recover causal structure from long, authentic documents where many links are only implied. The authors build ReCITE, a benchmark of 292 open-access MDPI and PLOS papers that use causal loop diagrams. In each paper, the human-drawn diagram was transcribed by annotators into a list of directed edges, which serves as the ground-truth causal graph. Given the full paper text, a model must produce the whole graph: events as nodes and causal links as edges. Outputs are scored by an LLM-as-a-Judge framework.

Ten state-of-the-art LLMs were tested zero-shot, and all performed weakly. The top average F1 was 0.535. Scores dropped further as causal statements became less explicit, and stayed low even when node names were supplied. This suggests the bottleneck is causal inference rather than spotting entities. The benchmark therefore exposes a concrete gap relevant to document-level reasoning.

Key points

  • Corpus construction: a keyword search for "causal loop diagram" returned 646 candidate papers. After manual filtering, 292 remained, each containing a single usable causal graph.
  • Scale and variety:
    • Graphs have 5–140 nodes and 6–205 edges.
    • Texts are 684–171,141 characters long.
    • About 87.7% of nodes are mentioned in the text.
    • Papers span 17 fields, with Environmental Science and Engineering at roughly 20% each.
  • Annotation quality: a second annotator re-labelled about 13% of the graphs. Edge-level agreement reached F1 0.984 and Cohen's κ 0.987.
  • Results: Claude Opus 4.5 was the strongest model overall, but no model did well in any domain. LLM judges correlated strongly with one another (r > 0.78). All experiments cost under $1000.
  • Limitations:
    • The data is English-only.
    • Models are told the target node count.
    • Evaluation is zero-shot with default settings.
    • All graphs come from system-dynamics-style diagrams.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AppleCupertino · United States23 works · 2,318 citations🎓Worcester Polytechnic InstituteWorcester · United States12 works · 380 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ryan Saklad, Aman Chadha, Oleg V. Pavlov and Raha Moraffah. 2026. Can Large Language Models Infer Causal Relationships from Real-World Text?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1003

@inproceedings{saklad2026can,
  title = {{Can Large Language Models Infer Causal Relationships from Real-World Text?}},
  author = {Ryan Saklad, Aman Chadha, Oleg V. Pavlov and Raha Moraffah},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1003},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1003}
}

References

46 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude Opus 4.5: Model card. Technical report. find
  2. Nikolay Babakov, Ehud Reiter, and Alberto Bugarín-Diz. 2025. CausalGraphBench: a benchmark for evaluating language models capabilities of causal graph discovery. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 24… find
  3. Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2025. Hallucination of multimodal large language models: A survey. Preprint, arXiv:2404.18930.
  4. T. Bratanic. 2024. Building knowledge graphs with llm graph transformer: A deep dive into langchain’s implementation of graph construction with llms. Towards Data Science. find
  5. Sirui Chen, Bo Peng, Meiqi Chen, Ruiqi Wang, Mengying Xu, Xingyu Zeng, Rui Zhao, Shengjie Zhao, Yu Qiao, and Chaochao Lu. 2024. Causal evaluation of language models. Preprint, arXiv:2405.00622. — [arXiv:2405.00622](https://arxiv.org/abs/2405.00622) find
  6. Haoang Chi, He Li, Wenjing Yang, Feng Liu, Long Lan, Xiaoguang Ren, Tongliang Liu, and Bo Han. 2025. Unveiling causal reasoning in large language models: Reality or mirage? Preprint, arXiv:2506.21215. — [arXiv:2506.21215](https://arxiv.org/abs/2506.21215) find
  7. DeepSeek-AI. 2025a. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. DeepSeek-AI. 2025b. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint. ArXiv:2512.02556.
  9. Xiaohan Ding, Kaike Ping, Buse Çarık, and Eugenia Rho. 2025. A multi-level benchmark for causal language understanding in social media discourse. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28764–28778, Suzhou, China. Association for Computational… find
  10. Prashant Garg and Thiemo Fetzer. 2025. Causal claims in economics. Preprint, arXiv:2501.06873. — [arXiv:2501.06873](https://arxiv.org/abs/2501.06873) find
  11. Google DeepMind. 2025. Gemini 3: Technical report. Technical report. find
  12. Donggyu Lee, Sungwon Park, Yerin Hwang, Hyoshin Kim, Hyunwoo Oh, Jungwon Kim, Meeyoung Cha, Sangyoon Park, and Jihee Kim. 2025. Benchmarking llm causal reasoning with scientifically validated relationships. Preprint, arXiv:2510.07231. find
  13. Alison Gopnik, Clark Glymour, David M. Sobel, Laura E. Schulz, Tamar Kushnir, and David Danks. 2004. A theory of causal learning in children: Causal maps and bayes nets. Psychological Review, 111(1):3–32. find
  14. Norman Meuschke, Apurva Jagdale, Timo Spinde, Jelena Mitrović, and Bela Gipp. 2023. A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi-domain Evaluation Framework for Academic Documents, page 383–405. Springer Nature Switzerland. find
  15. Arthur C. Graesser, Murray Singer, and Tom Trabasso. 1994. Constructing inferences during narrative text comprehension. Psychological Review, 101(3):371– 395. find
  16. Martina Miliani, Serena Auriemma, Alessandro Bondielli, Emmanuele Chersoni, Lucia Passaro, Irene Sucameli, and Alessandro Lenci. 2025. Explica: Evaluating explicit causal reasoning in large language models. Preprint, arXiv:2502.15487. find
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  18. Niyousha Hosseinichimeh, Aritra Majumdar, Ross Williams, and Navid Ghaffarzadegan. 2024. From text to map: a system dynamics bot for constructing causal loop diagrams. System Dynamics Review, 40(3):e1782. find
  19. Mistral AI Team. 2025. Mistral Small 3. find
  20. Moonshot AI. 2025. Kimi K2: Reasoning model card. Model card. find
  21. Sangmin S. Oh. 2025. Plausibly exogenous galore. Notion. Available at https://sangmino.notion.site/1a897b8106ca44eeaf31dcd5ae5a61b1?v= ff7dc75862c6427eb4243e91836e077e. ↗
  22. OpenAI. 2025a. GPT-5 system card. Official model documentation and system card. find
  23. OpenAI. 2025b. GPT-5.2 system card. System card. find
  24. Z. Jin, Y. Chen, F. Leeb, L. Gresele, O. Kamal, Z. Lyu, K. Blin, F. G. Adauto, M. Kleiman-Weiner, M. Sachan, and B. Schölkopf. 2024. CLadder: Assessing causal reasoning in language models. https://arxiv.org/abs/2312.04350. ↗
  25. OpenAI. 2025c. OpenAI o3-mini system card. OpenAI, January 31, 2025. https://cdn.openai.com/ o3-mini-system-card.pdf. ↗
  26. Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. 2024a. Cold: Causal reasoning in closed daily activities. Preprint, arXiv:2411.19500. — [arXiv:2411.19500](https://arxiv.org/abs/2411.19500) find
  27. Jason Priem, Heather Piwowar, and Richard Orr. 2022. Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. Preprint, arXiv:2205.01833. — [arXiv:2205.01833](https://arxiv.org/abs/2205.01833) find
  28. Nitish Joshi, Abulhair Saparov, Yixin Wang, and He He. 2024b. Llms are prone to fallacies in causal inference. Preprint, arXiv:2406.12158. find
  29. Hyunjung Kim and David F. Andersen. 2012. Building confidence in causal maps generated from purposive text data: mapping transcripts of the federal reserve. System Dynamics Review, 28(4):311–328. find
  30. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2024. Causal reasoning and large language models: Opening a new frontier for causality. Preprint, arXiv:2305.00050. find
  31. Uriel Anderson Lasheras and Vladia Pinheiro. 2025. CaLQuest.PT: Towards the collection and evaluation of natural causal ladder questions in Portuguese for AI agents. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 325–343, Abu Dhabi, United Arab Emirates. As… find
  32. Judea Pearl. 2009. Causality. Cambridge University Press, Cambridge. find
  33. Qwen Team. 2024. Qwen2.5: A party of foundation models! find
  34. Qwen Team. 2025. QwQ-32B: Embracing the power of reinforcement learning. find
  35. Ioannis Tsamardinos, Laura E. Brown, and Constantin F. Aliferis. 2006. The max-min hill-climbing bayesian network structure learning algorithm. Machine Learning, 65:31–78. find
  36. G. A. Veldhuis, D. Blok, M. H. T. de Boer, G. J. Kalk-man, R. M. Bakker, and R. P. M. van Waas. 2024. From text to model: Leveraging natural language processing for system dynamics model development. System Dynamics Review, 40(3): e1780. find
  37. Guangya Wan, Yunsheng Lu, Yuqi Wu, Mengxuan Hu, and Sheng Li. 2025. Large language models for causal discovery: Current landscape and future directions. Preprint, arXiv:2402.11068. — [arXiv:2402.11068](https://arxiv.org/abs/2402.11068) find
  38. Duo Wang, Yuan Zuo, Fengzhi Li, and Junjie Wu. 2024. Llms as zero-shot graph learners: Alignment of gnn representations with llm token embeddings. Preprint, arXiv:2408.14512. — [arXiv:2408.14512](https://arxiv.org/abs/2408.14512) find
  39. Jordan White and Pete Barbrook-Johnson. 2025. Using vision-language models to extract network data from images of system maps. INET Oxford Working Paper Series, (2025-26). find
  40. Khurram Yamin, Shantanu Gupta, Gaurav R. Ghosal, Zachary C. Lipton, and Bryan Wilder. 2024. Failure modes of llms for causal reasoning on narratives. Preprint, arXiv:2410.23884. find
  41. S. Yu, T. Huang, M. Liu, and Z. Wang. 2023. BEAR: Revolutionizing service domain knowledge graph construction with llm. In Monti, F., Rinderle-Ma, S., Ruiz Cortés, A., Zheng, Z., and Mecella, M. (Eds.), Service-Oriented Computing. find
  42. ICSOC 2023. Lecture Notes in Computer Science, vol 14419, pp. 339–346. Springer, Cham. find
  43. B. Zhang and H. Soh. 2024. Extract, define, canonicalize: An llm-based framework for knowledge graph construction. arXiv preprint arXiv:2404.03868.
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  45. Zhipu AI. 2025. GLM-4.7: General language model technical report. Technical report. find
  46. Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. 2024. Causalbench: A comprehensive benchmark for causal learning capability of llms. Preprint, arXiv:2404.06349. — [arXiv:2404.06349](https://arxiv.org/abs/2404.06349) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saklad-2026-can-large-language-models-infer/