Elamur Research⌕
☰
home / papers / 2026 / NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

Paper · 2026 · Zhi Xu and Yun Fu

Summary

Large language models often confuse correlation with causation. Most causal reasoning benchmarks (e.g., CausalQA, Cladder) use clean, fully observed scenarios where shallow heuristics are enough. Xu and Fu present NoisyCausal, a synthetic benchmark built in several steps. Each item starts from a sampled causal DAG, which is mapped onto a real-world story (medicine, education, economics). Values are drawn from a structural causal model (SCM), and the item is then corrupted with controlled noise: distractor variables, flipped values, hidden confounders and missing observations.

The authors also propose a modular graph-guided method. The LLM first identifies variables, then builds a causal graph from the text, and finally answers through a prompt organized around that graph rather than reasoning end to end. This approach outdoes plain prompting and reasoning baselines on NoisyCausal. It stays robust when misleading beliefs are inserted into prompts and transfers to Cladder without task-specific tuning. The work matters because it isolates concrete failure modes of LLM causal reasoning and shows that explicit structure makes inference sturdier and easier to interpret.

Key points

  • Generation pipeline. The pipeline has five stages: graph sampling, semantic grounding, SCM sampling, noise injection and question assembly.
    • DAGs have 3–7 nodes and cover chains, forks, colliders and multi-parent structures.
    • Each variable is tagged with a type (binary, categorical or continuous), its observability and its role.
  • Belief-Inconsistent Perturbation (BIP) test. False "common beliefs" were added to 1,000 clean instances, with the true causal relations and numbers left unchanged. Accuracy dropped as follows:
    • GPT-3.5: 57.9 → 51.2
    • GPT-4: 62.8 → 56.7
    • Causal CoT: 73.4 → 69.1
    • Graph-guided method: 80.7 → 79.5
  • Transfer to CausalQA. LLaMA-7B fine-tuned with LoRA on NoisyCausal improved on CausalQA causal questions from 44.2% to 49.3% (+5.1 points), without ever seeing CausalQA during training.
  • Error analysis. Numerical mistakes also appear when cues conflict, for example base rates versus conditional probabilities.
  • Limitations.
    • LLM-based variable and graph extraction can propagate errors to later steps.
    • The pipeline has no end-to-end optimization.
    • The method assumes the causal structure can be recovered from the text.
    • The authors state that real-world generalization still needs testing, despite the small CausalQA transfer experiment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityBoston · United States68 works · 7,316 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Zhi Xu and Yun Fu. 2026. NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1833

@inproceedings{xu2026noisycausal,
  title = {{NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise}},
  author = {Zhi Xu and Yun Fu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1833},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1833}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, Axel-Cyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Hagen, and Martin Potthast. 2022. Causalqa: A benchmark for causal question answering. In Proceedings of the 29th International Conference on Computational Lingu… find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Dexiong Chen, Leslie O’Bray, and Karsten Borgwardt. 2022. Structure-aware transformer for graph representation learning. In International conference on machine learning, pages 3469–3489. PMLR. find
  5. Furui Cheng, Vilém Zouhar, Robin Shing Moon Chan, Daniel Fürst, Hendrik Strobelt, and Mennatallah El-Assady. 2024. Interactive analysis of llms using meaningful counterfactuals. arXiv preprint arXiv:2405.00708. — [arXiv:2405.00708](https://arxiv.org/abs/2405.00708) find
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  7. Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. 2019. Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162. — [arXiv:1901.08162](https://arxiv.org/abs/1901.08162) find
  8. Haitao Jiang, Lin Ge, Yuhe Gao, Jianian Wang, and Rui Song. 2023. Large language model for causal decision making. arXiv preprint arXiv:2312.17122. — [arXiv:2312.17122](https://arxiv.org/abs/2312.17122) find
  9. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and 1 others. 2023. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems, 36:31038– 31065. find
  10. Uriel Lasheras, Elioenai Alves, and Vladia Pinheiro. 2025. Interventional and counterfactual causal reasoning for llm-based ai agents: A dataset and evaluation in portuguese. Procesamiento del Lenguaje Natural, 74:363–382. find
  11. Hang Luo, Jian Zhang, and Chujun Li. 2025. Causal graphs meet thoughts: Enhancing complex reasoning in graph-augmented llms. arXiv preprint arXiv:2501.14892. — [arXiv:2501.14892](https://arxiv.org/abs/2501.14892) find
  12. Daniel Malinsky and David Danks. 2018. Causal discovery algorithms: A practical guide. Philosophy Compass, 13(1):e12470. find
  13. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295.
  14. Judea Pearl. 2009. Causality. Cambridge university press. find
  15. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A Smith, and Yejin Choi. 2018. Event2mind: Commonsense inference on events, intents, and reactions. arXiv preprint arXiv:1805.06939. find
  16. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. find
  17. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–30… find
  18. Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634. find
  19. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and 1 others. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921.
  20. Aoqi Zuo, Susan Wei, Tongliang Liu, Bo Han, Kun Zhang, and Mingming Gong. 2022. Counterfactual fairness with partially known causal graph. Advances in Neural Information Processing Systems, 35:1238– 1252. find
  21. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  22. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford. edu/2023/03/13/alpaca. html, 3(6):… ↗
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  24. Lijing Wang, Aniruddha Adiga, Jiangzhuo Chen, Adam Sadilek, Srinivasan Venkatramanan, and Madhav Marathe. 2022. Causalgnn: Causal-based graph neural networks for spatio-temporal epidemic forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 12191–12199. find
  25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  26. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  27. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  28. Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. 2024. Easytool: Enhancing llm-based agents with concise tool instruction. arXiv preprint arXiv:2401.06201. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-noisycausal-benchmark-evaluating-causal-reasoning/