Elamur Research⌕
☰
home / papers / 2026 / Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study

Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study

Paper · 2026 · Yuanjun Zhang and Mourad Oussalah

Summary

Humanitarian situation reports are long and cover many topics at once, so it is hard to pull out comparable evidence about which interventions help and in which contexts. The authors build a two-stage LLM pipeline for ReliefWeb reports (2000–2024) retrieved with the query "cash assistance". The first stage is query-conditioned: it keeps only interventions of the requested class and collects outcome mentions with supporting snippets. The second stage uses those snippets to label each pair's direction (increase, decrease, no change) and evidence strength. This design counters the noise that recall-oriented keyword retrieval brings in. On an expert-annotated benchmark, Qwen-Plus gave the best closed-source result, and a LoRA-distilled Llama-3.1-8B-Instruct did better still. The authors also introduce context-preserving triangulation, which aggregates evidence within disaster×source cells rather than pooling raw counts and summarises cross-context agreement as a Level-of-Evidence (LoE) score. Applied to cash assistance, it shows consistently positive effects on food-related outcomes. The work matters because it offers auditable, scalable evidence synthesis for humanitarian decision support.

Key points

  • Corpus: 8,029 English ReliefWeb reports covering 4 disaster types (flood, drought, epidemic, earthquake) and 5 source types. The evaluation set has 100 expert-annotated reports with 220 relations, matched using BERTScore at a 0.8 threshold.
  • Query conditioning sharply reduces over-extraction. For Qwen-Plus, the average drops from 13.76 to 2.79 relations per document, against a ground truth of 2.20.
  • Qwen-Plus with two-step QC + SG reached 90.73% weighted F1 and had the best cost-efficiency of the three API models; the other two were GPT-4o-mini and DeepSeek-V3.
  • Llama-3.1-8B-Instruct, fine-tuned with LoRA on the teacher's outputs (one adapter per stage), improved from 75.12% to 94.15% weighted F1.
  • Triangulation used 19,568 extracted records and kept 19 of 20 disaster×source cells. For food outcomes it gave P_P = 0.91 and LoE = 0.865, with the cumulative LoE peaking in 2017 at 0.929.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OuluOulu · Finland3 works · 38 citations

Also on the byline: Lappeenranta-Lahti University of Technology.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 worksGeneral Social Sciences · 254 works

Cite

Cite as Yuanjun Zhang and Mourad Oussalah. 2026. Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1626

@inproceedings{zhang2026causal,
  title = {{Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study}},
  author = {Yuanjun Zhang and Mourad Oussalah},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1626},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1626}
}

References

45 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Jo Burton. 2020. “doing no harm” in the digital age: What the digitalization of cash means for humanitarian action. International Review of the Red Cross, 102(913):43–73. find
  5. Cash Learning Partnership (CaLP). 2015. Operational guidance and toolkit for multipurpose cash grants. Enhanced Response Capacity Project 2014–2015. find
  6. Zhiwei Chen, Yupeng Hu, Zhiheng Fu, Zixu Li, Jiale Huang, Qinlei Huang, and Yinwei Wei. 2026. Intent: Invariance and discrimination-aware noise mitigation for robust composed image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 40(25):20463–20471. find
  7. Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Xuemeng Song, and Liqiang Nie. 2025. Offset: Segmentation-based focus shift revision for composed image retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25, page 6113–6122, New York, NY, USA. Association for Computing… find
  8. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… find
  9. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4443–4458.
  10. Shannon Doocy and Hannah Tappis. 2017. Cash-based approaches in humanitarian emergencies: a systematic review. Campbell Systematic Reviews, 13(1):1– 200. find
  11. Julie Freccero, Audrey Taylor, Joanna Ortega, Zabihul-lah Buda, Paschal Kum Awah, Alexandra Blackwell, Ricardo Pla Cordero, and Eric Stover. 2019. Safer cash in conflict: exploring protection risks and barriers in cash programming for internally displaced persons in cameroon and afghanistan. Interna… find
  12. Xueren Ge, Sahil Murtaza, Anthony Cortez, and Homa Alemzadeh. 2026. Emsdialog: Synthetic multi-person emergency medical service dialogue generation from electronic patient care reports via multi-llm agents. Preprint, arXiv:2604.07549. find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  14. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.
  15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  16. Yupeng Hu, Zixu Li, Zhiwei Chen, Qinlei Huang, Zhiheng Fu, Mingzhu Xu, and Liqiang Nie. 2026. Refine: Composed video retrieval via shared and differential semantics enhancement. ACM Trans. Multimedia Comput. Commun. Appl. Just Accepted. find
  17. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, and 399 others. 2024. Gpt-4o…
  18. Todd D Jick. 1979. Mixing qualitative and quantitative methods: Triangulation in action. Administrative science quarterly, 24(4):602–611. find
  19. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation…
  20. Junxian Li, Xinyue Xu, Sai Ma, and Sichao Li. 2025. Faithact: Faithfulness planning and acting in mllms. arXiv preprint arXiv:2511.08409. — [arXiv:2511.08409](https://arxiv.org/abs/2511.08409) find
  21. Zixu Li, Yupeng Hu, Zhiwei Chen, Qinlei Huang, Guozhi Qiu, Zhiheng Fu, and Meng Liu. 2026a. Retrack: Evidence-driven dual-stream directional anchor calibration network for composed video retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 40(28):23373–23381. find
  22. Zixu Li, Yupeng Hu, Zhiwei Chen, Shiqi Zhang, Qinlei Huang, Zhiheng Fu, and Yinwei Wei. 2026b. Habit: Chrono-synergia robust progressive learning framework for composed image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 40(8):6762–6770. find
  23. Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, Juekai Lin, Xiaoran Shang, Conghui He, Wentao Zhang, and Lijun Wu. 2026. Mmfinereason: Closing the multimodal reasoning gap via open data-centric methods. Preprint, arXiv:2601.21821. — [arXiv:2601.21821](https://arxiv.org/abs/2601.21821) find
  24. Peiyang Liu, Ziqiang Cui, Di Liang, and Wei Ye. 2025a. Who stole your data? a method for detecting unauthorized rag theft. Preprint, arXiv:2510.07728. — [arXiv:2510.07728](https://arxiv.org/abs/2510.07728) find
  25. Peiyang Liu, Xi Wang, Ziqiang Cui, and Wei Ye. 2025b. Queries are not alone: Clustering text embeddings for video search. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, page 874–883, New York, NY, USA. Association for Co… find
  26. Peiyang Liu, Jinyu Yang, Lin Wang, Sen Wang, Yunlai Hao, and Huihui Bai. 2023. Retrieval-based unsupervised noisy label detection on text data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 4099–4104, New York, NY, USA. Association fo… find
  27. Zheng Liu, Hao Liang, Bozhou Li, Wentao Xiong, Chong Chen, Conghui He, Wentao Zhang, and Bin Cui. 2025c. Synthvlm: Towards high-quality and efficient synthesis of image-caption datasets for vision-language models. Preprint, arXiv:2407.20756. — [arXiv:2407.20756](https://arxiv.org/abs/2407.20756) find
  28. Yunbo Long, Yuhan Liu, and Liming Xu. 2026. Emomas: Emotion-aware multi-agent system for high-stakes edge-deployable negotiation with bayesian orchestration. Preprint, arXiv:2604.07003.
  29. Yunbo Long, Yuhan Liu, Liming Xu, and Alexandra Brintrup. 2025. Emodebt: Bayesian-optimized emotional intelligence for strategic agent-to-agent debt recovery. Preprint, arXiv:2503.21080. — [arXiv:2503.21080](https://arxiv.org/abs/2503.21080) find
  30. Kexin Ma, Ruochun Jin, Wang Haotian, Wang Xi, Huan Chen, Yuhua Tang, and Qian Wang. 2024. Context-driven index trimming: A data quality perspective to enhancing precision of RALMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4886–4901, Miami, Florida, USA. Associa… find
  31. Xiaoxu Ma, Xiangbo Zhang, and Zhenyu Weng. 2026. Stable and explainable personality trait evaluation in large language models with internal activations. Preprint, arXiv:2601.09833.
  32. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… find
  33. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  34. ReliefWeb. 2025. Reliefweb api documentation. https://apidoc.reliefweb.int/. Service provided by UN OCHA; accessed 2025-12-20. ↗
  35. Roberta Rocca, Nicolò Tamagnone, Selim Fekih, Ximena Contla, and Navid Rekabsaz. 2023. Natural language processing for humanitarian action: Opportunities, challenges, and the path toward humanitarian nlp. Frontiers in big Data, 6:1082787. find
  36. Xuanyu Shi, Wenjing Zhao, Ting Chen, Chao Yang, and Jian Du. 2025. Evidence triangulator: using large language models to extract and synthesize causal evidence across study designs. Nature Communications, 16(1):7355. find
  37. Qianli Wang, Van Bach Nguyen, Nils Feldhus, Luis Felipe Villa-Arenas, Christin Seifert, Sebastian Möller, and Vera Schmitt. 2025a. Truth or twist? optimal model selection for reliable label flipping evaluation in LLM-based counterfactuals. In Proceedings of the 18th International Natural Language Ge… find
  38. Zifeng Wang, Lang Cao, Benjamin Danek, Qiao Jin, Zhiyong Lu, and Jimeng Sun. 2025b. Accelerating clinical evidence synthesis with large language models. npj Digital Medicine, 8(1):509. find
  39. Qianyun Yang, Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, and Liqiang Nie. 2026. Stable: Efficient hybrid nearest neighbor search via magnitude-uniformity and cardinality-robustness. Preprint, arXiv:2604.01617. — [arXiv:2604.01617](https://arxiv.org/abs/2604.01617) find
  40. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777, Flor… find
  41. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  42. Qianchi Zhang, Hainan Zhang, Liang Pang, Yongxin Tong, Hongwei Zheng, and Zhiming Zheng. 2026a. Less is more: Compact clue selection for efficient retrieval-augmented generation reasoning. In Proceedings of the ACM Web Conference 2026, WWW ’26, page 1971–1982, New York, NY, USA. Association for Comp… find
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  44. Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, and Yongbin Li. 2026b. Expseek: Self-triggered experience seeking for web agents. Preprint, arXiv:2601.08605.
  45. Yuanjun Zhang, Fuzel Ahamed Shaik, Suvojit Acharjee, Fahad Khalid, and Mourad Oussalah. 2026c. Towards reliable multimodal disaster severity assessment through preference optimization and explainable vision-language reasoning. Reliability Engineering System Safety, page 112674. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-causal-evidence-extraction-triangulation-crisis/