Elamur Research⌕
☰
home / papers / 2026 / Benchmarking LLM's Capability in Reasoning over Conflicting Web References

Benchmarking LLM's Capability in Reasoning over Conflicting Web References

Paper · 2026 · Yizhen Yuan, Rui Kong, Dongze Li et al.

Summary

Web-search assistants built on RAG put every retrieved page into the model's context. These pages often disagree, so the LLM has to notice and handle conflicts that a search-engine user would normally sort out by hand. Existing contradiction datasets are small, synthetic, or limited to two short opposing sentences. The authors therefore built ConfRAG, which pairs 1,814 open-domain questions with real web paragraphs, averaging 9.58 per question. To create the gold annotations, an LLM pulled answers and justifications from each document, and annotators grouped them into viewpoint clusters with a human-in-the-loop process. Models are then tested on three tasks:

  • assigning documents to the correct viewpoint groups;
  • recovering every distinct answer;
  • reproducing the reasons behind each answer.

Seven proprietary and open models were evaluated, and all of them fell well short of the gold annotations. Even GPT-4.1, the best performer, recovered fewer than half of the answers and few of the rationales. The results show that factual accuracy alone is not enough. Models also need to be aware of contradictions and sensitive to sources. The construction pipeline can be reused to build benchmarks for new domains.

Key points

  • Dataset: ConfRAG has 1,814 questions drawn from Natural Questions, ELI5, Yahoo Answers and hand-curated examples. Documents were found by keyword search and converted to clean text with jina.ai. Strong contradictions appear in 57.2% of the questions.
  • Metrics: Answer clustering is scored with Normalized Mutual Information (NMI). Answer and reason coverage use bipartite maximum matching on keyword-based substring similarity. All scores fall between 0 and 1.
  • Models tested: GPT-4.1, GPT-4.1-mini, Claude-3-7-Sonnet, Llama-3.1-70B/8B-Instruct and Qwen2.5-72B/7B-Instruct.
  • Results:
    • GPT-4.1 led with NMI 0.47, answer coverage 0.45 and reason coverage 0.15.
    • The remaining models scored below 0.37 on answer coverage and generally under 0.14 on reason coverage.
    • Claude produced the most format-compliant outputs.
    • The 72B Qwen model clearly outperformed the 7B one.
  • Setup and availability: Models receive the true number of clusters k as input, so a setting where k is unknown is left for future work. The data is released under CC BY 4.0 on GitHub (XaiverYuan/ConfRAG).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Yizhen Yuan, Rui Kong, Dongze Li, Yuanchun Li and Yunxin Liu. 2026. Benchmarking LLM's Capability in Reasoning over Conflicting Web References. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.11

@inproceedings{yuan2026benchmarking,
  title = {{Benchmarking LLM's Capability in Reasoning over Conflicting Web References}},
  author = {Yizhen Yuan, Rui Kong, Dongze Li, Yuanchun Li and Yunxin Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.11},
  url = {https://doi.org/10.18653/v1/2026.acl-long.11}
}

References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2024. The claude 3 model family: Opus, sonnet, haiku. Technical report by Anthropic. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. Geo: Generative engine optimization. In KDD 2024 - Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Proceedings of the ACM SIGKDD International Co… find
  4. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. find
  5. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31.
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  7. Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. 2009. Normalized mutual information feature selection. IEEE Transactions on neural networks, 20(2):189–201. find
  8. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
  9. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.
  10. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1.
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, and Prasanna Sattigeri. 2024. Wikicontradict: A benchmark for evaluating llms on real-world knowledge conflicts from wikipedia. Advances in Neural Information Processing Systems, 37:… find
  13. Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. Won’t get fooled again: Answering questions with false premises. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5626– 5643, Toronto, Can… find
  14. Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. Boardgameqa: A dataset for natural language reasoning with contradictory information. Advances in Neural Information Processing Systems, 36:39052–39074. find
  15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Lin…
  16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  17. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… find
  18. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa…
  19. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  20. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations. find
  21. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and 1 others. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-benchmarking-llm-s-capability-reasoning/