Elamur Research⌕
☰
home / papers / 2026 / ReportLogic: Evaluating Logical Quality in Deep Research Reports

ReportLogic: Evaluating Logical Quality in Deep Research Reports

Paper · 2026 · Jujia Zhao, Zhaoxin Huan, Zihan Wang et al.

Summary

LLM-based Deep Research tools produce long analytical reports. Current evaluation mostly checks atomic facts, citations and fluency, so it misses whether the argument holds together and whether conclusions are actually backed. ReportLogic treats logical quality as auditability: can a reader follow, understand and check the reasoning? Its taxonomy has three layers, split into eight diagnostic dimensions:

  • Macro-Logic: staying on topic with a unified analytical arc.
  • Expositional-Logic: supplying context and keeping a Given→New flow, based on Information Flow Theory.
  • Structural-Logic: evidence and warrant links following the Toulmin model.

How much support counts as enough depends on the query. The authors therefore generate context-aware rubrics for each instance. Annotators then give pairwise preferences per dimension plus an overall verdict on Deep Research queries from varied sources. On this data they train an open-source LogicJudge, which agrees with humans better than general LLM judges and resists manipulation better. The work shows that popular judges are swayed by length or formatting and can silently fill in missing reasoning steps, which argues for dedicated logic evaluators.

Key points

  • Dimensions: the eight include Task Alignment, Global and Local Coherence, Concept Introduction, Evidence Sufficiency, Warrants, Qualifiers & Counterpoints, and Internal Consistency.

  • Rubrics: instance-specific, context-aware rubrics gave the most consistent human judgments, with per-dimension agreement reported as Fleiss' κ.

  • Robustness tests: run on N = 300 instances using two attack types, both measured by Attack Success Rate (ASR):

    • targeted-dimension attacks inject a single logical defect;
    • bias-type attacks change only surface cues.

  • Judge results: DeepSeek-V3 was the most vulnerable judge. o3 struggled on Warrants and Local Coherence. Length Bias was the strongest manipulation and Qualifier Wording the weakest. LogicJudge kept low ASR on most dimensions.

  • Failure patterns and availability: weak reports showed three recurring problems:

    • document-level inconsistency;
    • smooth transitions that lack real inferential links;
    • implicit or missing warrants.

    Code and data: github.com/Polaris-JZ/ReportLogic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Ant GroupHangzhou · China42 works · 1,131 citations🎓Leiden UniversityLeiden · Netherlands32 works · 1,074 citations🔬Helmholtz Center for Information SecuritySaarbrücken · Germany14 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHistory and Philosophy of Science · 53 works

Cite

Cite as Jujia Zhao, Zhaoxin Huan, Zihan Wang, Xiaolu Zhang, Jun Zhou, Suzan Verberne and Zhaochun Ren. 2026. ReportLogic: Evaluating Logical Quality in Deep Research Reports. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.384

@inproceedings{zhao2026reportlogic,
  title = {{ReportLogic: Evaluating Logical Quality in Deep Research Reports}},
  author = {Jujia Zhao, Zhaoxin Huan, Zihan Wang, Xiaolu Zhang, Jun Zhou, Suzan Verberne and Zhaochun Ren},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.384},
  url = {https://doi.org/10.18653/v1/2026.acl-long.384}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Boxi Cao, Mengjie Ren, Hongyu Lin, Xianpei Han, Feng Zhang, Junfeng Zhan, and Le Sun. 2024. Structeval: Deepen and broaden large language model assessment via structured evaluation. arXiv preprint arXiv:2408.03281. find
  2. Michael K Chen, Xikun Zhang, and Dacheng Tao. 2025. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851.
  3. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
  4. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763.
  5. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627.
  6. Michael Alexander Kirkwood Halliday and Christian MIM Matthiessen. 2013. Halliday’s introduction to functional grammar. Routledge. find
  7. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, and 1 others. 2024. Folio: Natural language reasoning with first-order logic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processi…
  8. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, and 1 others. 2025. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096. — [arXiv:2506.18096](https://arxiv.org/abs/2506.18096) find
  9. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and 1 others. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
  10. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. find
  11. John Lawrence and Chris Reed. 2019. Argument mining: A survey. Computational linguistics, 45(4):765–818.
  12. Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia. 2025. Reportbench: Evaluating deep research agents via academic survey tasks. arXiv preprint arXiv:2508.15804. — [arXiv:2508.15804](https://arxiv.org/abs/2508.15804) find
  13. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, and 1 others. 2025. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352.
  14. Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. arXiv preprint arXiv:2410.12832. find
  15. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  16. Jakob Mökander, Jessica Morley, Mariarosaria Taddeo, and Luciano Floridi. 2021. Ethics-based auditing of automated decision-making systems: Nature, scope, and limitations. Science and Engineering Ethics, 27(4):44. find
  17. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522.
  18. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, and 1 others. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191. — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) find
  19. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  20. Zhengliang Shi, Yiqun Chen, Haitao Li, Weiwei Sun, Shiyu Ni, Yougang Lyu, Run-Ze Fan, Bowen Jin, Yixuan Weng, Minjun Zhu, and 1 others. 2025. Deep research: A systematic survey. arXiv preprint arXiv:2512.02038. — [arXiv:2512.02038](https://arxiv.org/abs/2512.02038) find
  21. Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. 2024. Towards verifiable text generation with evolving memory and self-reflection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8211–8227. find
  22. Stephen E Toulmin. 2003. The uses of argument. Cambridge university press. find
  23. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Al-berdingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu…
  24. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, and 1 others. 2024. Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:80756–80827.
  25. Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. 2024. Longgenbench: Benchmarking long-form generation in long context llms. arXiv preprint arXiv:2409.02076. — [arXiv:2409.02076](https://arxiv.org/abs/2409.02076) find
  26. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. 2025. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507.16280. — [arXiv:2507.16280](https://arxiv.org/abs/2507.16280) find
  27. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-reportlogic-evaluating-logical-quality-deep/