Elamur Research⌕
☰
home / papers / 2026 / EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI

EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI

Paper · 2026 · Longfei Zuo, Barbara Plank and Siyao Peng

Summary

For NLI tasks, where a hypothesis is judged against a premise, the variation among human annotators' labels (HLV) means that several labels can be reasonable for the same premise–hypothesis pair. This makes it hard to tell real annotation errors apart from legitimate disagreement. The earlier VariErr framework handled this with two rounds of expert work: annotators wrote explanations, then judged whether those explanations still held up. That process is expensive and hard to scale. The authors propose EVADE, which hands both steps to LLMs. Models first write explanations for each candidate label, then give each explanation a validity score. Labels whose support scores poorly are treated as errors. Using the 500 VariErr items, which also appear in ChaosNLI, they compare LLM and human outputs on explanation distributions, overlap of validated labels, and downstream fine-tuning. LLM validation moves explanation distributions closer to human judgment distributions. Dropping LLM-flagged errors from training data also helps fine-tuned models more than dropping errors found by humans. This points to a cheaper, scalable route to cleaner datasets that still keep valid variation.

Key points

  • Data and models: The study uses the VariErr dataset (500 NLI items with labels, explanations and second-round validations from four expert annotators), with ChaosNLI's 100 crowdworker labels per item as the reference distribution. It tests Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct.
  • LLMs over-generate explanations compared with humans:
    • Llama-8B produced 8,845 explanations, while the other models produced roughly 4,000 each.
    • LLM explanations averaged 16–22 words, against 13.89 for humans.
    • LLMs explained nearly all three labels per item, while humans explained 1.76 on average.
  • Manual cleanup of generations: Defective outputs were removed by hand: 37 truncated Llama-8B outputs, 17 fallback responses from Llama-70B, and 6 Qwen-7B outputs containing Chinese text.
  • Fine-tuning results:
    • Downstream models were bert-base-uncased and roberta-base.
    • Removing human-detected errors (VariErr R2) did not improve alignment with human judgment distributions.
    • Setups informed by EVADE raised F1.
    • The all-llm validation setting, which gives the model all labels and explanations as context, performed best.
    • The one-expl setting ranked errors best but produced poorly calibrated label distributions.
  • Limitations and availability: The work covers only NLI and VariErr, and LLMs may introduce their own biases. Code and data are available at github.com/mainlp/LLM_AED.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Longfei Zuo, Barbara Plank and Siyao Peng. 2026. EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.65

@inproceedings{zuo2026evade,
  title = {{EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI}},
  author = {Longfei Zuo, Barbara Plank and Siyao Peng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.65},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.65}
}

References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013. ACM, 2013(2013). find
  2. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Mag., 36(1):15–24.
  3. Gabriel Bernier-Colborne and Sowmya Vajjala. 2024. Annotation errors and ner: A study with ontonotes 5.0. Preprint, arXiv:2406.19172. — [arXiv:2406.19172](https://arxiv.org/abs/2406.19172) find
  4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.…
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages…
  6. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Th…
  7. Beiduo Chen, Siyao Peng, Anna Korhonen, and Barbara Plank. 2025. A rose by any other name: Llm-generated explanations are good proxies for human explanations to collect label distributions on NLI. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - Augu…
  8. Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. "Seeing the big through the small": Can llms approximate human judgment distributions on NLI from a few explanations? In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Flori…
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singap… find
  11. Mononito Goswami, Vedant Sanil, Arjun Choudhry, Arvind Srinivasan, Chalisa Udompanyawit, and Artur Dubrawski. 2023. Aqua: A benchmarking tool for label quality assessment. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIP… find
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  13. Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2025. Litex: A linguistic taxonomy of explanations for understanding within-label variation in natural language inference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,…
  14. Chathuri Jayaweera and Bonnie J. Dorr. 2025. From disagreement to understanding: The case for ambiguity detection in NLI. In Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP, pages 37–46, Suzhou, China. Association for Computational Linguistics.
  15. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. Ecologically valid explanations for label variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 10622–10633. Association for Computational…
  16. Nanjiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natural language inference. Trans. Assoc. Comput. Linguistics, 10:1357–1374.
  17. Uthman Jinadu and Yi Ding. 2024. Noise correction on subjective datasets. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 5385– 5395. Association for Computational Linguistics. find
  18. Jan-Christoph Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2024. Analyzing dataset annotation quality management in the wild. Comput. Linguistics, 50(3):817–866.
  19. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation error detection: Analyzing the past and present for a more coherent future. Comput. Linguistics, 49(1):157–198.
  20. S. Kullback and R. A. Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. find
  21. Jenny Kunz and Marco Kuhlmann. 2024. Properties and challenges of LLM-generated explanations. In Proceedings of the Third Workshop on Bridging Human– Computer Interaction and Natural Language Processing, pages 13–27, Mexico City, Mexico. Association for Computational Linguistics. find
  22. Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, and Jonathan K. Kummerfeld. 2020. Inconsistencies in crowdsourced slot-filling annotations: A typology and identification methods. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spa… find
  23. Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of the 2019 Conference of the North American Chapte… find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  25. Arbi Haza Nasution and Aytug Onan. 2024. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language NLP tasks. IEEE Access, 12:71876–71900. find
  26. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9131–9143. Association for Co…
  27. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. find
  28. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Trans. Assoc. Comput. Linguistics, 7:677–694.
  29. Siyao Peng, Zihang Sun, Sebastian Loftus, and Barbara Plank. 2024. Different tastes of entities: Investigating human label variation in named entity annotations. In Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language, pages 73–81, Malta. Association for Computatio… find
  30. Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10671–10682. Association f…
  31. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  32. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 202… find
  33. Leon Weber, Robert Litschko, Ekaterina Artemova, and Barbara Plank. 2024. Donkii: Characterizing and detecting errors in instruction-tuning datasets. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 197–215, St. Julians, Malta. Association for Computational Linguistics. find
  34. Leon Weber and Barbara Plank. 2023. Activeaed: A human in the loop improves annotation error detection. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Findings of ACL, pages 8834–8845. Association for Computational Linguistics. find
  35. Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2024. Varierr NLI: separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, A…
  36. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018…
  37. Guoqing Zheng, Ahmed Hassan Awadallah, and Susan T. Dumais. 2021. Meta label correction for noisy label learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zuo-2026-evade-llm-based-explanation-generation/