Elamur Research⌕
☰
home / papers / 2026 / Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

Paper · 2026 · Pingjun Hong, Beiduo Chen, Siyao Peng et al.

Summary

NLI datasets often contain conflicting labels for the same premise–hypothesis pair. Earlier work with the LiTEx taxonomy studied only one kind of variation: annotators who pick the same label but justify it differently. This paper extends LiTEx to cases where the labels themselves diverge. One trained annotator categorized every free-text explanation in two English datasets, LiveNLI and VariErr. In both datasets, the same people produced the labels and the explanations, and items can carry several plausible labels. The authors then compared variation along three axes:

  • NLI label agreement,
  • agreement on taxonomy category,
  • semantic similarity of the explanation texts.

They also considered annotators' selection bias and followed individual annotators over many items. Two findings stand out. First, annotators sometimes choose different labels while giving nearly identical reasoning, so a label conflict can hide shared interpretation. Second, agreement on reasoning category tracks explanation similarity more closely than label agreement does. Individual annotators also show stable preferences in reasoning strategy and label choice. The work argues against treating single labels as ground truth and proposes reasoning categories as a diagnostic layer for building datasets and evaluating models.

Key points

  • LiTEx has eight categories in two groups. Six are Text-Based: Logic Conflict, Absence of Mention, Pragmatic, Semantic, Syntactic and Coreference. The other two, Factual Knowledge and Inferential Knowledge, make up the World-Knowledge group.
  • LiveNLI is an MNLI subset of 122 items, each labeled by at least 10 crowdworkers, giving 1,404 explanation–label pairs. Its labels (true/either/false) were mapped to entailment/neutral/contradiction.
  • VariErr contributes 1,933 explanations for 500 re-annotated MNLI items, plus 7,732 validity judgments.
  • A second annotator independently categorized 100 explanations per dataset. Inter-annotator agreement was Cohen's κ = 0.828 on LiveNLI and 0.792 on VariErr, comparable to the earlier e-SNLI result.
  • Annotator profiles cover four people who each explained more than 100 shared items. Category overlap was measured with Jaccard and text similarity with sentence-embedding cosine. The annotations are released at github.com/mainlp/LiTEx-NLI-extension.

Interesting ideas

  • I1. Measure agreement at several layers at once: final decision, type of justification, and similarity of the justification text. This separates real interpretive conflicts from conflicts that exist only at the output.
  • I2. Sort each justification by what it rests on: evidence in the text itself, or outside knowledge. Ties between such categories and disagreement rates show which kind of reasoning is fragile.
  • I3. "Absence of Mention" gets its own category, and the same rationale can yield either neutral or contradiction. This shows that some disagreements come from an implicit open- versus closed-world assumption.
  • I4. Profile each annotator across many items to expose stable preferences in reasoning and labeling, rather than treating every rater as interchangeable noise.
  • I5. Use reasoning categories to stratify evaluation sets, for a more diagnostic picture than one aggregate score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When witness parsers or LLM teacher tiers disagree on silver data, also compare the reasons behind each output. Items where witnesses differ only at the surface could be merged instead of discarded.
  • I2 (answer): Each answer from the readers could carry a tag saying whether it came from the text world or from compiled knowledge. Test results can then be broken down by that tag to locate failures.
  • I3 (world): Make the open- or closed-world stance an explicit, explainable switch in the world of the text. A fact that is simply not mentioned would then be handled predictably.
  • I4 (learn): Build similar profiles for each witness parser and teacher tier, such as habitual attachment or role choices. Agreement can then be weighted per source, and systematic biases can feed the self-induced repair rules.
  • I5 (answer): Split the fable QA and school-test benchmarks by the inference type each question needs, such as coreference, world knowledge or logic conflict. This shows which pipeline step is responsible for errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UCLouvainLouvain-la-Neuve · Belgium20 works · 3,741 citations🎓Munich Center for Machine LearningMunich · Germany62 works · 1,167 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth and Barbara Plank. 2026. Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1342

@inproceedings{hong2026agree,
  title = {{Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations}},
  author = {Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine {de Marneffe}, Benjamin Roth and Barbara Plank},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1342},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1342}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, Intensionality and Text Understanding. In Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning, pages 38–45. find
  2. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges Workshop, pages 177–190. find
  3. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did It Happen? The Pragmatic Complexity of Veridicality Assessment. Computational Linguistics, 38(2):301–333.
  4. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14349– 14371, Singapore. Association for Computational Linguistics. find
  5. Reto Gubelmann, Aikaterini-lida Kalouli, Christina Niklaus, and Siegfried Handschuh. 2023. When Truth Matters - Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs). In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 24–39, Toronto, Canada. Association for Computational Linguistics.
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  7. Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2025. LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34065–34085, Suzhou, China. Association for Computational Linguistics.
  8. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. ↗
  9. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating Reasons for Disagreement in Natural Language Inference. Transactions of the Association for Computational Linguistics, 10:1357– 1374. ↗
  10. Beiduo Chen, Yang Janet Liu, Anna Korhonen, and Barbara Plank. 2025a. Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33111–33135, Suzhou, China. Association for Computational Linguistics. find
  11. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. Ecologically Valid Explanations for Label Variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622–10633, Singapore. Association for Computational Linguistics.
  12. Beiduo Chen, Siyao Peng, Anna Korhonen, and Barbara Plank. 2025b. A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10777–10802, Aikaterini-Lida Kalouli, Hai Hu, Alexander F. Webb, Lawrence S. Moss, and Valeria de Paiva. 2023. Curing the SICK and Other NLI Maladies. Computational Linguistics, 49(1):199–243.
  13. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020b. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131– 9143, Online. Association for Computational Linguistics.
  14. Animesh Nighojkar, Antonio Laverghetta Jr., and John Licato. 2023. No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 199– 210, Toronto, Canada. Association for Computational Linguistics. find
  15. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  16. Barbara Plank. 2022. The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. ↗
  18. Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang, and Xiang Ren. 2024. Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10361–10386, Bangkok, Thailand. Association for Computational Linguistics. find
  19. Chenhao Tan. 2022. On the Diversity and Limits of Human Explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173–2188, Seattle, United States. Association for Computational Linguistics. ↗
  20. Alexandra Uma, Dina Almanea, and Massimo Poesio. 2022. Scaling and Disagreements: Bias, Noise, and Ambiguity. Frontiers in Artificial Intelligence, 5. ↗
  21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hong-2026-agree-disagree-explain-decomposing-human/