Elamur Research⌕
☰
home / papers / 2026 / Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference

Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference

Paper · 2026 · Artur Kulmizev, Erika Lombart, Patrick Watrin et al.

Summary

The paper asks whether LLMs used as annotators reproduce human label variation (HLV), the genuine disagreement that people show on ambiguous items, or whether they produce a false consensus. The task is English NLI. The authors give models the original human annotation instructions zero-shot and treat each model output as a separate annotator, instead of reading label probabilities from logits. They compare three ways to obtain a "crowd" of responses:

  • repeated runs of a single model, with different temperatures;
  • size variants within one model family;
  • mixtures of LLMs from different labs.

The resulting label distributions are compared with human annotations from LIVENLI, ChaosNLI and MNLI. The free-text explanations are compared as well.

Ensembles of models can produce label distributions that look human-like under aggregate metrics. However, on hard items they disagree in their own idiosyncratic ways, unlike humans. Model explanations are longer than human ones but much less varied in wording and style. The conclusion is that LLM annotators can add diversity but cannot simply stand in for people where real variation in judgment matters. All model responses are released as the PartyNLI dataset.

Key points

  • Human data. LIVENLI has 122 re-annotated MNLI items with 10 annotators each. Annotators could pick several labels and wrote an explanation for each label. Fifty of these items also have ChaosNLI labels (100 annotators) and original MNLI labels (5 annotators). For those comparisons, multi-label answers were collapsed to a single label: true or false only if chosen alone, otherwise either.
  • MODELMIX_hi contains the 10 top LMArena models as of July 2025, one per lab (e.g. claude-opus-4, gpt-4.1, gemini-2.5-pro, grok-4, deepseek-r1-671b, kimi-k2).
  • MODELMIX_lo contains 10 older or weaker counterparts released before December 2024 (e.g. claude-haiku-3.5, command-r). Open-source models here are in the 8B–20B range.
  • Other experiments. Size experiments use gemma3, deepseek-r1 and qwen3 (4, 6 and 7 sizes). Temperature experiments use grok-3-mini, qwen3, gemini-2.0-flash and kimi-k2, chosen as the most and least entropic labelers. Every model ran five trials, giving 610 annotations per model.
  • Findings. How much a model's labels vary depends heavily on which model it is. Variation grows when sampling across sizes within a family or across different models. On items with low human–model alignment, models introduce reasoning that humans do not use, such as remarks about tone, and they sometimes hallucinate.

Interesting ideas

  • I1. Treat every sampled generation as an independent annotator who receives the exact human instructions, rather than trusting logit-based probabilities, which earlier work found poorly calibrated.
  • I2. Build ensembles with one model per lab, so that members do not share pre- or post-training biases and their agreement means more.
  • I3. Do not stop at aggregate distribution similarity; check per item whether the disagreement matches the human pattern. A good average can hide idiosyncratic splits on hard items.
  • I4. In the analysed example, a negative tag question suggests the addressee's dispreference without asserting it. It actually works as a playful offer, which gpt-4.1 misread as an assertion.
  • I5. Allow the label "either" when both truth values are defensible, and require a written justification for each chosen label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could store the full spread of teacher votes per item as a soft silver label, so ambiguous sentences are flagged instead of being forced into one majority answer.
  • I2 (learn): witness parsers used to agree on silver UD trees should come from genuinely different architectures or training data; otherwise their consensus may only reflect a shared bias.
  • I3 (learn): evaluation could report, for each item, where silver annotators split versus where humans split, as a guard against misleadingly good overall scores.
  • I4 (frames): the speech-act and implicature layer should treat tag questions as questions or offers, and keep what is implied separate from what is asserted, so that the world model does not record an implicature as a fact.
  • I5 (answer): readers answering entailment-style test questions could output "undetermined" with an explicit reason, which fits the requirement that every step be explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksReasoning · 1,426 works

Cite

Cite as Artur Kulmizev, Erika Lombart, Patrick Watrin and Marie-Catherine de Marneffe. 2026. Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.752

@inproceedings{kulmizev2026label,
  title = {{Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference}},
  author = {Artur Kulmizev, Erika Lombart, Patrick Watrin and Marie-Catherine {de Marneffe}},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.752},
  url = {https://doi.org/10.18653/v1/2026.acl-long.752}
}

References

40 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, pages 1–16, New York, NY, USA. Association for Computing Machinery. find
  2. Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza-del Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. arXiv preprint. ArXiv:2509.08825 [cs]. ↗
  3. Beiduo Chen, Siyao Peng, Anna Korhonen, and Barbara Plank. 2025. A rose by any other name: LLM-generated explanations are good proxies for human explanations to collect label distributions on NLI. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10777–10802, Vienna, Austria. Association for Computational Linguistics.
  4. Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. “Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14396–14419, Miami, Florida, USA. Association for Computational Linguistics.
  5. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14349– 14371, Singapore. Association for Computational Linguistics. find
  6. Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. 1984. Studies on the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, University of Amsterdam. find
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638. find
  8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. arXiv preprint. ArXiv:1904.09751 [cs]. ↗
  9. Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2025. LiTEx: A linguistic taxonomy of explanations for understanding within-label variation in natural language inference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34065–34085, Suzhou, China. Association for Computational Linguistics.
  10. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. OCNLI: Original Chinese Natural Language Inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3512–3526, Online. Association for Computational Linguistics.
  11. Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609, Singapore. Association for Computational Linguistics. ↗
  12. Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xiangliang Zhang, Jianfeng Gao, Chaowei Xiao, and Lichao Sun. 2025. DataGen: Unified Synthetic Dataset Generation via Large Language Models. arXiv preprint. ArXiv:2406.18966 [cs]. ↗
  13. Nan-Jiang Jiang and Marie-Catherine De Marneffe. 2022. Investigating Reasons for Disagreement in Natural Language Inference. Transactions of the Association for Computational Linguistics, 10:1357– 1374. ↗
  14. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine De Marneffe. 2023a. Ecologically Valid Explanations for Label Variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622–10633, Singapore. Association for Computational Linguistics.
  15. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023b. Ecologically valid explanations for label variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622–10633, Singapore. Association for Computational Linguistics.
  16. Noah Lee, Na Min An, and James Thorne. 2023. Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569–4585, Singapore. Association for Computational Linguistics. find
  17. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  18. Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. 2025. Benchmarking distributional alignment of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 24–49, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  19. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 462–477, Red Hook, NY, USA. Curran Associates Inc. ↗
  20. Ramya Namuduri, Yating Wu, Anshun Asher Zheng, Manya Wadhwa, Greg Durrett, and Junyi Jessy Li. 2025. QUDsim: Quantifying Discourse Similarities in LLM-Generated Text. arXiv preprint. ArXiv:2504.09373 [cs]. ↗
  21. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131– 9143, Online. Association for Computational Linguistics.
  22. Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2025. StyleDistance: Stronger content-independent style embeddings with synthetic parallel examples. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8662–8685, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  23. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  24. Barbara Plank. 2022. The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  25. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942, Florence, Italy. Association for Computational Linguistics.
  26. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 930–957, Miami, Florida, USA. Association for Computational Linguistics. ↗
  27. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 Technical Report. arXiv preprint. ArXiv:2503.19786 [cs]. ↗
  28. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385– 1470. of LLM Labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–21, Honolulu HI USA. ACM. ↗
  29. Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating Annotation Error from Human Label Variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thailand. Association for Computational Linguistics.
  30. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing Human-AI Collaboration for Generating Free-Text Explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632–658, Seattle, United States. Association for Computational Linguistics. ↗
  31. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana. Association for Computational Linguistics.
  32. Yuxi Xia, Pedro Henrique Luz De Araujo, Klim Zaporojets, and Benjamin Roth. 2025. Influences on LLM calibration: A study of response agreement, loss functions, and prompt styles. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3740–3761, Vienna, Austria. Association for Computational Linguistics.
  33. Hitomi Yanaka and Koji Mineshima. 2022. Compositional evaluation on Japanese textual entailment and similarity. Transactions of the Association for Computational Linguistics, 10:1266–1284. ↗
  34. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Multilingual E5 Text Embeddings: A Technical Report. arXiv preprint. ArXiv:2402.05672 [cs]. ↗
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 Technical Report. arXiv preprint. ArXiv:2505.09388 [cs]. ↗
  36. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024b. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7407–7416, Bangkok, Thailand. Association for Computational Linguistics.
  37. Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022a. ZeroGen: Efficient Zero-shot Learning via Dataset Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653–11669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  38. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024c. find
  39. Human-LLM Collaborative Annotation Through Effective Verification Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2022b. ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3671– 3683, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  40. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large language model as attributed training data generator: A tale of diversity and bias. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, pages 55734–55784, Red Hook, NY, USA. Curran Associates Inc.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kulmizev-2026-label-explanation-variation-llm-based/