Elamur Research⌕
☰
home / papers / 2021 / Annotation Difficulties in Natural Language Inference

Annotation Difficulties in Natural Language Inference

Paper · 2021 · Aikaterini-Lida Kalouli, Livy Real, Annebeth Buis et al.

Summary

Modern models score very well on popular NLI benchmarks, yet they generalize poorly and ignore the fact that people often disagree about inference labels. The authors ran a small annotation study to see whether some premise–hypothesis pairs are hard to label no matter how the task is framed. Nine graduate students formed three teams of three, and all teams labelled the same 100 pairs taken from SICK and SNLI. One team used the SNLI guidelines, one used the improved KAL guidelines of Kalouli et al. 2019, and one had no guidelines. For each pair, annotators gave a label, a short justification and a 1–5 Difficulty Score. Pairs were split into clear-cut and controversial in two ways: by the authors' prior classification, and by each annotator's normalized difficulty ratings. Clear-cut pairs showed significantly higher agreement under both splits and every guideline condition, which suggests that some disagreement is inherent. Better guidelines (KAL) reduced it, while having no guidelines hurt agreement. The authors recommend that NLI datasets include a per-item Difficulty Score, which could serve to weight training examples, divide evaluation sets and detect dataset artifacts.

Key points

  • Data: 100 pairs, half from SICK and half from SNLI. Each half has 20 C, 20 N and 10 E pairs, split into 25 pairs the authors judged clear-cut and 25 judged controversial.
  • What counts as controversial: following Kalouli et al. 2019, pairs involving directionality, coreference or loosely defined terms.
  • Normalizing the Difficulty Score: ratings were z-scored per annotator, and a pair was marked controversial if its z-score exceeded 1.
    • This marked 20 pairs controversial under SNLI guidelines, 14 under KAL and 29 with no guidelines.
    • Mean difficulty was 2.14, 1.95 and 1.5 respectively.
  • Agreement by the authors' split (clear-cut vs. controversial): SNLI 72.1 vs. 51.9, KAL 76.7 vs. 52.5, none 50.8 vs. 38.9.
  • Agreement by the annotators' own split: SNLI 72.1 vs. 20.6, KAL 70.5 vs. 35.3, none 51.5 vs. 23.3.
    • Clear-cut pairs agree significantly more often (p < 0.05).
    • KAL beats the alternatives on controversial pairs.
    • Having no guidelines makes the task feel easy to annotators but yields poor agreement.

Interesting ideas

  • I1. Have annotators rate how hard each item felt, then z-normalize per rater so that personal scale habits cancel out; this gives a cheap, label-independent estimate of how reliable each item is.
  • I2. Keep "hard for humans to annotate" separate from "hard for machines": a passive paraphrase or a swapped-role sentence like the dog walking the man is trivial for people but trips up models.
  • I3. An item labelled identically and rated easy, whatever its premise, may signal a dataset artifact (for example, any hypothesis with "sleep" being a contradiction) rather than real inference.
  • I4. Asking annotators for a brief justification next to each label exposes where guideline gaps (coreference, vague words like "artist" or "small") cause splits.
  • I5. Clear guidelines measurably beat "annotate naturally"; the KAL instruction to treat the premise as a picture caption that fixes the truth reduced contested cases.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver trees and frames could carry a normalized "witness disagreement" score that down-weights contested items during training, instead of a plain keep/drop decision.
  • I2 (answer): build evaluation sets of human-obvious but structurally tricky pairs and report scores separately on clear-cut and contested items, so real reasoning failures are not hidden by ambiguous cases.
  • I3 (learn): treat such suspiciously uniform label–word correlations as negative controls, checking that Mova's frames → world reasoning reaches the label for the right reason.
  • I4 (learn): when using LLM teacher tiers, require a short rationale per decision and mine recurring rationale types to tighten the annotation conventions for roles and coreference.
  • I5 (world): adopt an explicit convention that the premise defines the text's world and the hypothesis is checked against it, which helps decide coreference between premise and hypothesis entities consistently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany44 works · 55,193 citations

Also on the byline: Topos Institute.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Aikaterini-Lida Kalouli, Livy Real, Annebeth Buis, Martha Palmer and Valeria Paiva. 2021. Annotation Difficulties in Natural Language Inference. Proceedings of the 13th Brazilian Symposium in Information and Human Language Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kalouli2021annotation,
  title = {{Annotation Difficulties in Natural Language Inference}},
  author = {Aikaterini-Lida Kalouli, Livy Real, Annebeth Buis, Martha Palmer and Valeria Paiva},
  booktitle = {Proceedings of the 13th Brazilian Symposium in Information and Human Language Technology},
  year = {2021},
  doi = {10.5753/stil.2021.17804},
  url = {https://doi.org/10.5753/stil.2021.17804}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  2. Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., and Blunsom, P. (2018). e-SNLI: Natural language inference with natural language explanations. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31, pages 9539–9549. Curran Associates, Inc. ↗
  3. Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S. J., and Goodman, N. D. (2018). Evaluating compositionality in sentence embeddings. CoRR, abs/1802.04302. ↗
  4. de Marneffe, M.-C., Rafferty, A. N., and Manning, C. D. (2008). Finding contradictions in text. In Proceedings of ACL-08.
  5. de Marneffe, M.-C., Simons, M., and Tonhauser, J. (2018). Factivity in doubt: Clause-embedding predicates in naturally occurring discourse (poster). Sinn und Bedeutung 23. find
  6. Fonseca, E. R., dos Santos, L. B., Criscuolo, M., and Aluı́sio, S. M. (2016). Assin: Avaliação de similaridade semântica e inferência textual. In Proceedings of PROPOR, pages 1–8. find
  7. Glockner, M., Shwartz, V., and Goldberg, Y. (2018). Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. Association for Computational Linguistics.
  8. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., and Smith, N. A. (2018). Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics. find
  9. Kalouli, A.-L., Buis, A., Real, L., Palmer, M., and de Paiva, V. (2019). Explaining simple natural language inference. In Proceedings of the 13th Linguistic Annotation Workshop, pages 132–143, Florence, Italy. Association for Computational Linguistics. find
  10. Kalouli, A.-L., Real, L., and de Paiva, V. (2017). Correcting Contradictions. In Proceedings of Computing Natural Language Inference (CONLI) Workshop, 19 September 2017. find
  11. Kumar, S. and Talukdar, P. (2020). NILE : Natural language inference with faithful natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730–8742, Online. Association for Computational Linguistics.
  12. Liu, X., He, P., Chen, W., and Gao, J. (2019). Multi-task deep neural networks for natural language understanding. CoRR, abs/1901.11504. ↗
  13. Majumder, B. P., Camburu, O., Lukasiewicz, T., and McAuley, J. J. (2021). Rationale-inspired natural language explanations with commonsense. CoRR, abs/2106.13876. ↗
  14. Manning, C. D. (2006). Local textual inference: it’s hard to circumscribe, but you know it when you see it–and nlp needs it. https://nlp.stanford.edu/manning/papers/Textual Inference.pdf. ↗
  15. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. (2014). A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC 2014.
  16. McCoy, T., Pavlick, E., and Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  17. Nie, Y., Wang, Y., and Bansal, M. (2018). Analyzing compositionality-sensitivity of NLI models. CoRR, abs/1811.07033. ↗
  18. Palomaki, J., Rhinehart, O., and Tseng, M. (2018). A case for a range of acceptable annotations. In SAD/CrowdBias@ HCOMP, pages 19–31. find
  19. Pavlick, E. and Callison-Burch, C. (2016). Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173, Berlin, Germany. Association for Computational Linguistics. find
  20. Pavlick, E. and Kwiatkowski, T. (2019). Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  21. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., and Van Durme, B. (2018). Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. find
  22. Real, L., Rodrigues, A., e Silva, A. V., Albiero, B., Guide, B., Thalenberg, B., Silva, C., Câmara, I. C. S., de Oliveira Lima, G., Souza, R., Stanojevic, M., and de Paiva, V. (2018). Sick-br: a portuguese corpus for inference. In PROPOR 2018. find
  23. Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Computing Machinery.
  24. Richardson, K., Hu, H., Moss, L. S., and Sabharwal, A. (2019). Probing natural language inference models through semantic fragments. CoRR, abs/1909.07521. ↗
  25. Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., and Zhou, X. (2019). Semantics-aware BERT for language understanding. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kalouli-2021-annotation-difficulties-natural-language-inference/