Elamur Research⌕
☰
home / papers / 2019 / Evaluating BERT for natural language inference: A case study on the CommitmentBank

Evaluating BERT for natural language inference: A case study on the CommitmentBank

Paper · 2019 · Nanjiang Jiang and Marie-Catherine de Marneffe

Summary

NLI benchmarks such as SNLI and MultiNLI were built by having annotators write hypotheses for given premises. This leaves artifacts: models can often guess the label from the hypothesis alone. Jiang and de Marneffe avoid this by converting the CommitmentBank (CB) into an NLI dataset. CB contains naturally occurring English items from Wall Street Journal, BNC fiction and Switchboard. In each item, a clause-embedding verb appears under negation, a modal, a question or a conditional. The premise is the context plus the target sentence, and the hypothesis is simply the embedded clause. Graded speaker-commitment ratings are mapped to entailment, neutral or contradiction, and only items with high annotator agreement are kept. PMI analysis finds no discriminating cues in the hypotheses.

The authors fine-tune BERT (bert large cased, via jiant) on CB, on MultiNLI, and on both, and compare it with CBOW and a five-rule linguistic heuristic. The best model reaches about 85 F1. Its gains come mostly on items the rules already solve, and probing shows that NLI training does not encode factivity or environment type. CB therefore remains a useful test of pragmatic inference.

Key points

  • The recast CB has 250 train, 56 dev and 250 test items. An item gets a label only if at least 80% of its (at least 8) ratings fall in one sub-range of [−3, 3]. The neutral class is tiny, with 16 train items. The dataset is part of SuperGLUE.
  • Hypothesis lengths are similar across classes (8.5, 6.6 and 7.3 tokens on average). Negation words do not signal contradiction, unlike in SNLI and MultiNLI.
  • The cues that do appear in premises match linguistic generalizations:
    • modals point to entailment;
    • factive verbs (know, notice) point to entailment;
    • "Do you think" questions point to neutral;
    • neg-raising "I don't think" and Switchboard fillers point to contradiction.
  • Results on CB:
    • MNLI+CB BERT scores 91.2 accuracy and 85.3 F1, against humans at 98.9 and 95.8.
    • The Heuristics baseline outperforms BERT trained only on MultiNLI.
    • Fine-tuning on CB hurts MultiNLI performance.
  • Comparison with the rules:
    • About 80% of the CB-tuned models' predictions coincide with Heuristics; the MultiNLI-only model agrees on 69.6%.
    • On the 47 test items where the rules fail, the models show no significant differences from each other.
    • Probes on the frozen NLI model output only negation and nonfactive.

Interesting ideas

  • I1. Build an inference test by taking the hypothesis directly from a sub-structure of the premise (the embedded clause), so no annotator-written hypothesis can leak label cues.
  • I2. Collect graded commitment scores and convert them to discrete labels only under strong agreement, discarding contested items.
  • I3. Encode a small rule set (modal → entailment, neg-raising → contradiction, factive → entailment, negation → contradiction, otherwise neutral) as a transparent baseline that rivals a large model.
  • I4. Split the test set by whether a simple rule baseline gets an item right, revealing whether a model only learns the shallow pattern.
  • I5. Probe a frozen task model versus one fine-tuned per feature, to tell "can learn the feature" apart from "actually uses it".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): We could auto-generate reading questions from complements of clause-embedding verbs in fables and tales, then test whether readers judge their truth correctly.
  • I2 (learn): Silver data from witness parsers or LLM teachers could be filtered with a similar consensus threshold before training.
  • I3 (frames): The prag crate could implement exactly these commitment rules over the UD tree, giving explainable speaker-commitment judgments for embedded clauses.
  • I4 (learn): Evaluation could report scores separately on rule-solvable and rule-hard items as a negative-control-style check.
  • I5 (tree): For gparse, probing frozen encoder layers for factivity or environment type would show whether the parser representation carries the cues downstream frames need.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The Ohio State UniversityColumbus · United States57 works · 17,793 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 510 worksEvaluation & benchmarks · 685 works

Cite

Cite as Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Evaluating BERT for natural language inference: A case study on the CommitmentBank. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1630

@inproceedings{jiang2019evaluating,
  title = {{Evaluating BERT for natural language inference: A case study on the CommitmentBank}},
  author = {Nanjiang Jiang and Marie-Catherine {de Marneffe}},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1630},
  url = {https://doi.org/10.18653/v1/d19-1630}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). ↗
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  3. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. find
  4. Valentine Hacquard and Jeffrey Lidz. 2019. Children’s attitude problems: Bootstrapping verb meaning from syntax and pragmatics. Mind & Language, 34(1):73–96. ↗
  5. Lauri Karttunen. 1971. Some observations on factivity. Papers in Linguistics, 4:55–69. ↗
  6. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations. ↗
  7. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2171–2179, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  9. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy. Association for Computational Linguistics. find
  10. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. ↗
  11. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. ↗
  12. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems.
  13. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F’evry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia, Nicolas Patry, Ellie Pavlick, and Samuel R. Bowman. 2019b. jiant 1.1: A software toolkit for research on general-purpose text understanding models. http://jiant.info/. ↗
  14. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  15. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗
  16. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2019-evaluating-bert-natural-language-inference/