Elamur Research
☰
home / papers / 2021 / DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.findings-acl.435.pdf

Summary

Most NLI resources pair one short premise sentence with one short hypothesis. Many downstream uses need more than that, such as checking whether a whole summary is faithful to its source or verifying a claim against a full article. The authors built DocNLI to support inference at document scale. They recast five existing benchmarks into a single binary entail / not-entail task:

  • ANLI, an NLI dataset;
  • SQuAD, via the QA2D conversion of question-answer pairs into statements;
  • three summarization corpora: DUC2001, CNN/DailyMail and Curation.

In every case the premise spans several sentences, usually a natural document like a news story. Hypotheses range from one sentence to passages of hundreds of words. For the summarization data, negative hypotheses are altered "fake" versions of the reference summaries. The construction was designed to minimize annotation artifacts, and a hypothesis-only baseline struggles to exploit such shortcuts.

A RoBERTa-large model trained on DocNLI transfers well even without task-specific tuning. It outperforms models trained on ANLI or MNLI on document-level tasks (FEVER-binary, MCTest) and holds up on sentence-level benchmarks. Fine-tuning improves results further. The data, code and checkpoints are publicly released.

Key points

  • Sources: DocNLI merges ANLI, QA2D/SQuAD, DUC2001, CNN/DailyMail and Curation.
    • ANLI's "neutral" and "contradict" classes are collapsed into "not entail".
    • QA2D is re-split 50k/7,236/8,275 (train/dev/test).
    • DUC2001's 600 document-summary pairs are split 400/50/150.
  • Hypothesis length: hypotheses go well beyond single sentences, with summary-based ones reaching hundreds of words. The authors present it as the first NLI dataset of this kind closely tied to end tasks.
  • Document-level transfer:
    • On MCTest, the DocNLI-trained model beats the prior state of the art by a large margin, both with and without fine-tuning.
    • On FEVER-binary, it generalizes better than models trained on MNLI or ANLI.
  • Sentence-level transfer: without fine-tuning, the model reaches 78.17 accuracy on SciTail, above fully supervised ESIM, De-Att and DGEM. After fine-tuning it comes close to the SciTail SOTA (DeBERTa).
  • Release: resources are available at github.com/salesforce/DocNLI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 56 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.435

@inproceedings{yin2021docnli,
  title = {{DocNLI: A Large-scale Dataset for Document-level Natural Language Inference}},
  author = {Yin, Wenpeng and Radev, Dragomir and Xiong, Caiming},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.435},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.435}
}

References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. ↗
  3. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. CoRR, abs/1910.12840. ↗
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632–642. find
  5. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL, pages 280– 290. find
  6. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In ACL, pages 1657– 1668. find
  7. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In AAAI, pages 6859–6866. find
  8. Curation. 2020. Curation corpus base. find
  9. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In ACL. find
  10. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches - erratum. Nat. Lang. Eng., 16(1):105. find
  11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP, pages 2249–2255. find
  12. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM@NAACL-HLT , pages 180–191. Association for Computational Linguistics. find
  13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392. find our card
  14. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203. find our card
  15. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT, pages 809–819. find our card
  16. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In LREC. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008. find
  18. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, pages 1112–1122. find
  19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. ↗
  20. Jianxing Yu, Zhengjun Zha, and Jian Yin. 2019. Inferential machine comprehension: Answering questions by recursively deducing the evidence chain from text. In Proceedings of ACL, pages 2241– 2251. find