Elamur Research⌕
☰
home / papers / 2022 / SciNLI: A Corpus for Natural Language Inference on Scientific Text

SciNLI: A Corpus for Natural Language Inference on Scientific Text

Paper · 2022 · Mobashir Sadat and Cornelia Caragea

Summary

Most NLI benchmarks are built from everyday language such as image captions, conversations and travel guides. They therefore say little about how well models reason over the dense, formal prose of research articles. The authors build SciNLI, an inference corpus drawn from NLP and computational linguistics papers. They observe that writers often open a sentence with a connective that signals how it relates to the previous one. These connectives serve as distant supervision: each adjacent sentence pair is labeled by its opening phrase, and the phrase is then removed. Besides ENTAILMENT and NEUTRAL, the corpus adds two classes typical of scientific writing. CONTRASTING widens contradiction to cover comparison, criticism and limitation. REASONING marks a first sentence that gives the cause or condition for a conclusion in the second. Training pairs stay noisy, while development and test pairs are checked by hand. Feature-based, neural and pre-trained transformer baselines all leave a clear gap, with XLNet reaching only about 78% Macro F1. SciNLI thus offers a harder, domain-specific benchmark for language understanding.

Key points

  • SciNLI has 107,412 sentence pairs and four labels: CONTRASTING, REASONING, ENTAILMENT, NEUTRAL. Data and code are released at github.com/msadat3/SciNLI.
  • Labels come from sentence-initial linking phrases:
    • "However" and "In contrast" signal CONTRASTING.
    • "Therefore", "Thus" and "As a result" signal REASONING.
    • "Specifically", "In particular" and "In other words" signal ENTAILMENT.
  • 6,000 pairs are manually annotated, split into 2,000 for development and 4,000 for testing.
  • Baselines include lexical/syntactic feature models, BiLSTM, CBOW, CNN, BERT, SciBERT, RoBERTa and XLNet.
    • XLNet is best, at 78.18% Macro F1 and 78.23% accuracy.
    • SciBERT consistently beats BERT (77.66 vs 75.36 F1).
  • Feeding only the second sentence sharply lowers scores (BERT 54.56 F1, SciBERT 58.16), so the labels are not guessable from hypothesis artifacts alone.

Interesting ideas

  • I1. Discourse connectives at sentence starts act as free, cheap labels for the relation to the previous sentence; deleting the cue forces a model to infer the relation from content alone.
  • I2. The label set is extended beyond the classic entailment/contradiction/neutral triad with CONTRASTING and REASONING, because those relations dominate argumentative text.
  • I3. A hypothesis-only baseline serves as a negative control, showing whether labels leak through one side of the pair.
  • I4. NEUTRAL pairs are built by sampling in several ways; pairs where one sentence comes from a related-class pair prove harder than fully random pairs. This makes it possible to grade negatives by difficulty.
  • I5. Error analysis of 100 test examples traces many failures to missing domain knowledge and to long sentences. Such manual inspection helps decide where to add knowledge rather than capacity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Connective-marked sentence pairs from English text could become silver data for training and checking inter-sentence relation detection, with the cue hidden from the learner.
  • I2 (world): Event ordering and state tracking could record explicit "because/therefore" and "in contrast" links between propositions, so answers can cite why one fact follows from another.
  • I3 (learn): Any Mova inference or QA evaluation should include a run that sees only part of the input, to show that the full pipeline really uses the premise.
  • I4 (answer): Distractors for school-test or fable QA could be drawn from semantically related sentences rather than random ones, giving tougher and more honest tests.
  • I5 (frames): Frame extraction on long, clause-heavy sentences deserves targeted diagnostics, and missing background facts could be flagged as such instead of producing a wrong frame.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States38 works · 19,844 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A Corpus for Natural Language Inference on Scientific Text. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.511

@inproceedings{sadat2022scinli,
  title = {{SciNLI: A Corpus for Natural Language Inference on Scientific Text}},
  author = {Mobashir Sadat and Cornelia Caragea},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.511},
  url = {https://doi.org/10.18653/v1/2022.acl-long.511}
}

References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beltagy, I.; Lo, K.; and Cohan, A. 2019. SciBERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3615–3620. Hong Kong, China: Association for Computational Linguistics. ↗
  2. Bird, S.; Dale, R.; Dorr, B. J.; Gibson, B.; Joseph, M. T.; Kan, M.-Y.; Lee, D.; Powley, B.; Radev, D. R.; and Tan, Y. F. 2008. The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics. In Proc. of the 6th International Conference on Language Resources and Evaluation Conference (LREC’08), 1755–1759.
  3. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 632–642. Lisbon, Portugal: Association for Computational Linguistics. ↗
  4. Conneau, A.; Kiela, D.; Schwenk, H.; Barrault, L.; and Bordes, A. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. ↗
  5. Dagan, I.; Glickman, O.; and Magnini, B. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, 177– 190. Springer. Louisiana: Association for Computational Linguistics. find
  6. Hall, D.; Jurafsky, D.; and Manning, C. D. 2008. Studying the history of ideas using topic models. In Proceedings of the conference on empirical methods in natural language processing, 363–371. Association for Computational Linguistics. find
  7. Hobbs, J. R. 1978. Why is discourse coherent. Technical report, SRI INTERNATIONAL MENLO PARK CA. find
  8. Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8): 1735– 1780. ↗
  9. Jernite, Y.; Bowman, S. R.; and Sontag, D. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557. ↗
  10. Khot, T.; Sabharwal, A.; and Clark, P. 2018. SciTaiL: A Textual Entailment Dataset from Science Question Answering. In AAAI, volume 17, 41–42. find
  11. Dagan, I.; Glickman, O.; and Magnini, B. 2006. The PASCAL Recognising Textual Entailment Challenge. In Quiñonero-Candela, J.; Dagan, I.; Magnini, B.; and d’Alché Buc, F., eds., Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, 177–190. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540-33428-6. find
  12. Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  13. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  14. Kuhn, T. S. 2012. The structure of scientific revolutions. University of Chicago press. find
  15. Do, Q.; Chan, Y. S.; and Roth, D. 2011. Minimally Supervised Event Causality Identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 294–303. Edinburgh, Scotland, UK.: Association for Computational Linguistics. find
  16. Dunietz, J.; Levin, L.; and Carbonell, J. 2017. The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations. In Proceedings of the 11th Linguistic Annotation Workshop, 95–104. Valencia, Spain: Association for Computational Linguistics. find
  17. Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 107–112. find
  18. New Orleans, Klein, D.; and Manning, C. D. 2003. Accurate Unlexicalized Parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 423–430. Sapporo, Japan: Association for Computational Linguistics.
  19. Landis, J. R.; and Koch, G. G. 1977. The measurement of observer agreement for categorical data. biometrics, 159–174. find
  20. Li, Z.; Ding, X.; Liu, T.; Hu, J. E.; and Van Durme, B. 2020. Guided Generation of Cause and Effect. In Bessiere, C., ed., Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, 3629–3636. International Joint Conferences on Artificial Intelligence Organization. Main track. find
  21. Liu, X.; He, P.; Chen, W.; and Gao, J. 2019a. Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4487–4496. Florence, Italy: Association for Computational Linguistics.
  22. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  23. Luukkonen, T. 1992. Is scientists’ publishing behaviour rewardseeking? Scientometrics, 24: 297– 319. ↗
  24. Marelli, M.; Menini, S.; Baroni, M.; Bentivogli, L.; Bernardi, R.; and Zamparelli, R. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 216–223. Reykjavik, Iceland: European Language Resources Association (ELRA).
  25. Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 1003–1011. Suntec, Singapore: Association for Computational Linguistics. find
  26. Nie, A.; Bennett, E.; and Goodman, N. 2019. Dis-Sent: Learning Sentence Representations from Explicit Discourse Relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4497–4510. Florence, Italy: Association for Computational Linguistics. find
  27. Nie, Y.; Williams, A.; Dinan, E.; Bansal, M.; Weston, J.; and Kiela, D. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4885–4901. Online: Association for Computational Linguistics. find
  28. Oh, J.-H.; Torisawa, K.; Hashimoto, C.; Sano, M.; De Saeger, S.; and Ohtake, K. 2013. Why-Question Answering using Intra- and Inter-Sentential Causal Relations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1733–1743. Sofia, Bulgaria: Association for Computational Linguistics. find
  29. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
  30. Pennington, J.; Socher, R.; and Manning, C. D. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), 1532–1543.
  31. Prasad, R.; Dinesh, N.; Lee, A.; Miltsakaki, E.; Robaldo, L.; Joshi, A.; and Webber, B. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). Marrakech, Morocco: European Language Resources Association (ELRA). ↗
  32. Pruksachatkun, Y.; Phang, J.; Liu, H.; Htut, P. M.; Zhang, X.; Pang, R. Y.; Vania, C.; Kann, K.; and Bowman, S. R. 2020. Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5231–5247. Online: Association for Computational Linguistics.
  33. Radev, D. R.; Muthukrishnan, P.; and Qazvinian, V. 2009. The ACL Anthology Network. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), 54–61. Suntec City, Singapore: Association for Computational Linguistics. find
  34. Radinsky, K.; Davidovich, S.; and Markovitch, S. 2012. Learning causality for news events prediction. In Proceedings of the 21st international conference on World Wide Web, 909–918. find
  35. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, 3266–3280.
  36. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353–355. Brussels, Belgium: Association for Computational Linguistics. ↗
  37. Webber, B.; Knott, A.; Stone, M.; and Joshi, A. 1999. Discourse relations: A structural and presuppositional account using lexicalised TAG. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, 41–48.
  38. Williams, A.; Nangia, N.; and Bowman, S. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1112–1122. Association for Computational Linguistics. find
  39. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, 5753–5763. ↗
  40. Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 93–104. Brussels, Belgium: Association for Computational Linguistics. find
  41. Zhu, Y.; Kiros, R.; Zemel, R.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sadat-2022-scinli-corpus-natural-language-inference/