Elamur Research⌕
☰
home / papers / 2019 / Exploiting Open IE for Deriving Multiple Premises Entailment Corpus

Exploiting Open IE for Deriving Multiple Premises Entailment Corpus

Paper · 2019 · Martin Víta and Jakub Klímek

Summary

Inference from several premises (MPE) asks whether a hypothesis follows from a set of sentences rather than from a single one. The task has very little training data: only one annotated corpus exists, built from Flickr30K captions, with 8000/1000/1000 items and exactly four premises each. The authors propose a way to derive MPE data automatically from an existing single-premise NLI corpus, with no new human annotation. An Open IE system splits each premise into textual tuples. Each tuple is flattened into a short pseudo-sentence, and the original hypothesis and label are kept. Filtering conditions decide whether a split is acceptable. Running the method on SNLI produces MPE(SNLI), a multi-premise dataset in which the count of premises differs from one item to the next. The authors describe its label distribution and its lexical biases. A hypothesis-only BERT classifier shows that the corpus carries roughly the same level of annotation artifacts as SNLI itself. The approach matters because it can turn any NLI corpus into multi-premise training data, in any language that has an Open IE tool.

Key points

  • An SNLI pair is kept only if all four filters pass:
    • the premise yields more than one tuple;
    • the tuples together cover every word type of the premise, ignoring "and";
    • no single tuple contains all the hypothesis words;
    • no tuple's word set is contained in another's.
  • MPE(SNLI) has 37,765 items with 2 premises, 5,013 with 3, 2,294 with 4, 253 with 5 and 297 with 6 or more.
  • The test labels are 23.1% entailment, 37.2% neutral and 39.6% contradiction.
  • Lexical bias was measured as p(label | word), counting only words seen at least 5 times in training.
  • A hypothesis-only MLP on BERT-base sentence embeddings (one hidden layer, d=100, chosen by grid search) reached 0.671 test accuracy against a 0.396 majority baseline. This is similar to what a premise-blind fastText achieves on SNLI.

Interesting ideas

  • I1. The original gold label is reused after the premise is rewritten into a meaning-equivalent form, so new task data costs no annotation, provided a coverage check guarantees no content was lost.
  • I2. Lossless splitting is checked with set conditions on word types: the pieces must jointly cover the sentence, and no piece may subsume another.
  • I3. Items are discarded when the hypothesis is fully contained in one fragment, so entailment cannot be trivially read off a single premise.
  • I4. A hypothesis-only classifier serves as a negative control that measures annotation artifacts in a dataset.
  • I5. Background knowledge can be supplied as extra premises, which links multi-premise inference to knowledge-aided NLI.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Gold NLI or QA labels could be carried over to sentences decomposed into frames, giving free supervision for multi-fact reasoning.
  • I2 (frames): The same test could flag frame extractions that drop arguments or produce redundant, nested frames from one clause.
  • I3 (answer): School-test or fable QA items could be filtered or graded by whether the answer requires combining several world facts rather than one.
  • I4 (learn): Answer-only or question-only baselines should be run on Mova's test sets to confirm that scores come from understanding the text.
  • I5 (world): Relevant facts from the knowledge graph could be injected as explicit, traceable premises when the world of a text is too thin to answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia108 works · 20,329 citations🎓Masaryk UniversityBrno · Czechia7 works · 432 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Information extraction · 109 works

Cite

Cite as Martin Víta and Jakub Klímek. 2019. Exploiting Open IE for Deriving Multiple Premises Entailment Corpus. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.26615/978-954-452-056-4_144

@inproceedings{vita2019exploiting,
  title = {{Exploiting Open IE for Deriving Multiple Premises Entailment Corpus}},
  author = {Martin Víta and Jakub Klímek},
  booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)},
  year = {2019},
  doi = {10.26615/978-954-452-056-4\_144},
  url = {https://doi.org/10.26615/978-954-452-056-4_144}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha and Dina Demner-Fushman. 2016. Recognizing question entailment for medical question answering. In AMIA Annual Symposium Proceedings. American Medical Informatics Association, volume 2016, page 310. find
  2. Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In IJ-CAI. volume 7, pages 2670–2676. our card
  3. Luisa Bentivogli, Ido Dagan, and Bernardo Magnini. 2017. The recognizing textual entailment challenges: Datasets and methodologies. In Handbook of Linguistic Annotation, Springer, pages 1119– 1147. ↗
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. ↗
  5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 . ↗ our card
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop. Springer, pages 177–190. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . ↗ our card
  8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324 . ↗
  9. Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. AAAI Press, pages 4074–4077. find
  10. Rodney D Nielsen, Wayne Ward, and James H Martin. 2009. Recognizing entailment in intelligent tutoring systems. Natural Language Engineering 15(4):479–501. ↗
  11. Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2018. A survey on open information extraction. arXiv preprint arXiv:1806.05599 . ↗
  12. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision. pages 2641–2649. ↗
  13. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042 . ↗
  14. Shan Jiang, Bohan Li, Chunhua Liu, and Dong Yu. 2018. Knowledge augmented inference network for natural language inference. In China Conference on Knowledge Graph and Semantic Computing. Springer, pages 129–135. find
  15. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pages 885– 895. find
  16. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. pages 427–431. find
  17. Martin Víta. 2018. From building corpora for recognizing faceted entailment to recognizing relational entailment. In Position Papers of the 2018 Federated Conference on Computer Science and Information Systems, FedCSIS 2018, Poznań, Poland, September 9-12, 2018.. pages 33–38. https://doi.org/10.15439/2018F381. ↗
  18. Daniel Z Korman, Eric Mack, Jacob Jett, and Allen H Renear. 2018. Defining textual entailment. Journal of the Association for Information Science and Technology 69(6):763–772. ↗
  19. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. arXiv preprint arXiv:1710.02925 . ↗
  20. Omer Levy and Ido Dagan. 2016. Annotating relation inference in context via question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). volume 2, pages 249–255. find
  21. Omer Levy, Torsten Zesch, Ido Dagan, and Iryna Gurevych. 2013. Recognizing partial textual entailment. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). volume 2, pages 451–455. find
  22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, pages 1112–1122. http://aclweb.org/anthology/N18-1101. ↗
  23. Han Xiao. 2018. bert-as-service. https://github.com/hanxiao/bert-as-service. ↗
  24. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2:67–78. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vita-2019-exploiting-open-ie-deriving-multiple/