Elamur Research⌕
☰
home / papers / 2019 / Understanding Roles and Entities: Datasets and Models for Natural Language Inference

Understanding Roles and Entities: Datasets and Models for Natural Language Inference

Paper · 2019 · Arindam Mitra, Ishan Shrivastava and Chitta Baral

Summary

Neural NLI models trained on SNLI often treat two sentences as equivalent when they differ only in which entity appears or which entity fills which role. They will judge that "Peter did X" follows from "John did X," or that swapping the giver and receiver changes nothing. The authors build two automatically generated stress datasets to probe this:

  • NER-CHANGED replaces names, places, numbers or dates. It draws on bAbI and AMR sentences.
  • ROLE-SWITCHED exchanges two same-type arguments of a verb. It is built from VerbNet classes whose distinct roles accept the same kind of entity, using VerbNet example sentences and QA-SRL annotations.

Both sets also include identical-sentence pairs labelled entailment. The authors then modify the word-to-word attention shared by DecAtt and ESIM. A learned gate, driven by NER-category features, blends embedding similarity with exact string match. The resulting Lambda models stay on par with the originals on SNLI and sharply improve on the entity-change test. This shows that a small symbolic signal can fix a blind spot that embeddings create.

Key points

  • Baseline failure: models trained on SNLI alone do badly on the new tests.
    • On NER-CHANGED, BERT (bert-large-uncased) scores 8.37%, ESIM 33.59% and DecAtt 59.34%.
    • On ROLE-SWITCHED, BERT scores 20.81%; DecAtt and ESIM stay around 54%.
  • Dataset sizes (train/dev/test):
    • NER-CHANGED has about 85.7k/4.4k/4.2k pairs.
    • ROLE-SWITCHED has about 10.4k/1.2k/1.2k pairs.
  • Dataset construction:
    • ROLE-SWITCHED uses 47 VerbNet classes, giving 646 candidate verbs and 89 VerbNet templates.
    • bAbI supplied 398 single-name templates, filled with 15 gender-neutral names.
    • AMR supplied 945 templates mentioning cities, countries or persons.
  • Results with SNLI plus both new sets as training data:
    • On NER-CHANGED, Lambda DecAtt reaches 97.24% and Lambda ESIM 95.88%. Plain ESIM reaches only 53.46% and BERT 80.55%.
    • On SNLI, accuracy stays roughly unchanged at about 84–88%.
  • Limits of the gate on role swaps: on ROLE-SWITCHED, the gate gives only modest gains (Lambda ESIM about 90.8% vs ESIM about 88.9%). It mainly targets entity identity rather than role order.

Interesting ideas

  • I1. Mine VerbNet for classes where two different thematic roles share the same selectional restriction (e.g. Agent and Recipient both Animate), then swap the fillers to get pairs that cannot be solved without role knowledge.
  • I2. When names differ, embeddings look alike, so attention degenerates into a near-diagonal "identical sentence" pattern; the fix is to let exact string identity count separately from vector closeness.
  • I3. The mixing weight between vector and symbolic match is conditioned on the pair of coarse NER categories (Name, Numeric, Date, Other), so string match matters strongly for names and numbers but less for ordinary words.
  • I4. Mapping QA-SRL questions onto VerbNet roles turns crowd question-answer annotations into role-labelled data without new manual labelling.
  • I5. Each stress set includes trivial entailment pairs where hypothesis equals premise, so a model cannot pass by always answering "not entailed".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): This gives an automatic negative control: a role-swapped sentence must yield a different frame, and any parser/frame step that maps both to the same who-did-what-to-whom fails the test.
  • I2 (world): Entity identity in the text world should rest on symbolic name/value equality, not similarity scores, so "Kendall" and "Peyton" never merge into one referent.
  • I3 (frames): Coreference and argument matching could apply category-dependent rules, demanding exact matches for named entities and quantities while allowing lexical relatedness for common nouns.
  • I4 (learn): QA-SRL could serve as an extra witness for VerbNet role assignment in silver data, checking whether our frames agree with question-based role answers.
  • I5 (answer): Test suites for readers should pair every perturbed question with its unperturbed twin, making sure the system's answer changes because of the edit and not because of a blanket bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Arizona State UniversityTempe · United States13 works · 1,572 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Arindam Mitra, Ishan Shrivastava and Chitta Baral. 2019. Understanding Roles and Entities: Datasets and Models for Natural Language Inference. arXiv:1904.09720.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.09720 · arXiv 1904.09720

@misc{mitra2019understanding,
  title = {{Understanding Roles and Entities: Datasets and Models for Natural Language Inference}},
  author = {Arindam Mitra, Ishan Shrivastava and Chitta Baral},
  year = {2019},
  doi = {10.48550/arxiv.1904.09720},
  eprint = {1904.09720},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.09720}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. CoRR, abs/1508.05326. ↗
  3. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. CoRR, abs/1603.06021. ↗
  4. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, and Hui Jiang. 2016. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR, abs/1609.06038. ↗
  5. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. CoRR, abs/1601.06733. ↗
  6. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, abs/1809.02922. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  8. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. CoRR, abs/1805.05377. ↗
  9. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. CoRR, abs/1805.02266. ↗
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. CoRR, abs/1803.02324. ↗
  11. Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard H. Hovy. 2018. Adventure: Adversarial training for textual entailment with knowledge-guided examples. CoRR, abs/1805.04680. ↗
  12. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI. find
  13. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. CoRR, abs/1901.11504. ↗
  14. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional LSTM model and inner-attention. CoRR, abs/1605.09090. ↗
  15. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2015. Recognizing entailment and contradiction by tree-based convolution. CoRR, abs/1512.08422. ↗
  16. Tsendsuren Munkhdalai and Hong Yu. 2016. Neural semantic encoders. CoRR, abs/1607.04315. ↗
  17. Tsendsuren Munkhdalai and Hong Yu. 2016. Neural Tree Indexers for Text Understanding. arXiv eprints, page arXiv:1607.04492. ↗
  18. Biswajit Paria, K. M. Annervaz, Ambedkar Dukkipati, Ankush Chatterjee, and Sanjay Podder. 2016. A neural architecture mimicking humans end-to-end for natural language inference. CoRR, abs/1611.04741. ↗
  19. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. CoRR, abs/1606.01933. ↗
  20. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, and Phil Blunsom. 2015. Reasoning about Entailment with Neural Attention. arXiv e-prints, page arXiv:1509.06664. ↗
  21. Karin Kipper Schuler. 2005. Verbnet: A Broad-coverage, Comprehensive Verb Lexicon. Ph.D. thesis, Philadelphia, PA, USA. AAI3179808. ↗
  22. Lei Sha, Baobao Chang, Zhifang Sui, and Sujian Li. 2016. Reading and thinking: Re-read lstm unit for textual entailment recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2870–2879, Osaka, Japan. find
  23. The COLING 2016 Organizing Committee. find
  24. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-Embeddings of Images and Language. arXiv e-prints, page arXiv:1511.06361. ↗
  25. Shuohang Wang and Jing Jiang. 2015. Learning natural language inference with LSTM. CoRR, abs/1512.08849. ↗
  26. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. ↗
  27. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. CoRR, abs/1704.05426. ↗
  28. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase Adversaries from Word Scrambling. arXiv e-prints, page arXiv:1904.01130. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mitra-2019-understanding-roles-entities-datasets-models/