Elamur Research⌕
☰
home / papers / 2016 / Probing for semantic evidence of composition by means of simple classification tasks

Probing for semantic evidence of composition by means of simple classification tasks

Summary

Sentence embeddings are usually judged by downstream task scores. Such scores can reward task-specific shortcuts instead of genuine composition. Even the SICK benchmark has cues, such as negation, that reveal most contradiction labels without composing meaning. Ettinger, Elgohary and Resnik propose a diagnostic alternative. They build controlled, carefully annotated sentence sets and test whether a simple classifier can recover a chosen semantic property from each sentence vector. They target semantic roles and negation scope. Their sentences vary voice, relative-clause position and negation placement while keeping the vocabulary nearly fixed, so lexical or word-order heuristics fail. In a proof-of-concept with pretrained GloVe, Paragram and Skip-Thought vectors, all three models encode which words occur. Only Skip-Thought lets the classifier tell whether "school" is the agent; the other two remain at chance. The authors call for fine-grained, task-independent test suites in the spirit of TSNLP. Such suites could be generated by a context-free grammar and could include implausible sentences to test composition of unseen phrases.

Key points

  • Motivation: in SICK, 86.4% of CONTRADICTION pairs can be detected from negation alone (per Bentivogli et al., 2016). Also, 98.9% of "add modifier" pairs are labeled ENTAILMENT, which makes per-phenomenon testing hard.
  • Example tasks: "professor is AGENT of recommend" and "sentence describes a professor recommending." Negative examples reuse the same words, and long-distance dependencies defeat adjacency cues.
  • Setup: logistic regression tuned with 5-fold cross-validation, 1000 training and 500 test sentences. Pretrained vectors were 300-dimensional (GloVe, Paragram) and 2400-dimensional (Skip-Thought).
  • Results: Has-school reached 100% for all models. Has-human scored 99.9 (GloVe), 90.5 (Paragram) and 99.0 (ST). School-as-agent scored 47.98, 48.57 and 91.15.
  • Proposed dataset design: grammar-based generation with automatic annotation, with implausible sentences seen as acceptable or even useful. Plausibility could be estimated with n-gram statistics or selectional preference models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 81 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2524

@inproceedings{ettinger2016probing,
  title = {{Probing for semantic evidence of composition by means of simple classification tasks}},
  author = {Ettinger, Allyson and Elgohary, Ahmed and Resnik, Philip},
  booktitle = {Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP},
  year = {2016},
  doi = {10.18653/v1/w16-2524},
  url = {https://doi.org/10.18653/v1/w16-2524}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 7th International Joint Conference on Natural Language Processing, pages 1681–1691. find
  2. Lorna Balkan, Doug Arnold, and Siety Meijer. 1994. Test suites for natural language processing. Translating and the Computer, pages 51–51. find
  3. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. find
  4. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou… find our card
  5. Alice Lai and Julia Hockenmaier. 2014. Illinois-lh: A denotational and distributional approach to semantics. Proc. SemEval. find
  6. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2015. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066. find our card
  7. Emily M Bender, Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 397– 408. find our card
  8. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Language Resources and Evaluation, pages 216– 223. find
  9. Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, and Roberto Zamparelli. 2016. SICK through the SemEval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.… find
  10. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532– 1543. find
  11. Phil Blunsom, Edward Grefenstette, and Karl Moritz Hermann. 2013. “not not bad” is not “bad”: A distributional account of negation. In Proceedings of the 2013 Workshop on Continuous Vector Space Models and their Compositionality. find
  12. Hannah Rashkin, Sameer Singh, and Yejin Choi. 2016. Connotation frames: A data-driven investigation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  13. Noam Chomsky. 1957. Syntactic structures. Mouton & Co. find
  14. Philip Resnik. 1996. Selectional constraints: An information-theoretic model and its computational realization. Cognition, 61(1):127–159. find
  15. Ahmed Elgohary and Marine Carpuat. 2016. Learning monolingual compositional representations via bilingual supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  16. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 813–821. find our card
  17. Katrin Erk. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 216–223. find
  18. Elizabeth Scarlett and Stan Szpakowicz. 2000. The power of the tsnlp: lessons from a diagnostic evaluation of a broad-coverage parser. In Advances in Artificial Intelligence, pages 138–150. Springer. find
  19. Steven M Frankland and Joshua D Greene. 2015. An architecture for encoding sentence meaning in left mid-superior temporal cortex. Proceedings of the National Academy of Sciences, 112(37):11732– 11737. find
  20. Diarmuid O Séaghdha. 2010. Latent variable models of selectional preference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 435–444. find
  21. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  22. Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015. Distributional vectors encode referential attributes. In Proceedingsof the Conference on Empirical Methods in Natural Language Processing. find
  23. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198. find our card
  24. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ettinger-2016-probing-semantic-evidence-composition-means/