Elamur Research⌕
☰
home / papers / 2014 / A Neural Network Approach to Selectional Preference Acquisition

A Neural Network Approach to Selectional Preference Acquisition

Summary

This EMNLP 2014 paper by Tim Van de Cruys addresses the automatic acquisition of selectional preferences, which is the tendency of predicates to favour semantically fitting arguments. A verb like "sing" fits a vocalist singing a ballad but not a tomato being sung. Corpus counts are sparse because of Zipf's law, so a model must generalize to predicate–argument combinations it never observed.

The author proposes an unsupervised feed-forward neural network that learns embeddings for predicates and arguments directly from unannotated text. Attested tuples serve as positive examples, and randomly corrupted tuples serve as negatives. Training uses a margin-based ranking loss adapted from Collobert and Weston (2008). There are two variants:

  • a two-way model for a verb and one argument;
  • a three-way model that scores a verb together with its subject and direct object, using a newly modified training objective.

On a pseudo-disambiguation task, both variants reach state-of-the-art results. They are compared against EM-based latent clustering, distributional similarity, and tensor factorization. This makes the paper an early demonstration that neural embeddings suit selectional preference modelling.

Key points

  • Architecture: predicate and argument vectors are concatenated, passed through one tanh hidden layer, and mapped to a single preference score.
  • Training: the score of an attested tuple must exceed that of a corrupted tuple by a margin of 1. Negatives are created by swapping in random words.
  • Baselines: Rooth et al. (1999) EM latent-variable clustering, Erk (2007) distributional similarity, and an earlier tensor factorization approach by the same author (Van de Cruys, 2009) that handles preferences involving several arguments at once.
  • Separate argument slots: the three-way model keeps distinct embedding matrices for subjects (S) and objects (O), so a word's meaning can depend on its position. For example, "virus" as a subject resembles "animal", while as an object it resembles "cell" or "device".
  • Future work: initializing the embeddings from large neural language models (Mikolov et al., 2013), sharing representations across argument slots, and trying neural tensor networks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

Cite as Van de Cruys, Tim. 2014. A Neural Network Approach to Selectional Preference Acquisition. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1004

@inproceedings{tim2014neural,
  title = {{A Neural Network Approach to Selectional Preference Acquisition}},
  author = {Van {de Cruys, Tim}},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1004},
  url = {https://doi.org/10.3115/v1/d14-1004}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. find
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. our card
  3. Shane Bergsma, Dekang Lin, and Randy Goebel. 2008. Discriminative learning of selectional preference from unlabeled text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 59–68. Association for Computational Linguistics. find
  4. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information & lexicography. Computational Linguistics, 16(1):22–29. find
  5. Stephen Clark and David Weir. 2001. Class-based probability estimation using a semantic hierarchy. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 95–102. Association for Computational Linguistics. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  7. Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723–763. find
  8. Katrin Erk. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 216–223, Prague, Czech Republic, June. Association for Computational Linguistics. find
  9. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational linguistics, 28(3):245–288. our card
  10. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Annual Meeting of the Association for Computational Linguistics (ACL). our card
  11. Daniel D. Lee and H. Sebastian Seung. 2000. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13, pages 556–562. find
  12. Hang Li and Naoki Abe. 1998. Generalizing case frames using a thesaurus and the MDL principle. Computational linguistics, 24(2):217–244. our card
  13. Tao Li and Chris Ding. 2006. The relationships among various nonnegative matrix factorization methods for clustering. In Data Mining, 2006. ICDM’06. Sixth International Conference on, pages 362–371. IEEE. find
  14. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503–528. find
  15. Diana McCarthy and John Carroll. 2003. Disambiguating nouns, verbs, and adjectives using automatically acquired selectional preferences. Computational Linguistics, 29(4):639–654. find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR 2013. our card
  17. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pages 641–648. ACM. find
  18. Joakim Nivre, Johan Hall, and Jens Nilsson. 2006. Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC-2006, pages 2216–2219. our card
  19. Diarmuid Ó Séaghdha and Anna Korhonen. 2012. Modelling selectional preferences in a lexical hierarchy. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth Intern… find
  20. Diarmuid Ó Séaghdha. 2010. Latent variable models of selectional preference. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 435–444. Association for Computational Linguistics. find
  21. Sebastian Padó, Ulrike Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 400–409, Prague, Cz… find
  22. Philip Resnik. 1996. Selectional constraints: An information-theoretic model and its computational realization. Cognition, 61:127–159, November. find
  23. Alan Ritter, Mausam, and Oren Etzioni. 2010. A latent dirichlet allocation method for selectional preferences. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 424–434, Uppsala, Sweden, July. Association for Computational Linguistics. find
  24. Mats Rooth, Stefan Riezler, Detlef Prescher, Glenn Carroll, and Franz Beil. 1999. Inducing a semantically annotated lexicon via em-based clustering. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, pages 104–111. Association for… find
  25. Ekaterina Shutova, Simone Teufel, and Anna Korhonen. 2013. Statistical metaphor processing. Computational Linguistics, 39(2):301–353. our card
  26. Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of HLT-NAACL 2003, pages 252– 259. our card
  27. Masashi Tsubaki, Kevin Duh, Masashi Shimbo, and Yuji Matsumoto. 2013. Modeling and learning semantic co-compositionality through prototype projections and neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 130–140, Seattle, Washington, U… find
  28. Tim Van de Cruys. 2009. A non-negative tensor factorization model for selectional preference induction. find
  29. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 83– 90, Athens, Greece, March. Association for Computational Linguistics. find
  30. Dong Yu, Li Deng, and Frank Seide. 2013. The deep tensor neural network with applications to large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 21(2):388–396. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cruys-2014-neural-network-approach-selectional-preference/