Elamur Research⌕
☰
home / papers / 2014 / Learning Grounded Meaning Representations with Autoencoders

Learning Grounded Meaning Representations with Autoencoders

Paper · 2014 · Carina Silberer and Mirella Lapata

Summary

The paper tackles grounding distributional word meaning in perception. Text-only vector spaces ignore the physical world, and earlier multimodal models often just glue together text and image vectors that have very different statistical properties. Silberer and Lapata describe each concrete noun by two automatically obtained attribute vectors, one inferred from text and one predicted from images. The two sources come from separate data, not paired documents. Each modality first goes through its own stacked denoising autoencoder. A joint layer then fuses them into a bimodal embedding. Fine-tuning is semi-supervised: the network reconstructs the attribute input and also predicts the object's class through a softmax. The embeddings are tested on word similarity, using a new set of 7,576 noun pairs rated for both semantic and visual similarity, and on concept categorization with Chinese Whispers clustering. The bimodal SAE beats SVD, kCCA, Bruni et al. and an RNN language model on both tasks. This suggests that learned higher-level attribute embeddings capture complementary textual and visual knowledge.

Key points

  • New similarity dataset: 7,576 word pairs, each with separate human ratings for semantic and visual similarity. Unlike WordSim353, this lets each modality be studied on its own.
  • Similarity results: the bimodal SAE (T+V) achieves ρ = 0.70 for semantic similarity and ρ = 0.64 for visual similarity, compared with inter-annotator agreement of 0.76 and 0.63, respectively. The SAE outperforms all other bimodal models.
  • Unimodal SAEs behave as expected:
    • The textual SAE is stronger on semantic similarity (0.65 vs 0.60 for the visual SAE).
    • The visual SAE is stronger on visual similarity (0.60 vs 0.52 for the textual SAE).
  • Categorization F-scores (T+V):
    • SAE 0.43, SVD 0.39, kCCA 0.37, Bruni 0.34.
    • Simple attribute concatenation scores 0.33; RNN-640 (text) scores 0.32.
    • The SAE comes close to the human McRae norms (0.42).
  • Models without attributes trail behind: Bruni et al. and the Mikolov RNN are outperformed by every attribute-based system on similarity.

Interesting ideas

  • I1. Use attributes as a shared interface between modalities. Expressing both image and text information as natural-language attributes puts them in a common, interpretable space before fusion. This could help merge heterogeneous sources, such as geographic or trade data, with text.
  • I2. Train modalities on decoupled data. Text and image attributes come from separate resources, linked only by the word. Fusion therefore does not require aligned multimodal corpora, which are often hard to obtain for lower-resource languages.
  • I3. Combine reconstruction with a classification signal. Adding object-class prediction to the autoencoder's reconstruction objective is an inexpensive form of weak supervision. It pushes embeddings to be task-relevant without abandoning unsupervised structure.
  • I4. Collect ratings per modality. Separate semantic and visual similarity judgments let you diagnose what each input source contributes. The same design fits any evaluation of multi-source embeddings.
  • I5. Build each modality's representation before fusing. Even the unimodal SAEs usually beat the raw attribute vectors. Learning a compressed encoding for each source first may pay off in purely textual NLP pipelines too.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

🧭 Topics 1 Semantic parsing & meaning representation · 135 works

Cite

Cite as Carina Silberer and Mirella Lapata. 2014. Learning Grounded Meaning Representations with Autoencoders. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1068

@inproceedings{silberer2014learning,
  title = {{Learning Grounded Meaning Representations with Autoencoders}},
  author = {Carina Silberer and Mirella Lapata},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1068},
  url = {https://doi.org/10.3115/v1/p14-1068}
}

References

55 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko and Aitor Soroa. 2007. SemEval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems. In Proceedings of the Fourth International Workshop on Semantic Evaluations. Prague, Czech Republic, pages 7–12. find
  2. Andrews, M., G. Vigliocco, and D. Vinson. 2009. Integrating Experiential and Distributional Data to Learn Semantic Representations. Psychological Review 116(3):463–498. find
  3. Baroni, M., B. Murphy, E. Barbu, and M. Poesio. 2010. Strudel: A Corpus-Based Semantic Model Based on Properties and Types. Cognitive Science 34(2):222–254. find
  4. Barsalou, Lawrence W. 2008. Grounded Cognition. Annual Review of Psychology 59:617–845. find
  5. Bengio, Y., P. Lamblin, D. Popovici, and H. Larochelle. 2007. Greedy Layer-Wise Training of Deep Networks. In Bernhard Schölkopf, John Platt, and Thomas Hoffman, editors, Advances in Neural Information Processing Systems 19. MIT Press, pages 153–160. find
  6. Bengio, Yoshua. 2009. Learning Deep Architectures for AI. Foundations and Trends in Machine Learning 2(1):1–127. find
  7. Biemann, Chris. 2006. Chinese Whispers – an Efficient Graph Clustering Algorithm and its Application to Natural Language Processing Problems. In Proceedings of TextGraphs: the 1st Workshop on Graph Based Methods for Natural Language Processing. New York, NY, pages 73–80. our card
  8. Blei, D. M., A. Y. Ng, and M. I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3:993–1022. our card
  9. Bruni, E., G. Boleda, M. Baroni, and N. Tran. 2012a. Distributional Semantics in Technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Jeju Island, Korea, pages 136–145. our card
  10. Bruni, E., U. Bordignon, A. Liska, J. Uijlings, and I. Sergienya. 2013. Vsem: An open library for visual semantics representation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Sofia, Bulgaria, pages 187–192. find
  11. Bruni, E., N. Tran, and M. Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res. (JAIR) 49:1–47. find
  12. Bruni, E., J. Uijlings, M. Baroni, and N. Sebe. 2012b. Distributional Semantics with Eyes: Using Image Analysis to Improve Computational Representations of Word Meaning. In Proceedings of the 20th ACM International Conference on Multimedia. Nara, Japan, pages 1219–1228. find
  13. Collobert, R., J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural Language Processing (almost) from Scratch. Journal of Machine Learning Research 12:2493–2537. our card
  14. Deng, J., W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Miami, Florida, pages 248–255. find
  15. Farhadi, A., I. Endres, D. Hoiem, and D. Forsyth. 2009. Describing Objects by their Attributes. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Miami Beach, Florida, pages 1778–1785. find
  16. Feng, Fangxiang, Ruifan Li, and Xiaojie Wang. 2013. Constructing Hierarchical Image-tags Bimodal Representations for Word Tags Alternative Choice. In Proceedings of the ICML 2013 Workshop on Challenges in Representation Learning. Atlanta, Georgia. find
  17. Feng, Yansong and Mirella Lapata. 2010. Visual Information in Semantic Representation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Los Angeles, California, pages 91–99. find
  18. Finkelstein, L., E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. 2002. Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems 20(1):116–131. find
  19. Fountain, Trevor and Mirella Lapata. 2010. Meaning Representation in Natural Language Categorization. In Proceedings of the 31st Annual Conference of the Cognitive Science Society. Amsterdam, The Netherlands, pages 1916– 1921. find
  20. Golub, Gene and Christian Reinsch. 1970. Singular Value Decomposition and Least Squares Solutions. Numerische Mathematik 14(5):403– 420. find
  21. Griffiths, T. L., M. Steyvers, and J. B. Tenenbaum. 2007. Topics in Semantic Representation. Psychological Review 114(2):211–244. find
  22. Hardoon, D. R., S. R. Szedmak, and J. R. Shawe-Taylor. 2004. Canonical Correlation Analysis: An Overview with Application to Learning Methods. Neural Computation 16(12):2639– 2664. find
  23. Harris, Zellig. 1970. Distributional Structure. In Papers in Structural and Transformational Linguistics, pages 775–794. find
  24. Hinton, Geoffrey E. and Ruslan R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313(5786):504– 507. find
  25. Hodosh, Micah, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. Journal of Artificial Intelligence Research 47:853–899. find
  26. Huang, Jing and Brian Kingsbury. 2013. Audiovisual Deep Learning for Noise Robust Speech Recognition. In Proceedings of the 38th International Conference on Acoustics, Speech, and Signal Processing. Vancouver, Canada, pages 7596–7599. find
  27. Jones, R., B. Rey, O. Madani, and W. Greiner. 2006. Generating Query Substititions. In Proceedings of the 15th International Conference on the World-Wide Web. Edinburgh, Scotland, pages 387–396. find
  28. Landau, B., L. Smith, and S. Jones. 1998. Object Perception and Object Naming in Early Development. Trends in Cognitive Science 27:19–24. find
  29. Landauer, Thomas and Susan T. Dumais. 1997. A Solution to Plato’s Problem: the Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. Psychological Review 104(2):211–240. find
  30. Lowe, D. 2004. Distinctive Image Features from Scale-invariant Keypoints. International Journal of Computer Vision 60(2):91–110. find
  31. Manning, C. D., P. Raghavan, and H. Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY. find
  32. McRae, K., G. S. Cree, M. S. Seidenberg, and C. McNorgan. 2005. Semantic Feature Production Norms for a Large Set of Living and Nonliving Things. Behavior Research Methods 37(4):547–559. find
  33. Mikolov, T., S. Kombrink, L. Burget, J. Černocký, and S. Khudanpur. 2011. Extensions of Recurrent Neural Network Language Model. In Proceedings of the 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing. Prague, Czech Republic, pages 5528– 5531. find
  34. Mikolov, T., Wen-tau Yih, and G. Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Atlanta, Georgia, pages 746–751. our card
  35. Miller, George A. and Walter G. Charles. 1991. Contextual Correlates of Semantic Similarity. Language and Cognitive Processes 6(1). find
  36. Nelson, D. L., C. L. McEvoy, and T. A. Schreiber. 1998. The University of South Florida Word Association, Rhyme, and Word Fragment Norms. find
  37. Ngiam, Jiquan, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Ng. 2011. Multimodal Deep Learning. In Proceedings of the 28th International Conference on Machine Learning. Bellevue, Washington, pages 689–696. find
  38. Patwardhan, Siddharth and Ted Pedersen. 2006. Using WordNet-based Context Vectors to Estimate the Semantic Relatedness of Concepts. In Proceedings of the EACL 2006 Workshop on Making Sense of Sense: Bringing Computational Linguistics and Psycholinguistics Together. Trento, Italy, pages 1–8. find
  39. Ranzato, Marc’Aurelio and Martin Szummer. 2008. Semi-supervised Learning of Compact Document Representations with Deep Networks. In Proceedings of the 25th International Conference on Machine Learning. Helsinki, Finland, pages 792–799. find
  40. Regier, Terry. 1996. The Human Semantic Potential. MIT Press, Cambridge, Massachusetts. find
  41. Roller, Stephen and Sabine Schulte im Walde. 2013. A Multimodal LDA Model integrating Textual, Cognitive and Visual Modalities. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, pages 1146–1157. find
  42. Sebastiani, Fabrizio. 2002. Machine Learning in Automated Text Categorization. ACM Computing Surveys 34:1–47. find
  43. Silberer, C., V. Ferrari, and M. Lapata. 2013. Models of Semantic Representation with Visual Attributes. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, pages 572–582. find
  44. Silberer, Carina and Mirella Lapata. 2012. Grounded Models of Semantic Representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Jeju Island, Korea, pages 1423–1433. find
  45. Socher, R., M. Ganjoo, C. D. Manning, and A. Y. Ng. 2013a. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems 26, pages 935–943. find
  46. Socher, R., Quoc V. Le, C. D. Manning, and A. Y. Ng. 2013b. Grounded Compositional Semantics for Finding and Describing Images with Sentences. In Proceedings of the NIPS Deep Learning Workshop. our card
  47. Socher, R., J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Edinburgh, Scotland, pages 151–161. our card
  48. Srivastava, Nitish and Ruslan Salakhutdinov. 2012. Multimodal Learning with Deep Boltzmann Machines. In Advances in Neural Information Processing Systems 25, pages 2231– 2239. find
  49. Steyvers, Mark. 2010. Combining Feature Norms and Text Data with Topic Models. Acta Psychologica 133(3):234–342. find
  50. Szumlanski, S. R., F. Gomez, and V. K. Sims. 2013. A New Set of Norms for Semantic Relatedness Measures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, pages 890– 895. find
  51. Turney, Peter D. and Patrick Pantel. 2010. From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research 37(1):141–188. our card
  52. Vincent, P., H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol. 2010. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research 11:3371–3408. von Ahn, Luis and Laura Dabbish. 2004. Labeling Images with… find
  53. Wu, Pengcheng, Steven C. H. Hoi, Hao Xia, Peilin Zhao, Dayong Wang, and Chunyan Miao. 2013. Online Multimodal Deep Similarity Learning with Application to Image Retrieval. In Proceedings of the 21st ACM International Conference on Multimedia. Barcelona, Spain, pages 153–162. find
  54. Yih, Wen-tau, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question Answering Using Enhanced Lexical Semantic Models. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, pages 1744–1753. find
  55. Yu, C. and D. H. Ballard. 2007. A Unified Model of Early Word Learning Integrating Statistical and Social Cues. Neurocomputing 70:2149– 2165. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/silberer-2014-learning-grounded-meaning-representations-autoencoders/