Elamur Research
☰
home / papers / 2016 / Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1608.04207

Summary

Fixed-length sentence vectors are widely used as features, but it is hard to say what information they actually hold. Usually they are compared only through downstream tasks such as sentiment or entailment, and those scores are too coarse to explain the differences. Adi et al. propose a probing framework instead. Each vector is fed to a classifier trained on a simple diagnostic task: predicting sentence length, checking whether a given word occurs in the sentence, and deciding the relative order of two words. A property counts as encoded if a classifier can recover it. They apply this to averaged word2vec vectors (CBOW) and LSTM encoder-decoder auto-encoders at sizes from 100 to 1000, and also test skip-thought vectors. The probe data come from Wikipedia sentences. Averaged vectors turn out to carry a lot of length information and some order information. Auto-encoders are strong on word content and order. Bigger vectors do not always help, and BLEU on the reconstructed sentence does not reliably reflect encoder quality. The method works with any encoder and became a foundation for later "probing" studies.

Key points

  • Probe data: 200,000 Wikipedia sentences, split into 150k for training and 25k each for development and test. Length is predicted as one of 8 bins; always guessing the most frequent bin gives 20.1% accuracy.
  • Probe classifier: an MLP with one ReLU hidden layer the same size as the input, trained with AdaGrad and dropout 0.8. The encoders were built in Torch7 (single-layer LSTM, reversed input) and trained for about 10 days on a Tesla K40.
  • Content vs. dimensionality: adding hidden units can make an auto-encoder worse at preserving word content, and this drop does not follow the decoder's BLEU score.
  • Order patterns: LSTM auto-encoders do not depend on word-order patterns learned in training when they encode new sentences, while skip-thought encoders do.
  • Significance: results were checked with paired t-tests. Most differences are highly significant; exceptions include ED-500 vs ED-750 on word order (p=0.11) and CBOW-750 vs CBOW-1000 on word content (p=0.3).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

304 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 283 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1608.04207 · arXiv 1608.04207

@misc{adi2016fine,
  title = {{Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks}},
  author = {Adi, Yossi and Kermany, Einat and Belinkov, Yonatan and Lavi, Ofer and Goldberg, Yoav},
  year = {2016},
  doi = {10.48550/arxiv.1608.04207},
  eprint = {1608.04207},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.04207}
}

References

32 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 238–247, Baltimore,… find our card
  3. Steven Bird. NLTK: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions, pp. 69–72. Association for Computational Linguistics, 2006. find
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. find
  5. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011. find
  6. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pp. 3061–3069, 2015. find
  7. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. find
  8. Jeffrey L Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195–225, 1991. find
  9. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 315–323, 2011. find
  10. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP, 2013. find
  11. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning Distributed Representations of Sentences from Unlabelled Data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1367–1377, San Diego, Californi… find our card
  12. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, 2012. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. find
  14. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. Representation of linguistic form and function in recurrent neural networks. arXiv preprint arXiv:1602.08952, 2016. find
  15. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015. find our card
  16. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems, pp. 3276–3284, 2015. find
  18. Nicholas Léonard, Sagar Waghmare, and Yang Wang. rnn: Recurrent library for torch. arXiv preprint arXiv:1511.07889, 2015. — [arXiv:1511.07889](https://arxiv.org/abs/1511.07889) find
  19. Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In Proc. of CONLL, pp. 171–180, Baltimore, Maryland, 2014. find
  20. Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3: 211–225, 2015. ISSN 2307-387X. URL https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/570. ↗
  21. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057, 2015. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013b. find
  24. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814, 2010. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics, 2002. find our card
  26. Donald B Rubin. Matching to remove bias in observational studies. Biometrics, pp. 159–183, 1973. find
  27. Allen Schmaltz, Alexander M Rush, and Stuart M Shieber. Word ordering without syntax. arXiv preprint arXiv:1604.08633, 2016. find
  28. Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024, 2011. find
  29. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. find
  30. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop. COURSERA: Neural networks for machine learning, 2012. find
  31. Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. find our card
  32. A PPENDIX I: E XPERIMENTAL S ETUP find