Elamur Research⌕
☰
home / papers / 2018 / Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

Summary

Text-encoding research often relies on recurrent or convolutional compositional functions. These add many parameters and slow down training, yet their real benefit over plain word embeddings had not been measured carefully. The authors run a task-by-task comparison between LSTM and CNN encoders and Simple Word-Embedding-based Models (SWEMs), which have no compositional parameters and only pool over word vectors. Besides plain averaging, they add three variants: max-pooling (SWEM-max), a concatenation of average and max features (SWEM-concat), and hierarchical pooling (SWEM-hier), which keeps local n-gram order. The evaluation covers 17 datasets in three groups: long-document classification, sentence matching (e.g., SNLI, WikiQA) and short-sentence classification/tagging (e.g., SST, TREC). In most settings SWEMs match or beat the heavier models. They fall behind mainly on short sentiment sentences, and hierarchical pooling closes much of that gap. A subspace-training analysis further suggests that the simple models need fewer effective parameters to reach good solutions. The result is a cheap, fast and interpretable baseline that benchmarks often omit, plus practical guidance on when extra architectural complexity is worth its cost.

Key points

  • The four SWEM variants (SWEM-aver, SWEM-max, SWEM-concat, SWEM-hier) are compared with LSTM and CNN encoders on 17 datasets. The datasets include Yahoo, Yelp, AG News, SNLI, WikiQA, MR, SST-1, SST-2, Subj and TREC.
  • Cost:
    • SWEM has zero compositional parameters, while CNN has n·K·d and LSTM has 4·d·(K+d).
    • SWEM's computational complexity is O(L·K).
    • SWEM needs O(1) sequential steps, unlike the O(L) steps of an LSTM, so it parallelizes well.
  • Where SWEMs do well and poorly:
    • Pooling works especially well for long documents with hundreds of words and for sentence matching.
    • Sentiment tasks depend more on word order than topic categorization does, which is where SWEM-hier helps.
  • Linear classifier on SWEM features: replacing the MLP with a linear classifier costs very little accuracy (Yahoo! Ans. 73.53% → 73.18%; Yelp P. 93.76% → 93.66%).
  • Sogou news (Chinese in Pinyin):
    • SWEM-hier with a window of 5 reaches 96.2%, versus 91.3% for SWEM-concat.
    • This is on par with the reported CNN (95.6%) and LSTM (95.2%) results.

Interesting ideas

  • I1. Take the maximum per embedding dimension. Pooling the maximum value of each dimension lets a few salient words dominate the representation while filler words fade out. The authors also find that single dimensions then group thematically related words, which gives interpretability almost for free.
  • I2. Concatenate average and max pooling. The two pooled vectors capture complementary information, so joining them is a nearly free gain for any embedding-based classifier or retrieval encoder.
  • I3. Pool hierarchically. Pooling first inside local windows and then across the whole sequence retains some word-order information without any learned compositional weights. This is useful for order-sensitive tasks, and the Sogou result suggests it may matter more for some languages than for English.
  • I4. Use subspace training to compare models. Restricting training to a low-dimensional parameter subspace, and measuring the intrinsic dimension needed to reach an accuracy threshold, shows how much capacity a task really demands. This is a reusable way to justify picking a simpler model.
  • I5. Probe representations with a linear classifier. Swapping the non-linear head for a linear one checks whether the encoder itself produces informative features. This sanity check transfers easily to evaluating other embeddings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

70 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1041

@inproceedings{shen2018baseline,
  title = {{Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms}},
  author = {Shen, Dinghan and Wang, Guoyin and Wang, Wenlin and Min, Martin Renqiang and Su, Qinliang and Zhang, Yizhe and Li, Chunyuan and Henao, Ricardo and Carin, Lawrence},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1041},
  url = {https://doi.org/10.18653/v1/p18-1041}
}

References

38 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. ICLR. find our card
  3. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations. find
  4. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016. A simple but tough-to-beat baseline for sentence embeddings. In ICLR. find
  5. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  6. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. JMLR, 3(Feb):1137–1155. find
  7. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science, 34(8):1388–1429. find
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12(Aug):2493–2537. find our card
  9. Yixin Nie and Mohit Bansal. 2017. Shortcut-stacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312. find
  10. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. EMNLP. find our card
  11. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In EMNLP, pages 79– 86. ACL. find our card
  12. Alexis Conneau, Holger Schwenk, Loı̈c Barrault, and Yann Lecun. 2016. Very deep convolutional networks for natural language processing. arXiv preprint arXiv:1606.01781. — [arXiv:1606.01781](https://arxiv.org/abs/1606.01781) find
  13. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. EMNLP. find our card
  14. Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. 2017. Learning generic sentence representations using convolutional neural networks. In EMNLP, pages 2380–2390. find
  15. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  16. Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. IEEE. find
  17. Dinghan Shen, Martin Renqiang Min, Yitong Li, and Lawrence Carin. 2017. Adaptive convolutional filter generation for natural language understanding. arXiv preprint arXiv:1709.08294. — [arXiv:1709.08294](https://arxiv.org/abs/1709.08294) find
  18. Dinghan Shen, Yizhe Zhang, Ricardo Henao, Qinliang Su, and Lawrence Carin. 2018. Deconvolutional latent-variable model for text sequence matching. AAAI. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  20. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In EMNLP, pages 1201–1211. Association for Computational Linguistics. find our card
  21. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In NIPS, pages 2042–2050. find
  22. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In ACL, volume 1, pages 16 81–1691. find our card
  23. Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. 2011a. Parsing natural scenes and natural language with recursive neural networks. In ICML, pages 129–136. find
  24. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
  25. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In EMNLP, pages 151– 161. Association for Computational Linguistics. find our card
  26. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188. find our card
  27. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958. find
  28. Yoon Kim. 2014. Convolutional neural networks for sentence classification. EMNLP. find our card
  29. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  30. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS. find
  32. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. ICLR. find our card
  33. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. find
  34. Shiliang Zhang, Hui Jiang, Mingbin Xu, Junfeng Hou, and Lirong Dai. 2015a. The fixed-size ordinally-forgetting encoding method for neural network language models. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on… find
  35. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In NIPS, pages 649–657. find our card
  36. Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017a. Adversarial feature matching for text generation. In ICML. find
  37. Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, and Lawrence Carin. 2017b. Deconvolutional paragraph representation learning. NIPS. find
  38. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In IJCAI, pages 4069–4076. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2018-baseline-needs-more-love-simple/