Elamur Research⌕
☰
home / papers / 2015 / Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

Summary

The paper asks how text classifiers can benefit from word order when bag-of-words models discard it and adding n-grams often fails to help, especially for topic categorization. Johnson and Zhang feed high-dimensional one-hot word vectors directly into a convolutional neural network instead of first looking up low-dimensional word vectors. The convolution layer therefore learns embeddings of short text regions directly. Efficient sparse computation on GPU makes this practical. They propose two variants. seq-CNN concatenates the one-hot vectors inside each region. bow-CNN collapses each region into a bag of words, which cuts the parameter count. A "parallel" extension combines several convolution layers so that different region embeddings complement each other. In experiments on sentiment data (e.g., Elec) and topic data (e.g., RCV1), seq-CNN performs better on sentiment and bow-CNN on topics. The better variant generally beats bag-of-n-gram baselines and more complex earlier text CNNs. The authors present this as the first successful use of word order to improve topic classification. Their code is publicly released.

Key points

  • Input is a document treated as a |D|×1 "image" with |V| channels, one one-hot vector per word. No pre-trained or separately learned word vectors are used, and the setting is purely supervised.
  • seq-CNN region vectors have p|V| dimensions, while bow-CNN regions have only |V| dimensions. With vocabularies of around 100K, bow-CNN keeps large regions affordable.
  • The authors argue that a word-vector lookup layer is just a convolution with region size one. They question this choice when bi-grams discriminate better than unigrams.
  • On Elec, an SVM with uni-, bi- and tri-gram features still relies mostly on unigrams. Its top 100 features contain only 28 bi-grams and 4 tri-grams. NB-weighting favors longer n-grams (60 tri-grams in its top 100) but does not always give the best accuracy.
  • A seq-CNN with region size 3 and 1000 neurons learns components that respond to test-set phrases never seen in training, such as "am entirely satisfied" and "best X ever".

Interesting ideas

  • I1. Embed regions, not words. Applying convolution straight to sparse one-hot input lets the model learn phrase-level features in one step. This could simplify pipelines for low-resource languages such as Ukrainian, where good pre-trained word vectors may be scarce.
  • I2. Bag-of-words inside small windows. Ignoring order only within a short region sits between full sequence models and plain bag-of-words. It is a cheap compromise for tasks like topic labeling, where exact local order matters less than which words appear together.
  • I3. Exploit sparsity rather than shrinking the vocabulary. Computing convolutions on sparse data efficiently avoids cutting the vocabulary or region size to save compute. This design lesson carries over to any model with huge categorical inputs.
  • I4. Combining several region embeddings in parallel. Layers with different region sizes or representations can run side by side and feed a single classifier. This is a simple ensemble-like trick for capturing patterns at multiple scales.
  • I5. Interpretability check through unseen n-grams. Listing test phrases that strongly activate predictive components but never occurred in training shows the model generalizes beyond memorized n-grams. This evaluation idea is useful whenever one needs to argue that a model captures compositional patterns.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

92 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1011

@inproceedings{johnson2015effective,
  title = {{Effective Use of Word Order for Text Categorization with Convolutional Neural Networks}},
  author = {Johnson, Rie and Zhang, Tong},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1011},
  url = {https://doi.org/10.3115/v1/n15-1011}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Bishop. 1995. Neural networks for pattern recognition. Oxford University Press. find
  2. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes, and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL. find our card
  3. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493– 2537. find our card
  4. Jianfeng Gao, Patric Pantel, Michael Gamon, Xiaodong He, and Li dent. 2014. Modeling interestingness with deep neural networks. In Proceedings of EMNLP. find
  5. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of ICML. find
  6. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580. find
  7. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modeling sentences. In Proceedings of ACL. find
  8. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. find our card
  9. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NIPS. find
  10. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of ICML. find our card
  11. Yann LeCun, León Bottou, Yoshua Bengio, and Patrick Haffner. 1986. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324. find
  12. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. Journal of Marchine Learning Research, 5:361–397. find our card
  13. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL. find our card
  14. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: Understanding rating dimensions with review text. In RecSys. find
  15. Grégoire Mesnil, Tomas Mikolov, Marc’Aurelio Ranzato, and Yoshua Bengio. 2014. Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews. arXiv:1412.5335v5 (4 Feb 2015 version). — [arXiv:1412.5335](https://arxiv.org/abs/1412.5335) find
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. find
  17. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2):1–135. find
  18. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 79–86. find our card
  19. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575. find our card
  20. Mehran Sahami, Susan Dumais, David Heckerman, and Eric Horvitz. 1998. A bayesian approach to filtering junk e-mail. In Proceedings of AAAI’98 Workshop on Learning for Text Categorization. find
  21. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mensnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of CIKM. find
  22. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. arXiv:1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  23. Chade-Meng Tan, Yuan-Fang Wang, and Chan-Do Lee. 2002. The use of bigrams to enhance text categorization. Information Processing and Management, 38:529–546. find
  24. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In Proceedings of ACL, pages 1555–1565. find our card
  25. Sida Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of ACL (short paper). find our card
  26. Jason Weston, Sumit Chopra, and Keith Adams. 2014. \#tagspace: Semantic embeddings from hashtags. In Proceedings of EMNLP, pages 1822–1827. find
  27. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In ASRU. find
  28. Liheng Xu, Kang Liu, Siwei Lai, and Jun Zhao. 2014. Product feature mining: Semantic clues versus syntactic constituents. In Proceedings of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/