Elamur Research
☰
home / papers / 2016 / Hierarchical Attention Networks for Document Classification

Hierarchical Attention Networks for Document Classification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1174.pdf

Summary

The paper looks at document classification, where a model must assign labels such as sentiment ratings or topics to long texts. Earlier neural models usually read a document as one flat sequence. The authors propose the Hierarchical Attention Network (HAN), which follows the natural structure of a document. A bidirectional GRU first encodes the words of each sentence. A learned word-level context vector then scores each word, and the weighted words form a sentence vector. A second bidirectional GRU encodes these sentence vectors. A sentence-level attention layer combines them into one document vector, which a softmax classifier labels. Because each word is weighted in its context, the same word can matter more in one document than in another. HAN was tested on six large datasets: three Yelp review sets, IMDB, Yahoo Answers and Amazon. It beat linear models, SVMs, CNNs, LSTMs and the hierarchical Conv-GRNN and LSTM-GRNN baselines by a wide margin. The attention weights can also be visualised to show which words and sentences drove each decision.

Key points

  • Architecture: HAN has four parts: a word encoder (BiGRU), word attention, a sentence encoder (BiGRU) and sentence attention. Both attention layers score importance against a context vector that starts random and is learned during training. The authors describe this as the first hierarchical attention mechanism.
  • Data: The datasets are Yelp 2013/2014/2015 (ratings 1–5), IMDB (ratings 1–10), Yahoo Answers (10 topic classes) and Amazon (ratings 1–5; 3,000,000 training and 650,000 test reviews). Most sets were split 80/10/10 into training, validation and test data.
  • Baselines: BOW and n-gram models (with and without TFIDF), bag-of-means, SVM variants (including SSWE and AverageSG), word-level and character-level CNNs, LSTM, Conv-GRNN and LSTM-GRNN. These results come from Zhang et al. (2015) and Tang et al. (2015).
  • Interpretability: On Yelp reviews, the attention picks out strongly emotional words such as "delicious" and "terrible". It also gives little weight to sentences that sound negative on their own but are positive in context. On Yahoo Answers it highlights words that point to the topic, such as "zebra" and "camouflage".
  • Missing numbers: The provided text skipped the middle of the paper, including the results tables, so this summary has no per-dataset accuracy figures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

697 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 679 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1174

@inproceedings{yang2016hierarchical,
  title = {{Hierarchical Attention Networks for Document Classification}},
  author = {Yang, Zichao and Yang, Diyi and Dyer, Chris and He, Xiaodong and Smola, Alex and Hovy, Eduard},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1174},
  url = {https://doi.org/10.18653/v1/n16-1174}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Phil Blunsom, Edward Grefenstette, Nal Kalchbrenner, et al. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistic… find our card
  3. Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J Smola, Jing Jiang, and Chong Wang. 2014. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 193– 202. find
  4. ACM. Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong He, Li Deng, and Yelong Shen. 2014. Modeling interestingness with deep neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Karl Moritz Hermann, Tomáš Kočiskỳ, Edward Grefenstette, L… find our card
  5. Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. find our card
  6. Springer. Rie Johnson and Tong Zhang. 2014. Effective use of word order for text categorization with convolutional neural networks. arXiv preprint arXiv:1412.1058. find
  7. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  8. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, pages 723– 762. find
  9. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285. find our card
  10. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Twenty-Ninth AAAI Conference on Artificial Intelligence. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition… find
  11. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057. find
  12. Rui Lin, Shujie Liu, Muyun Yang, Mu Li, Ming Zhou, and Sheng Li. 2015. Hierarchical recurrent neural network for document modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 899–907. Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudil… find our card
  13. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 142–150. Association… find our card
  14. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  16. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. find
  17. Mehran Sahami, Susan Dumais, David Heckerman, and Eric Horvitz. 1998. A bayesian approach to filtering junk e-mail. In Learning for Text Categorization: Papers from the 1998 workshop, volume 62, pages 98– 105. find
  18. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 101–110. find
  19. ACM. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP. find
  20. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. arXiv preprint arXiv:1503.08895. find
  21. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. find
  22. ACL. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1555–1565. find
  23. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432. find
  24. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2014. Grammar as a foreign language. arXiv preprint arXiv:1412.7449. find
  25. Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 90–94. Association for Computational Linguistics. find
  26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044. find our card
  27. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2015. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274. find
  28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. arXiv preprint arXiv:1509.01626. find our card
  29. Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis Lau. 2015. A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630. — [arXiv:1511.08630](https://arxiv.org/abs/1511.08630) find