Elamur Research⌕
☰
home / papers / 2014 / Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification

Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification

Summary

The paper addresses target-dependent sentiment classification on Twitter: given a tweet and one entity mentioned in it, the task is to decide whether the author's attitude toward that entity is positive, negative or neutral. One tweet can hold different opinions about different targets. The authors restructure each tweet's dependency parse so that the chosen target becomes the root. They then apply AdaRNN, a recursive neural network that passes sentiment from surrounding words up to the target node. Instead of a single composition matrix, AdaRNN has several. At each merge it computes a soft mixture over them, conditioned on the child vectors and the dependency relation type. The authors also release a manually labelled corpus of about 6,900 tweets. AdaRNN with dependency-type features reaches 65.8% accuracy and 65.5 macro-F1. This beats SVM baselines, including a re-implementation of the rule-based method of Jiang et al. (2011), and a plain RNN. The work shows that learned, syntax-aware composition can replace fragile hand-written rules on noisy social-media text.

Key points

  • Dataset:
    • Tweets were collected via Twitter API keyword queries such as "bill gates", "xbox" and "google".
    • The split is 6,248 training and 692 test tweets.
    • Classes were sampled to 25% negative, 50% neutral and 25% positive.
    • Two annotators agreed on about 82.5% of a sampled subset.
  • Pipeline:
    • Tokenization uses the Gimpel et al. Twitter tokenizer.
    • Dependency parses come from the Stanford Parser.
    • SVM baselines use Liblinear.
    • Training minimizes cross-entropy with L2 regularization using AdaGrad.
  • Model settings:
    • Word vectors have 25 dimensions and are updated during training.
    • The nonlinearity is tanh.
    • AdaRNN uses 10 composition matrices.
  • Results (accuracy / macro-F1):
    • SVM-indep: 62.7 / 60.2
    • SVM-dep: 63.4 / 63.3
    • RNN: 63.0 / 62.8
    • AdaRNN-w/oE: 64.9 / 64.4
    • AdaRNN-w/E: 65.8 / 65.5
    • AdaRNN-comb (AdaRNN root vector plus uni/bi-gram features in an SVM): 66.3 / 65.9
  • Error analysis: The hand-crafted target rules of Jiang et al. rarely fired on tweets because POS tags and parses were too noisy. This explains why SVM-dep gained little over SVM-indep.

Interesting ideas

  • I1. Re-root the parse tree at the query entity. Rebuilding the tree so the target sits at the top gives a separate composition path for each entity in the same sentence. This trick could serve any entity- or aspect-level task, such as opinions about specific products or places in Ukrainian texts.
  • I2. Mixture of composition functions instead of one. The model learns a gated, weighted blend of several transformation matrices at each node. This is an early mixture-of-experts style design, and it lets constructions like negation or comparison get their own behaviour without anyone writing rules.
  • I3. Feed syntactic labels only into the gate. Dependency relation types enter as a one-hot vector that decides which composition function to use, rather than being concatenated to word vectors. This is a lightweight way to inject linguistic annotation (e.g., UD relations) into a neural model, and it stays robust when the parser makes mistakes.
  • I4. Temperature β on the gating softmax. A single β moves selection smoothly from uniform (β = 0) through standard softmax to hard argmax as β grows large. The best results came at β = 2, while hard selection made training unstable. Sweeping β is a cheap diagnostic for whether soft routing actually helps.
  • I5. Balanced, target-annotated evaluation set with ablations. The paper compares a bag-of-words SVM restricted to words connected to the target against a plain RNN on the same tree. This isolates how much of the gain comes from structure versus adaptive composition, an evaluation design worth copying when testing new architectures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

137 papers of our library cite it (of the 140884 whose bibliographies we read); 19 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2009

@inproceedings{dong2014adaptive,
  title = {{Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification}},
  author = {Dong, Li and Wei, Furu and Tan, Chuanqi and Tang, Duyu and Zhou, Ming and Xu, Ke},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2009},
  url = {https://doi.org/10.3115/v1/p14-2009}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155, March. find
  2. Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2014. Adaptive multi-compositionality for recursive neural models with applications to sentiment analysis. In Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI). AAAI. find
  3. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121–2159, July. find
  4. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. J. Mach. Learn. Res., 9:1871–1874, June. find
  5. H.O. Georgii. 2011. Gibbs Measures and Phase Transitions. De Gruyter studies in mathematics. De Gruyter. find
  6. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Assoc… find our card
  7. Geoffrey E. Hinton. 1986. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pages 1–12. Hillsdale, NJ: Erlbaum. find
  8. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. find
  9. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 151–160, Stroudsburg, PA, USA. Assoc… find
  10. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 423– 430, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  11. Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using crfs with hidden variables. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 786–794. Associ… find
  12. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. find our card
  13. D.E. Rumelhart, G.E. Hinton, and R.J. Williams. 1986. Learning representations by back-propagating errors. Nature, 323(6088):533–536. find
  14. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. find
  15. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In EMNLP-CoNLL, pages 1201–1211. find our card
  16. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In EMNLP, pages 1631–1642. find
  17. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Comput. Linguist., 37(2):267–307, June. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/