Elamur Research⌕
☰
home / papers / 2013 / Open Domain Targeted Sentiment

Open Domain Targeted Sentiment

Summary

The paper addresses open domain targeted sentiment on microblogs. Most earlier targeted sentiment work assumes a fixed list of topics or targets. Here, the system must find any person or organization named in a tweet and decide whether positive, negative, or no sentiment is aimed at it. The key move is to treat sentiment as a label stretched over the tokens of the entity mention. This turns the task into sequence tagging, so named entity recognition and sentiment can be learned together. The authors train CRFs with minimum risk training and compare pipeline, joint, and collapsed label designs. The data is Spanish and English tweets annotated through Mechanical Turk. They deliberately avoid POS taggers and parsers, relying only on a sentiment lexicon, simple hand-written cues, and unsupervised features. In Spanish, the best models clearly beat a strong baseline and reach roughly 90% accuracy on the combined entity-plus-sentiment task. English results, from a much smaller dataset, are not significant. The work matters for low resource languages and noisy text, where supervised linguistic tools are scarce or poorly adapted.

Key points

  • Data: Starts from the tweet collection in Spanish and English released by Etter et al. (2013).
    • About 30,000 Spanish and 10,000 English tweets carry BIO named entity labels.
    • After removing retweets, 7,105 Spanish tweets contain 9,870 volitional (PERSON + ORGANIZATION) entities.
    • 2,350 English tweets contain 3,577 volitional entities.
  • Annotation: Sentiment labels come from Amazon Mechanical Turk.
    • Each tweet–entity pair was judged by three workers, and only majority-consensus items were kept.
    • Two control items were placed in every batch of six tweets to filter unreliable workers.
  • Model: CRFs trained to minimize empirical risk (Stoyanov and Eisner, 2012), in three variants: PIPE, JOINT, and COLL (collapsed).
  • Features: The only external resource is a sentiment lexicon (e.g., MPQA for English). The remaining features are:
    • Brown clusters
    • an orthography-based automatic syllabifier
    • hand-written cues such as curse words, laughs, emoticons, and negative or positive prefixes
  • Results and errors:
    • Spanish joint models reach about 90% accuracy and beat the baseline significantly; English does not.
    • For entities, the most frequent confusion was B-VOLITIONAL vs O.
    • For sentiment, positive was often predicted as neutral, and neutral as negative.
    • The tools and code were released with the paper.

Interesting ideas

  • I1. Sentiment as a span over the target. Projecting the opinion onto the entity tokens removes the need to find a separate opinion expression and link it to its target with a parser. This makes target-level opinion extraction feasible wherever a CRF or other sequence tagger can run, including on noisy Ukrainian social media.
  • I2. Comparing label architectures. The paper tests three designs: a pipeline, a joint model, and a single collapsed tag set combining entity and sentiment. The collapsed design did best on little data, while the pipeline and joint models gained more as data grew. Running this kind of comparison is a cheap way to choose a design based on how much annotation you have.
  • I3. Parser-free, unsupervised features. Brown clusters plus an orthography-driven syllabifier stand in for POS tags. The approach can be copied directly for languages or registers where existing taggers break down.
  • I4. Small, interpretable cue features. Hand-written features such as curse words, laughter tokens, emoticons, and affixes before or after a mention carried strong weights. A short list of language-specific cues like these can add real signal at low cost.
  • I5. Quality control and honest error analysis. The annotation used hidden control items and kept only majority-consensus labels. The authors also report confusion matrices and annotator disagreement, showing that the line between neutral and polar sentiment is hard even for humans. This is a useful practice for any subjective labeling project.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Part-of-speech tagging

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mitchell2013open,
  title = {{Open Domain Targeted Sentiment}},
  author = {Mitchell, Margaret and Aguilar, Jacqui and Wilson, Theresa and Durme, Benjamin Van},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1171},
  url = {https://doi.org/10.18653/v1/d13-1171}
}

References

52 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Agarwal, B. Xie, I. Vovsha, O. Rambow, and R. Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the Workshop on Language in Social Media. find our card
  2. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). find our card
  3. Luciano Barbosa and Junlan Feng. 2010. Robust sentiment detection on Twitter from biased and noisy data. In Proceedings of Coling: Posters. find our card
  4. Adam Bermingham and Alan F Smeaton. 2010. Classifying sentiment in microblogs: Is brevity an advantage? In Proceedings of CIKM-2010. find
  5. Albert Bifet and Eibe Frank. 2010. Sentiment knowledge discovery in Twitter streaming data. In Proceedings of the International Conference on Discovery Science (DS-2010). find
  6. Juliette Blevins. 1996. The syllable in phonological theory. In John A. Goldswmith, editor, The Handbook of Phonological Theory. Blackwell Publishing, Blackwell Reference Online. find
  7. N. N. Bora. 2012. Summarizing public opinions in tweets. In Proceedings of CICLing-2012. find
  8. Samuel Brody and Nicholas Diakopoulos. 2011. Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!: using word lengthening to detect sentiment in microblogs. In Proceedings of EMNLP-2011. find
  9. P. F. Brown, V. J. Della Pietra, P. V. deSouza, J.C. Lai, and R.L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  10. Pedro Henrique Calais Guerra, Adriano Veloso, Wagner Meira Jr, and Virgı́lio Almeida. 2011. From bias to opinion: a transfer-learning approach to real-time sentiment analysis. In Proceedings of the KDD-2011. find
  11. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with amazon’s mechanical turk. In Proceedings of the NAACL:HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  12. Lu Chen, Wenbo Wang, Meenakshi Nagarajan, Shaojun Wang, and Amit P. Sheth. 2012. Extracting diverse sentiment expressions with target-dependent polarity from twitter. In Proceedings of ICWSM-2012. find
  13. Yejin Choi, Eric Breck, and Claire Cardie. 2006. Joint extraction of entities and relations for opinion recognition. Proceedings of EMNLP 2006. find
  14. G. N. Clements. 1990. The role of the sonority cycle in core syllabification. In J. Kingston and M. Beckman, editors, Papers in Laboratory Phonology, pages 283– 333. CUP, Cambridge. find
  15. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using Twitter hashtags and smileys. In Proceedings of Coling: Posters. find our card
  16. Nicholas A Diakopoulos and David A Shamma. 2010. Characterizing debate performance via aggregated twitter sentiment. In Proceedings of CHI-2010. find
  17. David Etter, Francis Ferraro, Ryan Cotterell, Olivia Buzek, and Benjamin Van Durme. 2013. Nerit: Named entity recognition for informal text. Technical Report 11, Human Language Technology Center of Excellence, Johns Hopkins University, July. find
  18. Jenny Rose Finkel and Christopher D. Manning. 2010. Hierarchical joint learning: Improving joint parsing and named entity recognition with non-jointly labeled data. In Proceedings of ACL-2010. find
  19. Joan B. Hooper. 1976. The syllable in phonological theory. Language, 48(3):525–540. find
  20. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of KDD. find
  21. Xia Hu, Lei Tang, Jiliang Tang, and Huan Liu. 2013. Exploiting social relations for sentiment analysis in microblogging. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM-2013). find
  22. Niklas Jakob and Iryna Gurevych. 2010. Extracting opinion targets in a single-and cross-domain setting with conditional random fields. In Proceedings of EMNLP. find our card
  23. Long Jiang, Mo Yu, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of ACL-2011. find
  24. Wei Jin and Hung Hay Ho. 2009. A novel lexicalized hmm-based learning framework for web opinion mining. Proceedings of ICML 2009. find
  25. Soo-Min Kim and Eduard Hovy. 2006. Identifying and analyzing judgment opinions. Proceedings of NAACL 2006. find
  26. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proceedings of ACL/HLT. find our card
  27. Michal Kosinskia, David Stillwell, and Thore Graepel. 2013. Private trains and attributes are predictable from digital records of human behavior. Proc. of the National Academy of Sciences of the USA, 110(5). find
  28. Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The good the bad and the OMG! In Proceedings of ICWSM-2011. find
  29. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML-2001. find
  30. Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Ying-Ju Xia, Shu Zhang, and Hao Yu. 2010a. Structure-aware review mining and summarization. Proceedings of Coling 2010. find
  31. Guangxia Li, Steven CH Hoi, Kuiyu Chang, and Ramesh Jain. 2010b. Micro-blogging sentiment detection by collaborative online learning. In Proceedings of ICDM-2010. find
  32. Hao Li, Yu Chen, Heng Ji, Smaranda Muresan, and Dequan Zheng. 2012. Combining social cognitive theories with linguistic features for multi-genre sentiment analysis. In Proceedings of the Pacific Asia Conference on Language, Information and Computation (PACLIC-2012). find
  33. Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction, and web-enhanced lexicons. In Proceedings of CoNLL-2003. find our card
  34. Frida Morelli. 2003. The relative harmony of /s+stop/ onsets: Obstruent clusters and the sonority sequencing principle. In C. Fery and R. van de Vijver, editors, The syllable in optimality theory, pages 356–371. CUP, New York. find
  35. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of LREC-2010. find our card
  36. James W. Pennebaker, Roger J. Booth, and Martha E. Francis. 2007. Linguistic inquiry and word count: Liwc2007, operator’s manual. find
  37. Veronica Perez-Rosas, Carmen Banea, and Rada Mihalcea. 2012. Learning sentiment lexicons in spanish. Proceedings of the Conference on Language Resources and Evaluations (LREC 2012). find
  38. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of Coling:ACL-2006. find our card
  39. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In Proceedings of HLT:EMNLP-2005. find our card
  40. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1). find our card
  41. Hassan Saif, Yulan He, and Harith Alani. 2012. Alleviating data sparsity for twitter sentiment analysis. Proceedings of the WWW Workshop on Making Sense of Microposts (\# MSM2012). find
  42. Michael Speriosu, Nikita Sudan, Sid Upadhyay, and Jason Baldridge. 2011. Twitter polarity classification with label propagation over lexical links and the follower graph. In Proceedings of the EMNLP-2011 Workshop on Unsupervised Learning in NLP. find
  43. Veselin Stoyanov and Jason Eisner. 2012. Minimum-risk training of approximate crf-based nlp systems. In Proceedings of NAACL:HLT-2012. find
  44. Veselin Stoyanov, Alexander Ropson, and Jason Eisner. 2011. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AIStats. find
  45. Chenhao Tan, Lillian Lee, Jie Tang, Long Jiang, Ming Zhou, and Ping Li. 2011. User-level sentiment analysis incorporating social networks. In Proceedings of the KDD-2011. find
  46. Benjamin Van Durme. 2012. Jerboa: A toolkit for randomized and streaming algorithms. Technical report, Human Language Technology Center of Excellence, Johns Hopkins University. find
  47. Svitlana Volkova, Theresa Wilson, and David Yarowsky. 2013. Exploring sentiment in social media: Bootstrapping subjectivity clues from multilingual twitter streams. In Association for Computational Linguistics (ACL). find
  48. Xiaolong Wang, Furu Wei, Xiaohua Liu, Ming Zhou, and Ming Zhang. 2011. Topic sentiment analysis in Twitter: A graph-based hashtag sentiment classification approach. In Proceedings of CIKM-2011. find
  49. T. Wilson, J. Wiebe, and P. Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of HLT-EMNLP. find our card
  50. Theresa Wilson, Janyce Wiebe, and Paul Hoffman. 2009. Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Computational Linguistics, 35(3). find our card
  51. Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. Proceedings of ACL 2013. find
  52. Jeonghee Yi, Tetsuya Nasukawa, Razvan Bunescu, and Wayne Niblack. 2003. Sentiment analyzer: Extracting sentiments about a given topic using natural language processing techniques. In Proceedings of ICDM-2003. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mitchell-2013-open-domain-targeted-sentiment/