Elamur Research⌕
☰
home / papers / 2010 / Extracting Opinion Targets in a Single and Cross-Domain Setting with Conditional Random Fields

Extracting Opinion Targets in a Single and Cross-Domain Setting with Conditional Random Fields

Summary

The paper addresses opinion target extraction: finding the words in a sentence that an opinion is about (e.g., a product feature). The authors frame this as information extraction and solve it with Conditional Random Fields (CRF). They test it on user-generated text from four domains: movies, web-services, cars and cameras. The data is annotated with opinion targets at the sentence level. The CRF uses a small set of features:

  • the token string,
  • its POS tag,
  • whether the token is directly linked to an opinion expression in the dependency parse,
  • whether it lies in the noun phrase closest to an opinion expression.

The baseline is the supervised system of Zhuang et al. (2006). Within a single domain, the CRF beats the baseline in all four domains. The gain is much larger in a cross-domain setting, where the model is trained on some domains and tested on another. This matters because supervised opinion mining usually degrades on new domains. Domain portability had been studied for sentiment polarity, but not for target extraction.

Key points

  • Single-domain F-Measure gains over the Zhuang et al. (2006) baseline are 0.077, 0.126, 0.071 and 0.178 across the four domains.
  • Cross-domain gains are much larger: 0.409, 0.242, 0.294 and 0.343 F-Measure.
  • The feature set has short names in the result tables (tk = token, pos = POS, dLn = direct dependency to an opinion expression) plus a word-distance noun-phrase heuristic. Tagging uses the Stanford POS Tagger and parsing uses the Stanford Parser.
  • Development data was held out: 29 documents (movies), 23 (web-services) and 15 (cars & cameras).
  • The cameras dataset is the smallest, yet it is the best single training source for the other three domains. Shrinking the other training sets to the cameras size never improved F-Measure, and combining cars + cameras did not beat cameras alone.

Interesting ideas

  • I1. Encode linguistic heuristics as features: "directly attached to an opinion word" and "nearest noun phrase" become token-level features instead of hard rules. The CRF then learns how much to trust each cue, which is a cheap way to inject syntax into a sequence labeler for any span-extraction task.
  • I2. Drop lexical features to transfer: removing the token string feature kept results stable, or even helped, when the target vocabulary differs between domains. Lexically independent features (POS, dependency relations) are a natural first move for domain transfer, including transfer to new languages or text genres.
  • I3. Systematic train/test domain grid: training on every domain and domain combination, then testing on held-out domains, shows which sources actually transfer. Here more mixed data was not automatically better, which is a useful evaluation design for any multi-domain corpus.
  • I4. Check for overfitting by matching sizes: subsampling larger training sets down to the size of the smallest one separates data quantity from domain fit. This simple control is easy to reuse when comparing corpora of unequal size.
  • I5. Watch for gold-input dependence: three of the features rely on gold opinion expression annotations, and the authors flag this themselves. This is a reminder to measure pipeline error propagation before deploying such features in practice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jakob2010extracting,
  title = {{Extracting Opinion Targets in a Single and Cross-Domain Setting with Conditional Random Fields}},
  author = {Jakob, Niklas and Gurevych, Iryna},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1101/}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthony Aue and Michael Gamon. 2005. Customizing sentiment classifiers to new domains: A case study. In Proceedings of the 5th International Conference on Recent Advances in Natural Language Processing, Borovets, Bulgaria, September. find
  2. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440–447, Prague, Czech Republic, June. find our card
  3. Kenneth Bloom, Navendu Garg, and Shlomo Argamon. 2007. Extracting appraisal expressions. In Proceedings of Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, pages 308– 315, Rochester, New York, USA, April. find
  4. Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. Identifying sources of opinions with conditional random fields and extraction patterns. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 355–362, V… find
  5. James R. Cowie and Wendy G. Lehnert. 1996. Information extraction. Communications of the ACM, 39(1):80–91. find
  6. Hal Daumé III and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research (JAIR), 26:101–126. find
  7. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168–177, Seattle, Washington, USA, August. find
  8. Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264–271, Prague, Czech Republic, June. Association for Computational Linguistics. find our card
  9. Jason Kessler and Nicolas Nicolov. 2009. Targeting sentiment expressions through supervised ranking of linguistic configurations. In Proceedings of the Third International AAAI Conference on Weblogs and Social Media, pages 90–97, San Jose, California, USA, May. find
  10. Soo-Min Kim and Eduard Hovy. 2006. Extracting opinions, opinion holders, and topics expressed in online news media text. In Proceedings of the ACL Workshop on Sentiment and Subjectivity in Text, pages 1–8, Sydney, Australia, July. find our card
  11. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, pages 282–289, Williamstown, MA, USA, June. find
  12. Fuchun Peng and Andrew McCallum. 2006. Information extraction from research papers using conditional random fields. Information Processing and Management, 42(4):963–979, July. find
  13. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 339–346, Vancouver, Canada, October. find our card
  14. Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575–584, Uppsala, Sweden, July. find
  15. Jeonghee Yi, Tetsuya Nasukawa, Razvan Bunescu, and Wayne Niblack. 2003. Sentiment analyzer: Extracting sentiments about a given topic using natural language processing techniques. In Proceedings of the 3rd IEEE International Conference on Data Mining, pages 427–434, Melbourne, Florida, USA, December… find
  16. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In Proceedings of the ACM 15th Conference on Information and Knowledge Management, pages 43–50, Arlington, Virginia, USA, November. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jakob-2010-extracting-opinion-targets-single-cross/