Elamur Research⌕
☰
home / papers / 2005 / Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns

Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns

Paper · 2005 · Yejin Choi, Claire Cardie, Ellen Riloff et al.

Summary

The paper addresses automatically finding the source of an opinion: the person or entity holding or expressing a sentiment, belief or emotion in text. This includes both first-hand (direct) holders and second-hand (indirect) ones whose view is reported through someone else. The authors treat this as an information extraction problem and build a hybrid system with two parts:

  • A linear-chain Conditional Random Field that labels tokens as source start, source continuation, or outside. It uses capitalization, part-of-speech, opinion-lexicon, dependency-parse and semantic-class features.
  • AutoSlog-SE, a new fully automatic version of AutoSlog. It learns extraction patterns whose matches become extra CRF features.

Automatic induction of feature conjunctions is applied on top. Evaluated on the NRRC corpus, which carries MPQA-style annotations of direct and indirect sources, the combination beats either component alone. It reaches 81.2% precision and 60.6% recall under an overlap measure, and 79.3% precision and 59.5% recall under head-noun matching. This matters for opinion-oriented question answering and summarization, which must tell apart whose opinion is whose.

Key points

  • Tagging scheme: three labels, 'S' (first token of a source), 'T' (continuation) and '-' (outside), equivalent to IOB chunk tagging.

  • Feature sources: GATE supplies coarse POS tags in a ±2 window, the Collins (1999) dependency parser provides structural features, and the Sundance shallow parser yields 7 binary semantic-class features (authority, government, human, media, organization/company, proper name, other).

  • Opinion lexicon: it starts from roughly 500 words from Wiebe et al. (2002). It is extended with content words from the training data that appear inside opinion annotations at least 50% of the time.

  • Final results (basic features, IE-pattern features and feature induction combined):

    MeasureF-measurePrecisionRecall
    Overlap (OL)69.481.2%60.6%
    Head-noun matching (HM)68.079.3%59.5%
    Exact match (EM)62.072.7%54.1%

  • Error analysis: the main causes were sentence-splitting errors in GATE that made the parser fail, complex syntax, missed negation, and idiomatic opinion phrases absent from the lexicon.

Interesting ideas

  • I1. Patterns as features. Instead of running a pattern learner and a statistical tagger separately, the learned extraction patterns are fed into the CRF as features. This lets a symbolic, interpretable component and a probabilistic one be combined in other extraction tasks without hand-tuning how they interact.
  • I2. Phrase-level lexicon propagation. An opinion word marks its whole syntactic chunk as opinion-bearing, so a single-word lexicon can cover multi-word expressions. This is a cheap way to stretch any word-level lexicon to phrases.
  • I3. Data-driven lexicon expansion. Content words that mostly occur inside annotated spans in the training data are added to the lexicon. The same heuristic could grow domain lexicons wherever span annotations exist.
  • I4. Explicit conjunction features. Log-linear models cannot combine features on their own, so automatic induction of conjunctive features (e.g., human entity plus a parent chunk containing an opinion) gave consistent gains. This is worth trying for any CRF or logistic model with many sparse cues.
  • I5. Multiple evaluation measures. Reporting overlap, head-noun and exact-match scores shows how much error comes from span boundaries versus wrong entities. This is useful for any span-extraction evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🎓University of UtahSalt Lake City · United States29 works · 5,781 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Yejin Choi, Claire Cardie, Ellen Riloff and Siddharth Patwardhan. 2005. Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns. Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{choi2005identifying,
  title = {{Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns}},
  author = {Yejin Choi, Claire Cardie, Ellen Riloff and Siddharth Patwardhan},
  booktitle = {Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  doi = {10.3115/1220575.1220620},
  url = {https://doi.org/10.3115/1220575.1220620}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Baker, C. Fillmore & J. Lowe. 1998. The Berkeley FrameNet Project. In Proceedings of the COLING-ACL. our card
  2. S. Bethard, H. Yu, A. Thornton, V. Hativassiloglou & D. Jurafsky. 2004. Automatic extraction of opinion propositions and their holders. In Proceedings of AAAI Spring Symposium on Exploring Attitude and Affect in Text. find
  3. D. Bikel, S. Miller, R. Schwartz & R. Weischedel. 1997. Nymble: A High-Performance Learning Name-Finder. In Proceedings of the Fifth Conference on Applied Natural Language Processing. our card
  4. E. Breck & C. Cardie. 2004. Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions. In Proceedings of 20th International Conference on Computational Linguistics. find
  5. C. Cardie, J. Wiebe, T. Wilson & D. Litman. 2004. Low-level annotations and summary representations of opinions for multiperspective QA. In New Directions in Question Answering. AAAI Press/MIT Press. find
  6. M. Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find
  7. H. Cunningham, D. Maynard, K. Bontcheva & V. Tablan. 2002. GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics. find
  8. S. Das & M. Chen. 2001. Yahoo for amazon: Extracting market sentiment from stock message boards. In Proceedings of the 8th Asia Pacific Finance Association Annual Conference. find
  9. K. Dave, S. Lawrence & D. Pennock. 2003. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In International World Wide Web Conference. find
  10. J. Lafferty, A. K. McCallum & F. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of 18th International Conference on Machine Learning. find
  11. B. Levin. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press. find
  12. A. K. McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu. ↗
  13. A. K. McCallum. 2003. Efficiently Inducing Features of Conditional Random Fields. In Conference on Uncertainty in Artificial Intelligence. find
  14. A. K. McCallum & W. Li. 2003. Early Results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons. In Conference on Natural Language Learning. our card
  15. S. Morinaga, K. Yamanishi, K. Tateishi & T. Fukushima 2002. Mining Product Reputations on the Web. In Proceedings of the 8th Internatinal Conference on Knowledge Discover and Data Mining. find
  16. M. Palmer, D. Gildea & P. Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. In Computational Linguistics 31. our card
  17. B. Pang, L. Lee & S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing. our card
  18. B. Pang & L. Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. our card
  19. L. A. Ramshaw & M. P. Marcus. 1995. Nymble: A High-Performance Learning Name-Finder. In Proceedings of the 3rd Workshop on Very Large Corpora. our card
  20. E. Riloff. 1996a. An Empirical Study of Automated Dictionary Construction for Information Extraction in Three Domains. In Artificial Intelligence, Vol. 85. find
  21. E. Riloff. 1996b. Automatically Generating Extraction Patterns from Untagged Text. In Proceedings of the 13th National Conference on Artificial Intelligence. find
  22. E. Riloff & J. Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proceesings of 2003 Conference on Empirical Methods in Natural Language Processing. our card
  23. E. Riloff & W. Phillips. 2004. An Introduction to the Sundance and AutoSlog Systems Technical Report UUCS-04-015, School of Computing, University of Utah. find
  24. S. Sarawagi & W. W. Cohen. 2004. Semi-Markov Conditional Random Fields for Information Extraction 18th Annual Conference on Neural Information Processing Systems. find
  25. P. Turney. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. our card
  26. T. Wilson, J. Wiebe & R. Hwa. 2004. Just how mad are you? finding strong and weak opinion clauses. In Proceedings of the 9th National Conference on Artificial Intelligence. find
  27. T. Wilson, P. Hoffmann, S. Somasundaran, J. Kessler, J. Wiebe, Y. Choi, C. Cardie, E. Riloff & S. Patwardhan. 2005. OpinionFinder: A system for subjectivity analysis. Demonstration Description in Conference on Empirical Methods in Natural Language Processing. our card
  28. J. Yi, T. Nasukawa, R. Bunescu & W. Niblack. 2003. Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques. In Proceedings of the 3rd IEEE International Conference on Data Mining. find
  29. H. Yu & V. Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. our card
  30. J. Wiebe, E. Breck, C. Buckley, C. Cardie, P. Davis, B. Fraser, D. Litman, D. Pierce, E. Riloff & T. Wilson. 2002. NRRC Summer Workshop on Multiple-Perspective Question Answering: Final Report. find
  31. J. Wiebe & E. Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. Sixth International Conference on Intelligent Text Processing and Computational Linguistics. find
  32. J. Wiebe, T. Wilson & C. Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 1(2). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choi-2005-identifying-sources-opinions-conditional-random/