Elamur Research⌕
☰
home / papers / 2010 / Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

Paper · 2010 · Cigdem Toprak, Niklas Jakob and Iryna Gurevych

Summary

Most expression-level opinion annotation has targeted newswire (MPQA) and meeting dialogs. The few annotated review corpora, such as Hu & Liu (2004) and Zhuang et al. (2006), record only targets and polarities. They do not mark opinion spans, modifiers or anaphoric links, and they report no agreement figures. Toprak, Jakob and Gurevych address this with a corpus of consumer reviews from rateitall and eopinions, many of them about online universities, annotated with a two-stage scheme.

  1. Sentence level. Each sentence is checked for whether it is relevant to the review's topic and whether it carries an evaluation. An evaluation can be an explicit opinion or a "polar fact", meaning a verifiable statement that still implies judgment.
  2. Expression level. Evaluative on-topic sentences then receive:
    • opinion expression spans,
    • polarity and strength,
    • modifiers such as negation and intensifiers,
    • targets and holders, linked to their expressions and resolved through anaphora.

A two-annotator study shows that span marking and opinion polarity are fairly reliable. Strength and polar-fact polarity are harder. The resource supports fine-grained tasks like negation-aware polarity and target extraction.

Key points

  • The data comes from the rateitall and eopinions review sites. Examples cover schools such as Capella University, Devry and University of Phoenix.
  • Expression-level agreement in Table 4 ranges from 0.74 to 0.97 (raw) and 0.55 to 0.94 (κ) across subtasks. The lowest-agreement attributes are opinion strength and the polarity of polar targets.
  • Among annotated modifiers, about 70% are intensifiers and about 20% are negations.
  • Polarity differs by evaluation type:
    • 61% of agreed polar-fact targets were negative.
    • Only 35% of agreed opinion expressions were negative.
    • No instances were neutral.
  • Strength labels split into 6% weak, 54% average and 40% strong. Most strength disagreements were between weak and average. A third judge consolidates the annotations for the final corpus, which is announced for release on the UKP website.

Interesting ideas

  • I1. Filter sentences before annotating expressions. Doing a cheap sentence-level relevance and evaluation pass first narrows the costly span annotation and makes it more consistent. This cascaded design suits any expensive fine-grained labeling project.
  • I2. Annotate polar facts as a category. Objective statements that still imply praise or criticism get their own label. This captures evaluation that lexicon-based methods miss, and it applies equally to complaints, news comments or trade feedback.
  • I3. Store linked components rather than final scores. The scheme keeps opinion term, modifier, target and holder as separate units connected by links, including anaphoric ones. Errors can then be traced to a specific component, and new attributes can be added without redesigning the scheme.
  • I4. Use label distributions as findings. Comparing labels across categories exposed a pattern: reviewers tend to describe negative experiences factually but voice positive ones as explicit opinions. Such analyses can guide both model design and data sampling.
  • I5. Report agreement per subtask. Computing agreement separately for spans, polarity, strength and modifiers shows which attributes are reliable enough to train on and which need clearer guidelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany76 works · 35,008 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 394 works

Cite

Cite as Cigdem Toprak, Niklas Jakob and Iryna Gurevych. 2010. Sentence and Expression Level Annotation of Opinions in User-Generated Discourse. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{toprak2010sentence,
  title = {{Sentence and Expression Level Annotation of Opinions in User-Generated Discourse}},
  author = {Cigdem Toprak, Niklas Jakob and Iryna Gurevych},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1059/}
}

References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Asher, Farah Benamara, and Yvette Yannick Mathieu. 2008. Distilling opinion in discourse: A preliminary study. In Coling 2008: Companion volume: Posters, pages 7–10, Manchester, UK. find
  2. Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying expressions of opinion in context. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI-2007), pages 2683–2688, Hyderabad, India. find
  3. Xiwen Cheng and Feiyu Xu. 2008. Fine-grained opinion topic and polarity identification. In Proceedings of the 6th International Conference on Language Resources and Evaluation, pages 2710–2714, Mar-rekech, Morocco. find
  4. Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. Identifying sources of opinions with conditional random fields and extraction patterns. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 355–362,…
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  6. Xiaowen Ding, Bing Liu, and Philip S. Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the International Conference on Web Search and Web Data Mining, WSDM 2008, pages 231–240, Palo Alto, California, USA. find
  7. Andrea Esuli and Fabrizio Sebastiani. 2006. Senti-WordNet: A publicly available lexical resource for opinion mining. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 417–422, Genova, Italy. find
  8. Angela Fahrni and Manfred Klenner. 2008. Old wine or warm beer: Target-specific sentiment analysis of adjectives. In Proceedings of the Symposium on Affective Language in Human and Machine, AISB 2008 Convention, pages 60 – 63, Aberdeen, Scotland. find
  9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD’04: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168–177, Seattle, Washington. find
  10. Soo-Min Kim and Eduard Hovy. 2006. Extracting opinions, opinion holders, and topics expressed in online news media text. In Proceedings of the Workshop on Sentiment and Subjectivity in Text at the joint COLING-ACL Conference, pages 1–8, Sydney, Australia.
  11. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methology. Sage Publications, Thousand Oaks, California. find
  12. Rebecca J. Passonneau. 2004. Computing reliability for coreference. In Proceedings of LREC, volume 4, pages 1503–1506, Lisbon. find
  13. Randolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. find
  14. Ellen Riloff and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In EMNLP-03: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 105–112.
  15. Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2008. Discourse level opinion relations: An annotation study. In In Proceedings of SIGdial Workshop on Discourse and Dialogue, pages 129– 137, Columbus, Ohio. find
  16. Veselin Stoyanov and Claire Cardie. 2008. Topic identification for fine-grained opinion analysis. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 817–824, Manchester, UK. find
  17. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39:165–210. find
  18. Theresa Wilson and Janyce Wiebe. 2005. Annotating attributions and private states. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 53–60, Ann Arbor, Michigan. find
  19. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347–354, Vancouver, British Columbia, Canada.
  20. Theresa Wilson. 2008a. Annotating subjective content in meetings. In Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco. find
  21. Theresa Ann Wilson. 2008b. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States. Ph.D. thesis, University of Pittsburgh. find
  22. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In CIKM ’06: Proceedings of the 15th ACM international conference on Information and knowledge management, pages 43–50, Arlington, Virginia, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/