Elamur Research⌕
☰
home / papers / 2004 / Learning Subjective Language

Learning Subjective Language

Summary

The paper addresses how to automatically find language that expresses opinions, evaluations, emotions and speculation (what the authors call "private states"), as opposed to language that presents facts. No complete lexicon of such language exists, and many expressions are subjective only in some contexts, so the authors learn subjectivity clues directly from corpora. They use two kinds of supervision: a small set of Wall Street Journal and newsgroup texts with manual expression-level labels, and a much larger set of WSJ documents already labelled as opinion pieces (editorials, reviews) or non-opinion pieces. The candidate clues are unique words (hapax legomena), n-gram collocations including patterns with open slots, and adjectives and verbs selected with Lin's (1998) distributional similarity. Clues learned from one annotation type are tested on the other. All clue types rise and fall together across data sets, and clues that occur in a dense cluster of other clues are highly precise. The clues are then applied to opinion piece recognition. The work matters because it gives an early, corpus-driven foundation for opinion mining, question answering and information extraction that must separate fact from opinion.

Key points

  • Data:
    • Expression-level manual annotations of WSJ and Usenet newsgroup text.
    • Large existing document-level WSJ labels separating opinion pieces from non-opinion pieces.
  • Three clue families:
    • Hapax legomena (called "unique words").
    • N-gram collocations with fixed wording (e.g., of the century, get out of here), which are frequently built from stop-list words.
    • Distributionally similar adjectives and verbs based on Lin (1998). For example, seeding with bizarre yields tragic, sad and poignant.
  • Theoretical model: a subjective element has a source and usually a target. Sources can be nested, such as (writer, Maher) for quoted speech. The model draws on Banfield (1982), Uspensky, Kuroda and others.
  • Contextual disambiguation: clue density in the surrounding context strongly affects whether a clue instance is subjective. A separate annotation study found that most high-density clues lie in subjective text spans.
  • Missing numbers: the excerpt provided omits the numeric results. Precision figures and classification scores should be checked in the full paper before citing them.

Interesting ideas

  • I1. Cross-validating across annotation schemes. Train on fine-grained manual labels and test on coarse document labels made by outsiders, and vice versa. This shows whether results generalize, and it could be reused for Ukrainian tasks that combine a small hand-annotated corpus with large metadata-labelled collections such as news rubrics.
  • I2. Treating rare words as a single feature. Taken individually, hapax words are too sparse to use. Pooled together as one feature, they become frequent enough to signal subjectivity. This is a cheap trick for any low-resource setting.
  • I3. Collocations with open slots. Some positions in a pattern accept any word of the right part of speech, provided that word is unique in the data. This captures productive expressive patterns that fixed n-gram lists miss.
  • I4. Smoothing word selection with distributional neighbours. A word is kept or dropped based on the combined precision of the word and its nearest similar words. This reduces noise from words seen only a few times and could transfer to lexicon induction for any language.
  • I5. Using clue density as a contextual filter. A clue counts more when other clues surround it. This is a simple, model-free way to disambiguate words that can be subjective or objective, for example when pre-selecting sentences for annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/0891201041850885

@article{wiebe2004learning,
  title = {{Learning Subjective Language}},
  author = {Wiebe, Janyce and Wilson, Theresa and Bruce, Rebecca and Bell, Matthew and Martin, Melanie},
  journal = {Computational Linguistics},
  year = {2004},
  doi = {10.1162/0891201041850885},
  url = {https://doi.org/10.1162/0891201041850885}
}

References

76 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, Rakesh, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu. 2003. Mining newsgroups using networks arising from social behavior. In Proceedings of the 12th International World Wide Web Conference (WWW2003), Budapest, May 20-24. find
  2. Alvarado, Sergio J., Michael G. Dyer, and Margot Flowers. 1986. Editorial comprehension in oped through argument units. In Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86), Philadelphia, August 11–15, pages 250–256. find
  3. Anderson, Clifford W. and George C. McMaster. 1989. Quantification of rewriting by the Brothers Grimm: A comparison of successive versions of three tales. Computers and the Humanities, 23(4–5):341–346. find
  4. Aone, Chinatsu, Mila Ramos-Santacruz, and William J. Niehaus. 2000. Assentor: An NLP-based solution to e-mail monitoring. In Proceedings of the 12th Innovative Applications of Artificial Intelligence Conference (IAAI-2000), Austin, TX, August 1–3, pages 945–950. find
  5. Argamon, Shlomo, Moshe Koppel, and Galit Avneri. 1998. Routing documents according to style. In Proceedings of the First International Workshop on Innovative Internet Information Systems (IIIS-98), Pisa, Italy, June 8–9. find
  6. Banfield, Ann. 1982. Unspeakable Sentences. Routledge and Kegan Paul, Boston. find
  7. Barzilay, Regina, Michael Collins, Julia Hirschberg, and Steve Whittaker. 2000. The rules behind roles: Identifying speaker role in radio broadcasts. In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), Austin, TX, July 30–August 3, pages 679–684. find
  8. Biber, Douglas. 1993. Co-occurrrence patterns among collocations: A tool for corpus-based lexical knowledge acquisition. Computational Linguistics, 19(3):531–538. find
  9. Brill, Eric. 1992. A simple rule-based part of speech tagger. In Proceedings of the 3rd Conference on Applied Natural Language Processing (ANLP-92), Trenton, Italy, April 1–3 pages 152–155. find our card
  10. Bruce, Rebecca and Janyce Wiebe. 1999. Recognizing subjectivity: A case study of manual tagging. Natural Language Engineering, 5(2):187–205. find
  11. Carbonell, Jaime G. 1979. Subjective Understanding: Computer Models of Belief Systems. Ph.D. thesis, and Technical Report no. 150, Department of Computer Science, Yale University, New Haven, CT. find
  12. Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIG dial Workshop on Discourse and Dialogue (SIGdial-2001), Aalborg, Denmark, September 1–2, pages 30–39. find our card
  13. Chatman, Seymour. 1978. Story and Discourse: Narrative Structure in Fiction and Film. Cornell University Press, Ithaca, NY. find
  14. Church, Kenneth W. and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16:22–29. find our card
  15. Cohn, Dorrit. 1978. Transparent Minds: Narrative Modes for Representing Consciousness in Fiction. Princeton University Press, Princeton, NJ. find
  16. Copeck, Terry, Kim Barker, Sylvain Delisle, and Stan Szpakowicz. 2000. Automating the measurement of linguistic features to help classify texts as technical. In Proceedings of the Seventh Conference on Automatic NLP (TALN-2000), Lausanne, Switzerland, October 16–18, pages 101–110. find
  17. Dagan, Ido, Fernando Pereira, and Lillian Lee. 1994. Similarity-based estimation of word cooccurrence probabilities. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics (ACL-94), Las Cruces, NM, June 27–30, pages 272–278. find
  18. Dave, Kushal, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: Opinion extraction and semantic classification of produce reviews. In Proceedings of the 12th International World Wide Web Conference (WWW2003), Budapest, May 20–24. find
  19. Doležel, Lubomir. 1973. Narrative Modes in Czech Literature. University of Toronto Press, Toronto, Ontario, Canada. find
  20. Dyer, Michael G. 1982. Affect processing for narratives. In Proceedings of the Second National Conference on Artificial Intelligence (AAAI-82), Pittsburgh, August 18–20, pages 265–268. find
  21. Everitt, Brian S. 1977. The Analysis of Contingency Tables. Chapman and Hall, London. find
  22. Fludernik, Monika. 1993. The Fictions of Language and the Languages of Fiction. Routledge, London. find
  23. Fodor, Janet Dean. 1979. The Linguistic Description of Opaque Contexts, volume 13 of Outstanding Dissertations in Linguistics. Garland, New York and London. find
  24. General-Inquirer, The. 2000. Available at http://www.wjh.harvard.edu/ ˜ inquirer/spreadsheet guide.htm. ↗
  25. Gordon, Andrew, Abe Kazemzadeh, Anish Nair, and Milena Petrova. 2003. Recognizing expressions of commonsense psychology in English text. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-03), Sapporo, Japan, July 7–12, pages 208–215. find our card
  26. Hart, Roderick P. 1984. Systematic analysis of political discourse: The development of diction. In K. Sanders et al., editors, Political Communication Yearbook: 1984. Southern Illinois University Press, Carbondale, pages 97–134. find
  27. Hatzivassiloglou, Vasileios and Kathy McKeown. 1997. Predicting the semantic orientation of adjectives. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-97), Madrid, July 12, pages 174–181. find
  28. Heise, David. 2000. Affect control theory. Available at http://www.indiana.edu/socpsy/ACT/index.htm. ↗
  29. Hindle, Don. 1990. Noun classification from predicate-argument structures. In Proceedings of the 28th Annual Meeting of the Association for Computational Linguistics (ACL-90), Pittsburgh, June 6–9, pages 268–275. find our card
  30. Hovy, Eduard. 1987. Generating Natural Language under Pragmatic Constraints. Ph.D. thesis, Yale University, New Haven, CT. find
  31. Karlgren, Jussi and Douglass Cutting. 1994. Recognizing text genres with simple metrics using discriminant analysis. In Proceedings of the Fifteenth International Conference on Computational Linguistics (COLING-94), pages 1071–1075. find
  32. Karp, Daniel, Yves Schabes, Martin Zaidel, and Dania Egedi. 1994. A freely available wide coverage morphological analyzer for English. In Proceedings of the 15th International Conference on Computational Linguistics (COLING-94), Nantes, France pages 922–928. find
  33. Kaufer, David. 2000. Flaming: A white paper. Available at www.eudora.com. find
  34. Kessler, Brett, Geoffrey Nunberg, and Hinrich Schütze. 1997. Automatic detection of text genre. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-97), Madrid, July 7–12, pages 32–38. find
  35. Kuroda, S.-Y. 1973. Where epistemology, style and grammar meet: A case study from the Japanese. In P. Kiparsky and S. Anderson, editors, A Festschrift for Morris Halle. Holt, Rinehart & Winston, New York, pages 377–391. find
  36. Kuroda, S.-Y. 1976. Reflections on the foundations of narrative theory—from a linguistic point of view. In T. A. van Dijk, editor, Pragmatics of Language and Literature. North-Holland, Amsterdam, pages 107–140. find
  37. Lee, Lillian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, MD, pages 25–32. find
  38. Lee, Lillian and Fernando Pereira. 1999. Distributional similarity models: Clustering vs. nearest neighbors. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, MD, pages 33–40. find
  39. Lehnert, Wendy G., Michael Dyer, Peter Johnson, C. J. Yang, and Steve Harley. 1983. BORIS: An Experiment in In-Depth Understanding of Narratives. Artificial Intelligence, 20:15–62. find
  40. Lin, Dekang. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL-98), Montreal, August 10–14, pages 768–773. find our card
  41. Lin, Dekang. 1999. Automatic identification of non-compositional phrases. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, MD, pages 317–324. find our card
  42. Litman, Diane J. and Rebecca J. Passonneau. 1995. Combining multiple knowledge sources for discourse segmentation. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL-95), Cambridge, MA, June 26–30, pages 108–115. find
  43. Macleod, Catherine, Ralph Grishman, and Adam Meyers. 1998. Complex syntax reference manual. Technical report, New York University. find
  44. Marcu, Daniel, Magdalena Romera, and Estibaliz Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In Proceedings of the International Workshop on Levels of Representation in Discourse (LORID-99), Edinburgh, July 6–9 pages 71–78. find
  45. Marcus, Mitch, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  46. Mitchell, Tom. 1997. Machine Learning. McGraw-Hill, Boston. find
  47. Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), Philadelphia, July 6–7, pages 79–86. find our card
  48. Quirk, Randolph, Sidney Greenbaum, Geoffry Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. find
  49. Riloff, Ellen and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level Bootstrapping. In Proceedings of the 16th National Conference on Artificial Intelligence (AAAI-1999), Orlando, FL, July 18–22, pages 474–479. find
  50. Riloff, Ellen and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003), Sapporo, Japan, July 11–12, pages 105–112. find our card
  51. Riloff, Ellen, Janyce Wiebe, and Theresa Wilson. 2003. Learning subjective nouns using extraction pattern bootstrapping. In Proceedings of the Seventh Conference on Natural Language Learning (CoNLL-2003), Edmonton, Alberta, Canada, May 31–June 1, pages 25–32. find our card
  52. Sack, Warren. 1995. Representing and recognizing point of view. In Proceedings of the AAAI Fall Symposium on AI Applications in Knowledge Navigation and Retrieval, Cambridge, MA, page 152. find
  53. Samuel, Ken, Sandra Carberry, and K. Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL-98), Montreal, August 10–14, pages 1150–1156. find our card
  54. Smajda, Frank. 1993. Retrieving collocations from text: Xtract. Computational Linguistics, 19:143–177. find our card
  55. Spertus, Ellen. 1997. Smokey: Automatic recognition of hostile messages. In Proceedings of the Ninth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-97), Providence, RI, July 27–31, pages 1058–1065. find
  56. Stein, Dieter and Susan Wright, editors. 1995. Subjectivity and Subjectivisation. Cambridge University Press, Cambridge. find
  57. Terveen, Loren, Will Hill, Brian Amento, David McDonald, and Josh Creter. 1997. Building task-specific interfaces to high volume conversational data. In Proceedings of the Conference on Human Factors in Computing Systems (CHI-97), Los Angeles, April 18–23, pages 226–233. find
  58. Teufel, Simone and Marc Moens. 2000. What’s yours and what’s mine: Determining intellectual attribution in scientific texts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the Workshop on Very Large Corpora (EMNLP/VLC-2000), Hong Kong, October 7–8, pages 9–1… find
  59. Tong, Richard. 2001. An operational system for detecting and tracking opinions in on-line discussions. In Working Notes of the SIGIR Workshop on Operational Text Classification, New Orleans, September 9–13, pages 1–6. find
  60. Turney, Peter. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2000), Philadelphia, July 7–12, pages 417–424. find our card
  61. Uspensky, Boris. 1973. A Poetics of Composition. University of California Press, Berkeley, and Los Angeles. van Dijk, Teun A. 1988. News as Discourse. Erlbaum, Hillsdale, NJ. find
  62. Weeber, Marc, Rein Vos, and R. Harald Baayen. 2000. Extracting the lowest-frequency words: Pitfalls and possibilities. Computational Linguistics, 26(3):301–317. find
  63. Wiebe, Janyce and Theresa Wilson. 2002. Learning to disambiguate potentially subjective expressions. In Proceedings of the Sixth Conference on Natural Language Learning (CoNLL-2002), Taipei, Taiwan, pages 112–118. find
  64. Wiebe, Janyce. 1990. Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. Ph.D. thesis, State University of New York at Buffalo. find
  65. Wiebe, Janyce. 1994. Tracking point of view in narrative. Computational Linguistics, 20(2):233–287. find
  66. Wiebe, Janyce. 2000. Learning subjective adjectives from corpora. In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), Austin, TX, July 30–August 3, pages 735–740. find
  67. Wiebe, Janyce, Eric Breck, Chris Buckley, Claire Cardie, Paul Davis, Bruce Fraser, Diane Litman, David Pierce, Ellen Riloff, Theresa Wilson, David Day, and Mark Maybury. 2003. Recognizing and organizing opinions expressed in the world press. In Working Notes of the AAAI Spring Symposium in New Direc… find
  68. Wiebe, Janyce, Rebecca Bruce, Matthew Bell, Melanie Martin, and Theresa Wilson. 2001. A corpus study of evaluative and speculative language. In Proceedings of the Second ACL SIGdial Workshop on Discourse and Dialogue (SIGdial-2001), Aalborg, Denmark, September 1–2, pages 186–195. find
  69. Wiebe, Janyce, Rebecca Bruce, and Thomas O’Hara. 1999. Development and use of a gold standard data set for subjectivity classifications. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, MD, pages 246–253. find
  70. Wiebe, Janyce, Kenneth McKeever, and Rebecca Bruce. 1998. Mapping collocational properties into machine learning features. In Proceedings of the Sixth Workshop on Very Large Corpora (WVLC-98), Montreal, August 15–16, pages 225–233. find
  71. Wiebe, Janyce and William J. Rapaport. 1986. Representing de re and de dicto belief reports in discourse and narrative. Proceedings of the IEEE, 74:1405–1413. find
  72. Wiebe, Janyce and William J. Rapaport. 1988. A computational theory of perspective and reference in narrative. In Proceedings of the 26th Annual Meeting of the Association for Computational Linguistics (ACL-88), Buffalo, NY, pages 131–138. find
  73. Wiebe, Janyce M. and William J. Rapaport. 1991. References in narrative text. Noûs, 25(4):457–486. find
  74. Wiebe, Janyce, Theresa Wilson, and Matthew Bell. 2001. Identifying collocations for recognizing opinions. In Proceedings of the ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation, Toulouse, France, July 7, pages 24–31. find
  75. Wilson, Theresa and Janyce Wiebe. 2003. Annotating opinions in the world press. In Proceedings of the Fourth SIGdial Workshop on Discourse and Dialogue (SIGdial-2003), Sapporo, Japan, July 5–6, pages 13–22. find
  76. Yu, Hong and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003), Sapporo, Japan, July 11–12, pages 1… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wiebe-2004-learning-subjective-language/