Elamur Research⌕
☰
home / papers / 2001 / A Corpus Study of Evaluative and Speculative Language

A Corpus Study of Evaluative and Speculative Language

Paper · 2001 · Janyce Wiebe, Rebecca Bruce, Matthew Bell et al.

Summary

The paper looks at how to find subjective language in text. This covers evaluation (emotions, judgements, opinions) and speculation (wording that removes the presupposition that an event occurred or a state holds). Standard lexical resources do not mark these clues, so they must be learned from corpora. The authors annotate subjectivity at three granularities:

  • individual expressions inside sentences,
  • whole sentences,
  • whole documents, using editorials, reviews and hostile "flame" messages from Usenet and the Wall Street Journal.

They measure annotator agreement at each level and use the expression-level tags to describe what subjective language looks like. They then mine potential subjective elements (PSEs) from cheap, pre-existing document labels. Adjectives and verbs are clustered by distributional similarity, and the clustering parameters are tuned with a train–validate–test protocol.

The main results:

  • Document-level subjectivity is annotated reliably. Expression-level tagging is harder but feasible.
  • Words occurring only once are themselves a subjectivity clue.
  • Mined PSEs raise precision over baseline consistently across datasets.

This matters for extraction, retrieval and review mining, which must tell opinion from presented fact.

Key points

  • Flame corpus: 1140 Usenet messages, balanced across the alt, sci, comp and rec hierarchies, split into 778 training and 362 test messages.
  • Earlier sentence-level work on the Wall Street Journal Treebank Corpus:
    • Three judges reached an average pairwise Cohen's κ of .69 after two rounds.
    • Consensus tags were derived with EM (Dawid and Skene style).
    • A probabilistic classifier scored 72.17% in 10-fold cross validation, more than 20 points above the majority-class baseline.
  • Opinion-piece labels are "impure", and the authors work out how much this limits precision:
    • Assume 15% of sentences sit in opinion pieces, with 70% of those subjective versus 44% elsewhere.
    • Then the expected precision of subjective sentences for the opinion class is only about .22.
  • PSE mining procedure:
    • Lin's (1998) clustering, seeded with all training adjectives or verbs.
    • Cluster size C and filtering threshold FT are chosen on a validation set, keeping only sets with precision ≥ .25 and frequency ≥ 100.
    • Training and validation use existing WSJ opinion labels; testing uses manual annotations.
    • Test sets are W9-10, W9-22, W9-33 and W9-04.
  • Precision gains over baseline:
    • adjectives +.09 to +.21,
    • verbs +.07 to +.16,
    • unique words +.06 to +.10.
    • All three feature types rise and fall together across datasets.
    • Agreement is higher for adjectives than for determiners.

Interesting ideas

  • I1. Every subjective element carries a source (writer or a mentioned person) and a target, so a reported complaint is attributed to the complainer instead of being stored as fact, unlike a neutral "a judge said" report.
  • I2. The paper separates a potential subjective element (a lexicon entry such as eroding) from an actual subjective use, which depends on context.
  • I3. Words that appear only once in a text are treated as a positive clue for subjectivity, rather than discarded as noise.
  • I4. With noisy labels, the authors first compute the precision that is achievable at all from the class mixture, then judge features by their gain over a word-frequency baseline.
  • I5. EM-based consensus (Dawid–Skene style) merges several annotators into corrected tags, weighting each by estimated reliability instead of taking a simple vote.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could attach source/target slots to evaluative verbs like complain, so quoted content under them is marked as an attributed attitude rather than an asserted event.
  • I2 (world): the knowledge layer could store words as potentially evaluative, and the world builder would mark a state as opinion only when the context in the tree confirms it, keeping the decision explainable.
  • I3 (tokens): crate en could flag hapax tokens per document, giving prag's stance detection a cheap, interpretable signal.
  • I4 (learn): evaluation of silver-trained components could report an expected ceiling and baseline-relative gains, so impure supervision does not make good rules look bad.
  • I5 (learn): silver data from multiple witness parsers could be combined the same way, learning per-parser reliability for each relation type.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓New Mexico State UniversityLas Cruces · United States7 works · 5,672 citations🎓University of North Carolina at AshevilleAsheville · United States4 works · 1,502 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

Cite

Cite as Janyce Wiebe, Rebecca Bruce, Matthew Bell, Melanie Martin and Theresa Wilson. 2001. A Corpus Study of Evaluative and Speculative Language. Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wiebe2001corpus,
  title = {{A Corpus Study of Evaluative and Speculative Language}},
  author = {Janyce Wiebe, Rebecca Bruce, Matthew Bell, Melanie Martin and Theresa Wilson},
  booktitle = {Proceedings of the Second SIGdial Workshop on Discourse and Dialogue},
  year = {2001},
  doi = {10.3115/1118078.1118104},
  url = {https://doi.org/10.3115/1118078.1118104}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Aone, M. Ramos-Santacruz, and W. Niehaus. 2000. Assentor: An nlp-based solution to e-mail monitoring. In Proc. IAAI-2000 , pages 945{ 950. find
  2. A. Ban eld. 1982. Unspeakable Sentences . Routledge and Kegan Paul, Boston. find
  3. R. Barzilay, M. Collins, J. Hirschberg, and S. Whittaker. 2000. The rules behind roles: Identifying speaker role in radio broadcasts. In Proc. AAAI . ↗
  4. E. Brill. 1992. A simple rule-based part of speech tagger. In Proc. of the 3rd Conference on Applied Natural Language Processing (ANLP-92) , pages 152{155.
  5. R. Bruce and J. Wiebe. 1999. Recognizing subjectivity: A case study of manual tagging. Natural Language Engineering , 5(2). find
  6. J. Cohen. 1960. A coecient of agreement for nominal scales. Educational and Psychological Meas. , 20:37{46. find
  7. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics , 28:20{28. ↗
  8. M. Fludernik. 1993. The Fictions of Language and the Languages of Fiction . Routledge, London. find
  9. L. Goodman. 1974. Exploratory latent structure analysis using both identi able and unidenti - able models. Biometrika , 61:2:215{231. find
  10. E. Hovy. 1987. Generating Natural Language under Pragmatic Constraints . Ph.D. thesis, Yale University. find
  11. D. Karp, Y. Schabes, M. Zaidel, and D. Egedi. 1992. A freely available wide coverage morphological analyzer for English. In Proc. of the 14th International Conference on Computational Linguistics (COLING-92) . find
  12. D. Kaufer. 2000. Flaming: A White Paper . www.eudora.com. find
  13. B. Kessler, G. Nunberg, and H. Schutze. 1997. Automatic detection of text genre. In Proc. ACL-EACL-97 . ↗
  14. D. Lin. 1998. Automatic retrieval and clustering of similar words. In Proc. COLING-ACL '98 , pages 768{773.
  15. Diane J. Litman and R. J. Passonneau. 1995. Combining multiple knowledge sources for discourse segmentation. In Proc. 33rd Annual Meeting of the Association for Computational Linguistics (ACL-95) , pages 108{115. Association for Computational Linguistics, june. find
  16. D. Marcu, M. Romera, and E. Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In The Workshop on Levels of Representation in Discourse , pages 71{78. find
  17. M. Marcus, Santorini, B., and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: The penn treebank. Computational Linguistics , 19(2):313{330. ↗
  18. G. Nunberg, I. Sag, and T. Wasow. 1994. Idioms. Language , 70:491{538. find
  19. W. Sack. 1995. Representing and recognizing point of view. In Proc. AAAI Fall Symposium on AI Applications in Knowledge Navigation and Retrieval . find
  20. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. IAAI . find
  21. D. Stein and S. Wright, editors. 1995. Subjectivity and Subjectivisation . Cambridge University Press, Cambridge. find
  22. L. Terveen, W. Hill, B. Amento, D. McDonald, and J. Creter. 1997. Building task-speci c interfaces to high volume conversational data. In Proc. CHI 97 , pages 226{233. find
  23. S. Teufel and M. Moens. 2000. What's yours and what's mine: Determining intellectual attribution in scienti c texts. In Proc. Joint SIGDAT Converence on EMNLP and VLC . find
  24. J. Wiebe, K. McKeever, and R. Bruce. 1998. Mapping collocational properties into machine learning features. In Proc. 6th Workshop on Very Large Corpora (WVLC-98) , pages 225{ 233, Montreal, Canada, August. ACL SIGDAT. find
  25. J. Wiebe, R. Bruce, and T. O'Hara. 1999. Development and use of a gold standard data set for subjectivity classi cations. In Proc. 37th Annual Meeting of the Assoc. for Computational Linguistics (ACL-99) , pages 246{253, University of Maryland, June. ACL. find
  26. J. Wiebe. 1994. Tracking point of view in narrative. Computational Linguistics , 20(2):233{287. ↗
  27. J. Wiebe. 2000. Learning subjective adjectives from corpora. In 17th National Conference on Arti cial Intelligence (AAAI-2000) . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wiebe-2001-corpus-study-evaluative-speculative-language/