Elamur Research⌕
☰
home / papers / 1999 / Development and Use of a Gold-Standard Data Set for Subjectivity Classifications

Development and Use of a Gold-Standard Data Set for Subjectivity Classifications

Paper · 1999 · Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O'Hara

Summary

The paper builds a reliable gold standard for separating sentences that present facts objectively from sentences that convey opinions, evaluations or speculation. This distinction matters mostly in news text. The authors define the categories by the writer's primary intention, with special focus on sentences about private states and speech events. Judges assigned a subjective or objective tag to each sentence, plus a certainty rating from 0 to 3. Kappa alone was not enough, so the authors also fit log-linear models. Marginal homogeneity exposes relative bias between judges, and quasi-symmetry exposes symmetric, systematic disagreement. A latent class model, which has a naive Bayes form and is fit with EM in the CoCo package, then yields bias-corrected tags. These tags served two purposes. They guided a revision of the coding manual, which raised pairwise Kappa by up to 16 points. They also became training labels for a probabilistic classifier, which reached 72.17% accuracy against a 51% baseline. The workflow carries over to any manual annotation task where judges disagree in a biased but systematic way.

Key points

  • In the first round, four judges (B, D, J, M) each tagged a corpus in about four hours. The contingency table for D and J covers 504 sentences and shows that D leans objective and J leans subjective.
  • On the second data set, the tests show strong bias between judges and strong symmetric disagreement. For example, the D/J marginal homogeneity G² is 104.9, while quasi-symmetry fits well with significance near 0.99. As in the first round, the two-category latent class model gave the most consistent clusters and was used to define the corrected tags.
  • After the manual revision, pairwise Kappa among M, D and J is 0.87 or higher on sentences rated with certainty 2 or 3.
  • The candidate classifier structures were a naive Bayes model, a model treating all features as independent, a model treating them as fully interdependent, and variants of these found through forward and backward search. The final model was chosen by held-out accuracy within 10-fold cross-validation. It used simple features:
    • presence of a pronoun, adjective or cardinal number
    • a modal other than will, an adverb other than not
    • whether the sentence starts a paragraph
    • co-occurrence of word tokens and punctuation with each class
  • The classifier reached 72.17% overall and 81.5% on the 299 of 500 sentences that all judges rated as certain. The judges themselves agree with the corrected tags 89.5% of the time on average. The coding manual and data were made publicly available.

Interesting ideas

  • I1. Diagnose disagreement, not just measure it. Testing for marginal homogeneity and quasi-symmetry shows whether disagreement comes from one annotator's systematic tendency, which can be fixed, or from real noise. Any annotation project, such as UD or NER labeling for Ukrainian, could use this to decide whether to retrain annotators or rewrite the guidelines.
  • I2. Use a latent class model to produce one gold label from several annotators. Treat the true label as a hidden variable that explains the judges' tags, and estimate it with EM. This is a principled alternative to majority vote when annotators have consistent individual biases.
  • I3. Collect certainty ratings alongside labels. The ratings add the degrees of freedom the statistical tests need. They also allow evaluation on "confident" sentences only, where both human agreement and model accuracy turn out much higher.
  • I4. Run a data-driven loop for the coding manual. Annotate, analyze bias, revise the instructions, and annotate again. This iterative cycle is cheap and produced a large Kappa gain, so it fits any new tagging scheme.
  • I5. Report human agreement with the gold labels as a ceiling. Putting this figure next to model accuracy shows how much room for improvement remains, which is a useful habit when reporting classifier results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New Mexico State UniversityLas Cruces · United States6 works · 5,598 citations🎓University of North Carolina at AshevilleAsheville · United States2 works · 1,429 citations

📚 Shelves 8 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Decision Sciences · 85 works General Social Sciences · 87 works Statistics, Probability and Uncertainty · 32 works

Cite

Cite as Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O'Hara. 1999. Development and Use of a Gold-Standard Data Set for Subjectivity Classifications. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1034678.1034721

@inproceedings{wiebe1999development,
  title = {{Development and Use of a Gold-Standard Data Set for Subjectivity Classifications}},
  author = {Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O'Hara},
  booktitle = {Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics},
  year = {1999},
  doi = {10.3115/1034678.1034721},
  url = {https://doi.org/10.3115/1034678.1034721}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Badsberg. 1995. An Environment for Graphical Models. Ph.D. thesis, Aalborg University. find
  2. R. F. Bales. 1950. Interaction Process Analysis. University of Chicago Press, Chicago, ILL. find
  3. Ann Banfield. 1982. Unspeakable Sentences: Narration and Representation in the Language of Fiction. Routledge & Kegan Paul, Boston. find
  4. S. Bergler. 1992. Evidential Analysis o.f Reported Speech. Ph.D. thesis, Brandeis University. find
  5. Y.M. Bishop, S. Fienberg, and P. Holland. 1975. Discrete Multivariate Analysis: Theory and Practice. The MIT Press, Cambridge. find
  6. R. Bruce and J. Wiebe. 1998. Word sense distinguishability and inter-coder agreement. In Proc. 3rd Conference on Empirical Methods in Natural Language Processing (EMNLP-98), pages 53-60, Granada, Spain, June. ACL SIGDAT. find
  7. R. Bruce and J. Wiebe. 1999. Decomposable modeling in natural language processing. Computational Linguistics, 25(2). find
  8. R. Bruce and J. Wiebe. to appear. Recognizing subjectivity: A case study of manual tagging. Natural Language Engineering. find
  9. J. Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254. our card
  10. W. Chafe. 1986. Evidentiality in English conversation and academic writing. In Wallace Chafe and Johanna Nichols, editors, Evidentiality: The Linguistic Coding of Epistemology, pages 261-272. Ablex, Norwood, NJ. find
  11. P. Cheeseman and J. Stutz. 1996. Bayesian classification (AutoClass): Theory and results. In Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy, editors, Advances in Knowledge Discovery and Data Mining. find
  12. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Meas., 20:37-46. find
  13. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28:20-28. find
  14. A. Dempster, N. Laird, and D. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39 (Series B):1-38. find
  15. T. Dunning. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):75-102. our card
  16. L. Goodman. 1974. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 61:2:215-231. find
  17. V. Hatzivassiloglou and K. McKeown. 1997. Predicting the semantic orientation of adjectives. In ACL-EACL 1997, pages 174-181, Madrid, Spain, July. find
  18. Eduard Hovy. 1987. Generating Natural Language under Pragmatic Constraints. Ph.D. thesis, Yale University. find
  19. D. Jurafsky, E. Shriberg, and D. Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-01, University of Colorado Institute of Cognitive Science. find
  20. M.-Y. Kan, J. L. Klavans, and K. R. McKeown. 1998. Linear segmentation and segment significance. In Proc. 6th Workshop on Very Large Corpora (WVLC-98), pages 197-205, Montreal, Canada, August. ACL SIGDAT. find
  21. K. Krippendorf. 1980. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills. find
  22. P. Lazarsfeld. 1966. Latent structure analysis. In S. A. Stouffer, L. Guttman, E. Suchman, P.Lazarfeld, S. Star, and J. Claussen, editors, Measurement and Prediction. Wiley, New York. find
  23. D. Litman. 1996. Cue phrase classification using machine learning. Journal of Artificial Intelligence Research, 5:53-94. find
  24. M. Marcus, Santorini, B., and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: The penn treebank. Computational Linguistics, 19(2):313-330. our card
  25. Ted Pedersen and Rebecca Bruce. 1998. Knowledge lean word-sense disambiguation. In Proc. of the 15th National Conference on Artificial Intelligence (AAAI-98), Madison, Wisconsin, July. find
  26. R. Quirk, S. Greenbaum, G. Leech, and J. Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. find
  27. T. Read and N. Cressie. 1988. Goodness-of-fit Statistics for Discrete Multivariate Data. Springer-Verlag Inc., New York, NY. find
  28. K. Samuel, S. Carberry, and K. Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. In Proc. our card
  29. COLING-ACL 1998, pages 1150-1156, Montreal, Canada, August. find
  30. T.A. van Dijk. 1988. News as Discourse. Lawrence Erlbaum, Hillsdale, NJ. find
  31. J. Wiebe, R. Bruce, and L. Duan. 1997. Probabilistic event categorization. In Proc. Recent Advances in Natural Language Processing (RANLP-97), pages 163-170, Tsigov Chark, Bulgaria, September. find
  32. J. Wiebe, K. McKeever, and R. Bruce. 1998. Mapping collocational properties into machine learning features. In Proc. 6th Workshop on Very Large Corpora (WVLC-98), pages 225-233, Montreal, Canada, August. ACL SIGDAT. find
  33. J. Wiebe, J. Klavans, and M.Y. Kan. in preparation. Verb profiles for subjectivity judgments and text classification. find
  34. J. Wiebe. 1994. Tracking point of view in narrative. Computational Linguistics, 20(2):233-287. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wiebe-1999-development-use-gold-standard-data/