Elamur Research⌕
☰
home / papers / 2003 / Learning subjective nouns using extraction pattern bootstrapping

Learning subjective nouns using extraction pattern bootstrapping

Summary

The paper addresses sentence-level subjectivity classification: separating sentences that express opinions, emotions, evaluations or speculation ("private states") from factual ones in news text. The authors repurpose two extraction-pattern bootstrapping algorithms originally built for semantic lexicon induction, Meta-Bootstrapping and Basilisk. Starting from a few subjective seed nouns and a large unannotated FBIS news collection, the algorithms grow lists of subjective nouns. The hypothesis is that subjective words, like members of a semantic class, tend to appear in similar pattern contexts (e.g., as the object of "expressed").

The learned nouns, together with discourse features and subjectivity clues from earlier research, feed a Naive Bayes classifier. It is trained and tested on a small manually annotated FBIS corpus, where a sentence counts as subjective only if it contains a private state of at least medium strength. The bootstrappers produced over 1000 subjective nouns. The classifier reached 77% recall at 81% precision, improving on the earlier state of the art (Wiebe et al., 1999). The work shows that weakly supervised lexicon learning from raw text can strengthen supervised subjectivity models.

Key points

  • Data comes from English versions of foreign news collected by FBIS:
    • an experiment corpus of 109 documents (2197 sentences);
    • a tuning corpus of 33 documents (698 sentences);
    • a separate unannotated bootstrapping corpus of 950 texts.
  • Inter-annotator agreement on 178 sentences was 88% (κ = 0.71). Removing the 11% of sentences that were borderline (only low-strength private states) raised agreement to 94% (κ = 0.87).
  • Seed candidates were 850 nouns correlated with subjective sentences in another dataset. They were filtered by how often they occur in the bootstrapping corpus, because rare seeds cannot drive the bootstrapping.
  • The best Naive Bayes classifier combines the learned nouns, discourse features and prior subjectivity clues. It reaches 76.1% accuracy, significantly higher than the other classifiers tested.
  • A simple rule, labelling a sentence subjective if it contains any StrongSubjective noun, gives 87% precision at 26% recall.

Interesting ideas

  • I1. Reuse a semantic-class bootstrapper for a pragmatic category. Seeding with subjective words instead of words like LOCATION or VEHICLE turns a lexicon learner into a subjectivity-lexicon learner. Any category whose members share syntactic contexts could be learned the same way, such as hedges, threats or trade-related terms.
  • I2. Build the gold standard from annotation strength. The authors noticed that disagreements clustered on low-strength cases. They therefore folded weak subjectivity into the objective class, which gives a cleaner and more reliable target for the classifier.
  • I3. Choose seeds by their frequency in the target corpus. Ranking candidate seeds by how often they occur in the corpus being bootstrapped avoids a stalled start. This check is cheap and applies to any seed-based method.
  • I4. Combine unannotated and annotated resources. A large lexicon is learned without labels, then used as features in a classifier trained on a small annotated set. This suits low-resource settings, such as Ukrainian, where labelled data is scarce.
  • I5. Offer a high-precision lexicon-only mode. A plain "any strong noun present" rule trades recall for 87% precision. This is useful when false positives are costly, or for pre-filtering candidates before annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

85 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{riloff2003learning,
  title = {{Learning subjective nouns using extraction pattern bootstrapping}},
  author = {Riloff, Ellen and Wiebe, Janyce and Wilson, Theresa},
  booktitle = {Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003},
  year = {2003},
  doi = {10.3115/1119176.1119180},
  url = {https://doi.org/10.3115/1119176.1119180}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Baker, C. Fillmore, and J. Lowe. 1998. The berkeley framenet project. In Proceedings of the COLING-ACL. find our card
  2. T. Ballmer and W. Brennenstuhl. 1981. Speech Act Classification: A Study in the Lexical Analysis of English Speech Activity Verbs. Springer-Verlag. find
  3. A. Banfield. 1982. Unspeakable Sentences. Routledge and Kegan Paul, Boston. find
  4. S. Caraballo. 1999. Automatic Acquisition of a Hypernym-Labeled Noun Hierarchy from Text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 120–126. find
  5. C. Cardie. 1997. Empirical Methods in Information Extraction. AI Magazine, 18(4):65–79. find
  6. V. Hatzivassiloglou and K. McKeown. 1997. Predicting the semantic orientation of adjectives. In ACL-EACL 1997. find
  7. M. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proc. of the 14th International Conference on Computational Linguistics. find
  8. J. Karlgren and D. Cutting. 1994. Recognizing text genres with simple metrics using discriminant analysis. In COLING-94. find
  9. B. Kessler, G. Nunberg, and H. Schutze. 1997. Automatic detection of text genre. In Proc. ACL-EACL-97. find
  10. Beth Levin. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. find
  11. B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing. find
  12. R. Quirk, S. Greenbaum, G. Leech, and J. Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. find
  13. E. Riloff and R. Jones. 1999. Learning Dictionaries for Information Extraction by Multi-Level Bootstrapping. In Proceedings of the 16th National Conference on Artificial Intelligence. find
  14. E. Riloff and J. Shepherd. 1997. A Corpus-Based Approach for Building Semantic Lexicons. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pages 117–124. find
  15. B. Roark and E. Charniak. 1998. Noun-phrase Co-occurrence Statistics for Semi-automatic Semantic Lexicon Construction. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, pages 1110–1116. find
  16. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. IAAI. find
  17. M. Thelen and E. Riloff. 2002. A Bootstrapping Method for Learning Semantic Lexicons Using Extraction Pa ttern Contexts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing. find
  18. R. Tong. 2001. An operational system for detecting and tracking opinions in on-line discussion. In SIGIR 2001 Workshop on Operational Text Classification. find
  19. P. Turney. 2002. Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find
  20. J. Wiebe, R. Bruce, and T. O’Hara. 1999. Development and use of a gold standard data set for subjectivity classifications. In Proc. 37th Annual Meeting of the Assoc. for Computational Linguistics (ACL-99). find
  21. J. Wiebe, T. Wilson, and M. Bell. 2001. Identifying collocations for recognizing opinions. In Proc. ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation, July. find
  22. J. Wiebe. 1990. Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. Ph.D. thesis, State University of New York at Buffalo. find
  23. J. Wiebe. 2000. Learning subjective adjectives from corpora. In 17th National Conference on Artificial Intelligence. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/riloff-2003-learning-subjective-nouns-extraction-pattern/