📄 Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences

Summary

Yu and Hatzivassiloglou tackle a prerequisite for answering complex, opinion-oriented questions. A system must tell subjective material apart from factual reporting and decide whether an opinion is favourable or unfavourable. They work with Wall Street Journal articles and use the newspaper's own article-type metadata (Editorial, Letter to editor, Business, News) as weak supervision instead of hand-labelled training data. A word-based Naive Bayes model separates opinion-heavy documents from regular news. For sentences, they compare three techniques:

• similarity to opinion versus fact sentences on the same topic, measured with SIMFINDER;
• a Naive Bayes classifier with n-gram, part-of-speech and polarity features;
• an iterative scheme of several Naive Bayes classifiers that drops training sentences likely to be mislabelled.

For polarity, they grow a seed list of oriented words into a large lexicon covering adjectives, adverbs, nouns and verbs, then aggregate word scores per sentence. On a 400-sentence human-annotated set, the system reaches about 97% F-measure for documents, up to 91% for detecting opinion sentences, and 90% polarity accuracy. This shows that cheap document-level labels can bootstrap fine-grained opinion analysis.

Key points

• Document classification uses Naive Bayes (Rainbow implementation) over raw single words, with no stemming or stopword removal. It exceeds 97% precision and recall on WSJ editorials and letters versus news.
• Sentence-level training assumes every sentence inherits its article's label. Citing Wiebe et al. (2002), the authors note this holds for 75% of opinion-document sentences and 56% of factual-document sentences.
• The evaluation has two gold standards: Standard A, and Standard B, which keeps only sentences where two annotators agree. Scores are consistently higher on Standard B.
• Opinion sentences are detected far better (80–90% recall/precision) than factual ones (around 50%). The best feature set combines words, bigrams, trigrams, POS and polarity. Multiple classifiers give a small extra gain.
• The automatically built lexicon contains, for example, 39,652 positive and 65,773 negative adjectives, plus large sets of nouns, verbs and adverbs.
  • Lexicon quality improves with larger seed sets (1, 20 or 100 adjective pairs), checked against 1,336 hand-labelled adjectives.
  • Sentence polarity using adjectives, adverbs and verbs reaches 90% accuracy, against a 48% baseline.

Interesting ideas

• I1. Free labels from existing metadata. Section tags such as "Editorial" can serve as noisy training labels for both documents and sentences. This applies wherever publishers already categorise content, for example Ukrainian news sites with opinion or column sections.
• I2. Iterative self-cleaning of noisy training data. Train a classifier, remove the training items it contradicts, then retrain with a different feature subset until nothing more is dropped. This is a simple, general way to clean data whose labels were inherited from a coarser level.
• I3. Topic-conditioned similarity voting. A sentence is labelled by comparing it to opinion versus fact sentences retrieved for the same topic, using either average scores or counts above a threshold. Conditioning on topic keeps content words from dominating the stylistic signal.
• I4. Polarity beyond adjectives. Extending seed-based orientation lexicons to verbs, adverbs and nouns, and counting polarity sequences or POS-polarity combinations such as "JJ+", added useful evidence. This is relevant when building sentiment lexicons for lower-resource languages.
• I5. Reporting on an agreement-filtered gold set. Evaluating separately on the subset where annotators agree separates system errors from genuinely ambiguous cases. This is a cheap evaluation practice for any subjective annotation task.