📄 Detection of Puffery on the English Wikipedia

Summary

The paper addresses automatic detection of puffery, also called peacock phrases, on the English Wikipedia. Puffery is wording that promotes a subject without attributed, verifiable support, and it violates the Neutral Point of View policy. The positive examples are 284 sentences that editors had tagged as puffery or peacock. No tag marks a sentence as clean, so the authors compare three ways of choosing negatives. Naive random choices turn out to mislead. Sampling from the same article picks up untagged puffery. Sampling from random articles lets a classifier exploit topic differences instead. The chosen method queries a Lucene index with each tagged sentence and takes a sentence from the top-retrieved, topically similar article. On the resulting balanced 568-sentence set, they compare a bag-of-words linear SVM with fine-tuned RoBERTa, each with and without a CITE marker placed where citations occur. RoBERTa with citation markers reaches 0.963 F1. A lighter AlBERT variant runs a Wikipedia bot that lists likely problem pages for human editors. This shows how such a classifier can be deployed within community rules.

Key points

• Data: 284 tagged puffery sentences were paired with 284 retrieval-matched negatives. 10% of the shuffled set (57 sentences) was held out for testing. Code and data are on GitHub (abertsch72/wikipedia-puffery-detection).

• Negative-sampling probe with a BoW SVM:

  • Same-article negatives gave 0.542 F1, because many supposed negatives were puffery.
  • Random-article negatives gave 0.780 F1, but the top features revealed topic bias (education, film, business).
  • Information-retrieval negatives gave 0.658 F1, with peacock-like top features such as "great" and "impressive".

• Test results (F1):

  Model	Words only	Words + CITE
  BoW-SVM	0.658	0.680
  RoBERTa	0.726	0.963 (P 0.937, R 0.989)

• Deployment: AlBERT (0.899 F1) runs on Toolforge as PeacockPhraseFinderBot.

  • Each hour it checks the 1,000 most recently edited pages, scoring 15 random sentences from each.
  • A page is flagged when more than half of those sentences are predicted as puffery.
  • Bot policy forbids bots from editing for subjective issues, so the bot only maintains a review list.

• Limitations: the dataset is small and its balanced ratio is unrealistic, since puffery is rare on Wikipedia, so over-prediction is likely. The model also ignores what the citations contain, so fake citations could fool it.

Interesting ideas

• I1. When a label has no explicit "clean" counterpart, take negatives from retrieval-matched documents so that topic cannot separate the classes; random negatives quietly teach shortcuts.
• I2. Judge a data-construction choice by whether the learned features make sense, not only by its score; the authors accepted a lower F1 because the higher-scoring dataset was learning topic.
• I3. A non-lexical structural signal (where a citation sits) can be injected as a positional pseudo-token, letting the model link it to nearby words; position mattered more than mere presence.
• I4. Praise is not the problem in itself; the problem is evaluation lacking attribution, which separates puffery from plain positive sentiment.
• I5. When an automated judgment is subjective, the system should surface candidates for humans and aggregate over sampled sentences with a majority threshold, rather than act alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when assembling silver data and negative controls, draw contrast examples from topically similar texts, so learned repair rules or classifiers cannot win on domain cues alone.
• I2 (learn): add a feature/rule-inspection step to evaluation, rejecting training setups whose strongest signals are topical rather than linguistic, even when the metrics look better.
• I3 (frames): mark attribution and evidence anchors (citations, "according to X") at their exact spot, so frame-level analysis can tell which evaluative predicate is backed by a source.
• I4 (world): store evaluative claims in the text's world as opinions tied to a holder or source (or flagged unattributed), rather than as facts about the entity.
• I5 (answer): readers could report document-level judgments (bias, tone) from a majority over sampled sentence-level decisions, keeping each contributing sentence visible as the explanation.