Elamur Research⌕
☰
home / papers / 2013 / Linguistic Models for Analyzing and Detecting Biased Language

Linguistic Models for Analyzing and Detecting Biased Language

Summary

The paper asks how bias appears in text that is meant to be neutral, such as encyclopedia articles, and whether a system can find the exact word responsible for it. The authors built the NPOV corpus from Wikipedia articles that were ever placed in the NPOV-dispute category, along with their full revision histories. They kept only small edits whose revision comments mention (N)POV and compared each sentence before and after the edit. A manual study of these edits found two families of bias. Framing bias comes from subjective or one-sided wording. Epistemological bias comes from factive verbs, entailments, assertive verbs and hedges, which quietly shape how believable a claim seems. These cues became features for a logistic regression model. Given a sentence known to be biased, the model ranks its words to find the culprit. It reaches 34% top-1 and 59% top-3 accuracy, beats four baselines, and comes close to Amazon Mechanical Turk workers on the same sentences. This shows that a tool for flagging subtle bias could genuinely help editors of reference texts.

Key points

  • NPOV corpus size. The corpus covers about 2.7M revisions (~13G words) split into train, dev and test sets. It contains 13807, 1261 and 1751 bias-driven edits, and 1843, 163 and 230 sentences with a one-word edit.
  • Edit filtering. Edits were extracted with the word-mode diff from the Diff Match and Patch library. The authors kept only edits of at most five words whose revision comment mentions (N)POV. They dropped edits that only added a hyperlink or had a character Levenshtein distance below 4.
  • Bias types in a sample of 100 edits.
    • Framing bias: one-sided terms 38%, intensifiers 19%.
    • Epistemological bias: entailments 25%, assertives 11%, hedges 4%, factive verbs 3%.
  • Human comparison. Ten turkers labeled each sentence.
    • Individual turkers found the right word 30% of the time.
    • The majority answer matched the gold label in 37.39% of cases, slightly above the model.
    • Pairwise agreement among turkers was 40.73%, against 5.1% expected by chance.
    • About 34% of correct answers overlapped between the system and the humans.
  • Released resources. The NPOV data and a bias lexicon are publicly available.

Interesting ideas

  • I1. Comment-filtered revisions as free supervision. Filtering a revision history by editor comments yields aligned before/after pairs targeted at one phenomenon without hand annotation. The same trick could gather training data from Ukrainian Wikipedia or any versioned text with edit summaries.
  • I2. Size filters for cleaner edits. Keeping only short edits and dropping tiny changes (low edit distance, link-only edits) removes typo fixes and noise cheaply. This is a reusable recipe for mining revision data.
  • I3. Bias beyond sentiment. Epistemological cues such as factives, implicatives, assertive verbs and hedges are treated as a bias channel separate from sentiment. This is worth borrowing for detecting slanted reporting, where tone stays neutral but word choice signals how much a claim should be believed.
  • I4. A narrow, practical task. The task is framed as locating the single bias-inducing word in an already-flagged sentence and is scored with top-1/2/3 accuracy. This framing fits an editor-assist tool that highlights a few candidates rather than issuing a verdict.
  • I5. Humans on the identical test set. Running crowd workers on the same data, with agreement and chance baselines, shows how hard the task really is. This makes modest accuracy numbers interpretable, which is useful for any subjective annotation task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

111 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{recasens2013linguistic,
  title = {{Linguistic Models for Analyzing and Detecting Biased Language}},
  author = {Recasens, Marta and Danescu-Niculescu-Mizil, Cristian and Jurafsky, Dan},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2013},
  url = {https://aclanthology.org/P13-1162/}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of ACL-HLT 2011 Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, pages 1–9. find
  2. Jonathan Berant, Ido Dagan, Meni Adler, and Jacob Goldberger. 2012. Efficient tree-based approximation for entailment graph learning. In Proceedings of ACL 2012, pages 117–125. find
  3. Ewa Callahan and Susan C. Herring. 2011. Cultural bias in Wikipedia articles about famous persons. Journal of the American Society for Information Science and Technology, 62(10):1899–1915. find
  4. Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. Hedge detection as a lens on framing in the GMO debates: a position paper. In Proceedings of the ACL-2012 Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 70–79. find
  5. Alexander Conrad, Janyce Wiebe, and Rebecca Hwa. 2012. Recognizing arguing subjectivity and argument tags. In Proceedings of ACL-2012 Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 80–88. find
  6. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC 2006. find our card
  7. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? The pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301– 333. find our card
  8. Robert M. Entman. 2007. Framing bias: Media in the distribution of power. Journal of Communication, 57(1):163–173. find
  9. Viola Ganter and Michael Strube. 2009. Finding hedges by chasing weasels: Hedge detection using Wikipedia tags and shallow linguistic features. In Proceedings of ACL-IJCNLP 2009, pages 173–176. find our card
  10. Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of NAACL-HLT 2009, pages 503– 511. find
  11. Joan B. Hooper. 1975. On assertive predicates. In J. Kimball, editor, Syntax and Semantics, volume 4, pages 91–124. Academic Press, New York. find
  12. Ken Hyland. 2005. Metadiscourse: Exploring Interaction in Writing. Continuum, London and New York. find
  13. Lauri Karttunen. 1971. Implicative verbs. Language, 47(2):340–358. find
  14. Paul Kiparsky and Carol Kiparsky. 1970. Fact. In M. Bierwisch and K. E. Heidolph, editors, Progress in Linguistics, pages 143–173. Mouton, The Hague. find
  15. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proceedings of CoNLL 2006, pages 109–116. find
  16. Chenghua Lin, Yulan He, and Richard Everson. 2011. Sentence subjectivity detection with weakly-supervised learning. In Proceedings of AFNLP 2011, pages 1153–1161. find
  17. Bing Liu, Minqing Hu, and Junsheng Cheng. 2005. Opinion Observer: analyzing and comparing opinions on the Web. In Proceedings of WWW 2005, pages 342–351. find
  18. Aurélien Max and Guillaume Wisniewski. 2010. Mining naturally-occurring corrections and paraphrases from Wikipedia’s revision history. In Proceedings of LREC 2010, pages 3143–3148. find
  19. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: the new generation of linguistic data. In Proceedings of the NAACL-HLT 2010 Workshop on Creating Speech and Language Data Wi… find
  20. Rani Nelken and Elif Yamangil. 2008. Mining Wikipedias article revision history for training Computational Linguistics algorithms. In Proceedings of the 1st AAAI Workshop on Wikipedia and Artificial Intelligence. find
  21. Souneil Park, KyungSoon Lee, and Junehwa Song. 2011. Contrasting opposing views of news articles on contentious issues. In Proceedings of ACL 2011, pages 340–349. find
  22. Ellen Riloff and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proceedings of EMNLP 2003, pages 105–112. find our card
  23. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. find our card
  24. Peter D. Turney. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of ACL 2002, pages 417–424. find our card
  25. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational Linguistics, 30(3):277–308. find our card
  26. Wikipedia. 2013a. Wikipedia: Manual of style / Words to watch. http://en.wikipedia.org/wiki/Wikipedia:Words_to_avoid. [Retrieved February 5, 2013]. ↗
  27. Wikipedia. 2013b. Wikipedia: Neutral point of view. http:http://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view. [Retrieved February 5, 2013]. find
  28. Tae Yano, Philip Resnik, and Noah A. Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL-HLT 2010 Workshop on Creating Speech and Language Data With Amazons Mechanical Turk, pages 152–158. find
  29. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In Proceedings of NAACL-HLT 2010, pages 365–368. find our card
  30. Fabio M. Zanzotto and Marco Pennacchiotti. 2010. Expanding textual entailment corpora from Wikipedia using co-training. In Proceedings of the 2nd Coling Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources, pages 28–36. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/recasens-2013-linguistic-models-analyzing-detecting-biased/