Elamur Research⌕
☰
home / papers / 2015 / The Media Frames Corpus: Annotations of Frames Across Issues

The Media Frames Corpus: Annotations of Frames Across Issues

Summary

Social scientists usually study media framing with codebooks built for a single issue. This makes it hard to compare framing across topics, and earlier computational work mostly reduced framing to one dimension or one domain. The authors introduce the Media Frames Corpus (version 1), which contains several thousand U.S. newspaper articles on immigration, smoking, and same-sex marriage. The articles are annotated with 15 issue-independent "framing dimensions" from Boydstun et al. (2014), such as Economic, Morality, and Political. Undergraduate annotators tagged every dimension present in an article, highlighted the text spans that cue each one (spans may overlap), and chose a primary frame for the headline and for the whole article. Annotation proceeded in three stages, and multiple coding increased over time. Agreement was measured with Krippendorff's α on primary frames and with its unitizing variant α_U on spans. Span-level agreement is low but consistently above chance, and agreement rose as annotators gained experience. The annotations are released publicly, giving NLP researchers a shared resource for modeling framing across policy issues.

Key points

  • The articles were retrieved via Lexis-Nexis from 13 national U.S. newspapers published between 1990 and 2012. Immigration coverage goes back to 1969, though few articles predate 1980. Duplicates and near-duplicates were removed before random sampling.
  • The annotation was done in three stages, with the average number of annotators per article shown in parentheses:
    • Stage 1: 4,113 immigration and 4,077 smoking articles (about 1.2 annotators each).
    • Stage 2: 6,298 same-sex marriage articles (about 2.2 annotators each).
    • Stage 3: 5,549 immigration articles (about 2.2 annotators each).
  • In total, 19 undergraduate annotators took part. Across annotators, the average number of frames per article ranged from 2.0 to 3.7, and the average number of highlighted spans ranged from 3.4 to 10.0. The "Other" dimension appeared in fewer than 10% of cases.
  • Spans reached an average α_U of 0.16 in the stage 1 immigration articles, 0.23 in the tobacco articles, 0.08 in the same-sex marriage articles, and 0.20 in the stage 3 immigration articles.
  • Annotations, metadata, and tools for re-acquiring the licensed texts are available at github.com/dallascard/media_frames_corpus.

Interesting ideas

  • I1. A shared label set across topics. Using a single cross-cutting label set instead of issue-specific codebooks makes results comparable across topics. The same approach could suit any multi-domain annotation effort, for example tracking how Ukrainian media frame trade versus security.
  • I2. Disagreement as signal. The authors treat some annotator disagreement as legitimate differences in interpretation rather than noise, and they keep multiple annotations instead of forcing a gold label. This matters for subjective tasks, where collapsing labels to one answer can erase real variation.
  • I3. Documenting how the data was made. The release openly preserves process effects, such as annotators improving with practice and guidelines changing over time. Recording these effects alongside the data lets downstream users account for drift instead of assuming a uniform gold standard.
  • I4. Agreement for spans with fuzzy boundaries. α_U scores span agreement using the squared lengths of non-overlapping segments, and it is reported together with a simpler document-level α. Pairing a strict span metric with a coarse label metric gives a more honest picture for unitizing tasks such as NER or argument mining.
  • I5. Publishing under license constraints. Rather than distributing copyrighted texts, the authors release annotations and offsets plus scripts that rebuild the corpus for anyone holding a license. This is a practical model for sharing news-based datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

140 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 135 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-2072

@inproceedings{card2015media,
  title = {{The Media Frames Corpus: Annotations of Frames Across Issues}},
  author = {Card, Dallas and Boydstun, Amber E. and Gross, Justin H. and Resnik, Philip and Smith, Noah A.},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-2072},
  url = {https://doi.org/10.3115/v1/p15-2072}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amr Ahmed and Eric P. Xing. 2010. Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective. In Proc. of EMNLP. find
  2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  3. Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proc. of NAACL. find
  4. Frank R. Baumgartner, Suzanna L. De Boef, and Amber E. Boydstun. 2008. The decline of the death penalty and the discovery of innocence. Cambridge University Press. find
  5. Robert D. Benford and David A. Snow. 2000. Framing processes and social movements: An overview and assessment. Annual Review of Sociology, 26:611– 639. find
  6. Adam J. Berinsky and Donald R. Kinder. 2006. Making sense of issues through media frames: Understanding the Kosovo crisis. Journal of Politics, 68(3):640–656. find
  7. Amber E. Boydstun, Dallas Card, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2014. Tracking the development of media frames within and across policy issues. find
  8. APSA 2014 Annual Meeting Paper. find
  9. Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. Hedge detection as a lens on framing in the GMO debates: A position paper. In Proc of. ACL Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 70–79. find
  10. Dennis Chong and James N. Druckman. 2007. Framing theory. Annual Review of Political Science, 10(1):103–126. find
  11. Robert M. Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51–58. find
  12. William A. Gameson and Andre Modigliani. 1989. Media discourse and public opinion on nuclear power: A constructionist approach. American Journal of Sociology, 95(1):1–37. find
  13. Matthew Gentzkow and Jesse M. Shapiro. 2010. What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1):35–71. find
  14. Eric Hardisty, Jordan L. Boyd-Graber, and Philip Resnik. 2010. Modeling perspective using adaptor grammars. In Proc. of EMNLP. find
  15. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proc. of IJCNLP. find
  16. Mohit Iyyer, Peter Enns, Jordan L. Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proc. of ACL. find our card
  17. Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2010. Vocabulary choice as an indicator of perspective. In Proc. of ACL. find
  18. Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. Quality and Quantity, 38(6):787–800. find
  19. Klaus Krippendorff. 2012. Content Analysis: An Introduction to its Methodology. SAGE Publications. find
  20. Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In Proc. of KDD. find
  21. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proc. of CoNNL. find
  22. Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, and Kristina Miler. 2015. Tea party in the house: A hierarchical ideal point topic model and its application to Republican legislators in the 112th congress. In Proc. of ACL. find
  23. Zhongdang Pan and Gerald M. Kosicki. 1993. Framing analysis: An approach to news discourse. Political communication, 10(1):55–75. find
  24. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. In Proc. of ACL. find our card
  25. Stephen D. Rees, Oscar H. Gandy Jr., , and August E. Grant, editors. 2001. Framing public life: Perspectives on media and our understanding of the social world. Routledge. find
  26. Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proc. of EMNLP. find
  27. David A. Smith, Ryan Cordell, and Elizabeth Maddock Dillon. 2013. Infectious texts: modeling text reuse in nineteenth-century newspapers. In IEEE International Conference on Big Data. find
  28. David A. Smith, Ryan Cordel, Elizabeth Maddock Dillon, Nick Stramp, and John Wilkerson. 2014. Detecting and modeling local text reuse. In IEEE/ACM Joint Conference on Digital Libraries. find
  29. Nayda Terkildsen and Frauke Schnell. 1997. How media frames move public opinion: An analysis of the women’s movement. Political research quarterly, 50(4):879–900. find
  30. Oren Tsur, Dan Calacci, and David Lazer. 2015. Frame of mind: Using statistical models for detection of framing and agenda setting campaigns. In Proc. of ACL. find
  31. Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proc. of NAACL. find our card
  32. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational Linguistics, 30(3):277–308. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/card-2015-media-frames-corpus-annotations-frames/