Elamur Research⌕
☰
home / papers / 2013 / What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain

What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain

Paper · 2013 · Annie Louis and Ani Nenkova

Summary

Search and recommendation tools can already judge spelling, grammar, coherence, topical relevance and readability. They cannot yet recognize writing that is genuinely excellent: engaging, vivid and well crafted. Louis and Nenkova address this gap for science journalism. They build a corpus from the New York Times Annotated Corpus. Articles by writers featured in "The Best American Science Writing" anthologies are labeled VERY GOOD, and science articles by other authors are labeled TYPICAL. A topic-matched version of the corpus pairs each VERY GOOD article with similar TYPICAL ones.

The authors design genre-specific features for six facets of science writing: visual imagery, focus on people, beautiful language, sub-genre, sentiment, and depth of research description. They combine these with established readability, discourse-coherence, fiction-interest and bag-of-words features in SVM classifiers.

The genre-specific features alone beat general writing measures. Merging writing and content features reaches about 82–84% accuracy. This suggests that quality at this level can be modeled automatically and that topic and style contribute complementary signals.

Key points

  • The corpus covers NYT articles from 1999–2007. Of 23,709 science articles, 3,530 are VERY GOOD (from 63 anthology seed articles by about 40 authors) and 20,242 are TYPICAL. The topic-paired corpus has 35,300 pairs.
  • There are two tasks: "any topic" classification on a balanced test set, and "same topic" pairwise choice of the better article. The chance baseline for both is 50%.
  • The 41 interest-science features score 75.3% on any topic and 68.0% on same topic. This exceeds the readability (Pitler and Nenkova, 2008), Entity Grid/discourse, and McIntyre and Lapata (2009) fiction-interest feature sets.
  • All 102 writing features together score 76.7% and 74.7%. Bag-of-words with 1000 words scores 81.2% and 82.1%.
  • Combining writing and word features in one vector gives 82.6% and 84.0%, a significant gain over words alone. An oracle combination bound reaches 87.6% and 93.8%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Annie Louis and Ani Nenkova. 2013. What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00232

@article{louis2013what,
  title = {{What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain}},
  author = {Annie Louis and Ani Nenkova},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00232},
  url = {https://doi.org/10.1162/tacl_a_00232}
}

References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. The berkeley framenet project. In Proceedings of COLING-ACL, pages 86–90.
  2. Z. Bao, B. Kimelfeld, and Y. Li. 2011. A graph approach to spelling correction in domain-centric search. In Proceedings of ACL-HLT, pages 905–914. find
  3. R. Barzilay and M. Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34.
  4. R. Barzilay, N. Elhadad, and K. McKeown. 2002. Inferring strategies for sentence ordering in multi-document summar ization. Journal of Artificial Intelligence Research, 17:35–55. find
  5. D.M. Blei, A.Y. Ng, and M.I. Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research, 3:993–1022. D. Blum, M. Knudson, and R. M. Henig, editors. 2006. A field guide for science writers: the official guide of the national association of science writers.
  6. Oxford University Press, New York. E. Brill and R.C. Moore. 2000. An improved error model for noisy channel spelling correction. In Proceedings of ACL, pages 286–293.
  7. S. F. Chen and J. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL, pages 310–318.
  8. S. Cucerzan and E. Brill. 2004. Spelling correction as an iterative process that exploits the collective knowledge of web users. In Proceedings of EMNLP, pages 293– 300. find
  9. R. Dale and A. Kilgarriff. 2010. Helping our own: text massaging for computational linguistics as a new shared task. In Proceedings of INLG, pages 263–267. find
  10. M. C. De Marneffe, B. MacCartney, and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, volume 6, pages 449–454.
  11. P. Diederich. 1974. Measuring Growth in English. National Council of Teachers of English. find
  12. M. Elsner, J. Austerweil, and E. Charniak. 2007. A unified local and global model for discourse coherence. In Proceedings of NAACL-HLT, pages 436–443.
  13. J. R. Finkel, T. Grenager, and C. Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of ACL, pages 363–370.
  14. R. Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32:221 – 233. find
  15. A.C. Graesser, D.S. McNamara, M.M. Louwerse, and Z. Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods Instruments and Computers, 36(2):193–202. find
  16. H. Ji and D. Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person name detection. In Proceedings of PACLIC. find
  17. D. Klein and C.D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of ACL, pages 423–430.
  18. S.M. Kosslyn. 1980. Image and mind. find
  19. Harvard University Press. M. Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of ACL, pages 545–552.
  20. C. Lin and E. Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceedings of COLING, pages 495–501.
  21. D. Lin, K. W. Church, H. Ji, S. Sekine, D. Yarowsky, S. Bergsma, K. Patil, E. Pitler, R. Lathbury, V. Rao, K. Dalwani, and S. Narsale. 2010. New tools for web-scale n-grams. In Proceedings of LREC. find
  22. N. McIntyre and M. Lapata. 2009. Learning to tell tales: A data-driven approach to story generation. In Proceedings of ACL-IJCNLP, pages 217–225. find
  23. R. J. Mooney and L. Roy. 2000. Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital libraries, pages 195–204. find
  24. A. Nakhimovsky. 1988. Aspect, aspectual class, and the temporal structure of narrative. Computational Linguistics, 14(2):29–43, June.
  25. M. Pazzani, J. Muramatsu, and D. Billsus. 1996. Syskill & Webert: Identifying interesting web sites. In Proceedings of AAAI, pages 54–61. find
  26. E. Pitler and A. Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of EMNLP, pages 186–195.
  27. E. Pitler and A. Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of ACL-IJCNLP, pages 13–16.
  28. R Development Core Team, 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. find
  29. D. Ramage, D. Hall, R. Nallapati, and C.D. Manning. 2009. Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of EMNLP, pages 248–256.
  30. A. Rozovskaya and D. Roth. 2010. Generating confusion sets for context-sensitive error correction. In Proceedings of EMNLP, pages 961–970. find
  31. E. Sandhaus. 2008. The new york times annotated corpus. Corpus number LDC2008T19, Linguistic Data Consortium, Philadelphia. find
  32. S. Schwarm and M. Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of ACL, pages 523– 530.
  33. H.A. Semetko and P.M. Valkenburg. 2000. Framing european politics: A content analysis of press and television news. Journal of communication, 50(2):93–109. find
  34. V. Spandel. 2004. Creating Writers Through 6-Trait Writing: Assessment and Instruction. find
  35. Allyn and Bacon, Inc. S. H. Stocking. 2010. find
  36. The New York Times Reader: Science and Technology. CQ Press, Washington DC. P. J. Stone, J. Kirsh, and Cambridge Computer Associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. find
  37. MIT Press. J. R. Tetreault and M. Chodorow. 2008. The ups and downs of preposition error detection in esl writing. In Proceedings of COLING, pages 865–872.
  38. L. von Ahn and L. Dabbish. 2004. Labeling images with a computer game. In Proceedings of CHI, pages 319– 326. find
  39. R. L. Weide. 1998. The cmu pronunciation dictionary, release 0.6. http://www.speech.cs.cmu.edu/cgi-bin/cmudict. ↗
  40. T. Wilson, J. Wiebe, and P. Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of HLT-EMNLP, pages 347–354.
  41. M. Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. Behavior Research Methods, 20(1):6–10. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/louis-2013-what-makes-writing-great-first/