Elamur Research⌕
☰
home / papers / 2013 / What to do about bad language on the internet

What to do about bad language on the internet

Summary

This survey asks why language on social media departs so sharply from the edited text NLP systems are trained on. It also asks whether the field's two standard responses fit what researchers in computer-mediated communication know about online writing. Those responses are normalization (mapping non-standard forms to standard ones) and domain adaptation (treating social media as a target domain). Eisenstein reviews possible explanations for non-standard writing: illiteracy, length limits, input devices, pragmatic functions, and social identity. He tests the length-limit explanation on one million English tweets. He also measures lexical similarity between several Twitter samples, political blogs, Infinite Jest and the Penn Treebank using out-of-vocabulary bigram rates. Shortened forms turn out to appear in shorter tweets, not near the 140-character cap. All social media corpora are far from the PTB, but they also differ among themselves and change over time. He argues that normalization wrongly treats one variety as the norm, and that domain adaptation wrongly treats a medium as a single domain. He proposes using author metadata to guide more flexible, socially aware adaptation.

Key points

  • Performance drops sharply on Twitter:
    • The Stanford POS tagger falls from about 97% on WSJ to about 85%.
    • The CoNLL-trained Stanford NER falls from 86% to 44% F-measure.
    • Four parsers lose accuracy by double digits.
  • In a one-million-tweet dataset, messages containing ur, wit, goin, kno and bout are shorter on average than those containing the standard forms. For example, tweets with "bout" average about 74.5 characters, versus about 88.9 for "about". The gap is far larger than the one or two characters a shortening saves.
  • Existing literacy studies show that writers of non-standard text usually also command formal registers. Many non-standard features correlate with age, gender, race and location, and some (ctfu, hella, finna) carry regional or social identity.
  • OOV bigram rates make the Penn Treebank the clear outlier among all corpora compared.
    • Twitter subsets resemble each other most: the maximum is 33.4% between Twitter-# and Twitter-@.
    • Even this exceeds the 27.8% rate between two random Twitter samples from the same month.
    • Blog posts were no closer to the PTB than blog comments.
  • Named entities in tweets were replaced with a special token using the CMU Twitter POS tagger. This keeps changing entity mentions from inflating vocabulary drift.

Interesting ideas

  • I1. Test a folk explanation with a simple corpus statistic. If a constraint such as a character limit really caused a behaviour, the behaviour should cluster where the constraint binds. Checking the average length of messages that contain a form is a cheap falsification test that could be reused for many claimed causes of linguistic variation.
  • I2. Use a cross-corpus OOV matrix as a similarity map. One corpus supplies the dictionary and another supplies the tokens, giving an asymmetric distance measure. A baseline of two samples from the same source shows how much difference counts as large. This is a quick way to decide which source data is closest to a new target before annotating or adapting.
  • I3. Separate subsets of a medium by usage type. Splitting tweets that start with a hashtag from tweets that start with an @-mention shows that one platform contains distinct varieties. The same idea applies to any platform where a single model would otherwise average over very different communicative uses.
  • I4. Mask named entities before measuring vocabulary change. Replacing entities with a placeholder keeps measurements of language change from being dominated by shifting topics and names. This is useful for any study of lexical drift over time.
  • I5. Use metadata as an input to adaptation, not a prediction target. Instead of predicting who an author is from their language, models could condition on social metadata to adapt to many varieties at once. This avoids forcing all text into a single standard or a single target domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

150 papers of our library cite it (of the 140884 whose bibliographies we read); 22 of them have a page here, 128 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{eisenstein2013what,
  title = {{What to do about bad language on the internet}},
  author = {Eisenstein, Jacob},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1037/}
}

References

76 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Noor Ali-Hasan and Lada Adamic. 2007. Expressing social relationships on the blog through links and comments. In Proceedings of ICWSM. find
  2. American Society of Newspaper Editors. 1999. 1999 Newsroom Census: Minority Employment Inches up in Daily Newspapers. American Society of Newspaper Editors, Reston, VA. find
  3. Jannis Androutsopoulos. 2011. Language change and digital media: a review of conceptions and evidence. In Nikolas Coupland and Tore Kristiansen, editors, Standard Languages and Language Standards in a Changing Europe. Novus, Oslo. find
  4. Jacques Anis. 2007. Neography: Unconventional spelling in French SMS text messages. In Brenda Danet and Susan C. Herring, editors, The multilingual internet: Language, culture, and communication online, pages 87 – 115. Oxford University Press. find
  5. Shlomo Argamon, Moshe Koppel, James W. Pennebaker, and Jonathan Schler. 2007. Mining the blogosphere: age, gender, and the varieties of self-expression. First Monday, 12(9). find
  6. AiTi Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for SMS text normalization. In Proceedings of ACL, pages 33–40. find
  7. Lars Backstrom, Eric Sun, and Cameron Marlow. 2010. Find me if you can: improving geographical prediction with social and spatial proximity. In Proceedings of WWW, pages 61–70. find
  8. David Bamman, Jacob Eisenstein, and Tyler Schnoebelen. 2012. Gender in twitter: Styles, stances, and social networks. Technical Report 1210.4567, arXiv, October. find
  9. Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In Proceedings of ACL. find
  10. Shane Bergsma, Paul McNamee, Mossaab Bagdouri, Clayton Fink, and Theresa Wilson. 2012. Language identification for creating language-specific twitter Blog-body 47.1 47.8 49.6 find
  11. Blog-comment 48.6 49.9 51.0 27.2 Infinite-Jest 54.0 56.3 54.7 49.1 53.0 PTB 63.9 66.4 66.2 48.0 48.4 55.5 30.2 48.3 64.5 47.4 61.9 71.9 collections. In Proceedings of the Second Workshop on Language in Social Media, pages 65–74, June. find
  12. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python. O’Reilly Media, Incorporated. danah boyd and Kate Crawford. 2012. Critical questions for big data. Information, Communication & Society, 15(5):662–679, May. find
  13. Samuel Brody and Nicholas Diakopoulos. 2011. Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!: using word lengthening to detect sentiment in microblogs. In Proceedings of EMNLP. find
  14. Mary Bucholtz and Kira Hall. 2005. Identity and interaction: A sociocultural linguistic approach. Discourse studies, 7(4-5):585–614. find
  15. John D. Burger and John C. Henderson. 2006. An exploration of observable features related to blogger age. In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs. find
  16. John D. Burger, John C. Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on twitter. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  17. Monojit Choudhury, Rahul Saraf, Vijit Jain, Animesh Mukherjee, Sudeshna Sarkar, and Anupam Basu. 2007. Investigation and modeling of the structure of texting language. International Journal on Document Analysis and Recognition, 10(3):157–174. find
  18. Lauren B. Collister. 2011. *-repair in online discourse. Journal of Pragmatics, 43(3):918–921, February. find
  19. Lauren B. Collister. 2012. The discourse deictics ˆ and \<-- in a world of warcraft community. Discourse, Context & Media, 1(1):9–19, March. find
  20. Paul Cook and Suzanne Stevenson. 2009. An unsupervised model for text message normalization. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Linguistic Creativity, pages 71– 78. find
  21. Nikolas Coupland. 2007. Style (Key Topics in Sociolinguistics). Cambridge University Press, July. find
  22. William M. Darling, Michael J. Paul, and Fei Song. 2012. Unsupervised part-of-speech tagging in noisy and esoteric domains with a syntactic-semantic bayesian hmm. In Proceedings of EACL Workshop on Semantic Analysis in Social Media. find
  23. Eli Dresner and Susan C. Herring. 2010. Functions of the non-verbal in cmc: Emoticons and illocutionary force. Communication Theory, 20(3):249–268. find
  24. Michelle Drouin and Claire Davis. 2009. R u txting? is the use of text speak hurting your literacy? Journal of Literacy Research, 41(1):46–67. find
  25. John W. Du Bois. 2007. The stance triangle. In Robert Engelbretson, editor, Stancetaking in discourse, pages 139–182. John Benjamins Publishing Company, Amsterdam/Philadelphia. find
  26. Penelope Eckert and Sally McConnell-Ginet. 2003. Language and Gender. Cambridge University Press, New York. find
  27. Penelope Eckert. 2000. Linguistic variation as social practice. Blackwell. find
  28. Penelope Eckert. 2008. Variation and the indexical field. Journal of Sociolinguistics, 12(4):453–476. find
  29. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Proceedings of EMNLP. find
  30. Jacob Eisenstein, Noah A. Smith, and Eric P. Xing. 2011. Discovering sociolinguistic associations with structured sparsity. In Proceedings of ACL. find
  31. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2012. Mapping the geographical diffusion of new words. Technical Report 1210.5268, arXiv. find
  32. Jacob Eisenstein. 2013. Phonological factors in social media writing. In Proceedings of the NAACL Workshop on Language Analysis in Social Media. find
  33. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  34. Jenny R. Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of ACL, pages 363–370. find our card
  35. Jennifer Foster, Ozlem Cetinoglu, Joachim Wagner, Joseph Le Roux, Joakim Nivre, Deirdre Hogan, and Josef van Genabith. 2011. From news to comment: Resources and benchmarks for parsing the language of web 2.0. In Proceedings of IJCNLP. find
  36. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: annotation, features, and experiments. In Proceedings of ACL. find our card
  37. Scott A. Golder and Michael W. Macy. 2011. Diurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures. Science, 333(6051):1878–1881, September. find
  38. Stephan Gouws, Dirk Hovy, and Donald Metzler. 2011a. Unsupervised mining of lexical variants from noisy text. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 82–90, July. find
  39. Stephan Gouws, Donald Metzler, Congxing Cai, and Eduard Hovy. 2011b. Contextual bearing on linguistic variation in social media. In Proceedings of the ACL Workshop on Language in Social Media. find
  40. Lisa Green. 2002. African American English. Cambridge University Press. find
  41. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a\# twitter. In Proceedings of ACL, volume 1. find our card
  42. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of EMNLP. find
  43. Yuheng Hu, Kartik Talamadupula, and Subbarao Kambhampati. 2013. Dude, srsly?: The surprisingly formal nature of twitter’s language. In Proceedings of ICWSM. find
  44. Barbara Johnstone, Jennifer Andrus, and Andrew E Danielson. 2006. Mobility, indexicality, and the enregisterment of pittsburghese. Journal of English Linguistics, 34(2):77–104. find
  45. Lucy Jones. 2010. The changing face of spelling on the internet. Technical report, The English Spelling Society. find
  46. William Labov. 1972. Sociolinguistic patterns. Philadelphia: University of Pennsylvania Press. find
  47. William Labov. 2001. Principles of linguistic change. Vol.2 : Social factors. Blackwell Publishers, Oxford. find
  48. Fei Liu, Fuliang Weng, Bingqing Wang, and Yang Liu. 2011a. Insertion, deletion, or substitution? normalizing text messages without pre-categorization nor supervision. In Proceedings of ACL, pages 71–76. find
  49. Xiaohua Liu, Shaodian Zhang, Furu Wei, and Ming Zhou. 2011b. Recognizing named entities in tweets. In Proceedings of ACL. find
  50. Xiaohua Liu, Ming Zhou, Xiangyang Zhou, Zhongyang Fu, and Furu Wei. 2012. Joint inference of named entity recognition and normalization for tweets. In Proceedings of ACL. find
  51. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Proceedings of NAACL, pages 28–36, June. find
  52. Robert Munro and Christopher D. Manning. 2012. Short message communications: users, topics, and in-language processing. In Proceedings of the 2nd ACM Symposium on Computing for Development. find
  53. Mor Naaman, Hila Becker, and Luis Gravano. 2011. Hip and trendy: Characterizing emerging trends on twitter. Journal of the American Society for Information Science and Technology, 62(5):902–918. find
  54. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL. find our card
  55. John C. Paolillo. 1996. Language choice on soc.culture.punjab. Electronic Journal of Communication/La Revue Electronique de Communication, 6(3). find
  56. John C. Paolillo. 1999. The virtual speech community: Social network and language variation on irc. Journal of Computer-Mediated Communication, 4(4):0. find
  57. John C. Paolillo. 2001. Language variation on internet relay chat: A social network approach. Journal of Sociolinguistics, 5(2):180–213. find
  58. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  59. Daniel Ramage, Sue Dumais, and D. Liebling. 2010. Characterizing microblogs with topic models. In Proceedings of ICWSM. find
  60. Delip Rao, David Yarowsky, Abhishek Shreevats, and Manaswi Gupta. 2010. Classifying latent user attributes in twitter. In Proceedings of Workshop on Search and mining user-generated contents. find
  61. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Proceedings of NAACL. find our card
  62. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of EMNLP. find our card
  63. Sara Rosenthal and Kathleen McKeown. 2011. Age prediction in blogs: A study of style, content, and online behavior in pre- and Post-Social media generations. In Proceedings of ACL. find
  64. Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of WWW, pages 851–860. find
  65. Christina Sauper, Aria Haghighi, and Regina Barzilay. 2011. Content models with attitude. In Proceedings of ACL. find
  66. Beaux Sharifi, Mark-Anthony Hutton, and Jugal Kalita. 2010. Summarizing microblogs automatically. In Proceedings of NAACL. find
  67. Aaron Smith and Joanna Brewer. 2012. Twitter use 2012. Technical report, Pew Research Center, May. find
  68. Richard Sproat, Alan W Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christopher Richards. 2001. Normalization of non-standard words. Computer Speech & Language, 15(3):287–333. find
  69. Sali A. Tagliamonte and Derek Denis. 2008. Linguistic ruin? lol! instant messaging and teen language. American Speech, 83(1):3–34, March. find
  70. Mike Thelwall. 2009. Homophily in MySpace. J. Am. Soc. Inf. Sci., 60(2):219–231. find
  71. Crispin Thurlow. 2006. From statistical panic to moral panic: The metadiscursive construction and popular exaggeration of new media language in the print media. J. Computer-Mediated Communication, pages 667–701. find
  72. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of NAACL. find our card
  73. Byron Wallace. 2012. Multiple narrative disentanglement: Unraveling infinite jest. In Proceedings of NAACL. find
  74. Joseph B. Walther and Kyle P. D’Addario. 2001. The impacts of emoticons on message interpretation in computer-mediated communication. Social Science Computer Review, 19(3):324–347. find
  75. Benjamin Wing and Jason Baldridge. 2011. Simple supervised document geolocation with geodesic grids. In Proceedings of ACL. find
  76. Tae Yano, William W. Cohen, and Noah A. Smith. 2009. Predicting response to political blog posts with topic models. In Proceedings of NAACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/