Elamur Research⌕
☰
home / papers / 2013 / How Noisy Social Media Text, How Diffrnt Social Media Sources?

How Noisy Social Media Text, How Diffrnt Social Media Sources?

Summary

Social media text is often called "noisy," but this claim had rarely been measured, and work usually looked at one platform at a time. Baldwin and colleagues compare several social media sources with one another and with edited text. They build corpora from Twitter (two time snapshots), YouTube comments, vBulletin forums, blogs (Spinn3r) and English Wikipedia, and use the written BNC as a reference. All sources go through the same pipeline: language identification with langid.py, sentence splitting, then tokenisation and POS tagging with TweetNLP. The authors then compare language mix, lexical properties, grammaticality, and two kinds of corpus similarity (Kilgarriff's χ² measure and trigram language model perplexity).

The sources fall along a continuum, with Twitter at one end, Wikipedia at the other, and forums in the middle. Social media is less grammatical than edited text, but the gap is modest. Standard NLP steps such as language ID, normalisation and tagging reduce much of the noise. This makes social media look more tractable for NLP than its reputation suggests, and it gives guidance on choosing training data.

Key points

  • Datasets:
    • TWITTER-1 (22 Sep 2011) and TWITTER-2 (22 Feb 2012), plus FORUMS and BLOGS: 1M documents each.
    • COMMENTS: 874,772 documents.
    • WIKIPEDIA: 200,000 documents.
    • BNC: 3,141 documents.
  • Average document length (words): about 12 for tweets, 15.8 for comments, 23.2 for forums, 147.7 for blogs, 281.2 for Wikipedia, and over 31K for BNC.
  • English share (langid.py):
    • Twitter: only 40.6% and 43.9%, with Japanese second (14.4% and 12.4%).
    • Comments 75.7%, blogs 78.4%, forums 91.4%, Wikipedia 99.8%, BNC 100%.
  • Comparison setup: each corpus was sampled to 5M tokens and split into five 1M-token sub-corpora. Tokens that TweetNLP tags as #, @, ~, U or E were removed as non-linguistic.
  • χ² results:
    • The two Twitter snapshots differ very little (4.0 ×10³), while each is about 350 ×10³ away from Wikipedia.
    • Twitter is the most homogeneous social media corpus, and blogs are the most varied.
    • In the perplexity tests, FORUMS was the most "central" source, modelling the other corpora well except Wikipedia.

Interesting ideas

  • I1. Fixed-size sub-corpora: sampling equal token budgets and splitting them into several sub-corpora lets a measure that needs equal-sized inputs (Kilgarriff's χ²) compare corpora fairly. The same split also yields a self-similarity (homogeneity) score. This would work for comparing Ukrainian corpora of very different sizes.
  • I2. Cross-corpus perplexity matrix: training a language model on each source and testing it on every other source shows which single source transfers best. That is a cheap guide for picking training data before building taggers or parsers.
  • I3. Tagger-based filtering of platform tokens: using a tagger that tells a hashtag inside a sentence from a trailing tag lets you drop non-linguistic tokens without losing real words. This is better than deleting every hashtag or mention by regex.
  • I4. Two time snapshots of one platform: collecting the same source at two dates gives a direct check of how time-specific the data is. Here it showed Twitter to be stable at the lexical level.
  • I5. Sanity-checking language ID: running language ID on corpora known to be monolingual (BNC, Wikipedia) and manually inspecting the odd predictions helps validate it. In Wikipedia these turned out to be lists of foreign names, so the tags were actually correct. A related small trick is stripping invisible characters such as U+FEFF before detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baldwin2013how,
  title = {{How Noisy Social Media Text, How Diffrnt Social Media Sources?}},
  author = {Baldwin, Timothy and Cook, Paul and Lui, Marco and MacKinlay, Andrew and Wang, Li},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1041/}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Adolphs, Stephan Oepen, Ulrich Callmeier, Berthold Crysmann, Dan Flickinger, and Bernd Kiefer. 2008. Some fine points of hybrid natural language parsing. In European Language Resources Association (ELRA), editor, Proc. of the 6th International Conference on Language Resources and Evaluation (L… find
  2. Timothy Baldwin, Emily M. Bender, Dan Flickinger, Ara Kim, and Stephan Oepen. 2005. Beauty and the beast: What running a broad-coverage precision grammar over the BNC taught us about the grammar — and the corpus. In Stephan Kepser and Marga Reis, editors, Linguistic Evidence: Empirical, Theoretical,… find
  3. Hila Becker, Mor Naaman, and Luis Gravano. 2009. Event identification in social media. In Proceedings of the 12th International Workshop on the Web and Databases (WebDB 2009), Providence, USA. find
  4. Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 389–398, Portland, USA. find
  5. Lou Burnard. 2000. User Reference Guide for the British National Corpus. Technical report, Oxford University Computing Services. find
  6. Kevin Burton, Niels Kasch, and Ian Soboroff. 2011. find
  7. The ICWSM 2011 Spinn3r dataset. In Proceedings of the 5th International Conference on Weblogs and Social Media (ICWSM 2011), Barcelona, Spain. find
  8. Ulrich Callmeier. 2002. PET – a platform for experimentation with efficient HPSG processing techniques. In Stephan Oepen, Dan Flickinger, Jun’ichi Tsujii, and Hans Uszkoreit, editors, Collaborative Language Engineering. CSLI Publications, Stanford, USA. find
  9. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Proc. of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1277–1287, Cambridge, MA. find
  10. Jacod Eisenstein, Brendan O’Connor, Noad A. Smith, and Eric P. Xing. 2012. Mapping the geographical diffusion of new words. Arxiv preprint arXiv, 1210.5268. find
  11. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2013), pages 359–369, Atlanta, USA. find our card
  12. Dan Flickinger, Stephan Oepen, Hans Uszkoreit, and Jun’ichi Tsujii. 2000. On building a more efficient grammar by exploiting types. Journal of Natural Language Engineering (Special Issue on Efficient Processing with HPSG), 6(1):15–28. find our card
  13. Dan Flickinger, Stephan Oepen, and Gisle YtrestØol. 2010. WikiWoods: Syntacto-semantic annotation for English wikipedia. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta. find
  14. Jennifer Foster, Ozlem Cetinoglu, Joachim Wagner, Joseph Le Roux, Joakim Nivre, Deirdre Hogan, and Josef van Genabith. 2011. From news to comment: Resources and benchmarks for parsing the language of web 2.0. In Proc. of the 5th International Joint Conference on Natural Language Processing, pages 89… find
  15. Stephan Gouws, Dirk Hovy, and Donald Metzler. 2011. Unsupervised mining of lexical variants from noisy text. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 82–90, Edinburgh, UK. find
  16. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a \#twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL HLT 2011), pages 368–378, Portland, USA. find our card
  17. Bo Han, Paul Cook, and Timothy Baldwin. 2012a. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning 2012 (EMNLP-CoNLL 2012), pages 421–432, Jeju, Ko… find
  18. Bo Han, Paul Cook, and Timothy Baldwin. 2012b. Geolocation prediction in social media data by finding location indicative words. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pages 1045–1062, Mumbai, India. find
  19. Bo Han, Paul Cook, and Timothy Baldwin. 2013. A stacking-based approach to twitter user geolocation prediction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013): System Demonstrations, pages 7–12, Sofia, Bulgaria. find
  20. Lichan Hong, Gregorio Convertino, and Ed H. Chi. 2011. Language matters in Twitter: A large scale study. In Proceedings of the 5th International Conference on Weblogs and Social Media (ICWSM 2011), Barcelona, Spain. find
  21. Yuheng Hu, Kartik Talamadupula, and Subbarao Kambhampati. 2013. Dude, srsly?: The surprisingly formal nature of Twitters language. In Proceedings of the 7th International Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA. find
  22. Akshay Java. 2007. A framework for modeling influence, opinions and structure in social media. In Proceedings of the 22nd Annual Conference on Artificial Intelligence (AAAI-07), pages 1933–1934. find
  23. Adam Kilgarriff. 2001. Comparing corpora. International Journal of Corpus Linguistics, 6(1):97–133. find
  24. Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012) Demo Session, pages 25–30, Jeju, Republic of Korea. find our card
  25. Derek O’Callaghan, Martin Harrigan, Joe Carthy, and Pádraig Cunningham. 2012. Network analysis of recurring YouTube spam campaigns. In Proceedings of the 6th International Conference on Weblogs and Social Media (ICWSM 2012), pages 531–534, Dublin, Ireland. find
  26. Olutobi Owoputi, Brendan OConnor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguisti… find our card
  27. Sasa Petrovic, Miles Osborne, and Victor Lavrenko. 2012. Using paraphrases for improving first story detection in news and Twitter. In Proc. of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338–346, Montréal, C… find
  28. Daniel Preotiuc-Pietro, Sina Samangooei, Trevor Cohn, Nicholas Gibbins, and Mahesan Niranjan. 2012. Trendminer: An architecture for real time analysis of social media text. In Proceedings of the ICWSM 2013 Workshop on Real-Time Analysis and Mining of Social Streams, Dublin, Ireland. find
  29. Jonathon Read, Rebecca Dridan, Stephan Oepen, and Lars JØorgen Solberg. 2012a. Sentence boundary detection: A long solved problem? In Proceedings of COLING 2012: Posters, pages 985–994, Mumbai, India. find our card
  30. Jonathon Read, Dan Flickinger, Rebecca Dridan, Stephan Oepen, and Lilja Øvrelid. 2012b. The WeSearch corpus, treebank, and treecache – a comprehensive sample of user-generated content. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC 2012), pages 1829–183… find
  31. Luz Rello and Ricardo Baeza-Yates. 2012. Social media is NOT that bad! the lexical quality of social media. In Proceedings of the 6th International Conference on Weblogs and Social Media (ICWSM 2012), Dublin, Ireland. find
  32. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), pages 1524–1534, Edinburgh, UK. find our card
  33. Alan Ritter, Mausam, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from Twitter. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1104– 1112, Beijing, China. find
  34. Stephen Roller, Michael Speriosu, Sarat Rallapalli, Benjamin Wing, and Jason Baldridge. 2012. Supervised text-based geolocation using language models on an adaptive grid. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Lea… find
  35. Semiocast. 2010. Half of messages on twitter are not in English — Japanese is the second most used language. Technical report, Semiocast. find
  36. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proc. of the International Conference on Spoken Language Processing, volume 2, pages 901–904, Denver, USA. find
  37. Benjamin Wing and Jason Baldridge. 2011. Simple supervised document geolocation with geodesic grids. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 955–964, Portland, USA. find
  38. Jie Yin, Andrew Lampert, Mark Cameron, Bella Robinson, and Robert Power. 2012. Using social media to enhance emergency situation awareness. IEEE Intelligent Systems, 27(6):52–59. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baldwin-2013-how-noisy-social-media-text/