Elamur Research
☰
home / papers / 2016 / Demographic Dialectal Variation in Social Media: A Case Study of African-American English

Demographic Dialectal Variation in Social Media: A Case Study of African-American English

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1120.pdf

Summary

Dialects that were traditionally spoken rather than written are scarce in the text used to build NLP tools. They are now common on social media, where standard tools may handle them badly. The authors study African-American English (AAE) on Twitter. They collected geo-located US tweets from 2013 and linked each message to its US Census blockgroup. Each user received an averaged vector of four race/ethnicity proportions from the American Community Survey. A mixed-membership model, fitted with collapsed Gibbs sampling, ties each demographic category to its own unigram language model. Individual messages can still deviate from the author's neighborhood prior. This distant supervision separates AAE-aligned from white-aligned text without manual labeling, and the extracted language shows established AAE phonological and syntactic traits. Off-the-shelf language identification and dependency parsing perform clearly worse on AAE-aligned tweets than on white-aligned ones. A simple ensemble language identifier built on the model removes the identification gap. The work exposes demographic bias in NLP tools and releases a public corpus of AAE-like tweets.

Key points

  • Data pipeline: about 16 billion Gardenhose/Decahose messages from 2013 were reduced to 90 million that matched a Census blockgroup (typically 600–3000 residents). Once non-conversational tweets were filtered out, the final dataset held 59.2 million messages posted by 2.8 million users.
  • Model: the model uses four demographic topics (non-Hispanic white, non-Hispanic black, Hispanic, Asian) over a 191,873-word vocabulary. A Dirichlet prior centered on each user's Census proportions lets authors who mix dialects vary from message to message.
  • Released resource: a corpus of roughly 830,000 tweets aligned with African-American demographics, available at slanglab.cs.umass.edu/TwitterAAE.
  • Parsing gap: on 200 manually annotated messages (Stanford Dependencies), labeled recall was:
    • SyntaxNet: 64.0 on AA-aligned vs 80.4 on white-aligned tweets.
    • Stanford CoreNLP: 50.0 vs 71.0.
    • Both differences are statistically significant.
  • Language ID: langid.py has unequal recall across the two groups, and the ensemble removes this disparity. On the general 2.2M-message set, the baseline labels 88% of messages as English, and the ensemble adds at least 5.4% recall.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

181 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 164 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1120 · arXiv 1608.08868

@inproceedings{blodgett2016demographic,
  title = {{Demographic Dialectal Variation in Social Media: A Case Study of African-American English}},
  author = {Blodgett, Su Lin and Green, Lisa and O'Connor, Brendan},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1120},
  eprint = {1608.08868},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1120}
}

References

46 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042, 2016. ↗ our card
  2. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. How noisy social media text, how diffrnt social media sources? In International Joint Conference on Natural Language Processing, pages 356–364, 2013. find
  3. Shane Bergsma, Paul McNamee, Mossaab Bag-douri, Clayton Fink, and Theresa Wilson. Language identification for creating language-specific Twitter collections. In Proceedings of the Second Workshop on Language in Social Media, pages 65–74. Association for Computational Linguistics, 2012. find
  4. Marianne Bertrand and Sendhil Mullainathan. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. Technical report, National Bureau of Economic Research, 2003. find
  5. Danqi Chen and Christopher Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar, October 2014. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ D14-1082. guage Analysis in Social Media, pages 11–19, 2013. ↗ our card
  6. Jacob Eisenstein. Identifying regional dialects in online social media. In C. Boberg, J. Nerbonne, and D. Watt, editors, Handbook of Dialectology. Wiley, 2015. find
  7. Jacob Eisenstein, Amr Ahmed, and Eric P. Xing. Sparse additive generative models of text. In Proceedings of ICML, pages 1041–1048, 2011a. find
  8. Jacob Eisenstein, Noah A. Smith, and Eric P. Xing. Discovering sociolinguistic associations with structured sparsity. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1365–1374. Association for Computational Linguistics, 2011b. find
  9. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. Aida: Identifying code switching in informal Arabic text. Proceedings of EMNLP 2014, page 94, 2014. find
  10. Jennifer Foster, Ozlem Cetinoglu, Joachim Wagner, Joseph Le Roux, Stephen Hogan, Joakim Nivre, Deirdre Hogan, and Josef van Genabith. #hardtoparse: POS tagging and parsing the Twitterverse. In Proc. of AAAI-11 Workshop on Analysing Microtext, 2011. find
  11. Rayid Ghani, Rosie Jones, and Dunja Mladenić. Mining the web to create minority language corpora. In Proceedings of the Tenth International Conference on Information and Knowledge Management, pages 279–286. ACM, 2001. find
  12. M. C. de Marneffe and C. D. Manning. Stanford typed dependencies manual. Technical report, last revised April 2015 edition, 2008. find
  13. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeff Flanigan, and Noah A. Smith. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 42–47. Association for Computational Linguistics, 2011. find
  14. Gabriel Doyle. Mapping dialectal variation by querying social media. In Proceedings of EACL, pages 98–106, 2014. find
  15. Lisa J. Green. African American English: A Linguistic Introduction. Cambridge University Press, 2002. find
  16. Jacob Eisenstein. Phonological factors in social media writing. In Proc. of the Workshop on Lan-T.L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. An Algerian Arabic-French code-switched corpus. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme, page 34, 2014. find
  17. Sciences of the United States of America, 101 (Suppl 1):5228, 2004. sociation for Computational Linguistics. URL http://www.aclweb.org/anthology/D14-1108. ↗ our card
  18. Dirk Hovy and L. Shannon Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598. Association for Computational Linguistics, 2016. doi: 10.18653/ v1/P16-2096. URL http://aclweb.org/anthology/ P16-2096. ↗ our card
  19. M. Lui and T. Baldwin. langid. py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), Demo Session, Jeju, Republic of Korea, 2012. URL http://www.aclweb.org/ anthology-new/P/P12/P12-3005.pdf. ↗ our card
  20. Yuan Huang, Diansheng Guo, Alice Kasakoff, and Jack Grieve. Understanding US regional linguistic variation with Twitter data analysis. Computers, Environment and Urban Systems, 2015. find
  21. Baden Hughes, Timothy Baldwin, Steven Bird, Jeremy Nicholson, and Andrew MacKinlay. Reconsidering language identification for written language resources. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). European Language Resources Association (ELRA), 2006. URL http://aclweb.org/anthology/L06-1274. ↗
  22. Taylor Jones. Toward a description of African American Vernacular English dialect regions using “Black Twitter”. American Speech, 90(4): 403–440, 2015. find
  23. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. Learning a POS tagger for AAVE-like language. In Proceedings of NAACL. Association for Computational Linguistics, 2016. find our card
  24. Anna Katrine Jørgensen, Dirk Hovy, and Anders Søgaard. Challenges of studying and processing dialects in social media. In Proceedings of the Workshop on Noisy User-generated Text, pages 9– 18, 2015. find
  25. Ben King and Steven P Abney. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of HLT-NAACL, pages 1110–1119, 2013. find
  26. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A. Smith. A dependency parser for tweets. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1001–1012, Doha, Qatar, October 2014. As-Teresa Lynn, Kevin Scannell, and Eimear Maguire. Minority language Twitter: Part-of-speech tagging and analysis of Irish tweets. Proceedings of ACL-IJCNLP 2015, page 1, 2015. find our card
  27. Suraj Maharjan, Elizabeth Blair, Steven Bethard, and Thamar Solorio. Developing language-tagged corpora for code-switching tweets. In The 9th Linguistic Annotation Workshop held in conjuncion with NAACL 2015, page 72, 2015. find
  28. Shervin Malmasi, Eshrag Refaee, and Mark Dras. Arabic dialect identification using a parallel multidialectal corpus. In International Conference of the Pacific Association for Computational Linguistics, pages 35–53. Springer, 2015. find
  29. David Mimno and Andrew McCallum. Topic models conditioned on arbitrary features with Dirichlet-Multinomial regression. In Uncertainty in Artificial Intelligence, pages 411–418, 2008. find
  30. B. L. Monroe, M. P. Colaresi, and K. M. Quinn. Fightin’ Words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372, 2008. find
  31. Fred Morstatter, Jrgen Pfeffer, Huan Liu, and Kathleen Carley. Is the sample good enough? Comparing data from twitter’s streaming api with Twitter’s Firehose. In International AAAI Conference on Weblogs and Social Media, 2013. URL http://www.aaai.org/ocs/index.php/ ICWSM/ICWSM13/paper/view/6071. ↗
  32. Brendan O’Connor, Jacob Eisenstein, Eric P. Xing, and Noah A. Smith. A mixture model of demographic lexical variation. In NIPS Workshop on Machine Learning for Social Computing, 2010. find
  33. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL, 2013. find our card
  34. Umashanthi Pavalanathan and Jacob Eisenstein. Confounds and consequences in geotagged Twitter data. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), Lisbon, September 2015. URL http://www.aclweb.org/anthology/D/D15/D15-1256.pdf. ↗
  35. Daniel Ramage, Christopher D. Manning, and Susan Dumais. Partially labeled topic models for interpretable text mining. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 457–465, 2011. find
  36. John Russell Rickford. African American Vernacular English: Features, Evolution, Educational Implications. Wiley-Blackwell, 1999. find
  37. Margaret E Roberts, Brandon M Stewart, Dustin Tingley, and Edoardo M Airoldi. The structural topic model and applied social science. In Advances in Neural Information Processing Systems Workshop on Topic Models: Computation, Application, and Evaluation, 2013. find
  38. Mike Rosner and Paulseph-John Farrugia. A tagging algorithm for mixed language identification in a noisy domain. In Eighth Annual Conference of the International Speech Communication Association, 2007. find
  39. Thamar Solorio and Yang Liu. Learning to predict code-switching points. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 973–981. Association for Computational Linguistics, 2008. find
  40. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar, October 2014. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/W14-3907. ↗
  41. Sandeep Soni, Tanushree Mitra, Eric Gilbert, and Jacob Eisenstein. Modeling factuality judgments in social media text. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 415–420, Baltimore, Maryland, June 2014. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P14-2068. ↗
  42. Ian Stewart. Now we stronger than ever: African-American syntax in Twitter. Proceedings of EACL, page 31, 2014. find
  43. Latanya Sweeney. Discrimination in online ad delivery. ACM Queue, 11(3):10, 2013. find
  44. Matt Taddy. Multinomial inverse regression for text analysis. Journal of the American Statistical Association, 108(503):755–770, 2013. find
  45. Marcos Zampieri, Binyam Gebrekidan Gebre, and Sascha Diwersy. N-gram language models and pos distribution for the identification of Spanish varieties. Proceedings of TALN2013, pages 580– 587, 2013. find
  46. Arkaitz Zubiaga, Inaki San Vincente, Pablo Gamallo, Jose Ramom Pichel, Inaki Algeria, Nora Aranberri, Aitzol Ezeiza, and Victor Fresno. Overview of TweetLID: Tweet language identification at SEPLN 2014. In Proceedings of the Tweet Language Identification Workshop, Girona, Spain, September 2014. Spanish Society for Natural Language Processing. URL http://ceur-ws.org/Vol-1228/. ↗