Elamur Research⌕
☰
home / papers / 2022 / Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English

Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English

Paper · 2022 · Jamell Dacon

Summary

The paper tackles a fairness problem: part-of-speech taggers built on Mainstream American English (MAE) mislabel African-American English (AAE) tokens, such as "neva", "dat" and "finna", because they never saw them in training. The author samples 3000 tweets from the TwitterAAE corpus whose demographic-dialect probability exceeds 99%. The tweets are cleaned by collapsing letter runs and stripping punctuation, handles and emojis. Five bi-dialectal sociolinguist annotators from Amazon Mechanical Turk then label them, using slang dictionaries as aids. Twenty diglossic volunteers review a 300-tweet subset, matching AAE forms with MAE equivalents. The resulting labels are contrasted with NLTK's Penn Treebank tags, which systematically misclassify dialect features. Agreement among annotators is high (Krippendorff's α 0.88). The paper sets up CRF and Bi-LSTM taggers to test whether training on correctly tagged AAE reduces predictive bias, but the provided text reports no tagger scores. Its main contributions are the annotation approach, an AAE↔MAE feature mapping, and an argument for human-in-the-loop work on dialects.

Key points

  • Data: 3000 TwitterAAE tweets (Blodgett et al., 2016), averaging about 7 words each, selected only when the African-American dialect proportion confidence exceeds 99%.
  • Annotation aids: The Online Slang Dictionary (over 24,000 definitions for over 17,000 slang entries) and Word Type, a Wiktionary-based POS dictionary. Reference tags come from NLTK's fine-grained PTB tagset.
  • Annotators: 5 bi-dialectal AMT workers (HIT approval ≥95%, >10,000 HITs, US-based) reached Krippendorff's α = 0.88. Twenty volunteer diglossic annotators then checked 300 tweets.
  • Table 2 lists AAE forms that were tagged correctly, with their MAE equivalents, for example:
    • "tryna/finna" → trying to / fixing to
    • "ion/ian" → I don't
    • "iz" → is
    • "hw" → how
  • Model setup:
    • CRF with L1 = 0.25 and L2 = 0.3.
    • Bi-LSTM trained for 40 epochs with Adam at lr 0.001.
    • Evaluation uses a 70-30 split with 5-fold cross-validation, reporting per-tag precision, recall and F1.

Interesting ideas

  • I1. Show bias with a minimal pair: tag two sentences that mean the same thing, one in MAE and one in the dialect, and highlight where the tags diverge. This makes the failure visible without any training run.
  • I2. Ask annotators for the standard-variety equivalent of each dialect token, not just a label. This produces a dialect→standard substitution map that can be reused.
  • I3. Filter the corpus by dialect-membership confidence, here at 99%, so the annotated set is clearly the target variety and not a mixture.
  • I4. Use external lexicons, such as a slang dictionary and a Wiktionary-derived POS list, as annotator aids when tokens are lexically ambiguous.
  • I5. Watch for register-dependent meaning: in AAE "ass" in "long ass day" works as an emphatic intensifier, not an offensive noun.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Keep a suite of paired dialect/standard sentences and alert when the en tagger assigns different POS to tokens that should be equivalent.
  • I2 (tree): Such a map can drive UD dialect conversion, normalizing forms like "finna" or "ion" before parsing while keeping the original surface tokens in the tree.
  • I3 (learn): When building silver data for a variety, apply a high-confidence variety filter first so that agreement among witness parsers is measured on the right population.
  • I4 (world): Slang and dialect lexicon entries could be attached to the WordNet/Wiktionary-based knowledge layer, so that non-standard lemmas resolve to known senses.
  • I5 (frames): Pragmatics in prag should treat such tokens as intensity modifiers, not as insults or as participants in the frame, to avoid false offensive or irony readings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Michigan State UniversityEast Lansing · United States23 works · 1,077 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Jamell Dacon. 2022. Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English. Proceedings of the Second Workshop on Bridging Human–Computer Interaction and Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.hcinlp-1.8

@inproceedings{dacon2022deep,
  title = {{Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English}},
  author = {Jamell Dacon},
  booktitle = {Proceedings of the Second Workshop on Bridging Human–Computer Interaction and Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.hcinlp-1.8},
  url = {https://doi.org/10.18653/v1/2022.hcinlp-1.8}
}

References

40 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vernacular English. In Proceedings of the Student Research Workshop Associated with RANLP 2019, pages 16–20, Varna, Bulgaria. INCOMA Ltd. find
  2. Guy Bailey, John Baugh, Salikoko S. Mufwene, and John R. Rickford. 1998. African-American English: Structure, History and Use (1st ed.). Routledge. find
  3. Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21, Brussels, Belgium. Association for Computational Linguistics.
  4. John Baugh. 2008. Linguistic discrimination. In 1. Halbband, pages 709–714. De Gruyter Mouton. find
  5. Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3504–3519, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  6. Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and anti-racism in NLP. CoRR, abs/2106.11410. ↗
  7. Linda M. Bland-Stewart. 2005. Difference or deficit in speakers of african american english? ↗
  8. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. IJCNN 2005. https://leader.pubs.asha.org/doi/10. 1044/leader.FTR1.10062005.6 . ↗
  9. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗
  10. Jonathon Green. 2014. The vulgar tongue: Green’s history of slang. Oxford University Press, New York, USA. find
  11. Lisa J. Green. 2002. African American English: A Linguistic Introduction. Cambridge University Press. ↗
  12. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5877–5883, Online. Association for Computational Linguistics. ↗
  13. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics. ↗
  14. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. CoRR, abs/1707.00061. ↗
  15. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. New York, NY, USA. Association for Computing Machinery.
  16. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australia. Association for Computational Linguistics. ↗
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780. ↗
  18. Taylor Jones. 2015. Toward a description of african american vernacular english dialect regions using “black twitter”. American Speech, 90:403–440. find
  19. Herbert H. Clark and Michael F. Schober. 1992. Asking questions and influencing answers. In Russell Sage Foundation. find
  20. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1120, San Diego, California. Association for Computational Linguistics. ↗
  21. Jamell Dacon and Haochen Liu. 2021. Does gender matter in the news? detecting and examining gender bias in news articles. In Companion Proceedings of the Web Conference 2021, WWW ’21, page 385–392, New York, NY, USA. Association for Computing Machinery. find
  22. Sharese King. 2020. From african american vernacular english to african american language: Rethinking the study of race and language in african americans’ speech. Annual Review of Linguistics, 6(1):285–300. ↗
  23. Thomas Davidson and Debasmita Bhattacharya. 2020. Examining racial bias in an online abuse corpus with structural topic modeling. CoRR, abs/2005.13041. ↗
  24. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Rachel Dorn. 2019. Dialect-specific models for automatic speech recognition of African American Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. find
  25. Ian Stewart. 2014. Now we stronger than ever: African-American English syntax in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 31–37, Gothenburg, Sweden. Association for Computational Linguistics. ↗
  26. William Labov. 1975. Ralph fasold, tense marking in black english: a linguistic and social analysis. washington, d.c.: Center for applied linguistics, 1972. pp. 254. Language in Society, 4(2):222–227. find
  27. Spencer S. Swinton. 1981. Predictive bias in graduate admissions tests. ↗
  28. ETS Research Report Series, 1981. John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  29. Rachael Tatman and Conner Kasten. 2017. Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions. In Proc. Interspeech 2017, pages 934–938. find
  30. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramón Fermandez, Silvio Amir, Luı́s Marujo, and Tiago Luı́s. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1520–1530, Lisbon, Portugal. Association for Computational Linguistics. ↗
  31. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7–14, Online. Association for Computational Linguistics. find
  32. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Does gender matter? towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4403–4416, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  33. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397, Online. Association for Computational Linguistics. ↗
  34. Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63–70, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  35. Marcos Zampieri, Preslav Nakov, and Yves Scherrer. 2020. Natural language processing for similar languages, varieties, and dialects: A survey. Natural Language Engineering, 26(6):595–612. ↗
  36. Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020. Demographics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4134–4145, Online. Association for Computational Linguistics. find
  37. Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020. Hate speech detection and racial bias mitigation in social media based on bert model. PLOS ONE, 15:1–26. ↗
  38. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin, Germany. Association for Computational Linguistics. ↗
  39. Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3143–3155, Online. Association for Computational Linguistics. ↗
  40. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dacon-2022-deep-multi-layered-dialectal-language/