Elamur Research⌕
☰
home / papers / 2022 / Disambiguation of morpho-syntactic features of African American English – the case of habitual be

Disambiguation of morpho-syntactic features of African American English – the case of habitual be

Paper · 2022 · Harrison Santiago, Joshua Martin, Sarah Moeller et al.

Summary

The paper tackles a source of dialect bias in NLP. Habitual "be" in African American English (AAE), as in "I be in my office by 7:30", has the same surface form as ordinary uses of "be", and it is rare even in AAE data, so classifiers tend to miss it. The authors annotated 5,133 occurrences of "be" from the CORAAL interview corpus as habitual or non-habitual. Their pipeline has four steps:

  • tag the text with an NLTK POS tagger trained on Penn Treebank,
  • remove obviously non-habitual cases with hand-written rules over neighbouring POS tags,
  • generate synthetic habitual examples with nlpaug (Word2Vec and WordNet substitution and insertion) until the classes are balanced,
  • train simple scikit-learn classifiers on counts of POS tags in a window around "be".

A majority-vote ensemble trained on the filtered and augmented data reached a mean F1 of about 0.65, compared with roughly 0.44 without these steps. Results were also more stable across runs. The work shows that cheap syntactic filtering plus augmentation can make a rare dialect feature learnable from little data.

Key points

  • Data: 5,133 manually labelled "be" tokens from CORAAL (over 150 interviews, more than 127 hours of audio). Only 477 were habitual, about 9%.
  • Rule filter: it cut non-habitual instances to 994 and raised the habitual share to 30%. It also wrongly discarded 61 habitual tokens, an error rate of about 13%.
  • Augmentation: after augmentation the training data held 963 habitual and 944 non-habitual instances, a 50/50 split. Over 90% of augmentations introduced a word with a different POS. Ungrammatical outputs were deliberately kept.
  • Models: logistic regression, linear SVM, MLP (L-BFGS solver, hidden layers of 5 and 2), and an equal-vote ensemble of the three.
    • Mean F1 with filtering and augmentation: 0.648, 0.628, 0.627 and 0.652 respectively.
    • Mean F1 without them: 0.416, 0.542, 0.498 and 0.439.
    • Standard deviations shrank after balancing.
  • Evaluation protocol:
    • 70/30 split made before any augmentation.
    • The test set keeps the original class ratio.
    • 10 trials, with augmentation redone in each one.

Interesting ideas

  • I1. The rule filter does not try to find the rare class; it only removes cases that are clearly the common class, so the learned model handles just the hard residue
  • I2. The classifier sees only counts of POS tags in a context window rather than word embeddings, which sidesteps sparse vocabularies in a small dialect corpus
  • I3. Augmentation is applied only to the minority class and only after the train/test split, while the test set keeps the natural class ratio
  • I4. The authors rerun the random augmentation in every trial and report the spread across trials, not a single score
  • I5. Habitual "be" carries aspect (a recurring state or action), and a dialect form can map onto the same meaning as a GAE paraphrase with "usually"

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A morphology or tagger step for ambiguous forms such as invariant "be" could first apply high-precision exclusion rules and send only the undecided tokens to a statistical decision, keeping each exclusion traceable to a named rule.
  • I2 (learn): For rare constructions with little silver data, Mova could train lightweight disambiguators on tag-level features, which stay explainable and do not depend on the lexical coverage of the training text.
  • I3 (learn): Any synthetic boosting of rare phenomena in Mova's training data should follow the same protocol, so that evaluation reflects real-text frequencies and not the artificially balanced distribution.
  • I4 (learn): When Mova evaluates repair rules or silver data generated with randomness, repeated regeneration with reported variance would separate genuine gains from lucky samples.
  • I5 (world): Once the form is recognized as habitual, the world model could store it as a repeated or typical event rather than a single occurrence, which affects event ordering and answers to "when" and "how often" questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany34 works · 1,487 citations🎓University of FloridaGainesville · United States36 works · 364 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Harrison Santiago, Joshua Martin, Sarah Moeller and Kevin Tang. 2022. Disambiguation of morpho-syntactic features of African American English – the case of habitual be. Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.ltedi-1.9

@inproceedings{santiago2022disambiguation,
  title = {{Disambiguation of morpho-syntactic features of African American English – the case of habitual be}},
  author = {Harrison Santiago, Joshua Martin, Sarah Moeller and Kevin Tang},
  booktitle = {Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion},
  year = {2022},
  doi = {10.18653/v1/2022.ltedi-1.9},
  url = {https://doi.org/10.18653/v1/2022.ltedi-1.9}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Baugh. 2008. Linguistic discrimination. In Kon-taktlinguistik, pages 709–714. De Gruyter Mouton. find
  2. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2021. An empirical survey of data augmentation for limited data learning in nlp. ↗
  3. Rachel Dorn. 2019. Dialect-specific models for automatic speech recognition of african american vernacular english. In Student Research Workshop, pages 16–20.
  4. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567–573, Vancouver, Canada. Association for Computational Linguistics.
  5. Ralph W. Fasold. 1972. Tense Marking in Black English: A linguistic and social analysis. Number 8 in Urban Language Series. Center for Applied Linguistics, Arlington, VA. find
  6. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  7. Lisa J Green. 2002. African American English: a linguistic introduction. Cambridge University Press. ↗
  8. Tyler Kendall and Charlie Farrington. 2018. The corpus of regional African American language. Version, 6:1. find
  9. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. ↗
  10. Edward Ma. 2019. Nlp augmentation. https://github.com/makcedward/nlpaug. ↗
  11. Joshua L. Martin and Kevin Tang. 2020. Understanding Racial Disparities in Automatic Speech Recognition: The Case of Habitual “be”. In Proc. Interspeech 2020, pages 626–630. find
  12. B Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings of the 20th SIGNLL conference on computational natural language learning, pages 51–61. • If the word immediately preceding “be” is a modal, adjective, or “to”. • If the word immediately following “be” is a verbal noun, while the word immediately preceding is not a personal pronoun nor a noun.
  13. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR. ↗
  14. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM computing surveys (CSUR), 41(2):1– 69. • If the word immediately following “be” is an adjective, while the word immediately preceding “be” is not a personal pronoun nor a noun. find
  15. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. • If the word immediately following “be” is a preposition or subordinating conjunction, while the word immediately preceding “be” is a singular present verb. ↗
  16. M. Scott. 2020. Wordsmith tools version 8. Stroud: Lexical Analysis Software. • If the word immediately preceding “be” is a noun, and the word immediately preceding that noun is an adjective Rachael Tatman and Conner Kasten. 2017. Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions. In INTER-SPEECH, pages 934–938. • If the word immediately preceding “be” is an adverb, and the word immediately following “be” is either a personal pronoun or determiner. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. • If the word immediately preceding “be” is an adverb, and either the word immediately preceding the adverb is a verb, or modal Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In Computational Science – ICCS 2019, pages 84–95, Cham. Springer International Publishing.
  18. C Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. NIPS’15, page 649–657, Cambridge, MA, USA. MIT Press. Examples of augmenting occurrences of the habitual “be” • "they were like you should totally come here we be having so much fun So I tell my mom about it and" becomes "they were like you should totally come hither we be have got so much fun So I tell my mom astir it and" Zhi Zhong and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 System Demonstrations, pages 78–83, Uppsala, Sweden. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santiago-2022-disambiguation-morpho-syntactic-features-african/