📄 Investigating African-American Vernacular English in Transformer-Based Text Generation

Summary

The paper asks whether GPT-2 handles African-American Vernacular English (AAVE) as well as Standard American English (SAE). Social media has made AAVE common in writing, but language models are trained mostly on dominant varieties. The authors drew 2019 AAVE tweets from TwitterAAE and had Amazon Mechanical Turk workers rewrite each one in SAE. This produced intent-equivalent pairs that differ only in dialect-specific grammar and vocabulary.

Each tweet was split into a prompt and a held-out final five words. Small GPT-2 then continued both versions of the prompt. The authors compared the original endings and the generated continuations using three sentiment tools (DistilBERT, VADER, TextBlob), BLEU and ROUGE against the real ending, and pairwise human judgments.

Off-the-shelf sentiment tools rated AAVE as more negative than SAE even when the meaning was the same. GPT-2's continuations matched the reference less closely for AAVE and were judged lower in quality. The work shows that dialect bias enters at two points: in the classifiers that analyse text and in the model that generates it.

Key points

• Dataset construction: 2019 AAVE/SAE pairs. Source tweets had at least 15 words and an AAVE posterior of at least 99.9% in TwitterAAE. Hashtags and emoji were removed. Average length was about 21 words for AAVE and 22 for SAE.
• Translation check: Native AAVE speakers and code-switchers reviewed 156 sampled pairs. They judged 90% and 95% of them intent-equivalent, respectively.
• Classifier bias on human text: On the original endings, DistilBERT (fine-tuned on SST-2) produced 157 more negative than positive labels for AAVE, about 7.7% of the data. Its mean score was −0.1436 for AAVE versus 0.0066 for SAE. The averaged VADER/TextBlob score showed a smaller gap.
• Effect of generation on sentiment: Generated continuations were much less often neutral and mostly positive under VADER/TextBlob. Under DistilBERT, SAE scores fell (0.0066 → −0.0399) while AAVE scores rose (−0.1436 → −0.0769), though AAVE stayed more negative. McNemar-Bowker tests found significant shifts for most classifier and dialect combinations; DistilBERT on SAE was not significant (p = 0.11).
• Generation quality: BLEU and ROUGE favoured SAE, with ROUGE-1 and ROUGE-L significant under a Wilcoxon test (p = 0.001 and 0.003). Human raters preferred SAE continuations by 21.7 points for quality and 12 points for fit to context.

Interesting ideas

• I1. Build test pairs that keep meaning fixed and change only the dialect, so any difference in model behaviour can be traced to dialect features rather than content.
• I2. Before trusting human translations, have a small sample checked by native speakers of the source variety, because outsiders often misread it.
• I3. Run the same human judgment on original human-written text first, to expose the raters' own bias before judging generated text; here raters labelled real SAE as machine-made more often than real AAVE.
• I4. Measure sentiment with tools built on different principles, a lexicon-and-rule system and a neural classifier, and keep the binary tool separate because it has no neutral class.
• I5. Check result stability by re-computing the metric on 1000 random subsets of the data. For categorical label changes, use a paired test such as McNemar-Bowker rather than comparing raw percentages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): An AAVE/SAE pair set could show how much LAS drops on the vernacular side and which constructions break the perceptron parser or gparse, such as zero copula or habitual "be".
• I2 (learn): Silver data produced by LLM teachers or dialect conversion could get a similar small native-speaker spot check, giving a measured equivalence rate rather than an assumed one.
• I3 (back): When evaluating text produced from the meaning graph, also show raters the original texts as a negative control, so preference shifts reflect the generator rather than rater taste.
• I4 (frames): Mova's explainable polarity or speech-act judgments could be compared against both kinds of tool on dialect pairs, to find where Mova agrees or disagrees with them on the same meaning.
• I5 (learn): Mova's evaluation reports could add resampled variance and paired tests on before/after label shifts when a repair rule or retraining changes outputs.