📄 AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark

Summary

Standard NLU benchmarks such as GLUE and SuperGLUE are written almost entirely in Standard American English (SAE), so they cannot show how LLMs handle African American Vernacular English (AAVE). The earlier VALUE benchmark addressed this with fixed rule-based syntactic and morphological rewrites. The authors argue that such rewrites generalise poorly. Their benchmark, AAVENUE, instead translates 1000 sampled items from each of BoolQ, MultiRC, SST-2, COPA and WSC into AAVE. The translator is GPT-4o-mini, prompted with a few speaker-validated VALUE examples.

They checked translation quality in three ways:

• automatic scores (quality, fluency, coherence, understandability, BARTScore);
• pairwise preference judgments from five LLMs comparing AAVENUE and VALUE outputs;
• ratings from ten fluent AAVE speakers.

The new translations usually beat VALUE. When GPT-4o-mini, GPT-4-turbo, GPT-4o, Gemini-1.5-Flash and Gemini-1.5-Pro were run on both versions, accuracy was almost always lower on AAVE. MultiRC showed the largest drop. The work provides a reusable resource and evidence of dialect bias in current LLMs.

Key points

• Tasks and translation setup: five tasks (BoolQ, MultiRC, SST-2, COPA, WSC), 1000 items each, translated by few-shot GPT-4o-mini with VALUE exemplars. Code and site are at aavenue.live.
• Automatic metrics: on BoolQ passages, quality was 76.57 vs 58.21 for VALUE, and BARTScore was −1.44 vs −1.54. Some fields still favour VALUE, e.g. understandability on COPA second choices.
• LLM preference judgments: GPT-4-turbo picked AAVENUE for BoolQ passages 94.51% of the time vs 4.62% for VALUE. Gemini-1.5-Pro disagreed in places, favouring VALUE on MultiRC passages (85.71%) and BoolQ questions.
• Human ratings: ten AAVE speakers from the Bronx and Queens gave average scores between 7.02 (BoolQ) and 7.27 (MultiRC) on a 1–10 scale.
• Accuracy gap: SAE accuracy beat AAVE in nearly every model–task pair. MultiRC dropped by 12.5–16.2 points, and the only reversal was Gemini-1.5-Flash on COPA (+0.6).
• Error overlap: the intersection-over-union of wrong answers between SAE and AAVE was mostly low. WSC was the exception, reaching 49.54% for Gemini-1.5-Flash.

Interesting ideas

• I1. Instead of hand-coded rewrite rules, a dialect version of a dataset is produced by an LLM steered with a few human-checked example pairs; this scales across tasks without writing a rule per construction.
• I2. Each benchmark item is evaluated twice, in standard and variant form, and the per-task accuracy difference is reported rather than one overall score. This isolates the effect of surface form while keeping content fixed.
• I3. Intersection-over-union of wrong answers across the two versions separates items that are hard in themselves (errors overlap) from items broken by the variant (errors differ).
• I4. Translation quality is judged by several different LLMs in a forced binary choice, and their disagreements, such as Gemini-1.5-Pro's divergent votes, are reported openly instead of being averaged away.
• I5. Metrics are reported separately for each field of an item (passage, question, premise, each choice), which reveals that short fields can behave differently from long passages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): we could make dialect or informal-register twins of silver training sentences and check whether witness parsers still agree, as a robustness test for training data.
• I2 (answer): the fable QA and school-test readers could be run on paired standard/vernacular versions of a story, and any drop would show which pipeline step depends on surface wording.
• I3 (learn): computing error overlap between original and perturbed inputs would act as a negative control that separates parser brittleness from genuine reasoning failures.
• I4 (learn): the LLM teacher tiers could cast pairwise votes from multiple judges, with low-agreement items flagged for review rather than trusted as labels.
• I5 (tree): parser agreement on variant text should be measured per field type, since short questions and answer options may fail in ways that passage-level averages hide.