📄 That Ain't Right: Assessing LLM Performance on QA in African American and West African English Dialects

Summary

The paper asks whether popular chat LLMs answer factual questions equally well when a question uses a non-standard English variety instead of Standard American English (SAE). The authors drew 1,000 SQuAD questions at random and produced matching renditions of each in two other varieties: West African Pidgin English (WAPE) and African American Vernacular English (AAVE). AAVE versions came from a public dialect-conversion tool, WAPE versions from the PidginUNMT translation model, and a dialect speaker checked the results.

A 100-question subset (300 prompts) went to ChatGPT, Grok and Gemini without the SQuAD passage and with an instruction to answer briefly. Three human raters judged the 900 replies as right or wrong against the gold answer.

AAVE wording barely changed error rates, and no AAVE difference was significant. WAPE raised errors for every model, significantly so for ChatGPT and Grok. The study shows that speakers of under-represented varieties can quietly get worse answers. It also contributes a three-variety parallel QA set and a reusable cross-model testing procedure.

Key points

• Data: 1,000 SQuAD questions rendered in SAE, AAVE (online tool "Mr.Dialect") and WAPE (PidginUNMT). Only 100 per variety were scored, across history, sports, religion, politics and trivia.
• Setup: ChatGPT, Grok and Gemini were queried through public APIs. A system instruction, written in SAE for every variety, capped replies at under 20 words. Total API cost was under $15.
• Scoring: binary human judgment was used instead of EM/F1. Each rater scored 390 replies, 30% of items were secretly shared between raters, and Fleiss' Kappa on the shared items was 0.889.
• Error rates (SAE → AAVE → WAPE) and binomial-test p-values for WAPE:
  • Gemini: 0.45 → 0.48 → 0.53 (WAPE p = 0.075)
  • ChatGPT: 0.54 → 0.57 → 0.64 (p = 0.032)
  • Grok: 0.56 → 0.55 → 0.69 (p = 0.007)
• Qualitative findings: models seldom admitted confusion and instead misread dialect words. For example, the interjection "Chai" was taken as a scientist's name. Some replies switched into the input dialect, while others answered in SAE.

Interesting ideas

• I1. Hold the question content fixed, vary only the language variety, and test each system against its own standard-variety baseline with a paired significance test, which isolates robustness from overall skill.
• I2. Score "did the system understand the question" separately from "was the final answer correct," since these two failures need different fixes.
• I3. Unfamiliar dialect tokens tend to be silently forced into a wrong reading, such as an interjection becoming a person's name, rather than flagged.
• I4. Give annotators a hidden subset of overlapping items and measure agreement on it, which buys confidence in the non-overlapping majority at low cost.
• I5. Judge replies on whether they match the user's variety and register, not only on correctness.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): run the fable and school-test QA sets with dialectal or informal rewrites of the same questions and report per-variety drops relative to the SAE run.
• I2 (answer): because each step is explainable, check whether the frames and world built from the question were right before judging answer selection, so misreadings and reasoning errors are counted apart.
• I3 (tokens): mark out-of-vocabulary interjections and pidgin forms as uncertain instead of defaulting to proper-noun tags, and let later stages abstain or ask for clarification.
• I4 (learn): seed covert duplicate items into LLM-teacher or human labeling batches to audit silver-data consistency without doubling the annotation budget.
• I5 (back): add an optional generation mode that renders the answer graph in the dialect detected in the input, and evaluate that mode on comprehension as well as accuracy.