📄 The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline

Summary

Language models are known to perform worse on non-standard English dialects. It was unclear where in the pipeline this gap comes from, and the paper calls the resulting cost "the dialect tax." The study uses parallel corpora in which meaning stays fixed and only dialectal form changes: PARALLELAAVE (tweets with human SAE rewrites) and MULTIVALUE (rule-based conversions of CoQA into five dialects), plus the ReDial reasoning set. It first shows that EmbeddingGemma rates dialect pairs as near-equivalent in meaning. It then measures disparities at four stages:

• tokenization across major tokenizer families
• pre-training, through gradient comparisons
• post-training, through reward-model scores
• inference, through hidden states, entropy and accuracy

Replacing subword segmentation with a character-level counterfactual tokenizer narrows the input-side gap. However, accuracy and output behaviour stay largely the same, so the bias also lives in learned weights. The main conclusion is that no single component causes the tax; it builds up at every stage. Fixes should therefore be judged across the whole pipeline rather than one module at a time.

Key points

• Semantic equivalence. On MULTIVALUE, all five dialect rewrites score above 0.98 cosine similarity to SAE, which beats every perturbation and translation control (0.659–0.978). This was confirmed with one-sided Wilcoxon tests over 429 aligned sentences (Holm-adjusted p < 0.001). PARALLELAAVE pairs score 0.92, against a floor of 0.41 for unrelated pairs.
• Tokenizer coverage. The tokenizers tested were BPE (GPT-5, GPT-2, Gemma-3, Llama-3, Qwen-3), Unigram (T5) and WordPiece (BERT).
• Fertility gap. On every tokenizer, AAVE has higher fertility than its SAE counterpart, by about +0.07 tokens per word on average. On MULTIVALUE, Appalachian, AAVE and Chicano English show the largest token-length increases.
• Training-stage effects.
  • Gradients from an SAE/AAVE pair diverge more than gradients from two unrelated SAE documents.
  • Reward models give isolated AAVE-exclusive tokens higher scores than SAE-exclusive ones. Yet they penalize dialect in full reasoning contexts, in ways that depend on task and model.
• Inference on ReDial.
  • AAVE costs 3.09 pp of accuracy, versus 0.04–1.23 pp for noise and translations.
  • AAVE raises next-token entropy by +0.29 nats, close to character noise (+0.22), while translations barely change it (−0.03).
  • In 502 of 504 conditions, final-layer hidden-state similarity drops significantly below its mid-layer peak.

Interesting ideas

• I1. Similarity scores are interpreted against a calibration ladder: character noise, translations into high-, medium- and low-resource languages, and an unrelated-pair floor. A raw number alone says little until it sits among controls.
• I2. Parallel corpora that hold meaning fixed while form varies give a clean invariance test: whatever changes in the output must come from surface form.
• I3. An intervention at one stage, here a character-level tokenizer, is judged by whether the final downstream gap closes, not just the local metric. The paper found the local gap shrank while accuracy did not move.
• I4. Gradient similarity between paired inputs, set against the similarity between two unrelated texts, serves as a probe of how hard a variant is to learn from.
• I5. Close internal representations did not imply equal downstream results: AAVE had the highest hidden-state similarity among the transformations yet the lowest accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation could report agreement between witness parsers on dialect rewrites next to the same agreement on noise-perturbed and unrelated sentences, as built-in negative controls.
• I2 (frames): MULTIVALUE pairs could check whether the who-did-what-to-whom frames and VerbNet roles come out identical for an SAE sentence and its dialect rewrite.
• I3 (tokens): If we add normalization for forms like "buildin'" in the tagger or lemmatizer, we should accept it only if answer accuracy on dialect text actually improves, not merely token or tag statistics.
• I4 (learn): The perceptron parser's weight updates on paired SAE/dialect silver sentences could be compared the same way, to flag constructions the model treats as alien.
• I5 (answer): Fable QA and school tests should be rerun on dialect rewrites of the stories, measuring the answer drop directly rather than trusting that the tree or graph "looks similar."