Elamur Research⌕
☰
home / papers / 2025 / Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English

Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English

Paper · 2025 · Duke Nguyen, Aditya Joshi and Flora Salim

Summary

Most English NLU models are trained on Standard American English, and labeled data for other varieties is scarce. Even DialectBENCH covers only 19 of the 77 English dialects documented in eWAVE. The authors test whether test-time domain adaptation (TTDA), a technique from computer vision, can close this gap using only unlabeled text in the target dialect.

They port SHOT to text. A roberta-base encoder feeds a two-layer bidirectional GRU bottleneck and a feed-forward classifier. During adaptation the encoder stays frozen. Only the head is tuned, using an information-maximization objective and pseudo-labels cleaned up by weighted k-means clustering.

Multi-VALUE generates synthetic SAE, IndE, NgE and SingE versions of CoLA, SST-2 and RTE. Every source–target pairing is compared against both in-dialect and cross-dialect finetuning.

SHOT reliably helps models trained on SAE and often beats models finetuned on the target dialect itself. Its benefit grows with a newly defined "dialectal gap". This makes unlabeled adaptation a practical option for varieties that have no labeled data.

Key points

  • Setup: 4 dialects (SAE, IndE, NgE, SingE) × 3 GLUE tasks (CoLA scored by Matthews correlation, SST-2 by accuracy, RTE as a sentence-pair task). All dialect data is a synthetic Multi-VALUE transformation; STS-B was left out because SHOT does not handle regression.
  • In-dialect SAE baselines: CoLA 49.44, SST-2 91.97, RTE 58.12. Applying SHOT to SAE-trained models gave positive gains on every non-SAE target, for example +3.47 on CoLA for SAE→SingE.
  • Dialectal gap is defined as eval(A,A) − eval(A,B). Over all 27 source–target–task observations it correlates with TTDA gain at r = 0.4169 (p = 0.0305). For the 9 SAE-source cases alone, r = 0.8455 (p = 0.0041).
  • For non-SAE sources, the best score is usually on the dialect the model was trained on. Cross-dialect finetuning sometimes beats in-dialect finetuning.
  • Stated limitations: Multi-VALUE only changes syntax, so it misses lexical and other natural variation, and compute limits ruled out tasks such as QQP.

Interesting ideas

  • I1. Measuring a distribution shift by how much a model's score drops when moved from its training variety to another one gives a cheap, task-grounded distance between text varieties without any explicit corpus statistics.
  • I2. Checking whether the size of an adaptation's gain tracks the size of the gap it targets is a sanity test that the method fixes the shift rather than adding noise.
  • I3. Generating dialect data with rule-based syntactic transformations derived from typological feature catalogues allows controlled experiments when no natural corpus exists.
  • I4. Freezing the large encoder and adapting only a small head on unlabeled target text keeps adaptation cheap and limits damage to general knowledge.
  • I5. Pseudo-labels are refined by clustering target embeddings around class centroids, while an objective rewards confident individual predictions and diverse predictions overall to avoid collapse onto one class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Compute the parser's own score drop between standard treebanks and dialect-converted versions to rank which dialects most need silver data or repair rules.
  • I2 (learn): Add a correlation check between per-variety gap and per-variety improvement to evaluation, alongside negative controls on varieties with near-zero gap.
  • I3 (tree): The UD dialect conversion could reuse eWAVE-style feature rules to make dialect variants of treebanks, keeping the tree alignment so each transformed feature stays traceable.
  • I4 (tree): gparse could keep its transformer frozen and adapt only the arc/label scorer on unlabeled dialect text, keeping a per-variety head that can be inspected.
  • I5 (learn): When building silver data from witness parsers, a diversity term or centroid-based relabeling could stop agreement from collapsing onto the most frequent relation labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Duke Nguyen, Aditya Joshi and Flora Salim. 2025. Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English. arXiv:2503.12858.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.12858 · arXiv 2503.12858

@misc{nguyen2025harnessing,
  title = {{Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English}},
  author = {Duke Nguyen, Aditya Joshi and Flora Salim},
  year = {2025},
  doi = {10.48550/arxiv.2503.12858},
  eprint = {2503.12858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.12858}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Egoitz Laparra, Xin Su, Yiyun Zhao, Özlem Uzuner, Timothy Miller, and Steven Bethard. 2021. SemEval-2021 task 10: Source-free domain adaptation for semantic processing. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 348–356, Online. Association for Computational Linguistics. find
  2. Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. 2021. Self-supervised test-time learning for reading comprehension. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1200–1211, Online. Association for Computational Linguistics. find
  3. Jian Liang, Ran He, and Tieniu Tan. 2023. A comprehensive survey on test-time adaptation under distribution shifts. International Journal Of Computer Vision. ↗
  4. Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10:414– 433. ↗
  5. Jian Liang, Dapeng Hu, and Jiashi Feng. 2020. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6028–6039. PMLR. find
  6. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol-Yanchen Liu, William Held, and Diyi Yang. 2023. DADA: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore. Association for Computational Linguistics.
  7. Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. 2021. Efficient test time adapter ensembling for low-resource language varieties. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 730–737, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  9. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. ↗
  10. Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. 2017. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2208–2217. PMLR. ↗
  11. Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023. Task-agnostic low-rank adapters for unseen English dialects. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7857–7870, Singapore. Association for Computational Linguistics.
  12. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  13. Hai Ye, Yuyang Ding, Juntao Li, and Hwee Tou Ng. 2022. Robust question answering against distribution shifts with test-time adaption: An empirical study. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6179–6192, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  14. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. ↗
  15. Marcos Zampieri, Preslav Nakov, and Yves Scherrer. 2020. Natural language processing for similar languages, varieties, and dialects: A survey. Natural Language Engineering, 26(6):595–612. ↗
  16. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics. ↗
  17. Bo Zhang, Xiaoming Zhang, Yun Liu, Lei Cheng, and Zhoujun Li. 2021. Matching distributions between model and data: Cross-domain knowledge distillation for unsupervised domain adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5423–5433, Online. Association for Computational Linguistics. find
  18. Dipankar Srirag, Aditya Joshi, and Jacob Eisenstein. 2024. Predicting the target word of game-playing conversations using a low-rank dialect adapter for decoder models. arXiv preprint arXiv:2409.00358. ↗
  19. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.
  20. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi. Association for Computational Linguistics. ↗
  21. A Samson Tan, Shafiq Joty, Lav Varshney, and Min-Yen Kan. 2020. Mind your inflections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5647–5663, Online. Association for Computational Linguistics. find
  22. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2025-harnessing-test-time-adaptation-nlu/