Elamur Research⌕
☰
home / papers / 2025 / Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models

Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models

Paper · 2025 · Dipankar Srirag, Aditya Joshi and Jacob Eisenstein

Summary

Earlier dialect adapters, which help language models handle non-mainstream English varieties, were built for encoder models. This paper brings the idea to decoder LLMs with an architecture called LoRDD. The authors use MD-3, a corpus of transcribed taboo-style word-guessing dialogues in Indian (en-IN), Nigerian (en-NG) and US English (en-US). From successful games they hide the target word and ask the model to recover it, a task they call Target Word Prediction (TWP).

LoRDD stacks two LoRA adapters:

  • a task adapter, instruction-tuned on a mix of US and dialect conversations;
  • a dialect adapter underneath, trained contrastively so that dialect conversations land near frozen US representations of conversations about the same word.

The pairs come from natural conversations, not synthetic rewrites. With Mistral and Gemma, LoRDD beats an in-dialect baseline and three cross-dialect baselines. It also shrinks the gap to the US-English skyline considerably. This shows decoder models can be made more dialect-robust using real, loosely aligned dialect data.

Key points

  • Models and task: Mistral-7B-Instruct-v0.2 and Gemma2-9B-Instruct, evaluated on TWP over masked MD-3 dialogues.
  • Metrics: Sentence-BERT cosine similarity between predicted and gold word, plus exact-match accuracy.
  • Dialect adapter training: cosine embedding loss with a margin of 0.25. A pair is positive when both conversations share a target word and negative otherwise. Each adapter trains in about 25 minutes on one A100.
  • Baselines:
    • in-dialect fine-tuning;
    • en-US-only fine-tuning;
    • Multi-VALUE synthetic dialect data (IN-MV/NG-MV);
    • GPT-4 Turbo "de-dialected" en-IN (IN-TR).
  • Results (averaged over both models): LoRDD scores 59.9 similarity / 35.7 accuracy on en-IN and 63.5 / 41.9 on en-NG.
    • The en-IN gap to the skyline falls from 27.3% to 12% (similarity) and from 64.7% to 25% (accuracy).
    • The en-NG gap falls from 17.9% to 5.8% and from 43.1% to 4.5%.
  • Ablations:
    • Building the dialect adapter's pairs from synthetic corpora costs 1.0–2.3 similarity and 2.5–4.8 accuracy.
    • Dropping the dialect adapter costs up to 8.7 similarity and 12.2 accuracy.
    • Task-only training on synthetic augmentation performs worst.

Interesting ideas

  • I1. Instead of needing sentence-aligned translations, the authors build a pseudo-parallel corpus by pairing conversations that share the same hidden goal, which turns loosely related natural data into contrastive supervision.
  • I2. The standard-variety side is kept frozen as an anchor while only the dialect side is trained to move toward it, so the standard behaviour cannot drift.
  • I3. Natural dialect data clearly beat both rule-based (Multi-VALUE) and LLM-rewritten synthetic data, a warning that artificial dialect transforms can mislead training.
  • I4. Results are reported as the remaining gap to a standard-variety skyline, not only as absolute scores, which makes the fairness goal measurable.
  • I5. Guessing a masked target word from a describer's clues is a compact test of whether a system built an understanding of what was talked about.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Sentences from different English varieties or genres that resolve to the same frame or answer could be paired as weak parallel silver data for training parser or role-labeller robustness.
  • I2 (tree): When adapting gparse to non-standard English, the standard-trained encoder could serve as a fixed target, with a small adapter pulling dialect inputs toward it rather than retraining the whole parser.
  • I3 (learn): Synthetic dialect or noise augmentation should be checked against a small natural held-out set before being trusted, and kept as a negative control in evaluations.
  • I4 (learn): Evaluation reports for tagging, parsing or QA could show per-variety gap-to-skyline percentages, making robustness regressions visible.
  • I5 (answer): Taboo-style dialogues could become a reader test where Mova must infer the unnamed entity from the frames and properties it extracted, giving an explainable check of its world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia34 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Dipankar Srirag, Aditya Joshi and Jacob Eisenstein. 2025. Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-short.2

@inproceedings{srirag2025predicting,
  title = {{Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models}},
  author = {Dipankar Srirag, Aditya Joshi and Jacob Eisenstein},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-short.2},
  url = {https://doi.org/10.18653/v1/2025.naacl-short.2}
}

References

17 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗
  2. Nihal Jain, Dejiao Zhang, Wasi Uddin Ahmad, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, and Bing Xiang. 2023. ContraCLM: Contrastive learning for causal language model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6436–6459, Toronto, Canada. Association for Computational Linguistics. ↗
  3. Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. clembench: Using game play to evaluate chat-optimized language models as conversational agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11174–11219, Singapore. Association for Computational Linguistics. find
  4. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  5. Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315–2338, Online. Association for Computational Linguistics. ↗
  6. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Comput. Surv. Just Accepted. ↗
  7. C. Lange. 2012. The Syntax of Spoken Indian English. Varieties of English around the world. John Benjamins Publishing Company. ↗
  8. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-value: A framework for cross-dialectal english nlp. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Linguistics. ↗
  9. Yanchen Liu, William Held, and Diyi Yang. 2023. DADA: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore. Association for Computational Linguistics.
  10. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774. ↗
  11. Preprint, Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673.
  12. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  13. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi. Association for Computational Linguistics. ↗
  14. Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. 2023. Dialect-robust evaluation of generated text. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6010–6028, Toronto, Canada. Association for Computational Linguistics. ↗
  15. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. ↗
  16. Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023. Task-agnostic low-rank adapters for unseen English dialects. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7857–7870, Singapore. Association for Computational Linguistics.
  17. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srirag-2025-predicting-target-word-game-playing/