Elamur Research⌕
☰
home / papers / 2023 / Task-Agnostic Low-Rank Adapters for Unseen English Dialects

Task-Agnostic Low-Rank Adapters for Unseen English Dialects

Paper · 2023 · Zedian Xiao, William Held, Yanchen Liu et al.

Summary

Language models trained mostly on Standard American English (SAE) perform noticeably worse for speakers of other English dialects. Earlier fixes usually need dialect data for each task, and adapter-based approaches such as TADA still require a separate module for every dialect. The authors propose HyperLoRA, which removes the need for annotated data in a new dialect. Instead it uses expert typological descriptions of dialects: eWAVE feature vectors that say which morphosyntactic features a variety uses and how often. A small hypernetwork reads such a vector, plus a positional code for the layer and for query or value, and outputs LoRA weights for the attention matrices. It is trained on parallel SAE/pseudo-dialect text produced by Multi-VALUE. The training objective aligns dialect representations with SAE representations through optimal transport, so no task adapter is involved. At test time the generated LoRA is combined with an ordinary SAE task adapter. Across five unseen dialects on GLUE, HyperLoRA is best or close to best in a zero-shot setting. It adds under 0.5% parameters and no inference latency.

Key points

  • The hypernetwork generates the down- and up-projections of LoRA modules on query and value attention matrices. Its input is the dialect feature vector concatenated with an embedding of block position and projection type.
  • Pseudo-dialect training data comes from Multi-VALUE, which applies each eWAVE feature at a probability tied to its attestation: 100% if obligatory, 60% if neither pervasive nor rare, 30% if rare, 0% if absent or undocumented.
  • Training is task-agnostic: only hypernetwork weights are updated, under a morphosyntactic alignment objective based on an optimal transport plan between dialect and SAE representations.
  • The authors estimate that exploiting the expert feature vectors is worth roughly 250 annotations per dialect.
  • They introduce a feature-coverage metric: the share of a target dialect's weighted features that appear in the source dialects.
    • For CollSgE, choosing source dialects that are both close in Manhattan distance and high in coverage (MalaE, MaltE, JamE, IndSAE) gave +2.6% on RTE over the original source set.
    • The best configuration gave +0.7% mean improvement over the SAE baseline.

Interesting ideas

  • I1. Describe a variety as a weighted vector of expert-attested grammatical features and let a model condition on that description, rather than collecting data for each variety; new varieties then cost only a description.
  • I2. Sample each feature transformation at a rate tied to how pervasive the feature is (obligatory, medium, rare), so synthetic data reflects frequency instead of applying every rule always.
  • I3. Train the adaptation module only to pull dialect representations toward their SAE parallels, with no task head, so one module plugs into any downstream task.
  • I4. Measure, before training, what fraction of a target's features the training sources actually cover, alongside a distance measure, to choose source sets and to predict where zero-shot transfer will fail.
  • I5. Put the complexity of the variation space into a small generator that emits per-variety weights, keeping the main model free of cross-variety interference.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): eWAVE-style feature vectors could parameterise the UD dialect conversion rules, so parser training data for a new English variety is produced from its feature profile instead of hand-picked rewrites.
  • I2 (learn): silver-data generation for dialect robustness could use graded rule probabilities, which gives more realistic perturbations for testing witness parsers.
  • I3 (tree): gparse could be regularised so that a Multi-VALUE-transformed sentence and its SAE source yield matching encoder states, letting downstream frame and world stages stay unchanged.
  • I4 (learn): a coverage score over UD constructions or dialect features could act as an explainable diagnostic in evaluation, flagging test texts whose phenomena were never seen in training.
  • I5 (tokens): a tagger or lemmatizer could keep one core plus small generated per-variety corrections, such as spelling or morphology variants, selected by a variety profile.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Harvard UniversityCambridge · United States112 works · 42,023 citations🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Zedian Xiao, William Held, Yanchen Liu and Diyi Yang. 2023. Task-Agnostic Low-Rank Adapters for Unseen English Dialects. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.487

@inproceedings{xiao2023task,
  title = {{Task-Agnostic Low-Rank Adapters for Unseen English Dialects}},
  author = {Zedian Xiao, William Held, Yanchen Liu and Diyi Yang},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.487},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.487}
}

References

51 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1881–1890, Brussels, Belgium. Association for Computational Linguistics.
  2. Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4762–4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  3. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan. find
  4. Steven Bird. 2022. Local languages, third spaces, and other high-resource scenarios. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7817–7829, Dublin, Ireland. Association for Computational Linguistics. ↗
  5. Terra Blevins, Robert Kwiatkowski, Jamie MacBeth, Kathleen McKeown, Desmond Patton, and Owen Rambow. 2016. Automatically processing tweets from gang-involved youth: Towards detecting loss and aggression. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2196–2206, Osaka, Japan. find
  6. The COLING 2016 Organizing Committee. find
  7. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. ↗
  8. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australia. Association for Computational Linguistics. ↗
  9. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv.
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  11. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. ↗
  12. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc. find
  13. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Association for Computational Linguistics. ↗
  14. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. ↗
  15. Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun ichi Amari, Alain Trouvé, and Gabriel Peyré. 2018. Interpolating between optimal transport and mmd using sinkhorn divergences. find
  16. Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose language counts as high quality? measuring language ideologies in text data selection. ↗
  17. David Ha, Andrew Dai, and Quoc V. Le. 2016. Hypernetworks. find
  18. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021a. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. In Equity and Access in Algorithms, Mechanisms, and Optimization. ACM.
  19. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna M. Howard. 2021b. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. Equity and Access in Algorithms, Mechanisms, and Optimization. ↗
  20. Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. 2022. Hyperprompt: Prompt-based task-conditioning of transformers. ↗
  21. Will Held, Caleb Ziems, and Diyi Yang. 2023. Tada: Task-agnostic dialect adapters for english. ↗
  22. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR. ↗
  23. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. ↗
  24. Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.
  25. John Nerbonne. 2009. Data-driven dialectology. Language and Linguistics Compass, 3(1):175–198. ↗
  26. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453. ↗
  27. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023. Modular deep learning. ↗
  28. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
  29. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterhub: A framework for adapting transformers.
  30. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1120, San Diego, California. Association for Computational Linguistics. ↗
  31. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. Mad-x: An adapter-based framework for multi-task cross-lingual transfer.
  32. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51–57, Vancouver, Canada. Association for Computational Linguistics. ↗
  33. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  34. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  35. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  36. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. ↗
  37. Boris Knyazev, Michal Drozdzal, Graham W. Taylor, and Adriana Romero-Soriano. 2021. Parameter prediction for unseen deep architectures. ↗
  38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. ↗
  39. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. eWAVE. find
  40. Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023. Scaling down to scale up: A guide to parameter-efficient fine-tuning. ↗
  41. Yanchen Liu, William Held, and Diyi Yang. 2023. Dada: Dialect adaptation via dynamic aggregation of linguistic rules.
  42. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  43. Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. 2022. Hypertuning: Toward adapting large language models without back-propagation. ↗
  44. Anthony Rios. 2020. Fuzze: Fuzzy fairness evaluation of offensive language classifiers on african-american english. In AAAI Conference on Artificial Intelligence. find
  45. Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition of text representation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 815–825, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  46. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. max E [ℓ adv ] = max E d∼P DIAL [Adv(d; θ)]
  47. Cédric Villani. 2008. Optimal transport: Old and new. assume Adv is K-Lipschitz, Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438–4450, Online. Association for Computational Linguistics. Adv − E s∼P SAE [Adv(s; θ)] ≈ Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2021. Challenges in automated debiasing for toxic language detection. ArXiv, abs/2102.00086. ↗
  48. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics. ↗
  49. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023a. Can large language models transform computational social science? ↗
  50. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023b. Multi-value: A framework for cross-dialectal english nlp.
  51. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. 2022. Hyper-x: A unified hypernetwork for multi-task multilingual transfer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2023-task-agnostic-low-rank-adapters/