Elamur Research⌕
☰
home / papers / 2026 / Why Does Low-Rank Adaptation Work for Hindi-English Code-Mixing? A Geometric Analysis

Why Does Low-Rank Adaptation Work for Hindi-English Code-Mixing? A Geometric Analysis

Paper · 2026 · Shashank Vishwakarma and Rakesh Kumar

Summary

The paper asks why LoRA needs so few trainable dimensions when multilingual encoders are adapted to Hindi-English (Hinglish) code-mixed text. The authors propose a geometric explanation. Code-mixed inputs fall into a compact cross-lingual subspace that pre-training has already shaped, so fine-tuning only has to adjust existing structure. To test this, they filter the COMI-LINGUA corpus (Hindi as matrix language) into a controlled subset for token-level language identification. They then study mBERT and MuRIL with three tools:

  • SVD-based effective rank of pre-trained Query/Key matrices;
  • Centered Kernel Alignment (CKA) between Hinglish and monolingual embeddings;
  • effective rank of trained LoRA updates.

Pre-trained attention weights are close to full-rank, while the learned LoRA updates shrink to a handful of dimensions. Hinglish representations also sit much closer to Hindi than to English or a monolingual baseline. The authors read this agreement as evidence that LoRA builds on multilingual structure the model already has. They derive a testable rule for choosing ranks on other language pairs, but acknowledge that the analysis covers one pair, one task and a small dataset.

Key points

  • Data: COMI-LINGUA, LID labels {hi, en, other}. Filtering leaves 1,666 sentences (1332/166/168 train/dev/test; about 61% Hindi tokens). Macro-F1 is the main metric.
  • Pre-trained Query/Key matrices in mBERT and MuRIL have effective ranks of 437–441, and roughly 90% of spectral energy lies in about 301 dimensions.
  • LoRA updates trained with r = 4, 8 and 16 have effective ranks of only 2.1–5.9 (mean 3.9). This is about 136× compression (Wilcoxon p < 10⁻¹⁹).
  • CKA of Hinglish embeddings is 0.279 with Hindi and 0.093 with English, against a monolingual baseline of 0.074. That makes Hindi alignment 3.77× the baseline, confirmed by permutation tests.
  • Prediction for future work: language pairs with stronger pre-trained cross-lingual alignment should need lower LoRA ranks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Gautam Buddha University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksMultilingual & cross-lingual NLP · 828 worksEfficient models & compression · 242 works

Cite

Cite as Shashank Vishwakarma and Rakesh Kumar. 2026. Why Does Low-Rank Adaptation Work for Hindi-English Code-Mixing? A Geometric Analysis. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/55yuuwkijgj9

@inproceedings{vishwakarma2026why,
  title = {{Why Does Low-Rank Adaptation Work for Hindi-English Code-Mixing? A Geometric Analysis}},
  author = {Shashank Vishwakarma and Rakesh Kumar},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/55yuuwkijgj9},
  url = {https://doi.org/10.63317/55yuuwkijgj9}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
  2. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association.
  3. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Asso…
  4. Dan Biderman, Jose Javier Gonzalez Ortiz, Ranganath Jacob, Matthew Sullivan, Sainbayar Sukhbaatar, Martin Wistuba, et al. 2024. LoRA learns less and forgets less. arXiv preprint arXiv:2405.09673. find
  5. John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. 2024. Rlhf can speak many languages: Unlocking multilingual preference optimization for llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13134–13156, Miami,… find
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. A survey of code-switching: Linguistic and social perspectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International…
  9. Deepak Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2020. A semi-supervised approach to generate the code-mixed text using pre-trained encoder and transfer learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2267–2280, Online. Association for Computational Ling… find
  10. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In Proceedings of the International Conference on Learning Representations (ICLR).
  11. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790– 2799. PMLR. find
  12. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR).
  13. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Agrawal, Pooja Hase, Pushpak Gupta, Palak Singh, et al. 2021. MuRIL: Multilingual representations for Indian languages. arXiv preprint arXiv:2103.10730.
  14. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575– 3585, Online. Association for Computat…
  15. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529. PMLR.
  16. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  17. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916.
  18. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  19. Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. 2023. A kernel-based view of language model fine-tuning. In Proceedings of the 40th International Conference on Machine Learning, pages 23610–23641. PMLR. find
  20. Charles H. Martin and Michael W. Mahoney. 2019. Traditional and heavy-tailed self regularization in neural network models. In arXiv preprint, volume abs/1901.08276. — [arXiv:1901.08276](https://arxiv.org/abs/1901.08276) find
  21. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1971– 1982, Vancouver, Canada. Association for Computational Linguistics. find
  22. Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, volume 31. find
  23. Kaushal Sharma, Ajit Kembhavi, Aniruddha Kembhavi, T Sivarani, Sheelu Abraham, and Kaustubh Vaghmare. 2020. Application of convolutional neural networks for stellar spectral classification. Monthly Notices of the Royal Astronomical Society, 491(2):2280–2300. find
  24. Pieter Muysken. 2000. Bilingual Speech: A Typology of Code-Mixing. Cambridge University Press, Cambridge. find
  25. Carol Myers-Scotton. 1993. Duelling Languages: Grammatical Structure in Codeswitching. Oxford University Press, Oxford. find
  26. Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5414–5426, Online and Pun… find
  27. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503…
  28. Ameya Prabhu, Aditya Joshi, Manish Shrivastava, and Vasudeva Varma. 2016. Towards subword level compositions for sentiment analysis of Hindi-English code mixed text. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2482–2491, Osak… find
  29. The COLING 2016 Organizing Committee. find
  30. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  31. Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. 2018. Language identification and named entity recognition in Hinglish code mixed tweets. In Proceedings of ACL 2018, Student Research Workshop, pages 52–58, Melbourne, Australia. Association for Computational Linguistics. find
  32. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Work…
  33. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15, Punta Cana, Dominican Republic. Association for Computat…
  34. Language Resource References Sheth, Rajvee and Beniwal, Himanshu and Singh, Mayank. 2025. COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi–English Code-Mixing. Association for Computational Linguistics. PID https://aclanthology.org/2025.findings-emnlp.422/. ↗
  35. Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on Twitter with a novel generalized word-level language detection technique. In Proceedings of the 55th Annual find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vishwakarma-2026-why-does-low-rank-adaptation/