Elamur Research⌕
☰
home / papers / 2026 / Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering

Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering

Paper · 2026 · Qiming Li, Xiaocheng Feng, Yixuan Ma et al.

Summary

Large language models and vision-language models reason noticeably worse in non-English languages, especially low-resource ones such as Thai and Swahili. Current fixes either need expensive multilingual fine-tuning or depend on external translation, which adds latency and is sensitive to translation quality. The authors propose MRRE, a training-free method that edits hidden states at inference time and needs no extra data or tools.

MRRE adds two precomputed vectors to the last-token representation at chosen decoder layers:

  • A mid-layer vector is the average gap between hidden states for parallel English and non-English reasoning responses. It pushes non-English reasoning toward the stronger English representation.
  • A later-layer vector is the difference between prompts that force the target language and prompts that force English. It moves the output back to the target language, because the first vector alone tends to make the model answer in English.

Across six LLMs and LVLMs on four reasoning benchmarks, MRRE improves non-English reasoning and keeps answers in the input language. The cost is almost zero, which makes it a practical way to reduce language inequity in deployed open-weight models.

Key points

  • Non-English reasoning accuracy rises by 5.48% on average and by up to 7.54% for Thai and Swahili. Input–output language consistency improves by 3.78%.
  • The steering vectors are built from only 100 samples. With Qwen2.5-7B on one A100, MRRE adds about 0.4 ms to both TTFT and TPOT.
  • The best intervention layers are about 20 and 23 for the LLM and 22 and 24 for the LVLM. The best strengths are α1 = 1 and α2 = 0.75.
  • The ablation shows a trade-off between the two vectors. Using only the first vector raises accuracy but drops language consistency for Qwen2.5-7B from 84.7 to 26.2. The full MRRE restores consistency to 92.7 while keeping most of the accuracy gain.
  • A "debiasing" variant reverses the first vector. It helps Qwen2.5-7B-Instruct but hurts Qwen2.5-VL-7B-Instruct, which the authors attribute to LVLMs' English-heavy image–text alignment. MRRE also requires access to model internals, so it cannot be used with closed-source models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Qiming Li, Xiaocheng Feng, Yixuan Ma, Ruihan Chen, Zihe Tong, Zekai Ye, Xiachong Feng, Libo Qin, Haoyu Ren, Kun Chen, Yunfei Lu, Dandan Tu and Bing Qin. 2026. Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1138

@inproceedings{li2026unlocking,
  title = {{Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering}},
  author = {Qiming Li, Xiaocheng Feng, Yixuan Ma, Ruihan Chen, Zihe Tong, Zekai Ye, Xiachong Feng, Libo Qin, Haoyu Ren, Kun Chen, Yunfei Lu, Dandan Tu and Bing Qin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1138},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1138}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gaurav Arora, Srujana Merugu, Shreya Jain, and Vaibhav Saxena. 2024. Towards robust knowledge representations in multilingual llms for equivalence and in-Xiang Geng, Ming Zhu, Jiahuan Li, Zhejian Lai, Wei Zou, Shuaijie She, Jiaxin Guo, Xiaofeng Zhao, Yinglu Li, Yuang Li, et al. 2024. Why not transfo… — [arXiv:2405.13923](https://arxiv.org/abs/2405.13923) find
  2. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  3. Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, and Fei Yuan. 2024. Mindmerger: Efficiently boosting llm reasoning in non-english languages. Advances in Neural Information Processing Systems, 37:34161– 34187. find
  4. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  5. Aditi Khandelwal, Utkarsh Agarwal, Kumar Tanmay, and Monojit Choudhury. 2024. Do moral judgment and reasoning capability of llms change with language? a study using the multilingual defining issues test. arXiv preprint arXiv:2402.02135. find
  6. Huiyuan Lai and Malvina Nissim. 2024. mcot: Multilingual instruction tuning for reasoning consistency in language models. arXiv preprint arXiv:2406.02301.
  7. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  8. Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2023. Improving in-context learning of multilingual generative language models with cross-lingual alignment. arXiv preprint arXiv:2311.08089. find
  9. Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, and Xiachong Feng. 2025a. Causal tracing of object representations in large vision language models: Mechanistic interpretability and hallucination mitigation. arXiv preprint arXiv:2511.05923. — [arXiv:2511.05923](https://arxiv.org/abs/2511.05923) find
  10. Hongyu Wang, Jiayu Xu, Senwei Xie, Ruiping Wang, Jialin Li, Zhaojie Xie, Bin Zhang, Chuyan Xiong, and Xilin Chen. 2024. M4u: Evaluating multilingual understanding and reasoning for large multimodal models. arXiv preprint arXiv:2405.15638. — [arXiv:2405.15638](https://arxiv.org/abs/2405.15638) find
  11. Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Libo Qin, Ruihan Chen, Baohang Li, Kui Jiang, Yaowei Wang, Ting Liu, et al. 2025b. Cai: Caption-sensitive attention intervention for mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2506.23590. — [arXiv:2506.23590](https://arxiv.org/abs/2506.23590) find
  12. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. 2025a. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  13. Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258.
  14. Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, et al. 2025b. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprint arXiv:2504.18428. — [arXiv:2504.18428](https://arxiv.org/abs/2504.18428) find
  15. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  16. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695–2709.
  17. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1). find
  18. Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2024. mcsqa: Multilingual commonsense reasoning dataset with unified creation strategy by language models and humans. arXiv preprint arXiv:2406.04215. find
  19. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.
  20. Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Wayne Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. 2025. Unlocking general long chain-of-thought reasoning capabilities of large language models via representation engineering. arXiv preprint arXiv:2503.11314.
  21. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  23. Junhao Xiao, Zhiyu Wu, Hao Lin, Yi Chen, Yahui Liu, Xiaoran Zhao, Zixu Wang, and Zejiang He. 2026. Not just what’s there: Enabling clip to comprehend negated visual descriptions without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 10978–10986. find
  24. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  25. Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, et al. 2025. Claim: Mitigating multilingual object hallucination in large vision-language models with cross-lingual attention intervention. arXiv preprint arXiv:2506.11073. find
  26. Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, and Min Zhang. 2024a. Lingualift: An effective two-stage instruction tuning framework for low-resource language tasks. arXiv e-prints, pages arXiv–2412. find
  27. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. 2024b. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer.
  28. Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, et al. 2025. When less language is more: Language-reasoning disentanglement makes llms better multilingual reasoners. arXiv preprint arXiv:2505.15257. — [arXiv:2505.15257](https://arxiv.org/abs/2505.15257) find
  29. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? Advances in Neural Information Processing Systems, 37:15296– 15319.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2026-unlocking-multilingual-reasoning-capability-llms/