Elamur Research⌕
☰
home / papers / 2026 / Layer-aware Dual-directional Modulation for Low-resource Machine Translation

Layer-aware Dual-directional Modulation for Low-resource Machine Translation

Paper · 2026 · Siqi Zhang, Ran Song, Shuting Jiang et al.

Summary

LLM-based machine translation still performs much worse for low-resource languages than for high-resource ones, largely because pre-training data is unevenly distributed. The authors look for the source of this gap inside the network, layer by layer. They compare how many neurons fire when translating Chinese into English versus Chinese into low-resource targets such as Lao, Khmer and Bengali. From this comparison they define a metric called Activation Disparity (∆R). Layers with positive ∆R are labelled Task-Adaptive Layers (TAL) and sit mostly deeper in the network. Layers with negative ∆R are labelled Legacy-Inert Layers (LIL); these are mainly middle layers still governed by pre-training bias.

Building on this analysis, they introduce LaDM, which plugs into LoRA. LaDM uses learnable gates to scale up the outputs of a few TAL and scale down a few LIL. Across seven Chinese-to-low-resource directions on Llama-3.1, Qwen2.5 and Gemma-2, it beats standard LoRA by 1.73 spBLEU on average. The main lesson is that adapting to low-resource languages means both learning new signals and actively suppressing entrenched pre-trained behaviour.

Key points

  • MLPs are the most translation-sensitive part. Probing sub-modules (MLP, Q, K, V, O) with a cross-layer activation variance criterion shows MLP activations vary most across depth. Attention sub-modules stay fairly flat, so MLPs are chosen as the target module.
  • A second signal supports the TAL/LIL split. An SVD-based Entropy Gap (∆H) shows LIL form less concentrated representations in low-resource settings, while TAL remain stable.
  • Main result. LaDM gains an average of 1.73 spBLEU over standard LoRA on Llama-3.1, Qwen2.5 and Gemma-2. The paper reports that it also outperforms other PEFT baselines, including uniform and weight-decomposed methods (DoRA is discussed among related LoRA variants).
  • Small, sparse modulation works best. In a sensitivity test on Qwen2.5-7B for zh-bn, the best boost range was 1.03–1.07 and the best mute range was 0.91–0.96, applied to only k = 3 layers. Stronger scaling or more layers reduced BLEU.
  • Limitations. LaDM depends on predefined modulation bounds and targets data-scarce settings; gains may shrink in moderate-resource cases. Code is available at github.com/zzssqqq/LaDM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kunming University of Science and TechnologyKunming · China2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,105 worksLow-resource languages · 287 works

Cite

Cite as Siqi Zhang, Ran Song, Shuting Jiang, Yuxin Huang and Zhengtao Yu. 2026. Layer-aware Dual-directional Modulation for Low-resource Machine Translation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1054

@inproceedings{zhang2026layer,
  title = {{Layer-aware Dual-directional Modulation for Low-resource Machine Translation}},
  author = {Siqi Zhang, Ran Song, Shuting Jiang, Yuxin Huang and Zhengtao Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1054},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1054}
}

References

39 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Priyobroto Acharya, Haranath Mondal, Dipanjan Saha, Dipankar Das, and Sivaji Bandyopadhyay. 2025. Junlp: Improving low-resource indic translation system with efficient lora-based adaptation. In Proceedings of the Tenth Conference on Machine Translation, pages 1201–1209. find
  2. Divyanshu Aggarwal, Ashutosh Sathe, Ishaan Watts, and Sunayana Sitaram. 2024. Maple: Multilingual evaluation of parameter efficient finetuning of large language models. ArXiv, abs/2401.07598. find
  3. Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, and André FT Martins. 2023. Steering large language models for machine translation with finetuning and in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pag… find
  4. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2025. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112.
  5. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and 1 others. 2023a. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701.
  6. Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. 2023b. Beyond surface: Probing llama across scales and layers. arXiv preprint arXiv:2312.04333, 1(3). — [arXiv:2312.04333](https://arxiv.org/abs/2312.04333) find
  7. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 6022– 6034.
  8. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. 2024. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.02181. — [arXiv:2403.02181](https://arxiv.org/abs/2403.02181) find
  11. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. find
  12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  13. Tianze Hua, Tian Yun, and Ellie Pavlick. 2024. mothello: When do cross-lingual representation alignment and cross-lingual transfer emerge in multilingual models? arXiv preprint arXiv:2404.12444. find
  14. Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi, Jaesung Lee, Keigo Shibata, and Jun Suzuki. 2025. Layerwise importance analysis of feed-forward networks in transformer-based language models. arXiv preprint arXiv:2508.17734. — [arXiv:2508.17734](https://arxiv.org/abs/2508.17734) find
  15. Shuting Jiang, Ran Song, Yuxin Huang, Yan Xiang, Yantuan Xian, Shengxiang Gao, and Zhengtao Yu. 2026. Consensus-aligned neuron efficient fine-tuning large language models for multi-domain machine translation. arXiv preprint arXiv:2602.05694. — [arXiv:2602.05694](https://arxiv.org/abs/2602.05694) find
  16. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745.
  17. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, and 1 others. 2025. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the 31st international conferenc…
  18. Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2023. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454.
  19. Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2025. Language ranker: A metric for quantifying llm performance across high and low-resource languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28186–28194.
  20. Amartya Hatua. 2025. Mechanistic interpretability of gpt-2: Lexical and contextual layers in sentiment analysis. arXiv preprint arXiv:2512.06681. — [arXiv:2512.06681](https://arxiv.org/abs/2512.06681) find
  21. Xiao Liang, Yen-Min Jasmina Khaw, Soung-Yue Liew, Tien-Ping Tan, and Donghong Qin. 2025. Towards low-resource languages machine translation: A language-specific fine-tuning with lora for specialized large language models. IEEE Access. find
  22. Linyang He, Peili Chen, Ercong Nie, Yuanning Li, and Jonathan R Brennan. 2024. Decoding probing: Revealing internal linguistic structures in neural language models using minimal pairs. arXiv preprint arXiv:2403.17299. find
  23. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. find
  24. Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Beerel. 2024b. Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Paper… find
  25. Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024c. Fantastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics. ArXiv, abs/2403.01509. find
  26. Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang, Murun Yang, Dingyang Lin, Kaiyan Chang, Tong Zheng, Bei Li, Peinan Feng, Quan Du, Tong Xiao, and Jingbo Zhu. 2025. Beyond english: Toward inclusive and scalable multilingual machine translation with llms. arXiv preprint arXiv:2511.07003. — [arXiv:2511.07003](https://arxiv.org/abs/2511.07003) find
  27. Cunli Mao, Xiaofei Gao, Ran Song, Shizhu He, Shengxiang Gao, Kang Liu, and Zhengtao Yu. 2025. Multilingual knowledge graph completion via efficient multilingual knowledge sharing. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10882–10896. find
  28. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.
  29. Qwen Team and 1 others. 2024b. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). find
  30. Hetong Wang, Pasquale Minervini, and Edoardo Ponti. 2024. Probing the emergence of cross-lingual alignment during llm training. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12159–12173. find
  31. Zhipeng Yang, Junzhuo Li, Siyu Xia, and Xuming Hu. 2025. Internal chain-of-thought: Empirical evidence for layer-wise subtask scheduling in llms. arXiv preprint arXiv:2505.14530.
  32. Ran Zhang, Wei Zhao, and Steffen Eger. 2025. How good are llms for literary translation, really? literary translation evaluation with humans and llms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technol… find
  33. Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. 2024. Investigating layer importance in large language models. arXiv preprint arXiv:2409.14381. find
  34. Dawei Zhu, Pinzhen Chen, Miaoran Zhang, Barry Haddow, Xiaoyu Shen, and Dietrich Klakow. 2024. Fine-tuning large language models to translate: Will a touch of noisy data in misaligned languages suffice? arXiv preprint arXiv:2404.14122. find
  35. Toshiki Nakai, Ravi Kiran Chikkala, Lena Sophie Oberkircher, Nicholas Jennings, Natalia Skachkova, Tatiana Anikina, and Jesujoba Oluwadara Alabi. 2025. Treplina: Layer-wise cka+ repina alignment improves low-resource machine translation in aya-23 8b. arXiv preprint arXiv:2510.06249. find
  36. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395.
  37. Hammam Riza, Michael Purwoadi, Teduh Uliniansyah, Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai, Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea, Sethserey Sam, and 1 others. 2016. Introduction of the asian language treebank. In 2016 Conference of The Oriental Chapter of International Committee fo… find
  38. Xinyuan Song, Keyu Wang, Pengxiang Li, Lu Yin, and Shiwei Liu. 2025. Demystifying the roles of llm layers in retrieval, knowledge, and reasoning. arXiv preprint arXiv:2510.02091. find
  39. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024a. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-layer-aware-dual-directional-modulation/