Elamur Research⌕
☰
home / papers / 2026 / Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

Paper · 2026 · Zeli Su, Ziyin Zhang, Zhou Liu et al.

Summary

When LLMs are adapted to a poorly supported language, they often lose ground on skills they already had. This is known as the alignment tax. The authors blame supervised fine-tuning (SFT): its token-level likelihood objective forces the model to copy small, skewed datasets, which overwrites pretrained knowledge. They instead frame language expansion as alignment in semantic space. A brief cold-start SFT first teaches the model to write in the target script. Training then continues with GRPO, which scores outputs by embedding similarity to the reference plus a language-consistency check. Any wording that keeps the meaning is rewarded, and policy updates stay constrained. Evaluated on two tasks, translating between Tibetan and Chinese and producing headlines in Tibetan, the method gains Tibetan ability with almost no loss of Chinese competence, and it outscores SFT on the CMRC benchmark. Its outputs share fewer n-grams with references, yet LLM judges prefer them, and they transfer better in few-shot settings. For teams working on low-resource languages, this suggests that careful reward design may be a safer route than additional SFT.

Key points

  • Two-stage pipeline: cold-start SFT on 5k instances for two epochs, then a single epoch of GRPO on the remaining training data, using embedding-similarity and language-consistency (LC) rewards.

  • Machine translation: on the dominant-language CMRC benchmark, the RL model beats the SFT baseline by +5.15 points, indicating much less forgetting. An appendix analysis of OOD token NLL and KL divergence to the base model supports this.

  • Headline generation: despite lower n-gram overlap, the RL model achieves a +16.1% higher win rate than SFT under LLM-based judges.

  • Reward ablation (semantic similarity, MT):

    • Embedding + LC: 0.7164
    • BLEU + LC: 0.6375
    • BLEU + Embedding + LC: 0.6175
    • BLEU + Embedding: 0.2312

    Removing LC leads to mixed Tibetan–Chinese output, and adding BLEU brings back SFT-like rigidity.

  • Length reward: an extra length-constraint reward gave no quality gain and cut CMRC by about 2 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of AutomationBeijing · China87 works · 14,883 citations🎓University of MacauMacao · China43 works · 2,368 citations🏢Ant GroupHangzhou · China51 works · 1,211 citations🎓Minzu University of ChinaBeijing · China7 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 287 works

Cite

Cite as Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu and Wentao Zhang. 2026. Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.880

@inproceedings{su2026reinforcement,
  title = {{Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax}},
  author = {Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu and Wentao Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.880},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.880}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thales Sales Almeida, Rodrigo Nogueira, and Hélio Pedrini. 2025. Curió-edu 7b: Examining data selection impacts in llm continued pretraining. Preprint, arXiv:2512.12770. — [arXiv:2512.12770](https://arxiv.org/abs/2512.12770) find
  2. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Ami… — ALLaM: Large Language Models for Arabic and English
  3. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261.
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  6. Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching large language models new languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–2… find
  7. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for Chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf…
  8. DeepSeek-AI and 1 others. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. Preprint, arXiv:2512.02556.
  9. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  10. Danni Liu and Jan Niehues. 2025. Conditions for catastrophic forgetting in multilingual translation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 347–359. Association for Computational Linguistics. find
  11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  12. OpenAI. 2025. Introducing gpt-4.1 in the api. OpenAI Blog. Accessed: 2025-12-25. find
  13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
  14. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  15. Fabian David Schmidt. 2025. Robust and Scalable Cross-Lingual Transfer. Ph.D. thesis, Bayerische Julius-Maximilians-Universitaet Wuerzburg (Germany). find
  16. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. arXiv preprint arXiv:1502.05477. find
  17. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  18. Zhihong Shao and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  19. Kimi Team, Yifan Bai, and 1 others. 2025. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534.
  20. Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, and 1 others. 2019. Large-scale datasets for going deeper in image understanding. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1480–1485. IEEE. find
  21. Guixian Xu, Zeli Su, Ziyin Zhang, Jianing Liu, Xu Han, Ting Zhang, and Yushuang Dong. 2025. CMHG: A dataset and benchmark for headline generation of minority languages in China. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12350–12357, Suzhou, Chin… find
  22. Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, and Nikolaos Aletras. 2025. Mitigating catastrophic forgetting in target language adaptation of llms via source-shielded updates. Preprint, arXiv:2512.04844.
  23. An Yang and 1 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  24. Menglin Yang, Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Qianru Zhang, Min Zhou, Irwin King, and Rex Ying. 2025b. Low-rank adaptation for foundation models: A comprehensive review. CoRR, abs/2501.00365. — [arXiv:2501.00365](https://arxiv.org/abs/2501.00365) find
  25. Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. 2022. CINO: A Chinese minority pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3937– 3949, Gyeongju, Republic of Korea. International Committee o… find
  26. Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. CoRR, abs/2401.01055.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2026-reinforcement-learning-semantic-rewards-enables/