Elamur Research⌕
☰
home / papers / 2026 / Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation

Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation

Paper · 2026 · Jiarui Wan, Songming Zhang and Yufeng Chen

Summary

Chain-of-thought (CoT) reasoning models often produce long, redundant traces. Existing compression techniques are tuned mostly on English. In other languages they either shorten outputs very little or cost accuracy, and low-resource languages suffer most. The authors propose MCD (Multilingual Chain-of-thought Compression via Cross-lingual Distillation), which addresses both data and training. They build a corpus from DeepMath-103K in German, French, Japanese, Russian and Chinese. Qwen3.5-27B translates each problem while solving it, and only items whose answer matches the ground truth are kept. Stratified sampling preserves the AoPS difficulty profile per language. The same model then rewrites traces into shorter versions that must still lead to the correct answer. Qwen3 student models are first fine-tuned on these compressed traces (SFT) and then trained with DPO to favour brief over verbose reasoning. On MGSM and MMATH, MCD cuts output length while keeping accuracy close to the original. It is notably more stable for Bengali, Swahili and Telugu, which supports cheaper and more equitable multilingual reasoning.

Key points

  • Motivating gap. The SFT-based SEER baseline on Qwen3-4B (MGSM) shows how unevenly English-centric compression transfers:
    • English: average tokens fall from 2126 to 1660.
    • Bengali: tokens fall only from 2645 to 2486, with an accuracy drop of 3.20 points.
    • Swahili: tokens fall only from 4371 to 4194.
  • Data pipeline. Translation-with-verification uses rejection sampling against the ground truth, and difficulty-aware stratified sampling follows the AoPS levels. Each final record holds the problem, difficulty, compressed trace, original trace, answer and ground truth.
  • Setup. The backbones are Qwen3-1.7B, Qwen3-4B and Qwen3-8B. SEER is the baseline, and SFT-Only and DPO-Only serve as ablations. Evaluation covers MGSM and MMATH.
  • Ablations.
    • SFT-Only compresses strongly but loses accuracy, especially in Swahili, Telugu and Bengali.
    • DPO-Only keeps accuracy but barely shortens outputs.
    • Combining the two stages recovers most of the accuracy while retaining compression.
  • Adaptive behaviour. Savings are largest on easy problems and shrink as difficulty rises. At the hardest AoPS levels, MCD can match or slightly exceed the original token count, indicating it keeps reasoning depth where the problem requires it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing Jiaotong UniversityBeijing · China22 works · 1,754 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Jiarui Wan, Songming Zhang and Yufeng Chen. 2026. Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.8

@inproceedings{wan2026multilingual,
  title = {{Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation}},
  author = {Jiarui Wan, Songming Zhang and Yufeng Chen},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.8},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.8}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  2. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. find
  3. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  4. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing… find
  5. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, and 1 others. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.
  6. Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, and Xin Xia. 2025a. Reasoning efficiently through adaptive chain-of-thought compression: A self-optimizing framework. arXiv preprint arXiv:2509.14093. — [arXiv:2509.14093](https://arxiv.org/abs/2509.14093) find
  7. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  8. Shulin Huang, Yiran Ding, Junshu Pan, and Yue Zhang. 2025b. Beyond english-centric training: How reinforcement learning improves cross-lingual reasoning in llms. arXiv preprint arXiv:2509.23657. find
  9. Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. 2025. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266. find
  10. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  11. Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. 2025. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067.
  12. Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. 2025. C3ot: Generating shorter chain-of-thought without compromising effectiveness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24312–24320. find
  13. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  15. Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, and Kai Chen. 2025. Not all tokens are what you need in thinking. arXiv preprint arXiv:2505.17827. — [arXiv:2505.17827](https://arxiv.org/abs/2505.17827) find
  16. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
  17. Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit Iyyer, Amir Houmansadr, and Eugene Bagdasarian. 2025. Overthink: Slowdown attacks on reasoning llms. arXiv preprint arXiv:2502.02542. find
  18. Chengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Shaochu Zhang, Shengchao Liu, Guoxin Ma, Yu Lan, and Chao Shen. 2025. Upfront chain-of-thought: A find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wan-2026-multilingual-chain-thought-compression-cross/