Elamur Research⌕
☰
home / papers / 2024 / Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation

Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation

Paper · 2024 · Tianyu Zheng; Shuyue Guo; Xingwei Qu; Jiawei Guo; Xinrun Du; Qi Jia; Chenghua Lin; Wenhao Huang; Jie Fu; and Ge Zhang

Summary

The paper tackles the shortage of large, high-quality, openly usable Chinese instruction-tuning data. Manual annotation scales poorly, and distillation from stronger LLMs caps a model at its teacher's level. Kun builds on Humpback-style instruction back-translation, where a model infers an instruction for raw web text. It adds Answer Polishment (AP), because raw passages often fit their generated instructions poorly.

Yi-6B is fine-tuned on hand-checked seed pairs in two directions. A backward "label model" writes instructions for unlabelled segments from WuDao, Wanjuan and SkyPile. A forward "primary chat" model then filters the resulting pairs and rewrites the answers so they match the instructions more closely. The curated data trains a final chat model.

The pipeline yields over a million Chinese instruction pairs. In human preference tests, Kun-52k beats models tuned on COIG and Belle. On C-EVAL and CMMLU, Kun variants and a Kun+COIG mix are competitive. The approach offers a cheap route to Chinese alignment data with little dependence on annotators or proprietary teachers.

Key points

  • Seed data and models: 26,000 manually annotated instruction-output pairs were used to fine-tune Yi-6B into two models. One is a backward label model, p(x|y). The other is a forward primary chat model, p(y|x).
  • Filtering and polishing: Unlabelled sources were WuDao, Wanjuan and SkyPile. Candidates were filtered by perplexity and length, with anything over 512 tokens discarded, and purely descriptive text was dropped before AP refinement.
  • Output: More than one million Chinese instruction data points, released as COIG-Kun (github.com/Zheng0428/COIG-Kun).
  • Evaluation:
    • Data quality: 1,000 pairs per source were rated for clarity, feasibility, practicality and alignment.
    • Model quality: humans compared responses on 500 ShareGPT-zh prompts against baselines COIG-38k, Belle-52k and OL-CC-10k.
  • Results: Kun-52k won the human evaluation, followed by the Kun-52k + COIG-one-38k mix. Benchmark results (perplexity and generation modes) were mixed across variants. The extracted text is internally inconsistent, naming "Kun-39k" as best in places, so these figures need checking against the original paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Tianyu Zheng; Shuyue Guo; Xingwei Qu; Jiawei Guo; Xinrun Du; Qi Jia; Chenghua Lin; Wenhao Huang; Jie Fu; and Ge Zhang. 2024. Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation. arXiv:2401.06477.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.06477 · arXiv 2401.06477

@misc{fu2024kun,
  title = {{Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation}},
  author = {Tianyu Zheng; Shuyue Guo; Xingwei Qu; Jiawei Guo; Xinrun Du; Qi Jia; Chenghua Lin; Wenhao Huang; Jie Fu; and Ge Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2401.06477},
  eprint = {2401.06477},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.06477}
}

References

37 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631, 2023.
  2. BAAI. Coig-pc, 2023a. URL https://huggingface.co/datasets/BAAI/COIG-PC. ↗
  3. BAAI. Coig-pc-lite, 2023b. URL https://huggingface.co/datasets/BAAI/COIG-PC-Lite. ↗
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
  5. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024. find
  6. Wanyun Cui and Qianle Wang. Ada-instruct: Adapting instruction generators for complex reasoning. arXiv preprint arXiv:2310.04484, 2023. find
  7. Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. arXiv preprint arXiv:2309.08730, 2023.
  8. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  9. Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023. find
  10. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2305.08322, 2023. doi: 10.48550/ARX…
  11. Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases, 2023.
  12. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. arXiv preprint arXiv:2310.00752, 2023.
  13. Dichucheng Li, Yinghao Ma, Weixing Wei, Qiuqiang Kong, Yulun Wu, Mingjin Che, Fan Xia, Emmanouil Benetos, and Wei Li. Mertech: Instrument playing technique detection using self-supervised pretrained model with multi-task finetuning. arXiv preprint arXiv:2310.09853, 2023a. — [arXiv:2310.09853](https://arxiv.org/abs/2310.09853) find
  14. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. CoRR, abs/2306.09212, 2023b. doi: 10.48550/ARXIV.2306.09212. URL https://doi.org/10.48550/arXiv.2306.09212. ↗
  15. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023c.
  16. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023d. ↗
  17. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. find
  18. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL, 2022. find
  19. OL-CC. Openlabel-chinese conversations dataset (ol-cc), 2023. URL https://data.baai.ac.cn/details/OL-CC. ↗
  20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  21. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
  22. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chan… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  23. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, and Xipeng Qiu. Moss: Train… find
  24. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023a. ↗
  25. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford. edu/2023/03/13/alpaca. html, 3(6):7, 202… ↗
  26. Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975, 2023a.
  27. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022a. find
  28. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP, 2022b. find
  29. Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, et al. Interactive natural language processing. arXiv preprint arXiv:2305.13246, 2023b. — [arXiv:2305.13246](https://arxiv.org/abs/2305.13246) find
  30. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
  31. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yong… — Skywork: A More Open Bilingual Foundation Model
  32. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023a.
  33. Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. Superclue: A comprehensive chinese large language model benchmark. arXiv preprint arXiv:2307.15020, 2023b.
  34. Zhao Xue, Hanyu Zhao, Sha Yuan, and Yequan Wang. WuDaoCorpora Text, December 2022. find
  35. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
  36. Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. Chinese open instruction generalist: A preliminary release, 2023. find
  37. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2024-kun-answer-polishment-chinese-self/