Elamur Research⌕
☰
home / papers / 2026 / Generation of Instruction and Preference Dataset for Improving Japanese Instruction Following in LLMs

Generation of Instruction and Preference Dataset for Improving Japanese Instruction Following in LLMs

Summary

Instruction tuning needs large annotated datasets. These are costly to build and mostly exist in English. LLM-based augmentation has rarely been tested on languages with their own constraint types, such as Japanese script rules that require or forbid kanji. The authors build an automatic pipeline that produces both a Japanese instruction dataset and a preference dataset from a small human-written seed set organised by constraint categories.

An LLM creates new instructions by either adding a constraint to a seed instruction or rewriting it to fit a target category. Candidates are filtered using ROUGE-L, which removes near-duplicates to keep diversity, and LLM-as-a-Judge scores for quality. Responses are then generated and judged again. For the preference data, the validated responses become the chosen answers, and the LLM writes deliberately flawed rejected answers, which are also screened.

Models trained with SFT and DPO on this data follow instructions better, especially in Japanese, and also improve on other NLP benchmarks. This shows synthetic data can address language-specific alignment gaps.

Key points

  • Instructions are generated in two modes, "Add" (insert a new constraint) and "Rewrite" (adapt to a given constraint category). Example categories include punctuation rules and CSV table formatting.
  • Filtering works in two ways:
    • ROUGE-L is computed against both previously accepted instructions and the seed instruction.
    • LLM-as-a-Judge uses 5-point Likert scales with few-shot examples. It rates relevance, fluency and verbosity for instructions, and adherence, fluency, verbosity and completeness for responses.
  • Rejected responses come in two types:
    • "content" negatives obey the constraints but are off-topic;
    • "format" negatives are on-topic but break the constraints.
  • Several LLMs were fine-tuned with SFT and DPO and evaluated on instruction following plus eleven other NLP tasks. Gains appeared in Japanese and English instruction following and on downstream tasks.
  • The released resource is available at huggingface.co/datasets/llm-jp/synth-if (from NII LLMC / llm-jp).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3w8ceszaj7m9

@inproceedings{moriyama2026generation,
  title = {{Generation of Instruction and Preference Dataset for Improving Japanese Instruction Following in LLMs}},
  author = {Moriyama, Kei and Kodama, Takashi and Nakayama, Kouta},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3w8ceszaj7m9},
  url = {https://doi.org/10.63317/3w8ceszaj7m9}
}

References

62 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv [cs.LG]. find our card
  3. Bekki Daisuke and zoeai. Jsem: Japanese semantic test suite (japanese fracas and extensions). find
  4. Antoine Dussolle, Andrea Cardeña Díaz, Shota Sato, and Peter Devine. 2025. M-IFEval: Multilingual instruction-following evaluation. arXiv [cs.CL]. find
  5. Masatsugu Hangyo, Daisuke Kawahara, and Sadao Kurohashi. 2014. Building and analyzing a diverse document leads corpus annotated with semantic relations. Journal of Natural Language Processing, 21(2):213–247. find
  6. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), p… find
  7. Kan Hatakeyama. 2024a. kanhatakeyama/automultiturnbycalm3-22b. find
  8. Kan Hatakeyama. 2024b. kanhatakeyama/ramdom-to-fixed-multiturn-calm3. find
  9. Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2024), volume 38, pages 18188–18196. find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the Ninth International Conference on Learning Representations (ICLR 2021). find our card
  11. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), pages 14409–14428. find our card
  12. Shijia Huang, Jianqiao Zhao, Yanyang Li, and Liwei Wang. 2023. Learning preference model for LLMs via automatic preference data generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 9187–9199. find
  13. Ryosuke Ishigami. 2024. cyberagent/calm3-22bchat. find
  14. Ai Ishii, Naoya Inoue, Hisami Suzuki, and Satoshi Sekine. 2024. JEMHopQA: Dataset for Japanese explainable multi-hop question answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9515–9525. find
  15. Horio Kaito, Murata Eiki, Wang Hao, Ide Tatsuya, Kawahara Daisuke, Yamazaki Takato, Shinzato Kenta, Nakamachi Akifumi, Li Shengzhe, and Sato Toshinori. 2023. Verification of chain-of-thought prompting in japanese. The 37th Annual Conference of the Japanese Society for Artifical Intelligence (JSAI 20… find
  16. Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schuetze. 2024. LongForm: Effective instruction tuning with reverse instructions. In Findings of the Association for Computational Linguistics (ACL 2024), pages 7056–7078. find
  17. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), pages 2957–2966. find our card
  18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. find
  19. Suhyun Lee and Changheon Han. 2025. Sentimatic: Sentiment-guided automatic generation of preference datasets for customer support dialogue system. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  20. Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023. Synthetic data generation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 10443– 10461, Singapore. find our card
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. find
  22. Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. 2023. MUFFIN: Curating multi-faceted instructions for improving instruction following. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2023). find
  23. Suzuki Masatoshi, Suzuki Jun, Matsuda Kouji, Nishida Kyousuke, and Inoue Naoya. 2020. JAQKET: Construction of japanese qa datset based on the quiz. In 26th Annual Meeting of the Association for Natural Language Processing (NLP 2020). In Japanese. find
  24. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), pages 3470–3487. find
  25. NICT. Japanese-english bilingual corpus of wikipedia’s kyoto articles. find
  26. Kazumasa Omura, Fei Cheng, and Sadao Kurohashi. 2024. An empirical study of synthetic data generation for implicit discourse relation recognition. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1073–… find our card
  27. Kazumasa Omura, Daisuke Kawahara, and Sadao Kurohashi. 2023. Building a Commonsense Inference Dataset based on Basic Events and its Application. Journal of Natural Language Processing, 30(4):1206–1239. In Japanese. find
  28. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In Proceedings of the 40th Internation… find
  29. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. In-FoBench: Evaluating instruction following ability in large language models. In Findings of the Association for Computational Linguistics (ACL 2024), pages 13025–130… find
  30. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… find
  31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). find
  32. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 2685– 2702. find our card
  33. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. find our card
  34. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pages 6943–6951. find
  35. Maximilian Schmidt, Andrea Bartezzaghi, and Ngoc Thang Vu. 2024. Prompting-based synthetic data generation for few-shot question answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13168–13178. find
  36. Satoshi Sekine. 2003. Development of a question answering system focused on an encyclopedia. In 9th Annual Meeting of the Association for Natural Language Processing (NLP2003). In Japanese. find
  37. Satoshi Sekine, Ando Maya, Goto Michiko, Suzuki Hisami, Kawahara Daisuke, Inoue Naoya, and Inui Kentaro. 2024. ichikara-isntruction construction of japanese instruction dataset for llms. In In Proceedings of The Thirtieth Annual Meeting of The Association for Natural Language Processing (NLP 2024),… find
  38. Tomoki Sugimoto, Yasumasa Onoe, and Hitomi Yanaka. 2023. Jamp: Controlled Japanese temporal inference dataset for evaluating generalization capacity of language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL SRW 2023), pages 57–68. find
  39. Yikun Sun, Zhen Wan, Nobuhiro Ueda, Sakiko Yahata, Fei Cheng, Chenhui Chu, and Sadao Kurohashi. 2024. Rapidly developing high-quality instruction data and evaluation benchmark for large language models with minimal human effort: A case study on japanese. In Proceedings of the 2024 Joint Internationa… find
  40. Hisami Suzuki, Satoru Katsumata, Takashi Kodama, Tetsuro Takahashi, Kouta Nakayama, and Satoshi Sekine. 2025. Answercarefully: A dataset for improving the safety of japanese llm output. find
  41. Kubo Takahiro and Nakayama Hiroki. chabsa: Aspect based sentiment analysis dataset in japanese. find
  42. Masashi Takeshita, Rafal Rzpeka, and Kenji Araki. 2023. Jcommonsensemorality: Japanese dataset for evaluating commonsense morality understanding. In In Proceedings of The Twenty Nineth Annual Meeting of The Association for Natural Language Processing (NLP 2023), pages 357–362. In Japanese. find
  43. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. find
  44. Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Introducing the Asian language treebank (ALT). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1574–1578. find
  45. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  46. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), pages… find
  47. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… find
  48. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv [cs.CL]. find
  49. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Benchmarking complex instruction-following with multiple constraints composition. In Proceedings of the 38th Annual Conferenc… find
  50. Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. FOFO: A benchmark to evaluate LLMs’ format-following capability. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 680–699. find
  51. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empowering large pre-trained language models to follow complex instructions. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024)… find our card
  52. Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 6268–6278. find our card
  53. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang, Huitao Li, Xin Li, Kunyu Yu, Nan Liu, Qingyu Chen, Douglas Teodoro, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, and Irene Li. 2025. Mmlu-prox: A multilingual benchmark for advanced large language… find
  54. Hitomi Yanaka and Koji Mineshima. 2021. Assessing the generalization capacity of pre-trained language models through japanese adversarial natural language inference. In Proceedings of the 2021 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP2021). find
  55. Hitomi Yanaka and Koji Mineshima. 2022. Compositional evaluation on japanese textual entailment and similarity. Transactions of the Association for Computational Linguistics, 10:1266–1284. find
  56. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. In Findings of the Association for Computational Linguistics (ACL 2020), pages 1008–1025. find
  57. Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, Tao Gui, Qi Zhang, Zhongchao Shi, Jianping Fan, and Xuanjing Huang. 2025. A multi-dimensional constraint framework for evaluating and improving instruction following in larg… find
  58. Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 4031–4047. find
  59. Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, and Satoshi Sekine. 2024. Should we respect LLMs? a cross-lingual study on the influence of prompt politeness on LLM performance. In Proceedings of the Second Workshop on Social Influence in Conversations (SICon 2024), pages 9–35. find
  60. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). find
  61. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv [cs.CL]. find
  62. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv [cs.CL]. find our card

↑ back to top