Elamur Research⌕
☰
home / papers / 2026 / PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

Paper · 2026 · Kai Xiong, Yanwei Huang, Rongjunchen Zhang et al.

Summary

Reasoning datasets with checkable answers are needed to train LLMs, but manually written puzzle collections are small. LLM-driven augmentation can be unreliable, repetitive and expensive. PuzzleClone tackles this by turning each seed puzzle, by hand, into a specification written in a purpose-built, human-readable DSL plus a config file holding the seed's concrete values. The specification declares variables with domains and difficulty factors, symbols to solve, a pool of constraint templates, and query templates. A generator then randomizes values and constraint combinations, computes ground-truth answers with symbolic or custom solvers, and renders natural-language text. Correctness is checked by requiring that the original seed question and answer can be regenerated from its specification. The resulting PC-83K set covers SMT, linear programming and classic logic puzzles such as Sudoku, and remains hard for strong models. Post-training Qwen2.5-7B-Instruct on PC-83K brings large in-domain gains and transfers to other logic and math benchmarks. This offers a scalable, verifiable route to reasoning training data.

Key points

  • PC-83K holds 83,657 programmatically validated puzzles in varied formats; a single puzzle can include several sub-questions with different answer and evaluation types.
  • ChatGPT-4o and DeepSeek-R1 find the benchmark challenging. After SFT and RL, Qwen2.5-7B-Instruct rises from 14.5 to 66.0 on PC-83K.
  • The trained model improves on 7 logic and math benchmarks by up to 18.4 absolute points, with SATBench going from 51.6 to 70.0.
  • The authors favour "backward" specifications, which sample a valid solution first and build constraints around it, over "forward" ones, because fewer unsolvable instances are wasted. Instance names are generated with Faker.
  • Each specification can only produce a limited number of unique puzzles. For the 9-vase seed with a narrow domain, output plateaued near 3.3K distinct items after 100K generations.
  • Stated limitations:
    • the difficulty measure is coarse (diff_factor takes only +1, 0 or −1);
    • generated puzzles skew toward easier instances;
    • writing specifications requires substantial manual work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China170 works · 26,341 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Kai Xiong, Yanwei Huang, Rongjunchen Zhang, Kun Chen, Haipang WU and Yingcai Wu. 2026. PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1669

@inproceedings{xiong2026puzzleclone,
  title = {{PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data}},
  author = {Kai Xiong, Yanwei Huang, Rongjunchen Zhang, Kun Chen, Haipang WU and Yingcai Wu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1669},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1669}
}

References

63 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet. Announcement; see also model card addendum. ↗
  2. Anthropic. 2025. Claude opus 4 & claude sonnet 4: System card. https://www.anthropic.com/ claude-4-system-card. Model referenced in paper as Claude-4-sonnet. ↗
  3. Clark Barrett, Aaron Stump, Cesare Tinelli, and 1 others. 2010. The smt-lib standard: Version 2.0. In Proceedings of the 8th international workshop on satisfiability modulo theories (Edinburgh, UK), volume 13, page 14. find
  4. Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2022. A survey on data augmentation for text classification. ACM Computing Surveys, 55(7):1–39. find
  5. ByteDance. 2025. Seed1.6 tech introduction. https://seed.bytedance.com/en/seed1_6. Model referenced in paper as Seed1.6. ↗
  6. Guizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan, Chaoqun Liu, Lidong Bing, Deli Zhao, Anh Tuan Luu, and Yu Rong. 2025a. FINEREA-SON: Evaluating and Improving LLMs’ Deliberate Reasoning through Reflective Puzzle Solving. arXiv preprint. ArXiv:2502.20238 [cs]. — [arXiv:2502.20238](https://arxiv.org/abs/2502.20238) find
  7. Jianghao Chen, Zhenlin Wei, Zhenjiang Ren, Ziyong Li, and Jiajun Zhang. 2025b. LR 2 Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems. arXiv preprint. ArXiv:2502.17848 [cs]. — [arXiv:2502.17848](https://arxiv.org/abs/2502.17848) find
  8. Michael K Chen, Xikun Zhang, and Dacheng Tao. 2025c. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851.
  9. William F Clocksin and Christopher S Mellish. 2003. Programming in PROLOG. Springer Science & Business Media. find
  10. Bálint Csanády, Lajos Muzsai, Péter Vedres, Zoltán Nádasdy, and András Lukács. 2024. Llambert: Large-scale low-cost data annotation in nlp. arXiv preprint arXiv:2403.15938. — [arXiv:2403.15938](https://arxiv.org/abs/2403.15938) find
  11. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: an efficient smt solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’08/ETAPS’08, page 337–340, Berlin, Heidelberg. Springer-Verlag. find
  12. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.
  13. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
  14. Gemini Team. 2025a. Gemini 2.0 flash. https://cloud.google.com/vertex-ai/generative-ai/ docs/models/gemini/2-0-flash. Model referenced in paper as Gemini-2.0-flash. ↗
  15. Gemini Team. 2025b. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, Google DeepMind. Covers Gemini 2.5 Pro (and 2.5 Flash); model referenced in paper as Gemini-2.5-pro.
  16. Gemini Team. 2025c. Gemini 3 pro: the frontier of vision ai. https://blog.google/technology/ developers/gemini-3-pro-vision/. Model referenced in paper as Gemini-3-pro. ↗
  17. Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. Logicgame: Benchmarking rule-based reasoning abilities of large language models. arXiv preprint arXiv:2408.15778. find
  18. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Zongfang Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, DeepSeek-AI Team, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability…
  19. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021a. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  21. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. NeurIPS.
  22. Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and 1 others. 2024. Datagen: Unified synthetic dataset generation via large language models. In The Thirteenth International Conference on Learning Representations. find
  23. Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Mehta, Lalit D. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri S. Shalit, Yi Tay, Vinh Q. Tran, Quoc V. Le, and Orhan Firat. 2025. Big-bench extra hard. arXi…
  24. Joshua Ong Jun Leang, Giwon Hong, Wenda Li, and Shay B. Cohen. 2025. Theorem Prover as a Judge for Synthetic Data Generation.
  25. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F Chen, Zhengyuan Liu, and Diyi Yang. 2023. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638.
  26. Naiqi Li, Peiyuan Liu, Zheng Liu, Tao Dai, Yong Jiang, and Shu-Tao Xia. 2025. Logic-of-thought: Empowering large language models with logic programs for solving puzzles in natural language. arXiv preprint arXiv:2505.16114. — [arXiv:2505.16114](https://arxiv.org/abs/2505.16114) find
  27. Junqi Liu, Xiaohan Lin, Jonas Bayer, Yael Dillies, Weijie Jiang, Xiaodan Liang, Roman Soletskyi, Haiming Wang, Yunzhou Xie, Beibei Xiong, and 1 others. 2025a. Combibench: Benchmarking llm capability for combinatorial mathematics. arXiv preprint arXiv:2505.03171. find
  28. Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, and Junxian He. 2025b. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. Prepri… — [arXiv:2505.19641](https://arxiv.org/abs/2505.19641) find
  29. Yingzhou Lu, Minjie Shen, Huazheng Wang, Xiao Wang, Capucine van Rechem, Tianfan Fu, and Wenqi Wei. 2023. Machine learning for synthetic data generation: a review. arXiv preprint arXiv:2302.04062. — [arXiv:2302.04062](https://arxiv.org/abs/2302.04062) find
  30. Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational…
  31. Man Luo, Shrinidhi Kumbhar, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, and 1 others. 2023. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00836.
  32. MAA and Benchmark users. 2023. find
  33. Amc 2023 benchmark (american mathematics competition 2023). Used as evaluation benchmark in LLM reasoning research. find
  34. MAA and Benchmark users. 2024. find
  35. Aime 2024 benchmark (american invitational mathematics examination 2024). Used as evaluation benchmark in LLM reasoning research. find
  36. MAA and Benchmark users. 2025. find
  37. Aime 2025 benchmark (american invitational mathematics examination 2025). Used in recent benchmarks like SAT-Bench analysis, etc. find
  38. Anirudh Maiya, Razan Alghamdi, Maria Leonor Pacheco, Ashutosh Trivedi, and Fabio Somenzi. 2025. find
  39. Explaining puzzle solutions in natural language: An exploratory study on 6x6 sudoku. arXiv preprint arXiv:2505.15993. find
  40. Math-AI. 2025. Aime25 dataset. Accessed: 2025-07-01. find
  41. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229.
  42. Chinmay Mittal, Krishna Kartik, Mausam, and Parag Singla. 2024. PuzzleBench: Can LLMs Solve Challenging First-Order Combinatorial Reasoning Problems? arXiv preprint. ArXiv:2402.02611 [cs] version: 2. — [arXiv:2402.02611](https://arxiv.org/abs/2402.02611) find
  43. OpenAI. 2024. Gpt-4o system card. https://arxiv.org/abs/2410.21276. Model referenced in paper as ChatGPT-4o. ↗
  44. OpenAI. 2025a. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. Model referenced in paper as ChatGPT-5. ↗
  45. OpenAI. 2025b. Openai o3 and o4-mini system card. https://openai.com/index/ o3-o4-mini-system-card/. Model referenced in paper as ChatGPT-o3. ↗
  46. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824, Singapore. Association for Computational Lingui…
  47. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin…
  48. THUDM. 2025. Glm-z1-32b-0414. https://huggingface.co/THUDM/GLM-Z1-32B-0414. GLM-Z1 reasoning model series (incl. 9B/32B); model card. ↗
  49. Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024a. A Survey on Data Synthesis and Augmentation for Large Language Models, volume 1. Association for Computing Machinery. find
  50. Yifan Wang, David Stevens, Pranay Shah, Wenwen Jiang, Miao Liu, Xu Chen, Robert Kuo, Na Li, Boying Gong, Daniel Lee, and 1 others. 2024b. Model-in-the-loop (milo): Accelerating multimodal ai data annotation with llms. arXiv preprint arXiv:2409.10702. — [arXiv:2409.10702](https://arxiv.org/abs/2409.10702) find
  51. Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, and Alex Aiken. 2025a. Satbench: Benchmarking llms’ logical reasoning via automated puzzle generation from sat formulas. arXiv preprint arXiv:2505.14615. find
  52. Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, and Alex Aiken. 2025b. Satbench: Benchmarking llms’ logical reasoning via automated puzzle generation from sat formulas. arXiv preprint arXiv:2505.14615. find
  53. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522.
  54. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024a. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
  55. Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. 2024. AI-Assisted Generation of Difficult Math Questions. find
  56. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464.
  57. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Introduces Group Relative Policy Optimizatio…
  58. Amane Sugiyama and Naoki Yoshinaga. 2019. Data augmentation using back-translation for context-aware neural machine translation. In Proceedings of the fourth workshop on discourse in machine translation (DiscoMT 2019), pages 35–44. find
  59. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 42 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  60. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 24 others. 2024. Qwen2.5 technical report. arXiv preprint arXi… find
  61. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. Llmaaa: Making large language models as active annotators. arXiv preprint arXiv:2310.19596. find
  62. Shikun Zhang, Omid Jafari, and Parth Nagarkar. 2021. A survey on machine learning techniques for auto labeling of video, audio, and text data. arXiv preprint arXiv:2109.03784. — [arXiv:2109.03784](https://arxiv.org/abs/2109.03784) find
  63. Zhen Zhu, Yibo Wang, Shouqing Yang, Lin Long, Runze Wu, Xiu Tang, Junbo Zhao, and Haobo Wang. 2024. Coral: Collaborative automatic labeling system based on large language models. Proc. VLDB Endow., 17(12):4401–4404. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2026-puzzleclone-dsl-powered-framework-synthesizing/