Elamur Research⌕
☰
home / papers / 2025 / JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

Paper · 2025 · Saibo Geng, Hudson Cooper, Michał Moskal et al.

Summary

Constrained decoding has become the standard way to make language models emit machine-readable output, and JSON Schema is the usual way to specify the required structure. However, frameworks rarely get compared systematically. This paper fills that gap. The authors propose an evaluation along three axes:

  • efficiency: how fast compliant output is produced;
  • coverage: how many schema features a framework actually handles;
  • quality: effect on downstream task accuracy.

To support this, they assembled JSONSchemaBench, roughly 10K real-world schemas grouped into 10 datasets of differing difficulty. Sources include GitHub, JSON Schema Store, GlaiveAI function calls and Kubernetes configurations. They complement it with the official JSON Schema Test Suite.

Six systems were tested: Guidance, Outlines, Llamacpp, XGrammar, OpenAI and Gemini. Constrained decoding can make generation about 50% faster than unconstrained sampling. Real-world schema support differs widely, with the strongest framework handling twice as many schemas as the weakest. Constraints also slightly improve reasoning-task accuracy. Guidance performs best overall. The benchmark shows that structured generation is far from solved and gives practitioners a basis for choosing tools.

Key points

  • The benchmark has 9,558 schemas across 10 datasets, including GlaiveAI-2K, Kubernetes, Snowplow, Washington Post, JSONSchemaStore and GitHub splits by size. GitHub-Trivial and GitHub-Ultra were left out of the experiments, being too easy or too hard; Ultra is kept as a future target.

  • On the JSON Schema Test Suite, failures were counted by category as compile errors, over-constraining or under-constraining:

    • Guidance had the fewest total failures, with only one under-constrained category.
    • XGrammar had just 3 compile-error categories but 38 under-constrained ones, meaning it leans toward permissiveness.

  • Quality was measured with Llama-3.1-8B-Instruct on three tasks: Last Letter, Shuffle Objects and GSM8K. Every constrained framework beat the unconstrained baseline.

  • Guidance gained about 3 points on each task compared with the unconstrained model:

    TaskUnconstrainedGuidance
    GSM8K80.1%83.8%
    Last Letters50.7%54.0%
    Shuffle Objects52.6%55.9%

    The authors attribute this partly to Guidance's token healing.

  • The benchmark is publicly available at github.com/guidance-ai/jsonschemabench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland44 works · 7,050 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz and Harsha Nori. 2025. JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. arXiv:2501.10868.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.10868 · arXiv 2501.10868

@misc{geng2025jsonschemabench,
  title = {{JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models}},
  author = {Saibo Geng, Hudson Cooper, Michal Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz and Harsha Nori},
  year = {2025},
  doi = {10.48550/arxiv.2501.10868},
  eprint = {2501.10868},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.10868}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Snowplow Analytics. Iglu central. https://github.com/snowplow/iglucentral, 2022. Commit hash 726168e. Retrieved 19 September 2022. ↗
  2. Lyes Attouche, Mohamed-Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger. Witness Generation for JSON Schema. Proceedings of the VLDB Endowment, 15(13):4002–4014, September 2022. ISSN 2150-8097. doi: 10.14778/3565838.3565852. URL https://dl.acm.org/doi/10.14778/3… — [doi:10.14778/3565838.3565852](https://doi.org/10.14778/3565838.3565852) ↗
  3. Mohamed Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger. A json schema corpus, 2021. https://github.com/sdbs-uni-p/ json-schema-corpus. ↗
  4. Julian Berman. python-jsonschema. https://github.com/python-jsonschema/ jsonschema, 2025. URL https://github.com/python-jsonschema/jsonschema. Accessed: 2025-01-05. ↗
  5. Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting Is Programming: A Query Language for Large Language Models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969, June 2023. ISSN 2475-1421. doi: 10.1145/3591300. URL http://arxiv.org/abs/2212.06094. arXiv:2212.06094 [cs]. ↗
  6. Bowtie. Bowtie: A meta-validator of the json schema specification, 2025. URL https://github.com/bowtie-json-schema/bowtie/. DOI: 10.5281/zenodo.14646449. — [doi:10.5281/zenodo.14646449](https://doi.org/10.5281/zenodo.14646449) ↗
  7. Daniel Deutsch, Shyam Upadhyay, and Dan Roth. A General-Purpose Algorithm for Constrained Sequential Inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 482–492, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/K… find
  8. Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models, November 2024. URL http://arxiv.org/abs/2411.15100. arXiv:2411.15100 [cs]. ↗
  9. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained decoding for structured NLP tasks without finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10932–10952, Singa… find
  10. Saibo Geng, Sankalp Gambhir, Chris Wendler, and Robert West. Byte bpe tokenization as an inverse string homomorphism, 2024. URL https://arxiv.org/abs/2412.03160. — [arXiv:2412.03160](https://arxiv.org/abs/2412.03160) ↗
  11. Georgi Gerganov and al. Llama.cpp: A port of facebook’s llama model in c++. https://github.com/ggerganov/llama.cpp, 2023. Accessed: 2025-01-16. ↗
  12. GlaiveAI. Glaive function calling dataset. https://huggingface.co/datasets/ glaiveai/glaive-function-calling, 2024. Accessed: 2024-12-21. ↗
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  14. Guidance AI. Guidance: A language model programming framework, 2023. URL https://github.com/guidance-ai/guidance. Accessed: 2024-12-18. ↗
  15. GuidanceAI. Prompt boundaries and token healing. https://github.com/guidance-ai/ guidance/blob/main/notebooks/art_of_prompt_design/prompt_boundaries_ and_token_healing.ipynb, 2024a. Accessed: 2024-12-21. ↗
  16. GuidanceAI. Guidance acceleration tutorial. https://guidance.readthedocs.io/en/ stable/example_notebooks/tutorials/guidance_acceleration.html, 2024b. Accessed: 2025-01-16. ↗
  17. JSON Schema Org. Json schema test suite. https://github.com/json-schema-org/ JSON-Schema-Test-Suite, 2024. URL https://github.com/json-schema-org/ JSON-Schema-Test-Suite. Accessed: 2024-12-19. ↗
  18. Kubernetes. Kubernetes json schemas. https://github.com/instrumenta/ kubernetes-json-schema, 2022. Commit hash 133f848. ↗
  19. Michael Kuchnik, Virginia Smith, and George Amvrosiadis. Validating Large Language Models with ReLM, May 2023. URL http://arxiv.org/abs/2211.15458. arXiv:2211.15458 [cs]. — [arXiv:2211.15458](https://arxiv.org/abs/2211.15458) ↗
  20. Will Kurt. Coalescence: Making llm inference 5x faster. https://blog.dottxt.co/ coalescence.html, 2024a. Accessed: 2024-12-21. ↗
  21. Will Kurt. Say what you mean: A response to ’let me speak freely’, 2024b. URL https://.txt.co/blog/say-what-you-mean-a-response-to-let-me-speak-freely. ↗
  22. Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. "We Need Structured Output": Towards User-centered Constraints on Large Language Model Output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1… — [doi:10.1145/3613905.3650756](https://doi.org/10.1145/3613905.3650756) find
  23. Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models, January 2022. URL http://arxiv.org/abs/2201.11227. arXiv:2201.11227 [cs]. ↗
  24. Maciej P. Polak and Dane Morgan. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communications, 15(1), February 2024. ISSN 2041-1723. doi: 10.1038/s41467-024-45914-8. URL http://dx.doi.org/10.1038/s41467-024-45914-8. — [doi:10.1038/s41467-024-45914-8](https://doi.org/10.1038/s41467-024-45914-8) ↗
  25. The Washington Post. ans-schema. https://github.com/washingtonpost/ans-schema, 2022. Commit hash abdd6c211. Retrieved 19 September 2022. ↗
  26. Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing, January 2024. URL http://arxiv.org/abs/2206. 10668. arXiv:2206.10668 [cs]. — [arXiv:2206.10668](https://arxiv.org/abs/2206.10668) ↗
  27. Schema Store Org. The largest collection of independent json schemas in the world. https://www.schemastore.org/json/, 2014. A universal JSON schema store where schemas for popular JSON documents can be found. Contributions are welcome; see CONTRIBUTING.md for more information. ↗
  28. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in N…
  29. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural… find
  30. Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Constrained Language Models Yield Few-Shot Semantic Parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language… find
  31. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conf… find
  32. Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Amer… find
  33. Vivien. Llm decoding with regex constraints. https://vivien000.github.io/blog/ journal/llm-decoding-with-regex-constraints.html, 2024. Accessed: 2024-12-21. ↗
  34. Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A Saurous, and Yoon Kim. Grammar Prompting for Domain-Specific Language Generation with Large Language Models. Brandon T. Willard and Rémi Louf. Efficient Guided Generation for Large Language Models, August 2023. URL http://arxiv.org/abs/2307.09702. a… ↗
  35. Austin Wright, Henry Andrews, Ben Hutton, and Greg Dennis. Draft 2020-12: Json schema core specification. https://json-schema.org/draft/2020-12/ json-schema-core.html, 2022. Published 16 June 2022. Metaschema available at https://json-schema.org/draft/2020-12/schema. ↗
  36. Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. COLLIE: Systematic Construction of Constrained Text Generation Tasks, July 2023a. URL http://arxiv.org/abs/2307.08689. arXiv:2307.08689 [cs]. ↗
  37. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b.
  38. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient Execution of Structured Language Model Programs, June 2024. URL http://arxiv.org/abs/2312.07104. arXiv:231… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geng-2025-jsonschemabench-rigorous-benchmark-structured-outputs/