Elamur Research⌕
☰
home / papers / 2026 / Explain the Synth: Interpretable Evaluation of LLM Data Synthesis

Explain the Synth: Interpretable Evaluation of LLM Data Synthesis

Paper · 2026 · Yue Yang, Fan Yang, Yu Bai et al.

Summary

LLMs are increasingly used to create synthetic tabular data. Standard checks, such as distributional similarity tests (e.g., C2ST) and train-on-synthetic/test-on-real (TSTR) utility scores, can flag that a synthetic table is off but cannot show which feature–label dependencies the generator distorted. The authors propose an auditing framework that uses an independent, transparent rule learner as a common vocabulary for comparing real and synthetic data. A rule ensemble trained on real data (Rule set A) is first scored on the synthetic table. It is then warm-started and trained further on the synthetic data to obtain Rule set B. Orthogonal Gradient Boosting (OGB) forces each new rule to be orthogonal to earlier ones, so the A→B difference separates reweighting of known patterns from genuinely new structure. Four metrics quantify utility transfer and drift, and a theoretical analysis links these gaps to distribution mismatch within the rule hypothesis class. Experiments on six tabular benchmarks with several LLM generators show that score-only evaluation can mislead, while rule diffs pinpoint concrete mechanism changes.

Key points

  • Pipeline: Rule set A is learned on real data and used as a baseline audit of the synthetic data. Warm-started OGB training on the synthetic data then yields Rule set B. Rules are threshold conjunctions that practitioners can read directly.
  • Metrics and theory: Four metrics capture utility transfer and synthetic drift. The analysis assumes an exponential-family / canonical GLM setting and gives guarantees for when drift can be corrected by reweighting existing rules, plus bounds on cross-domain utility transfer.
  • California housing case study (HARMONIC):
    • In the real-data rules, latitude below roughly 34° (Southern California) is a positive signal across three rules (weights +0.4285, +0.3753, +0.3399).
    • The HARMONIC rules instead contain a negative rule (−0.2492) over an overlapping latitude range, which inverts the geographic mechanism.
    • HARMONIC preserves local dependencies and predictive utility but loses the global explanatory structure, so passing predictive tests does not imply explanation fidelity.
  • Limitations:
    • The method is meant for inspecting generators and does not serve to certify any of them.
    • It does not address privacy leakage, memorization, or fairness.
    • A large-scale human meta-evaluation linking the metrics to rule interpretability is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash UniversityMelbourne · Australia64 works · 54,971 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yue Yang, Fan Yang, Yu Bai and Hao Wang. 2026. Explain the Synth: Interpretable Evaluation of LLM Data Synthesis. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1995

@inproceedings{yang2026explain,
  title = {{Explain the Synth: Interpretable Evaluation of LLM Data Synthesis}},
  author = {Yue Yang, Fan Yang, Yu Bai and Hao Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1995},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1995}
}

References

38 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Advances in neural information processing systems, 31. find
  2. Arthur Asuncion and David Newman. 2007. Uci machine learning repository. find
  3. Clément Bénard, Gérard Biau, Sébastien Da Veiga, and Erwan Scornet. 2021. Sirus: Stable and interpretable rule set for classification. find
  4. Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. 2022. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280. — [arXiv:2210.06280](https://arxiv.org/abs/2210.06280) find
  5. William W Cohen. 1995. Fast effective rule induction. In Proceedings of the twelfth international conference on machine learning, pages 115–123. find
  6. Joao Fonseca and Fernando Bacao. 2023. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1):115. find
  7. Jerome H Friedman and Bogdan E Popescu. 2008. Predictive learning via rule ensembles. find
  8. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097– 1179.
  9. Mauro Giuffrè and Dennis L Shung. 2023. Harnessing the power of synthetic data in healthcare: innovation, application, and privacy. NPJ digital medicine, 6(1):186. find
  10. Mikel Hernandez, Pablo A Osorio-Marulanda, Mikel Catalina, Lorea Loinaz, Gorka Epelde, and Naiara Aginako. 2025. Comprehensive evaluation framework for synthetic tabular data in health: fidelity, utility and privacy analysis of generative models with and without privacy guarantees. Frontiers in Digi… find
  11. Jan Kapar, Niklas Koenen, and Martin Jullum. 2025. What’s wrong with your synthetic tabular data? using explainable ai to evaluate generative models. In World Conference on Explainable Artificial Intelligence, pages 19–43. Springer. find
  12. Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. 2023. Tabddpm: Modelling tabular data with diffusion models. In International conference on machine learning, pages 17564–17579. PMLR. find
  13. Sean Kulinski and David I Inouye. 2023. Towards explaining distribution shifts. In International Conference on Machine Learning, pages 17931–17952. PMLR. 43062 9 find
  14. Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2019. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1096. find
  15. Anton D Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp. 2025. Syntheval: a framework for detailed utility and privacy evaluation of tabular synthetic data. Data Mining and Knowledge Discovery, 39(1):6. find
  16. David Lopez-Paz and Maxime Oquab. 2016. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545. — [arXiv:1610.06545](https://arxiv.org/abs/1610.06545) find
  17. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
  18. Peter McCullagh. 2019. Generalized linear models. Routledge. find
  19. Ravid Shwartz-Ziv and Amitai Armon. 2022. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90. find
  20. Aivin V Solatorio and Olivier Dupriez. 2023. Realtabformer: Generating realistic relational and tabular data using transformers. arXiv preprint arXiv:2302.02041. — [arXiv:2302.02041](https://arxiv.org/abs/2302.02041) find
  21. Marieke Stolte, Franziska Kappenberg, Jörg Rahnen-führer, and Andrea Bommert. 2024. Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statistic Surveys, 18:163–298. find
  22. Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2024. Harmonic: Harnessing llms for tabular data synthesis and privacy protection. Advances in Neural Information Processing Systems, 37:100196–100212. find
  23. Marko Miletic and Murat Sariyar. 2024. Challenges of using synthetic data generation methods for tabular microdata. Applied Sciences, 14(14):5975. find
  24. Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. 2019. Modeling tabular data using conditional gan. Advances in neural information processing systems, 32. find
  25. Carlos Mougan, Klaus Broelemann, David Masip, Gjergji Kasneci, Thanassis Thiropanis, and Steffen Staab. 2023. Explanation shift: How did the distribution shift impact the model? arXiv preprint arXiv:2303.08081. — [arXiv:2303.08081](https://arxiv.org/abs/2303.08081) find
  26. Fan Yang, Pierre Le Bodic, Michael Kamp, and Mario Boley. 2024. Orthogonal gradient boosting for simpler additive rule ensembles. In International Conference on Artificial Intelligence and Statistics, pages 1117–1125. PMLR. find
  27. Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909.09223. — [arXiv:1909.09223](https://arxiv.org/abs/1909.09223) find
  28. Yue Yang, Fan Yang, Yu Bai, and Hao Wang. 2025. Self-interpretable reinforcement learning via rule ensembles. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’25, page 2235–2243, Richland, SC. International Foundation for Autonomous Agents and M… find
  29. Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. 2016. The synthetic data vault. In 2016 IEEE international conference on data science and advanced analytics (DSAA), pages 399–410. IEEE. find
  30. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and 1 others. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830. find
  31. Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32. find
  32. Pol G Recasens, Alberto Gutierrez, Jordi Torres, Josep Berral, Anisa Halimi, Kieran Fraser, and 1 others. 2025. In-context bias propagation in llm-based tabular data generation. arXiv preprint arXiv:2506.09630. — [arXiv:2506.09630](https://arxiv.org/abs/2506.09630) find
  33. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215.
  34. Yue Yang, Fan Yang, Yu Bai, and Hao Wang. 2026. Neural+symbolic approaches for interpretable actor-critic reinforcement learning. In The Fourteenth International Conference on Learning Representations. find
  35. Zexi Yao, Nataša Krčo, Georgi Ganev, and Yves-Alexandre de Montjoye. 2025. The dcr delusion: Measuring the privacy risk of synthetic data. In European Symposium on Research in Computer Security, pages 469–487. Springer. find
  36. Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. 2023. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656. — [arXiv:2310.09656](https://arxiv.org/abs/2310.09656) find
  37. Jun Zhang, Graham Cormode, Cecilia M Procopiuc, Divesh Srivastava, and Xiaokui Xiao. 2017. Privbayes: Private data release via bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1– 41. 43063 10 find
  38. Zilong Zhao, Robert Birke, and Lydia Y Chen. 2025. Tabula: Harnessing language models for tabular data synthesis. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 247–259. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-explain-synth-interpretable-evaluation-llm/