Elamur Research⌕
☰
home / papers / 2026 / Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning

Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning

Paper · 2026 · Max Schaffelder and Albert Gatt

Summary

LLM developers increasingly train on machine-generated text, and such data is known to narrow a model's output distribution. This paper tests whether drawing synthetic fine-tuning data from several generator models, instead of only one, reduces this harm. The authors took the human-written Databricks-Dolly-15K dataset and had LLMs from three size brackets (5–15B, 50–150B, 400B+) regenerate its responses. For each bracket they built a single-source version (Llama-3.1 only) and a multi-source version (four different models). The original human data served as a control. Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct were then LoRA fine-tuned on each variant and evaluated for distribution collapse, jailbreak robustness and self-preference bias when acting as judges. Mixing sources helped preserve the breadth and textual diversity of outputs. Both human and synthetic data weakened safety guardrails, but models tuned on synthetic data tended to give higher-quality harmful answers, which makes those answers more usable and therefore riskier. Fine-tuning reduced self-preference bias most with human data, followed by multi-source synthetic data. These results matter for anyone building fine-tuning sets from model outputs.

Key points

  • Generator models:
    • Small: Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-2-9B, Falcon3-10B-instruct.
    • Medium: Llama-3.1-70B-Instruct, Qwen-2.5-72B-Instruct, Mistral-Large-Instruct-2411, Cohere Command R+.
    • Large: Llama-3.1-405B-Instruct, Gemini 2.0 Flash, GPT-4o, DeepSeek-V3.
  • Data generation: 90% of the Dolly-15k prompts were used for training. Responses were generated with temperature 0.7, top_p 0.9 and at most 1024 output tokens.
  • Fine-tuning setup: LoRA (r = 16, α = 32) in 16-bit precision on H100 GPUs, run for 3 epochs. Training used learning rate 5e-5, AdamW, 3% warm-up, cosine decay and weight decay 0.01.
  • Robustness finding: adversarial robustness depends jointly on how diverse the synthetic sources are and how large the generating models are. Llama-3.1-70B also served as the judge model.
  • Stated limitations:
    • English-only data and single-turn interactions.
    • LoRA SFT only, with no DPO or RLHF.
    • Target models from a single family.
    • Reliance on LLM-as-judge scoring.
    • Source diversity cannot be fully separated from general heterogeneity of the data.
  • Code: available at github.com/maxschaffelder/synthetic_data_diversity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Max Schaffelder and Albert Gatt. 2026. Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.360

@inproceedings{schaffelder2026synthetic,
  title = {{Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning}},
  author = {Max Schaffelder and Albert Gatt},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.360},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.360}
}

References

63 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah I Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio CT Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim… — Phi-4 Technical Report
  2. Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. 2024. Self-consuming generative models go MAD. In The Twelfth International Conference on Learning Representations. find
  3. Aaron Chatterji, Thomas Cunningham, David J. Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How People Use ChatGPT. Working Paper 34255, National Bureau of Economic Research. find
  4. Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah Abdin. 2024. On the diversity of synthetic data and its impact on training large language models. ArXiv, abs/2410.15226.
  5. Cohere. 2024. Command R and command R+ model card. https://docs.cohere.com/docs/ responsible-use. Accessed: 2025-01-08. ↗
  6. Cohere. 2025. Cohere chat api. https://docs.cohere.com/v2/docs/chat-api. Accessed: 2025-06-10. ↗
  7. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in Language Models Is Mediated by a Single Direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  8. Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned LLM. Databricks blog post. find
  9. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2025. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122. find
  10. Deepinfra. 2025. Deepinfra. https://deepinfra.com/. Accessed: 2025-06-10. ↗
  11. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  12. DeepSeek-AI, Daya Guo, Dejian Yang, and et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  13. Alexander Bick, Adam Blandin, and David J. Deming. 2024. The Rapid Adoption of Generative AI. Working Paper 32966, National Bureau of Economic Research. find
  14. Gemini Team, Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
  15. Abeba Birhane, Sepehr Dehdashtian, Vinay Uday Prabhu, and Vishnu Boddeti. 2024. The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FaCCT), pages 1229–1244. ArXiv:2405.04623 [cs]. find
  16. Thomas F. Burns, Letitia Parcalabescu, Stephan Wäldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, and Björn Deiseroth. 2025. Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation. arXiv preprint. ArXiv:2…
  17. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  18. Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. 2024. A Tale of Tails: Model Collapse as a Change of Scaling Laws. arXiv preprint. ArXiv:2402.07043 [cs]. — [arXiv:2402.07043](https://arxiv.org/abs/2402.07043) find
  19. Falcon-LLM Team. 2024. The falcon 3 family of open models. https://huggingface.co/blog/ falcon3. Accessed: 2025-10-08. ↗
  20. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. find
  21. Google. 2025. Gemini developer api. https://ai.google.dev/gemini-api/docs. Accessed: 2025-06-10. ↗
  22. Groq. 2025. Groq. https://groq.com/. Accessed: 2025-06-10. ↗
  23. Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2024. The curious decline of linguistic diversity: Training language models on synthetic text. Preprint, arXiv:2311.09807. find
  24. Harold Stanley Heaps. 1978. Information Retrieval: Computational and Theoretical Aspects. Academic Press, London. find
  25. Meta AI. 2024b. Llama 3 Use Policy. https://www.llama.com/llama3/use-policy/. Accessed: 2025-06-11. ↗
  26. Mark Hennings. 2023. Lora fine-tuning & hyperparameters explained (in plain english). https://www.entrypointai.com/blog/lora-fine-tuning/. Accessed: 2025-06-30. ↗
  27. Mistral AI. 2024. Mistral large instruct 2411. https://huggingface.co/mistralai/ Accessed: Mistral-Large-Instruct-2411. 2025-10-08. ↗
  28. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  29. Mistral AI. 2025. Mistral ai api (version 1.0.0). https://docs.mistral.ai/api. Accessed: 2025-06-10. ↗
  30. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  31. OpenAI. 2025. Openai api. https://openai.com/ api/. Accessed: 2025-06-10. ↗
  32. Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D Hwang, Kai-Wei Chang, and Ranjay Krishna. 2025. Scale Can’t Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning. find
  33. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv preprint. ArXiv:2001.08361 [cs, stat].
  34. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Metho…
  35. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B. Preprint, arXiv:2310.20624. find
  36. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2024. Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators. arXiv preprint. ArXiv:2403.16950 [cs].
  37. The AI Index 2025 Annual Report. Technical report, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA. find
  38. Meta AI. 2024a. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  39. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. Preprint, arXiv:1602.06023.
  40. OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774.
  41. Preprint, Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. find
  42. Juan N Pava, Caroline Meinhardt, Haifa Badi Uz Zaman, Toni Friedman, Sang T Truong, Daniel Zhang, Elena Cryst, Vukosi Marivate, and Sanmi Koyejo. 2025. Mapping the Challenges of LLM Development in Low-Resource Language Contexts. Technical report, HAI: Stanford University Human-Centered Artificial In… find
  43. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  44. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations.
  45. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2025. A survey of multilingual large language models. Patterns, 6(1). find
  46. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
  47. Sebastian Raschka. 2023. Practical tips for fine-tuning llms using lora (low-rank adaptation). https://magazine.sebastianraschka.com/ p/practical-tips-for-finetuning-llms. Accessed: 2025-06-30. ↗
  48. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084. rubend18. 2023. Chatgpt-jailbreak-prompts. https://huggingface.co/datasets/rubend18/ ChatGPT-Jailbreak-Prompts. Dataset available at Hugging Face. ↗
  49. Isik Baran Sandan, Tu Anh Dinh, and Jan Niehues. 2025. Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons. arXiv preprint. ArXiv:2506.03785 [cs]. find
  50. Rylan Schaeffer, Joshua Kazdan, Alvan Caleb Aru-landu, and Sanmi Koyejo. 2025. Position: Model collapse does not mean what you think. Preprint, arXiv:2503.03150. — [arXiv:2503.03150](https://arxiv.org/abs/2503.03150) find
  51. Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  52. Jiaxu Zhao, Meng Fang, Fanghua Ye, Ke Xu, Qin Zhang, Joey Tianyi Zhou, and Mykola Pechenizkiy. 2025. Understanding Large Language Model Vulnerabilities to Social Bias Attacks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17… find
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  54. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759. find
  55. Alan Zhu, Parth Asawa, Jared Quincy Davis, Lingjiao Chen, Boris Hanin, Ion Stoica, Joseph E. Gonzalez, and Matei Zaharia. 2025. BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation. arXiv preprint. ArXiv:2502.01697 [cs]. find
  56. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Will we run out of data? limits of llm scaling based on human-generated data. Preprint, arXiv:2211.04325. find
  57. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631.
  58. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. Preprint, arXiv:2410.21819.
  59. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A Benchmarking Platform for Text Generation Models. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097–1100. ACM.
  60. Peter West and Christopher Potts. 2025. Base Models Beat Aligned Models at Randomness and Creativity. arXiv preprint. ArXiv:2505.00047 [cs]. find
  61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  62. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474–15492, Bangkok, Thail…
  63. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schaffelder-2026-synthetic-eggs-many-baskets-impact/