Elamur Research⌕
☰
home / papers / 2026 / LLMSurgeon: Diagnosing Data Mixture of Large Language Models

LLMSurgeon: Diagnosing Data Mixture of Large Language Models

Paper · 2026 · Yaxin Luo, Jiacheng Cui, Xiaohan Zhao et al.

Summary

LLM developers rarely disclose the composition of their pretraining data. This makes it hard to audit models for bias, copyright exposure or uneven performance across domains. Membership inference attacks work one document at a time and scale poorly to corpus-level questions. The authors define Data Mixture Surgery (DMS), the task of estimating the domain proportions of a model's training corpus, under a fixed taxonomy, from generated text alone.

Their method, LLMSurgeon, relies on the label-shift assumption: domain frequencies may differ between training data and generations, but each domain's linguistic profile stays stable. It works in three steps:

  • A DistilBERT classifier trained on reference data yields a calibrated soft confusion matrix.
  • The target model is sampled with neutral prompts and its outputs are classified.
  • The skewed class counts are corrected by solving a constrained inverse problem.

For evaluation, they release LLMScan, a benchmark of open models with published data recipes. LLMSurgeon outperforms aggregation-based baselines and transfers to held-out models, though accuracy falls when domains overlap. It is a lightweight black-box auditing tool that needs neither weights nor training data.

Key points

  • LLMScan covers 8 open models from 1B to 65B parameters: LLaMA-1, OLMo, Amber, Pythia, GPT-Neo and StarCoder. It tests three granularity levels: coarse (K=6, SlimPajama-DC domains), mid (K=17, The Pile) and fine (The Stack).
  • In a held-out test, evaluation settings were frozen on earlier OLMo releases and then applied to OLMo-3. LLMSurgeon reached 86.41% overlap accuracy and recovered the Web share closely (76.88% true vs. 75.37% predicted).
  • In a controlled GPT-2 sandbox with 7 domains, overlap accuracy was 75.62% for the balanced mix, 87.53% for the web-heavy mix and only 50.15% for the book-heavy mix. The authors attribute the weak book-heavy result to semantic overlap between domains.
  • In a safety test, 5%, 10% or 20% of GPT-2 training tokens were replaced with RealToxicityPrompts. LLMSurgeon estimated the toxic share at 7.90%, 12.00% and 22.73%, which tracks the true amounts in the right order.
  • Stated limitations:
    • The method may be unreliable for heavily aligned models (RLHF or instruction tuning).
    • It works only within a fixed set of categories and cannot discover new domains.
    • It is unstable for near-identical domains, such as C vs. C++.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li and Zhiqiang Shen. 2026. LLMSurgeon: Diagnosing Data Mixture of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1964

@inproceedings{luo2026llmsurgeon,
  title = {{LLMSurgeon: Diagnosing Data Mixture of Large Language Models}},
  author = {Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li and Zhiqiang Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1964},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1964}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318. find
  2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on… find
  3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. find
  4. Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S Weld. 2020. Tldr: Extreme summarization of scientific documents. arXiv preprint arXiv:2004.15011.
  5. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. find
  6. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE. find
  7. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021a. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
  8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021b. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–265…
  9. Bowen Chen, Namgi Han, and Yusuke Miyao. 2025. A statistical and multi-perspective revisiting of the membership inference attack in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22854–22874. find
  10. Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, and Jinyang Gao. 2024a. Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International Conference on Management of Data, pages 120–134. find
  11. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jartu Olmo, Jelena Diakonikolas, and Christopher Re. 2024b. Skill-it! a data-driven skills framework for understanding and training language models. In International Conference on Learning Representations (ICLR). find
  12. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841.
  13. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  14. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the association for computational linguistics: EMNLP 2020, pages 3356–3369.
  15. Gemini Team. 2025. Gemini 3 technical report. find
  16. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024. Olmo: Accelerating the science of language models. arXiv:2402.00838. arXiv preprint Robert Kirk, Ishita Mediratta, Christoforos Na… find
  17. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and 1 others. 2022. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533. find
  18. Raymond Li, Loubna Ben Allal, Denis Kocetkov, Chenghao Mou, Carlos Muñoz Ferrandis, Sean Santacroce, Sean Hughes, Younes Belkada, and 1 others. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. find
  19. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  20. Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Paik, Atanc Li, Gen Li, Xinyang Geng, Renrubin Wang, Yiran Sun, and 1 others. 2023. Llm360: Towards fully transparent open-source llms. In arXiv preprint arXiv:2312.06550. Reference for Amber. find
  21. Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. 2021. Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706. — [arXiv:2104.10706](https://arxiv.org/abs/2104.10706) find
  22. Justus Mattern, Fatemeh Shao, Alexandre Sablayrolles, and Pierre Kochems. 2023. Membership inference against language models via neighborhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330–11343. find
  23. Matthieu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marek Rei, and Yves-Alexandre de Montjoye. 2025. Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it). In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 385–401. IEEE. find
  24. OpenAI. 2025. Introducing GPT-5.2. OpenAI Blog. find
  25. Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and 1 others. 2023. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818.
  26. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Dennis Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. find
  27. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18.
  28. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, and 1 others. 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd annual meeting… find
  29. Yao Tong, Jiayuan Ye, Sajjad Zarifzadeh, and Reza Shokri. 2025. How much of my dataset did you use? quantitative data usage inference in machine learning. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models. find
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  31. Jeffrey G Wang, Jason Wang, Marvin Li, and Seth Neel. 2024. Pandora’s white-box: Precise training data detection and extraction in large language models. arXiv preprint arXiv:2402.17012. — [arXiv:2402.17012](https://arxiv.org/abs/2402.17012) find
  32. Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. 2022. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations. find
  33. Zehao Xiao, Jiayi Shen, Mohammad Mahdi Der-akhshani, Shengcai Liao, and Cees GM Snoek. 2024. Any-shift prompting for generalization over distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13849–13860. find
  34. Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. 2024. Recall: Membership inference via relative conditional log-likelihoods. arXiv preprint arXiv:2406.15968. find
  35. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifan Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023a. Doremi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS. find
  36. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023b. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems (NeurIPS). find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  38. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE.
  39. Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. 2024a. Membership inference attacks cannot prove that a model was trained on your data. arXiv preprint arXiv:2409.19798. — [arXiv:2409.19798](https://arxiv.org/abs/2409.19798) find
  40. Jingyang Zhang, Yaming Bao, Deming Wen, Houqiang Fan, Wenyuan Lin, Kai Liu, Wayne Xin Zhao, and Ji-Rong Wen. 2024b. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936. — [arXiv:2404.02936](https://arxiv.org/abs/2404.02936) find
  41. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024c. Pre-training data detection for large language models: A divergence-based calibration method. arXiv preprint arXiv:2409.14781. — [arXiv:2409.14781](https://arxiv.org/abs/2409.14781) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-llmsurgeon-diagnosing-data-mixture-large/