Elamur Research⌕
☰
home / papers / 2026 / Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training

Paper · 2026 · Lei Liu, Hao Zhu, Xiaoyan Yang et al.

Summary

Continual pre-training (CPT) adapts general LLMs to specialised domains such as medicine. Simply adding more domain tokens quickly stops paying off, because classical scaling laws treat every token as equally useful. This paper proposes a perplexity-aware data scaling law. The frozen base model runs one forward pass over unlabeled domain text, and the resulting perplexity distribution (the "perplexity landscape") serves as a cheap signal of where the model's knowledge gaps lie. The mean and variance of PPL become an importance factor Q, combined with dataset size D in a Chinchilla-style loss term. The law is fitted on small pilot runs and then used to choose training subsets of moderate difficulty, avoiding both redundant low-PPL text and noisy high-PPL text. The authors test this selection strategy (DOS) on Qwen3 and Deepseek-V3-Base against random, quantity-based, low-PPL and high-PPL sampling. DOS gives the best medical scores and slightly improves general-domain results. The method therefore offers a label-free, retraining-free way to get more out of fixed CPT token budgets.

Key points

  • Scaling-law form: loss is modeled as E + (D_c / (Q·D))^α_D, where Q(µ, σ) = µ^α_µ · σ^α_σ combines the mean and variance of the PPL distribution. Fitting is illustrated on Qwen3-0.6B-Base and Qwen3-14B-Base with 40B tokens.
  • Medical CPT results (100B tokens): DOS averages 72.48, versus 71.34 for random sampling (RS) and 71.22 for low-PPL sampling (LPS). This is +3.16 over the base model, with notable gains on DiagnosisArena and GPQA-Med.
  • Deepseek-V3-Base: DOS-CPT reaches 75.26 on medical benchmarks, versus 74.24 for RS-CPT and 71.79 for LPS-CPT. It also beats quantity-based sampling (QBS).
  • General domain (CEVAL, CMMLU, MMLU): DOS averages 84.43 against a base of 84.16, which suggests reduced catastrophic forgetting.
  • Budgets and limitations: DOS leads at every budget tested (10B, 50B and 100B tokens). Its effectiveness depends on how well the base model is calibrated, and it ignores document structure and topic diversity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🏢Ant GroupHangzhou · China42 works · 1,131 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Lei Liu, Hao Zhu, Xiaoyan Yang, Yue Shen, Zhixuan Chu, Jian Wang, Jinjie Gu and Kui Ren. 2026. Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.999

@inproceedings{liu2026perplexity,
  title = {{Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training}},
  author = {Lei Liu, Hao Zhu, Xiaoyan Yang, Yue Shen, Zhixuan Chu, Jian Wang, Jinjie Gu and Kui Ren},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.999},
  url = {https://doi.org/10.18653/v1/2026.acl-long.999}
}

References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. 2024. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541. — [arXiv:2405.20541](https://arxiv.org/abs/2405.20541) find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Francesco Cagnetta, Hyunmo Kang, and Matthieu Wyart. 2025. Learning curves theory for hierarchically compositional data with power-law distributed features. arXiv preprint arXiv:2505.07067. — [arXiv:2505.07067](https://arxiv.org/abs/2505.07067) find
  4. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2025a. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Languag…
  5. Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, and 1 others. 2025b. Towards effective and efficient continual pre-training of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Comp…
  6. Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, and 1 others. 2025. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739. find
  7. Logan Engstrom, Axel Feldmann, and Aleksander Madry. 2024. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926. find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  9. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and 1 others. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  10. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991–63010…
  11. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  12. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  13. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  14. Uriel Katz, Eran Cohen, Eliya Shachar, Jonathan Somer, Adam Fink, Eli Morse, Beki Shreiber, and Ido Wolf. 2024. Gpt versus resident physicians—a benchmark based on official board scores. Nejm Ai, 1(5):AIdbp2300192. find
  15. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.
  16. Lei Liu, Xiaoyan Yang, Junchi Lei, Yue Shen, Jian Wang, Peng Wei, Zhixuan Chu, Zhan Qin, and Kui Ren. 2024. A survey on medical large language models: Technology, application, trustworthiness, and future directions. arXiv preprint arXiv:2406.03712. — [arXiv:2406.03712](https://arxiv.org/abs/2406.03712) find
  17. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR.
  18. Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, and 1 others. 2024. D-cpt law: Domain-specific continual pre-training scaling law for large language models. Advances in Neural Information Processing Systems, 37:90318– 903… find
  19. Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. 2025. Diagnosisarena: Benchmarking diagnostic reasoning for large language models. arXiv preprint arXiv:2505.14107.
  20. George Kingsley Zipf. 2013. The psycho-biology of language: An introduction to dynamic philology. Routledge. find
  21. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  22. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  23. Jiachen T Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. 2024. Greats: Online selection of high-quality data for llm training in every iteration. Advances in Neural Information Processing Systems, 37:131197–131223. find
  24. Xingjin Wang, Howe Tissue, Lu Wang, Linjing Li, and Daniel Dajun Zeng. 2025. Learning dynamics in continual pre-training for large language models. arXiv preprint arXiv:2505.07796. — [arXiv:2505.07796](https://arxiv.org/abs/2505.07796) find
  25. Sondre Wold, Petter Mæhlum, and Oddbjørn Hove. 2024. Estimating lexical complexity from document-level distributions. arXiv preprint arXiv:2404.01196. find
  26. Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, and 1 others. 2025. From selection to generation: A survey of llm-based active learning. In Proceedings of the 63rd Annual Meeting of the Association for Computati…
  27. Zichun Yu, Spandan Das, and Chenyan Xiong. 2024. Mates: Model-aware data selection for efficient pre-training with data influence models. Advances in Neural Information Processing Systems, 37:108735– 108759. find
  28. Xuemiao Zhang, Feiyu Duan, Liangyu Xu, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. 2025. Frame: Boosting llms find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-perplexity-aware-data-scaling-law/