Elamur Research⌕
☰
home / papers / 2025 / SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation

SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation

Paper · 2025 · Gayathri Saranathan, Cong Xu, Mahammad Parwez Alam et al.

Summary

Running every new LLM on every benchmark has become very expensive. Existing data-efficient evaluation methods usually rely on many prior model results, which a newly added benchmark does not have. SubLIME, from Hewlett Packard Labs, addresses this cold-start case. It picks a small subset, 1–20% of a benchmark's items, whose model rankings closely match those from the full benchmark.

Several sampling strategies (random, clustering, quality-, difficulty- and readability-based) produce candidate subsets. A Rank Correlation Prediction (RCP) model then estimates how well each candidate will preserve full-set rankings. It uses intrinsic dataset features (Difficulty, Quality, Distributional Dispersion) together with full results from only 5–20 anchor LLMs. The top-scoring "winner" subset is then used to evaluate unseen models.

Across ten Open LLM Leaderboard benchmarks and 100 held-out LLMs, SubLIME generally beats TinyBench and random sampling on rank and score correlation, while cutting evaluation cost by 80–99%. The approach is useful for leaderboard curators and for teams benchmarking models on limited compute, including on newly built benchmarks.

Key points

  • Benchmarks: The ten benchmarks come from Hugging Face Open LLM Leaderboard v1/v2: TruthfulQA, ARC, Winogrande, GSM8k, Hellaswag, GPQA, MUSR, BBH, MATH and MMLU-Pro.
  • Models: Sample-level results were gathered for 313 models. Of these, 213 trained the RCP model and 100 served as a held-out test set.
  • Quality metrics: Subset quality is measured by Spearman ρ (rank preservation) and Pearson r (score-distribution preservation). During subset search, the target is both values ≥ 0.9.
  • Results with 20 anchors and 20% subsets:
    • Rank correlation is about 96–99.5% on most benchmarks.
    • It is much lower on GPQA (74.35% vs. 64.33% for TinyBench) and MUSR (77.89% vs. 69.69%). The authors attribute such drops to datasets with extreme distributional dispersion.
  • Stability and limitations:
    • Hard benchmarks such as GPQA need about 20 anchor models for stable predictions, while GSM8K works with 5.
    • The anchor models still require full evaluation.
    • The method assumes the anchors are representative of the wider model population.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Hewlett-PackardPalo Alto · United States3 works · 691 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Gayathri Saranathan, Cong Xu, Mahammad Parwez Alam, Tarun Kumar, Martin Foltin, Soon Yee Wong and Suparna Bhattacharya. 2025. SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1477

@inproceedings{saranathan2025sublime,
  title = {{SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation}},
  author = {Gayathri Saranathan, Cong Xu, Mahammad Parwez Alam, Tarun Kumar, Martin Foltin, Soon Yee Wong and Suparna Bhattacharya},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1477},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1477}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vladimir Bochkarev, Anna Shevlyakova, and Valery Solovyev. 2012. Average word length dynamics as indicator of cultural changes in society. Social Evolution and History, 14:153–175. find
  2. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, and Luming Liang. 2023. The efficiency spectrum of large language models: An algorithmic survey. Preprint, arXiv:2312.00678. — [arXiv:2312.00678](https://arxiv.org/abs/2312.00678) find
  5. Chengcheng Guo, Bo Zhao, and Yanbing Bai. 2022. Deepcore: A comprehensive library for coreset selection in deep learning. arXiv preprint arXiv:2204.08499. — [arXiv:2204.08499](https://arxiv.org/abs/2204.08499) find
  6. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
  7. Wenhao Hu, Dong Xu, and Zhihua Niu. 2021. Improved k-means text clustering algorithm based on bert and density peak. In 2021 2nd Information Communication Technologies Conference (ICTC), pages 260–264. find
  8. Yifei Hu, Xiaonan Jing, Youlim Ko, and Julia Taylor Rayz. 2024. Misspelling correction with pre-trained contextual language model. find
  9. Hugging Face. 2022. Open llm leaderboard. https://huggingface.co/open-llm-leaderboard. Retrieved February 3, 2022. ↗
  10. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  11. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. Preprint, arXiv:2109.07958.
  12. H. P. Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159–165. find
  13. Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. When less is more: Investigating data pruning for pre-training llms at scale. Preprint, arXiv:2309.04564. find
  14. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. Version 3.
  15. Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. 2024. Efficient benchmarking of language models. Preprint, arXiv:2308.11696. find
  16. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. Preprint, arXiv:2402.14992. find
  17. Ameya Prabhu, Vishaal Udandarao, Philip Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. 2024. Lifelong benchmarks: Efficient model evaluation in an era of rapid progress. Preprint, arXiv:2402.19472. 18th Conference of the European Chapter of the Association for Computational Linguistics (Volum… find
  18. Tomasz Walkowiak and Mateusz Gniewkowski. 2019. Evaluation of vector embedding models in clustering of text documents. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1304– 1311, Varna, Bulgaria. INCOMA Ltd. find
  19. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark…
  20. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023. Data selection for language models via importance resampling. In Thirty-seventh Conference on Neural Information Processing Systems. find
  21. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  22. Abdelrahman Zayed, Prasanna Parthasarathi, Gonçalo Mordido, Hamid Palangi, Samira Shabanian, and Sarath Chandar. 2023. Deep learning on a healthy data diet: finding important examples for fairness. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Confe… find
  23. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. find
  24. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830. find
  25. Philip Sedgwick. 2014. Spearman’s rank correlation coefficient. Bmj, 349. find
  26. Fred Zenker and Kristopher Kyle. 2021. Investigating minimum text lengths for lexical diversity indices. Assessing Writing, 47:100505. find
  27. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2023. Beyond neural scaling laws: beating power law scaling via data pruning. Preprint, arXiv:2206.14486.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saranathan-2025-sublime-subset-selection-rank-correlation/