Elamur Research⌕
☰
home / papers / 2026 / From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation

From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation

Paper · 2026 · Aviya Maimon, Amir David Nisan Cohen, Gal Vishne et al.

Summary

LLMs are usually compared through long lists of benchmark scores or a single averaged number. It is unclear what these numbers measure and how much benchmarks overlap. The authors borrow tools from psychometrics. They treat models as respondents and benchmarks as test items, then build a leaderboard of 60 LLMs scored on 44 public tasks, with all metrics rescaled to a common 0–10 range. They apply exploratory factor analysis to this matrix using Principal Axis Factoring with Varimax rotation, which keeps only variance shared across tasks.

The matrix turns out to be strongly low-rank. Eight latent factors account for most of its structure, and an LLM-assisted procedure assigns each factor a skill-like name. These skills stay stable when tasks or models are removed. They also expose differences between models that look nearly identical on Chatbot Arena. Built on this skill space, the authors offer three tools:

  • detecting redundant benchmarks,
  • profiling a new model from a few tasks,
  • choosing models for unseen tasks.

The work matters because it offers an interpretable, cheaper alternative to single-score leaderboards.

Key points

  • The performance matrix covers 60 models and 44 benchmarks, 25 of them classification-based. It mixes short- and long-context tasks and both classification and generation formats.
  • The authors chose PAF over PCA because it separates shared from task-specific variance. In their comparison, it produced more stable and more interpretable factors.
  • The eight recovered skills range from general NLI to instruction-following and domain reasoning. They split the broad "understanding" and "reasoning" factors reported in earlier work (Burnell et al., 2023; Ilić & Gignac, 2024).
  • Two models rated 1322 and 1316 on Chatbot Arena have clearly different skill profiles.
  • In leave-one-task-out tests, scores on held-out tasks were predicted from a few pilot models. For well-covered tasks such as GLUE_RTE, BoolQ and CoQA, normalized MSE stayed below 0.20 and Pearson r exceeded 0.85. Errors were larger on underrepresented tasks like bAbI. Code, leaderboard and matrices are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Bar-Ilan UniversityRamat Gan · Israel141 works · 23,225 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksEfficient models & compression · 242 works

Cite

Cite as Aviya Maimon, Amir David Nisan Cohen, Gal Vishne, Shauli Ravfogel and Reut Tsarfaty. 2026. From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.745

@article{maimon2026benchmarks,
  title = {{From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation}},
  author = {Aviya Maimon, Amir David Nisan Cohen, Gal Vishne, Shauli Ravfogel and Reut Tsarfaty},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.745},
  url = {https://doi.org/10.1162/tacl.a.745}
}

References

69 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre-Antoine Absil, Alan Edelman, and Plamen Koev. 2006. On the largest principal angle between random subspaces. Linear Algebra and its Applications, 414(1):288–294. find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset.
  3. Tanya Y. Berger-Wolf and Jared Saia. 2007. Maxmin diversity. Computational Geometry, 36:67–85. find
  4. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer, New York. find
  5. Åke Björck and Gene H. Golub. 1973. Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27(123):579–594. find
  6. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. find
  7. Ryan Burnell, Han Hao, Andrew R. A. Conway, and Jose Hernandez Orallo. 2023. Revealing the structure of language model capabilities. find
  8. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac : Question answering in context.
  9. Paul T. Costa and Robert R. McCrae. 1992. Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO-FFI) Professional Manual. Psychological Assessment Resources, Odessa, FL. find
  10. Anna B. Costello and Jason W. Osborne. 2005. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(7):1–9. find
  11. Lee J. Cronbach. 1951. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297–334. find
  12. Lee J. Cronbach and Paul E. Meehl. 1955. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302. find
  13. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
  14. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. 2016. Evaluating prerequisite qualities for learning end-to-end dialog systems. find
  15. Arpad E. Elo. 1978. The Rating of Chessplayers, Past and Present. Arco Publishing, New York. find
  16. Leandre R Fabrigar, Duane T Wegener, Robert C MacCallum, and Erin J Strahan. 1999. Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3):272–299. find
  17. Dan Feldman. 2020. Introduction to core-sets: an updated survey. find
  18. Clémentine Fourrier, Nathan Habib, Hynek Ky-dlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation. find
  19. Gemini-Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
  20. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessic… — LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  21. Harry H. Harman. 1976. Modern factor analysis. University of Chicago Press. find
  22. D. M. Hawkins. 1980. Identification of outliers. Journal of the American Statistical Association, 75(372):432–439. find
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding.
  24. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset.
  25. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend.
  26. HuggingFace. 2024. Hugging face lmsys leaderboard. https://huggingface.co/spaces/ lmarena-ai/chatbot-arena-leaderboard. ↗
  27. David Ilić and Gilles E. Gignac. 2024. Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement? Intelligence, 106:101858. find
  28. Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning. Springer. find
  29. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.
  30. Ian T Jolliffe. 2002. Principal Component Analysis. Springer. find
  31. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
  32. Henry F. Kaiser. 1960. The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1):141– 151. find
  33. Yuta Koreeda and Christopher D. Manning. 2021. Contractnli: A dataset for document-level natural language inference for contracts.
  34. Michael H Kutner, Christopher J Nachtsheim, John Neter, and William Li. 2005. Applied Linear Statistical Models, 5th edition. McGraw-Hill Education. find
  35. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset.
  36. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods.
  37. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning.
  38. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon…
  39. Prasanta Chandra Mahalanobis. 1936. On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2(1):49– 55. find
  40. Roderick P. McDonald. 1999. Test Theory: A Unified Treatment. Lawrence Erlbaum, Mahwah, NJ. find
  41. G. Mclachlan. 1999. Mahalanobis distance. Resonance, 4:20–26. find
  42. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. find
  43. Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. 2020. Coresets for data-efficient training of machine learning models. find
  44. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
  45. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.
  46. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2022. Quality: Question answering with long input texts, yes!
  47. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. Bbq: A hand-built bias benchmark for question answering.
  48. Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, and Mikhail Yurochkin. 2025. Sloth: scaling laws for llm skills to predict multi-benchmark performance across families. find
  49. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text.
  50. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge.
  51. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark.
  52. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. Arb: Advanced reasoning benchmark for large language models. find
  53. Peter H. Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10. find
  54. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Comput… find
  55. Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  56. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  57. Louis L Thurstone. 1947. Multiple-Factor Analysis. University of Chicago Press. find
  58. Louis Leon Thurstone. 1931. Multiple factor analysis. Psychological review, 38(5):406. find
  59. Ledyard R. Tucker. 1951. A method for synthesis of factor analysis studies. Psychometrika, 16(2):169–192. find
  60. Ernest C. Tupes and Raymond E. Christal. 1961. Recurrent personality factors based on trait ratings. Technical Report USAF ASD Technical Report No. 61-97, U.S. Air Force, Lackland Air Force Base, TX. find
  61. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. find
  62. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding.
  63. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2023. Blimp: The benchmark of linguistic minimal pairs for english. find
  64. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
  65. Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. Docnli: A large-scale dataset for document-level natural language inference.
  66. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? find
  67. Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multi-level benchmark for examining large language models. find
  68. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
  69. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. Qmsum: A new benchmark for query-based multi-domain meeting summarization.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maimon-2026-benchmarks-skills-low-rank-factors/