Elamur Research⌕
☰
home / papers / 2026 / Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists

Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists

Paper · 2026 · Michał Pietruszka, Łukasz Borchmann, Aleksander Jędrosz et al.

Summary

Most LLM benchmarks test knowledge recall or ask a model to turn a clear specification into code. Few check whether a model can decide for itself which data transformations would make a predictive model stronger. The authors introduce FeatEng, a benchmark that treats feature engineering on tabular data as a code-generation problem.

For each task, the LLM receives:

  • a general instruction and a worked example,
  • column descriptions with their most frequent values,
  • a short problem statement.

It must then write a Python transform function that operates on Pandas train and test DataFrames. The output is scored by how much it improves an XGBoost model compared with training on the original data. An automated feature engineering baseline from EvalML serves as a non-LLM reference. Tasks come from 103 manually curated Kaggle datasets spanning domains such as healthcare, finance and entertainment.

Features grounded in domain knowledge noticeably raise scores, and model rankings on FeatEng closely follow Chatbot Arena standings. FeatEng therefore offers a cheap, automated way to measure a practically important data science skill.

Key points

  • FeatEng contains 103 Kaggle datasets covering classification and regression tasks with permissive licenses. Their selection and preparation (target choice, leakage removal) took over 100 hours of manual work.
  • The evaluation metric is the error reduction of a fixed XGBoost model trained on LLM-transformed features versus the untransformed baseline. EvalML AutoFE serves as a comparison point.
  • Generating "strong domain knowledge-based" features has a statistically significant positive effect on scores (elasticity 0.1717, p < 0.001). One example is combining diabetes, hypertension and coronary artery disease flags into a single risk indicator.
  • FeatEng results correlate strongly with Chatbot Arena rankings, suggesting the benchmark is a proxy for general LLM capability.
  • The authors acknowledge several limitations:
    • The benchmark covers only single-pass, non-iterative generation, though separate train/test splits allow future agentic setups.
    • It is restricted to tabular data and Pandas, with XGBoost as the only evaluator.
    • Temperature sampling causes run-to-run variance.
    • Results may partly reflect model familiarity with Kaggle-style data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Polish Academy of SciencesWarsaw · Poland11 works · 520 citations🏢Snowflake Inc.Bozeman · United States3 works · 17 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Michał Pietruszka, Łukasz Borchmann, Aleksander Jędrosz and Paweł Morawiecki. 2026. Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.308

@inproceedings{pietruszka2026can,
  title = {{Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists}},
  author = {Michał Pietruszka, Łukasz Borchmann, Aleksander Jędrosz and Paweł Morawiecki},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.308},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.308}
}

References

31 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alteryx. 2023. EvalML. https://evalml.alteryx.com/en/stable/. ↗
  2. Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. Preprint, arXiv:2108.07732.
  4. Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. 2013. API design for machine learning software: experie… find
  5. Mark Chen and 1 others. 2021. Evaluating Large Language Models Trained on Code. ArXiv, abs/2107.03374.
  6. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA. Association for Computing Machinery. find
  7. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024a. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Preprint, arXiv:2403.04132.
  8. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024b. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ArXiv, abs/2403.04132.
  9. DeepSeek-AI and 1 others. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. Preprint, arXiv:2405.04434. find
  10. Abhimanyu Dubey and 1 others. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  12. Noah Hollmann, Samuel Müller, and Frank Hutter. 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. In Advances in Neural Information Processing Systems, volume 36, pages 44753– 44775. Curran Associates, Inc. find
  13. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
  14. Kaggle. 2024. Kaggle. https://www.kaggle.com/. ↗
  15. James Max Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE International Conference on Data Science and Advanced Analytics, DSAA 2015, Paris, France, October 19-21, 2015, pages 1–10. IEEE. find
  16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  17. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. Preprint, arXiv:2211.11501. find
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  19. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, and 3 others. 2024. Jamba:…
  20. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. Preprint, arXiv:2209.14610.
  21. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, and 3 others. 2021. Codexglue: A machine learning bench… find
  22. Wes McKinney and 1 others. 2010. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, volume 445, pages 51–56. Austin, TX. find
  23. Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias Boutros Khalil, and Deepak S. Turaga. 2017. Learning feature engineering for classification. In International Joint Conference on Artificial Intelligence. find
  24. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures. Preprint, arXiv:2406.06565.
  25. OpenAI and 1 others. 2024. GPT-4 Technical Report. Preprint, arXiv:2303.08774.
  26. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. Preprint, arXiv:1508.00305.
  27. Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. 2024. Benchmarks as microscopes: A call for model metrology. Preprint, arXiv:2407.16711. — [arXiv:2407.16711](https://arxiv.org/abs/2407.16711) find
  28. Aarohi Srivastava and 1 others. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Preprint, arXiv:2206.04615.
  29. Gemini Team and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  30. Melvyn Weeks. 2002. Introductory Econometrics: A Modern Approach. find
  31. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. Preprint, arXiv:1809.08887.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pietruszka-2026-can-models-help-us-create/