Elamur Research⌕
☰
home / papers / 2026 / Generalization or Memorization? Multi-Agent vs. Baseline LLMs and AutoML Models for Tabular Classification

Generalization or Memorization? Multi-Agent vs. Baseline LLMs and AutoML Models for Tabular Classification

Paper · 2026 · Aida Sanatizadeh, Sorouralsadat Fatemi, Reza Mousavi et al.

Summary

The paper asks whether LLMs that perform well on tabular classification are actually reasoning or simply recalling benchmarks seen during pre-training. The authors built a modular multi-agent system (MALLM) on GPT-4o and LangGraph. Separate agents handle feature analysis, feature engineering, retrieval, and prediction, and the system runs in zero-shot, few-shot, and RAG modes. They compared it with the tabular LLM baselines TABLET and TABLLM and with the AutoML systems AutoGluon, MLJAR, and H2O.

The test set has eleven binary tasks. Five are well-known benchmarks that predate the model's knowledge cutoff. Six are new datasets released afterwards, covering healthcare, education, housing, and workplace settings. Evaluation goes beyond accuracy to include calibration, class imbalance, robustness to renamed features, and instance hardness and typicality.

MALLM matches or beats AutoML on the older benchmarks but falls clearly behind on the unseen data. There it favours the majority class, is poorly calibrated, and varies a lot between runs, especially on difficult cases. AutoML stays stable across both groups. The authors conclude that LLM agents are not yet a substitute for discriminative tabular models in high-stakes use, and that contamination-free benchmarks are needed.

Key points

  • MALLM is built on GPT-4o-2024-08-06. The post-cutoff datasets were all released after September 2024, which reduces, though cannot fully rule out, training-data contamination.
  • The six post-cutoff datasets are Student Depression (1000 rows), Postpartum Depression (1491), Employee Satisfaction (1000 rows, 64 features), Students AI Tools (2000), Rental Properties (482), and ICU Admission (254). They come from Kaggle or private sources, and positive-class shares range from 48% to 65%.
  • AutoGluon was run with the best-quality preset and a 1-hour limit. H2O used a stacked ensemble optimized for AUC.
  • On novel data, AutoML confidence tracks instance hardness more closely, and its results vary less across runs than MALLM's.
  • Stated limitations include:
    • small datasets (N < 2000);
    • binary classification only;
    • no subgroup fairness analysis;
    • higher latency and API cost for MALLM;
    • privacy concerns from sending records to external APIs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of VirginiaCharlottesville · United States36 works · 9,730 citations🎓University of Notre DameNotre Dame · United States51 works · 36,717 citations🎓Northern Illinois UniversityDeKalb · United States2 works · 256 citations

Also on the byline: California State University, Monterey Bay.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Aida Sanatizadeh, Sorouralsadat Fatemi, Reza Mousavi and Ahmed Abbasi. 2026. Generalization or Memorization? Multi-Agent vs. Baseline LLMs and AutoML Models for Tabular Classification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1994

@inproceedings{sanatizadeh2026generalization,
  title = {{Generalization or Memorization? Multi-Agent vs. Baseline LLMs and AutoML Models for Tabular Classification}},
  author = {Aida Sanatizadeh, Sorouralsadat Fatemi, Reza Mousavi and Ahmed Abbasi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1994},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1994}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sebastian Bordt, Harsha Nori, Vasco Rodrigues, Besmira Nushi, and Rich Caruana. 2024. Elephants never forget: Memorization and learning of tabular data in large language models. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS). find
  2. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szuto, Nazanin Mohajer, Vahid Nouriborji, Eugene Cheplygina, Veronika anderos, and 1 others. 2021. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, vol… find
  3. Balderton Capital. 2025. Prior labs raises C9m to revolutionise how businesses interact with their tabular data. https://www.balderton.com/news/prior-labs-r aises-e9m-to-revolutionise-how-businesses-interac t-with-their-tabular-data/. Accessed: 24.09.2025. ↗
  4. Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 1… find
  5. Hyeong Kyu Choi and Yixuan Li. 2024. Picle: Eliciting diverse behaviors from large language models with persona in-context learning. In International Conference on Machine Learning (ICML). find
  6. Ryan A Cook, John P Lalor, and Ahmed Abbasi. 2025. No simple answer to data complexity: An examination of instance-level complexity metrics for classification tasks. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human L… find
  7. David Cyrus, Dany Varghese, and Alireza Tamaddoni-Nezhad. 2024. An inductive logic programming approach for feature-range discovery. In International Joint Conference on Learning and Reasoning (IJ-CLR), pages 203–217. Springer. find
  8. Romain Dillet. 2025. Neuralk-ai is developing ai models specifically designed for structured data. https://techcrunch.com/2025/02/03/neuralk-ai-is-develop ing-ai-models-specifically-designed-for-structure d-data/. Accessed: 24.09.2025. ↗
  9. Tuan Dinh, Yuchen Zeng, Ruiqi Zhang, Zihao Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:11763–11784. find
  10. Nicholas Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. 2020. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505. — [arXiv:2003.06505](https://arxiv.org/abs/2003.06505) find
  11. Xi Fang, Weijie Xu, Fiona Anting Tan, Jianfeng Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socol-insky, Srinivasan Sengamedu, and Christos Faloutsos. 2024. Large language models(llms) on tabular data: Prediction, generation, and understanding – a survey. arXiv preprint arXiv:2402.17944. — [arXiv:2402.17944](https://arxiv.org/abs/2402.17944) find
  12. Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023. Tabllm: Few-shot classification of tabular data with large language models. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR. find
  13. HopesB. 2023. Student depression dataset. https://ww w.kaggle.com/datasets/hopesb/student-depressio n-dataset. Kaggle dataset. Licensed under Apache License 2.0. find
  14. Shubham Jaitly, Tanay Shah, Anshul Shugani, and Rajat Singh Grewal. 2023. Towards better serialization of tabular data for few-shot classification with large language models. arXiv preprint arXiv:2312.12464. — [arXiv:2312.12464](https://arxiv.org/abs/2312.12464) find
  15. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xu Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237–9251, Singapore. Association fo… find
  16. James Max Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 1–10. IEEE. find
  17. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines.
  18. Hyunjin Kwon, Matthew Greenberg, Colin Bruce Josephson, Joon Lee, and 1 others. 2024. Measuring the prediction difficulty of individual cases in a dataset using machine learning. Scientific Reports, 14(1):10474. find
  19. Erin LeDell and Sebastien Poirier. 2020. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML. find
  20. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K"uttler, Mike Lewis, Wen-tau Yih, Tim Rockt"aschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, vol…
  21. Shang-Ching Liu, ShengKun Wang, Tsungyao Chang, Wenqi Lin, Chung-Wei Hsiung, Yi-Chen Hsieh, Yu-Ping Cheng, Sian-Hong Luo, and Jianwei Zhang. 2023. JarviX: A LLM no code platform for tabular data analysis and optimization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language… find
  22. Tianyang Liu, Fei Wang, and Muhao Chen. 2024. Rethinking tabular data understanding with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 450–482, Me… find
  23. Hariharan Manikandan, Yiding Jiang, and J. Zico Kolter. 2023. Language models are weak learners. arXiv preprint arXiv:2306.14101. — [arXiv:2306.14101](https://arxiv.org/abs/2306.14101) find
  24. Md Mahadi Hasan Nahid and Davood Rafiei. 2024. NormTab: Improving symbolic reasoning in LLMs through tabular data normalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3569–3585, Miami, Florida, USA. Association for Computational Linguistics. find
  25. Parvez Al Muqtadir. 2023. Postpartum depression dataset. https://www.kaggle.com/datasets/parv ezalmuqtadir2348/postpartum-depression. Kaggle dataset. Licensed under the MIT License. ↗
  26. Aleksandra Płońska and Piotr Płoński. 2021. Mljar: State-of-the-art automated machine learning framework for tabular data (version 0.10.3). https://github.com/mljar/mljar-supervised. ↗
  27. Rakesh Kapilavai. 2025. Ai tool usage by indian college students 2025. https://www.kaggle.com/datasets/ rakeshkapilavai/ai-tool-usage-by-indian-college-s tudents-2025. Kaggle dataset. Licensed under CC BY-NC-SA 4.0. ↗
  28. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215.
  29. Ravid Shwartz-Ziv and Amitai Armon. 2021. Tabular data: Deep learning is not all you need. arXiv preprint arXiv:2106.03253. — [arXiv:2106.03253](https://arxiv.org/abs/2106.03253) find
  30. Dylan Slack and Sameer Singh. 2023. Tablet: Learning from instructions for tabular data. arXiv preprint arXiv:2304.13188. — [arXiv:2304.13188](https://arxiv.org/abs/2304.13188) find
  31. Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. 2024. TAP4LLM: Table provider on sampling, augmenting, and packing semi-structured data for LLM reasoning. In The Twelfth International Conference on Learning Representations (ICLR). find
  32. Gustavo P. Torquette, Victor S. Nunes, Pedro Y. A. Paiva, Lourenço B. C. Neto, and Ana C. Lorena. 2022. Characterizing instance hardness in classification and regression problems. arXiv preprint arXiv:2212.01897. — [arXiv:2212.01897](https://arxiv.org/abs/2212.01897) find
  33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837.
  34. Xiaokang Zhang, Sijia Luo, Bohan Zhang, Zeyao Ma, Jing Zhang, Yang Li, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, and Jie Tang. 2025. Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios. Preprint, arXiv:2403.19318. find
  35. Pedro Yuri Arbs Paiva, Camila Castro Moreno, Kate Smith-Miles, Maria Gabriela Valeriano, and Ana Carolina Lorena. 2022. Relating instance hardness to classification performance in a dataset: a visual approach. Machine Learning, 111(7):3085–3123. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sanatizadeh-2026-generalization-or-memorization-multi-agent/