Elamur Research⌕
☰
home / papers / 2024 / DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning

DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning

Paper · 2024 · Siyuan Guo, Cheng Deng, Ying Wen et al.

Summary

Agents built on large language models often fail at automated machine learning because they propose poorly grounded experiment plans and hallucinate, even with GPT-4. DS-Agent addresses this with case-based reasoning (CBR) rather than costly fine-tuning. The authors built a case bank from recent Kaggle competitions covering text, time-series and tabular data. It holds cleaned winning-team write-ups and top leaderboard code, which GPT-3.5 condenses into textual insights.

In a development stage, the agent:

  • pulls up relevant cases,
  • re-ranks them based on execution feedback,
  • drafts and runs a plan,
  • revises it over several iterations,
  • stores successful solutions.

A cheaper deployment stage skips the iteration loop. It adapts a stored solution from a similar task and generates code in a single pass, which places less demand on the underlying model.

Across 30 tasks, DS-Agent ranks best with both GPT-4 and GPT-3.5 and costs little per run. The work shows that reusing expert and past solutions can substitute for parameter updates when building practical data-science agents.

Key points

  • Benchmark: 30 tasks in total, 12 for development and 18 for deployment. Baselines include ResearchAgent, AutoGPT and LangChain.
  • Development stage: DS-Agent with GPT-4 completes all 12 tasks (100% success rate).
  • Deployment stage, one-pass rate:
    • GPT-3.5 reaches 85% and GPT-4 reaches 99%, against 56% and 60% for the strongest baseline.
    • Mixtral-8x7b-Instruct improves from 6% to 31%.
  • Cost per run:
    • Development: about $1.60 with GPT-4 and $0.06 with GPT-3.5.
    • Deployment: roughly $0.135 with GPT-4 and $0.0045 with GPT-3.5.
  • Retrieval and ablation findings:
    • Retrieval ranks cases by cosine similarity of embeddings.
    • Reusing past full solutions works better than reusing textual Kaggle insights.
    • Zero-shot generation performs worst, and adding more in-context cases lowers performance.
    • Code and data are released on GitHub (guosyjlu/DS-Agent).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China232 works · 20,305 citations🎓University College LondonLondon · United Kingdom119 works · 23,545 citations🎓Jilin UniversityChangchun · China31 works · 5,302 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang and Jun Wang. 2024. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. arXiv:2402.17453.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.17453 · arXiv 2402.17453

@misc{guo2024ds,
  title = {{DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning}},
  author = {Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang and Jun Wang},
  year = {2024},
  doi = {10.48550/arxiv.2402.17453},
  eprint = {2402.17453},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.17453}
}

References

53 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aamodt, A. and Plaza, E. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI communications, 7(1):39–59, 1994. find
  2. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.
  3. Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287– 318. PMLR, 2023.
  4. Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research… find
  5. Chase, H. LangChain, October 2022. URL https://github.com/langchain-ai/langchain. ↗
  6. Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023.
  7. Christianos, F., Papoudakis, G., Zimmer, M., Coste, T., Wu, Z., Chen, J., Khandelwal, K., Doran, J., Feng, X., Liu, J., et al. Pangu-agent: A fine-tunable generalist agent with structured reasoning. arXiv preprint arXiv:2312.14878, 2023. — [arXiv:2312.14878](https://arxiv.org/abs/2312.14878) find
  8. De Bie, T., De Raedt, L., Hernández-Orallo, J., Hoos, H. H., Smyth, P., and Williams, C. K. Automating data science. Communications of the ACM, 65(3):76–87, 2022. find
  9. Deng, C., Zhang, T., He, Z., Chen, Q., Shi, Y., Zhou, L., Fu, L., Zhang, W., Wang, X., Zhou, C., Lin, Z., and He, J. K2: A foundation language model for geoscience knowledge understanding and utilization. 2023. URL https://api.semanticscholar.org/CorpusID:259108887. ↗
  10. Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020. — [arXiv:2003.06505](https://arxiv.org/abs/2003.06505) find
  11. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.
  12. Guo, C., Tian, Z., Tang, J., Wang, P., Wen, Z., Yang, K., and Wang, T. A case-based reasoning framework for adaptive prompting in cross-domain text-to-sql. arXiv preprint arXiv:2304.13301, 2023. — [arXiv:2304.13301](https://arxiv.org/abs/2304.13301) find
  13. Hollmann, N., Müller, S., and Hutter, F. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. find
  14. Hong, S., Lin, Y., Liu, B., Wu, B., Li, D., Chen, J., Zhang, J., Wang, J., Zhang, L., Zhuge, M., et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024a.
  15. Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=VtmBAG… ↗
  16. Huang, Q., Vora, J., Liang, P., and Leskovec, J. Benchmarking large language models as ai research agents. arXiv preprint arXiv:2310.03302, 2023.
  17. Hutter, F., Kotthoff, L., and Vanschoren, J. Automated machine learning: methods, systems, challenges. Springer Nature, 2019. find
  18. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  19. Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., and Veeramachaneni, K. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1–36, 2021. find
  20. Kim, G., Baldi, P., and McAleer, S. M. Language models can solve computer tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  21. Kolodner, J. L. An introduction to case-based reasoning. Artificial intelligence review, 6(1):3–34, 1992. find
  22. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023.
  23. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319–18345. PMLR, 2023. find
  24. LeDell, E. and Poirier, S. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020. find
  25. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  26. Lin, Z., Deng, C., Zhou, L., Zhang, T., Xu, Y., Xu, Y., He, Z., Shi, Y., Dai, B., Song, Y., Zeng, B., Chen, Q., Shi, T., Huang, T., Xu, Y., Wang, S., Fu, L., Zhang, W., He, J., Ma, C., Zhu, Y., Wang, X., and Zhou, C. Geogalactica: A scientific large language model in geoscience. ArXiv, abs/2401.0043… — [arXiv:2401.00434](https://arxiv.org/abs/2401.00434) find
  27. Liu, Y., Tang, X., Cai, Z., Lu, J., Zhang, Y., Shao, Y., Deng, Z., Hu, H., Yang, Z., An, K., et al. Ml-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv preprint arXiv:2311.09835, 2023.
  28. Mahadi Hassan, M., Knipper, A., and Kanti Karmaker Santu, S. Chatgpt as your personal data scientist. arXiv e-prints, pp. arXiv–2305, 2023. find
  29. OpenAI. Introducing chatgpt. 2022. URL https://openai.com/blog/chatgpt. ↗
  30. OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023. arXiv preprint Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12…
  31. Pei, H., Yang, B., Liu, J., and Chang, K. C.-C. Active surveillance via group sparse bayesian learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1133–1148, 2020. find
  32. Pei, H., Chen, T., Chen, A., Deng, H., Tao, J., Wang, P., and Guan, X. Hago-net: Hierarchical geometric massage passing for molecular representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14572–14580, 2024a. find
  33. Pei, H., Xiong, Y., Wang, P., Tao, J., Liu, J., Deng, H., Ma, J., and Guan, X. Memory disagreement: A pseudo-labeling measure from training dynamics for semi-supervised graph learning. In Proceedings of the ACM on Web Conference 2024, pp. 434–445, 2024b. find
  34. Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, pp. 1–3, 2023.
  35. Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2671, 2022.
  36. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems, 2023. find
  37. Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
  38. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  39. Significant Gravitas. AutoGPT, 2023. URL https://github.com/Significant-Gravitas/ AutoGPT. ↗
  40. Sourati, Z., Ilievski, F., Sandlin, H.-Â., and Mermoud, A. Case-based reasoning with language models for classification of logical fallacies. arXiv preprint arXiv:2301.11879, 2023. — [arXiv:2301.11879](https://arxiv.org/abs/2301.11879) find
  41. Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14918–14937, 2023.
  42. Wang, L., Yang, N., and Wei, F. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164, 2023a.
  43. Wang, L., Yang, N., and Wei, F. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164, 2023b.
  44. Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c. URL https://openreview.net/forum?id=KtvPdGb31Z. ↗
  45. Watson, I. and Marir, F. Case-based reasoning: A review. The knowledge engineering review, 9(4):327–354, 1994. find
  46. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. 2023.
  47. Yang, Z., Du, X., Cambria, E., and Cardie, C. End-to-end case-based reasoning for commonsense knowledge base completion. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3491–3504, 2023. find
  48. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
  49. Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., and Tang, J. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023.
  50. Zhang, L., Zhang, Y., Ren, K., Li, D., and Yang, Y. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv preprint arXiv:2304.14979, 2023a.
  51. Zhang, P., Xiao, S., Liu, Z., Dou, Z., and Nie, J.-Y. Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554, 2023b. — [arXiv:2310.07554](https://arxiv.org/abs/2310.07554) find
  52. Zhang, S., Gong, C., Wu, L., Liu, X., and Zhou, M. Automl-gpt: Automatic machine learning with gpt. arXiv preprint arXiv:2305.02499, 2023c.
  53. Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., and Huang, G. Expel: Llm agents are experiential learners. Thirty-Eighth AAAI Conference on Artificial Intelligence, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2024-ds-agent-automated-data-science/