Elamur Research⌕
☰
home / papers / 2024 / DiscoveryBench: Towards Data-Driven Discovery with Large Language Models

DiscoveryBench: Towards Data-Driven Discovery with Large Language Models

Paper · 2024 · Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal et al.

Summary

DiscoveryBench tests whether LLM-based agents can independently form and check hypotheses using only supplied datasets, a setting the authors call data-driven discovery. They model a hypothesis as a combination of a context, a set of variables and a relationship. A task hides one of these and gives a natural-language goal plus dataset(s) with metadata. The real portion, DB-REAL, has 264 tasks built by hand from discovery workflows in more than 20 published studies across six fields, including sociology and engineering. DB-SYNTH adds 903 LLM-generated tasks spanning 48 domains, with controllable difficulty.

Because answers are open-ended, scoring compares predicted and gold sub-hypotheses facet by facet. Several agent frameworks were run with both open and closed LLMs, and the strongest system reaches only about 25%. Errors grow with workflow length and with the need for specialized statistics or domain knowledge. The benchmark exposes how far autonomous scientific analysis still has to go and offers a reproducible way to track progress.

Key points

  • Resource and evaluation: DB-REAL (264 tasks) and DB-SYNTH (903 tasks) are released on GitHub and Hugging Face under allenai/discoverybench. The evaluation metric is the Hypothesis Matching Score (HMS), which is built from context F1, variable F1 and relationship accuracy.
  • Baselines: The tested agents were CodeGen, React, DataVoyager, Reflexion (Oracle) and a NoDataGuess memorization check. They were paired with GPT-4o, GPT-4p and Llama-3. React prompting and DataVoyager's planner-critic design did not beat plain CodeGen. Only Reflexion, which receives oracle feedback, improved substantially.
  • Task types: Correlation analysis was the best-handled workflow category for the top non-oracle agent, at about 55%. Spatial analysis, pollen dating and ecological modeling all scored 0%.
  • Domains: Economics (25%) and sociology (23%) did best, while biology (0%) and engineering (7%) did worst. Recovering a relationship when context and variables are given is easier than finding the context or the variables.
  • Domain hints: Adding domain-knowledge hints to archaeology tasks raised DataVoyager (GPT-4p) from 9.9% to 17.5%. Scores also drop steeply as gold workflow length or semantic-tree height increases. Results on synthetic and real tasks were similar.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksDecision Sciences · 249 worksGeneral Social Sciences · 254 worksInformation Systems and Management · 52 works

Cite

Cite as Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal and Peter Clark. 2024. DiscoveryBench: Towards Data-Driven Discovery with Large Language Models. arXiv:2407.01725.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.01725 · arXiv 2407.01725

@misc{majumder2024discoverybench,
  title = {{DiscoveryBench: Towards Data-Driven Discovery with Large Language Models}},
  author = {Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal and Peter Clark},
  year = {2024},
  doi = {10.48550/arxiv.2407.01725},
  eprint = {2407.01725},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.01725}
}

References

55 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. L. Alexander, C. Riordan, J. Fennessey, and A. M. Pallas. Social background, academic resources, and college graduation: Recent evidence from the national longitudinal survey. American Journal of Education, 90(4):315–333, 1982. find
  2. A. P. S. Alves, M. Kalinowski, G. Giray, D. Mendez, N. Lavesson, K. Azevedo, H. Villamizar, T. Escovedo, H. Lopes, S. Biffl, et al. Status quo and problems of requirements engineering for machine learning: Results from an international survey. In International Conference on Product-Focused Software… find
  3. R. Apel and G. Sweeten. The impact of incarceration on employment during the transition to adulthood. Social problems, 57(3):448–479, 2010. find
  4. E. N. Appiah. The effect of education expenditure on per capita gdp in developing countries. International Journal of Economics and Finance, 9(10):136–144, 2017. find
  5. J. Brinkmann. Copper output, demand for wood and energy expenditure–evaluating economic aspects of bronze age metallurgy. How’s life, pages 11–34, 2019. find
  6. J. P. Brozio, J. Müller, M. Furholt, W. Kirleis, S. Dreibrodt, I. Feeser, W. Dörfler, M. Weinelt, H. Raese, and A. Bock. Monuments and economies: What drove their variability in the middle-holocene neolithic? The Holocene, 29(10):1558–1571, 2019. find
  7. J. P. Brozio, J. Kneisel, S. Schaefer-Di Maida, J. Laabs, I. Feeser, A. Ribeiro, and S. Schultrich. Patterns of socio-economic cultural transformations in neolithic and bronze age societies in the central northern european plain: Human-environmental interaction concerning bourdieu’s forms of capital… find
  8. F. O. Cerezer, C. S. Dambros, M. T. Coelho, F. A. Cassemiro, E. Barreto, J. S. Albert, R. O. Wüest, and C. H. Graham. Accelerated body size evolution in upland environments is correlated with recent speciation in south american freshwater fishes. Nature Communications, 14(1):6070, 2023. find
  9. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  10. M. Dal Corso, W. Kirleis, J. Kneisel, N. Taylor, M. Wieckowska-Lüth, and M. Zanon. How’s Life? Living Conditions in the 2nd and 1st Millennia BCE. Sidestone Press, 2019. find
  11. C. Dougherty. Numeracy, literacy and earnings: Evidence from the national longitudinal survey of youth. Economics of education review, 22(5):511–521, 2003. find
  12. I. Feeser, W. Dörfler, J. Kneisel, M. Hinz, and S. Dreibrodt. Human impact and population dynamics in the neolithic and bronze age: Multi-proxy evidence from north-western central europe. The Holocene, 29(10):1596–1606, 2019. find
  13. M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In NeurIPS, 2015. find
  14. J. Fu, S.-K. Ng, Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire. ArXiv, abs/2302.04166, 2023. URL https://api.semanticscholar.org/CorpusID:256662188. ↗
  15. P. Gijsbers, M. L. P. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. Amlb: an automl benchmark. ArXiv, abs/2207.12560, 2022. URL https://api.semanticscholar.org/CorpusID:251066648. — [arXiv:2207.12560](https://arxiv.org/abs/2207.12560) ↗
  16. D. J. Glass and N. Hall. A brief history of the hypothesis. Cell, 134(3):378–381, 2008. find
  17. R. Heyard and L. Held. Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. Technical report, Center for Open Science, 2024. find
  18. H. Jin, F. Chollet, Q. Song, and X. Hu. Autokeras: An automl library for deep learning. J. Mach. Learn. Res., 24:6:1–6:6, 2023. find
  19. J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  20. E. Kaiser and W. Schier. Time and Materiality: Periodization and Regional Chronologies at the Transition from Bronze to Iron Age in Eurasia (1200-600 BCE). Edited by Elke Kaiser and Wolfram Schier. Verlag Marie Leidorf GmbH, 2021. find
  21. H. Kitano. Artificial intelligence to win the nobel prize and beyond: Creating the engine for scientific discovery. AI magazine, 37(1):39–49, 2016. find
  22. J. Kneisel. Chronology and transformation. the transition from bronze to iron age in northern europe. Time and materiality: Periodization and regional chronologies at the transition from Bronze to Iron Age in Eurasia (1200–600 BCE), pages 237–263, 2021. find
  23. P. Langley. Data-driven discovery of physical laws. Cogn. Sci., 5:31–54, 1981. URL https://api.semanticscholar.org/CorpusID:39694251. ↗
  24. E. LeDell and S. Poirier. H2O AutoML: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, 2020. find
  25. M. Y. Li, E. B. Fox, and N. D. Goodman. Automated statistical model discovery with language models. ArXiv, abs/2402.17879, 2024. URL https://api.semanticscholar.org/ CorpusID:268041863. ↗
  26. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/ tatsu-lab/alpaca_eval, 2023. ↗
  27. B. Y. Lin, K. Chandu, F. Brahman, Y. Deng, A. Ravichander, V. Pyatkin, R. L. Bras, and Y. Choi. Wildbench: Benchmarking language models with challenging tasks from real users in the wild, 2024. URL https://huggingface.co/spaces/allenai/WildBench. ↗
  28. X. Liu, Z. Wu, X. Wu, P. Lu, K.-W. Chang, and Y. Feng. Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv:2402.17644, 2024. find
  29. L. Lorenz. Kommunikationsstrukturen mittelneolithischer Gesellschaften im nordmitteleuropäis-chen Tiefland. Verlag Dr. Rudolf Habelt GmbH, in Kommission, 2018. find
  30. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Proc… find
  31. S.-D. Maida et al. Unter hügeln: Bronzezeitliche transformationsprozesse in schleswig-holstein am beispiel des fundplatzes von mang de bargen (bornhöved, kr. segeberg) band 1, 2023. find
  32. B. P. Majumder, B. D. Mishra, P. Jansen, O. Tafjord, N. Tandon, L. Zhang, C. Callison-Burch, and P. Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. arXiv preprint arXiv:2310.10134, 2023.
  33. B. P. Majumder, H. Surana, D. Agarwal, S. Hazra, A. Sabharwal, and P. Clark. Data-driven discovery with large generative models. ICML, 2024.
  34. G. I. P. Ottaviano, G. Peri, and G. C. Wright. Immigration, offshoring, and american jobs. American Economic Review, 103(5):1925–1959, 2013. find
  35. L. C. Pal. Impact of education on economic development. Khazanah Pendidikan Islam, 5(1): 10–19, 2023. find
  36. A. Palmisano, A. Bevan, A. Kabelindde, N. Roberts, and S. Shennan. Long-term demographic trends in prehistoric italy: Climate impacts and regionalised socio-ecological trajectories. Journal of world prehistory, 34(3):381–432, 2021. find
  37. E. W. Parkinson, T. R. McLaughlin, C. Esposito, S. Stoddart, and C. Malone. Radiocarbon dated trends and central mediterranean prehistory. Journal of world prehistory, 34:317–379, 2021. find
  38. N. Rambeli, D. A. A. Marikan, J. M. Podivinsky, R. Amiruddin, and I. Ismail. The dynamic impact of government expenditure in education on economic growth. International Journal of Business and Society, 22(3):1487–1507, 2021. find
  39. M. Riera, J. Pino, L. Sáez, P. Aymerich, and Y. Melero. Effect of introduction pathways on the invasion success of non-native plants along environmental gradients. Biological Invasions, pages 1–20, 2024. find
  40. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  41. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
  42. Z. Shao, F. Wang, Y. Xu, W. Wei, C. Yu, Z. Zhang, D. Yao, G. Jin, X. Cao, G. Cong, C. S. Jensen, and X. Cheng. Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis. ArXiv, abs/2310.06119, 2023. — [arXiv:2310.06119](https://arxiv.org/abs/2310.06119) find
  43. S. Shashidhar, A. Chinta, V. Sahai, Z. Wang, and H. Ji. Democratizing llms: An exploration of cost-performance trade-offs in self-refined open-source models. ArXiv, abs/2310.07611, 2023. URL https://api.semanticscholar.org/CorpusID:263834891. ↗
  44. N. Shinn, F. Cassano, B. Labash, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023. URL https://api.semanticscholar.org/CorpusID:258833055. ↗
  45. P. K. Smith, B. Bogin, and D. Bishai. Are time preference and body mass index associated?: Evidence from the national longitudinal survey of youth. Economics & Human Biology, 3(2): 259–270, 2005. find
  46. C. Sommerfeld. Gerätegeld Sichel: studien zur monetären Struktur bronzezeitlicher Horte im nördlichen Mitteleuropa, volume 19. Walter de Gruyter, 2013. find
  47. W. H. Thompson and S. Skau. On the scope of scientific hypotheses. Royal Society Open Science, 10(8):230607, 2023. find
  48. H. Weatherly, K. Lopez, and C. Tierra. The impact of education on gdp per capita. 2022. find
  49. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 – 61, 2010. doi: 10.25080/Majora-92bf1922-00a.
  50. Z. Yang, X. Liu, T. Li, D. Wu, J. Wang, Y. Zhao, and H. Han. A systematic literature review of methods and datasets for anomaly-based network intrusion detection. Comput. Secur., 116: 102675, 2022. find
  51. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023. URL https://openreview.net/ forum?id=WE_vluYUL-X. ↗
  52. Z. Yuan, J. Liu, Q. Zi, M. Liu, X. Peng, and Y. Lou. Evaluating instruction-tuned large language models on code comprehension and generation. ArXiv, abs/2308.01240, 2023. URL https://api.semanticscholar.org/CorpusID:260379087. — [arXiv:2308.01240](https://arxiv.org/abs/2308.01240) ↗
  53. K. Zaw, D. Hamilton, and W. Darity. Race, wealth and incarceration: Results from the national longitudinal survey of youth. Race and Social Problems, 8:103–115, 2016. find
  54. Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. ArXiv, abs/2310.07641, 2023. URL https://api.semanticscholar.org/CorpusID:263834884. ↗
  55. S. Zhang, C. Gong, L. Wu, X. Liu, and M. Zhou. AutoML-GPT: Automatic machine learning with GPT. ArXiv, abs/2305.02499, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/majumder-2024-discoverybench-data-driven-discovery/