Elamur Research⌕
☰
home / papers / 2023 / Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

Paper · 2023 · Simran Arora, Brandon Yang, Sabri Eyuboglu et al.

Summary

Turning messy document collections such as web pages, PDFs and plain text into queryable tables is a long-standing data-management goal. Earlier systems usually depend on format- or domain-specific assumptions, hand-written rules, or annotated training data. The authors build Evaporate, a prototype that uses the in-context learning of LLMs to infer a schema and populate it without training or per-task customization. They compare two designs. In Evaporate-Direct, the model reads every document and returns values. In Evaporate-Code, the model writes extraction functions (e.g., Python) from a small sample, and those functions are then run over the whole collection. Direct reading is accurate, but its cost grows with corpus size. Generated code is cheap but clearly weaker. Evaporate-Code+ narrows this gap: it produces many candidate functions, discards those that conflict with LLM outputs, and merges the remainder with a new weak-supervision algorithm built for machine-written functions that never abstain. Across 16 real-world settings, Code+ outperforms prior state-of-the-art systems and direct extraction while the LLM processes about 110× fewer documents. This makes LLM-based structuring of data lakes affordable at scale.

Key points

  • The evaluation includes 16 settings drawn from 5 domains and covering 3 kinds of document format, including NBA player bios, movie and university websites, and FDA 510(k) medical-device reviews. Code is released at github.com/HazyResearch/evaporate.
  • Plain Evaporate-Code trails Evaporate-Direct by 13.8 F1 points on average (24.9% relative). About 40% of the generated functions score below 25% accuracy, which violates the standard assumptions of weak supervision.
  • The authors estimate that running OpenAI models directly over the 55M Wikipedia articles would cost over $110k with gpt-3.5 and about $1.1M with text-davinci-003 (March 2023 prices). This motivates a sublinear, code-based pass.
  • Code+ favours precision across model families. Its average precision is 70.9 (GPT-4), 67.6 (Claude-V1) and 50.9 (Jurassic Jumbo-2-Instruct), versus 61.9, 55.1 and 49.9 for Direct.
  • Schema identification is not the main bottleneck (average attribute F1@k of 71.9 with text-davinci-003). Most errors come from failing to synthesize a function for some attributes and from chatbot-style non-answers during direct extraction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Cornell UniversityIthaca · United States120 works · 59,940 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer and Christopher Re. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes. arXiv:2304.09433.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.09433 · arXiv 2304.09433

@misc{arora2023language,
  title = {{Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes}},
  author = {Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer and Christopher Re},
  year = {2023},
  doi = {10.48550/arxiv.2304.09433},
  eprint = {2304.09433},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.09433}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. April 2023. Wikipedia Statistics. https://en.wikipedia.org/wiki/Special:Statistics ↗
  2. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large Language Models are Few-Shot Clinical Information Extractors. The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2022).
  3. Simran Arora, Patrick Lewis, Angela Fan, Jacob Kahn, and Christopher Ré. 2023. Reasoning over Public and Private Data in Retrieval-Based Systems. Transactions of Computational Linguistics (TACL) (2023). find
  4. Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2023. Ask Me Anything: A simple strategy for prompting language models. International Conference on Learning Representations (ICLR) (2023). find
  5. Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes. (2023). https://www.dropbox.com/scl/fi/3gt3ixdbvp986ptyz5j4t/VLDB_Revision. pdf?rl… ↗
  6. Amanda Askell, Yushi Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, and Nicholas Joseph et al. 2021. A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861v3 (2021). find
  7. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew G Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. IJCAI (2007).
  8. David W Bates, David M Levine, Hojjat Salmasian, Ania Syrowatka, David M Shahian, Stuart Lipsitz, Jonathan P Zebrowski, Laura C Myers, Merranda S Logan, Christopher G Roy, et al. 2023. The Safety of Inpatient Health Care. New England Journal of Medicine 388, 2 (2023), 142–153. find
  9. Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. 2021. Interactive weak supervision: Learning useful heuristics for data labeling. find
  10. Rideout J. R. Dillon M. R. Bokulich N. A. Abnet C. C. Al-Ghalith G. A. Alexander H. Alm E. J. Arumugam M. et al. Bolyen, E. 2019. Reproducible, interactive, scalable and extensible microbiome data science using qiime 2. In Nature biotechnology. find
  11. Rishi Bommasani, Drew A. Hudson, E. Adeli, Russ Altman, Simran Arora, S. von Arx, Michael S. Bernstein, Jeanette Bohg, A. Bosselut, Emma Brunskill, and et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258 (2021).
  12. S. Brin. 1998. Extracting patterns and relations from the WorldWide Web. In WebDB. find
  13. Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. 2013. Extraction and integration of partially overlapping web sources. PVLDB (2013). find
  14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  15. Michael J. Cafarella, Christopher Re, Dan Suciu, Oren Etzioni, and Michele Banko. 2007. Structured Querying of Web Text. In Conference on Innovative Data Systems Research (CIDR). find
  16. Michael J Cafarella, Dan Suciu, and Oren Etzioni. 2007. Navigating Extracted Data with Schema Discovery.. In WebDB. 1–6. find
  17. Zui Chen, Zihui Gu, Lei Cao, Ju Fan, Sam Madden, and Nan Tang. 2023. Symphony: Towards Natural Language Query Answering over Multi-modal Data Lakes. CIDR (2023). find
  18. Eric Chu, Akanksha Baid, Ting Chen, AnHai Doan, and Jeffrey Naughton. 2007. A Relational Approach to Incrementally Extracting and Querying Structure in Unstructured Data. In VLDB. find
  19. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR). find
  20. W. Cohen. 2004. Information extraction and integration: An overview. IJCAI (2004). find
  21. Lei Cui, Furu Wei, and Ming Zhou. 2022. Neural Open Information Extraction. (2022). find
  22. Xiang Deng, Prashant Shiralkar, Colin Lockard, Binxuan Huang, and Huan Sun. 2022. DOM-LM: Learning Generalizable Representations for HTML Documents. (2022). find
  23. Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Commun. ACM 51, 12 (2008), 68–74.
  24. Oren Etzioni, Michael Cafarella, Doug Downey, Stanley Kok, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2004. Web-scale information extraction in knowitall: (preliminary results). In Proceedings of the 13th international conference on World Wide Web. 100–110. find
  25. Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. 2004. Unsupervised named-entity extraction from the Web: An experimental study. In AAAI. find
  26. J. H. Faghmous and V Kumar. 2014. A big data guide to understanding climate change: The case for theory-guided data science. In Big data. [53] Laurel Orr. 2022. Manifest. https://github.com/HazyResearch/manifest. [54] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishki… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arora-2023-language-models-enable-simple-systems/