Elamur Research⌕
☰
home / papers / 2024 / OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

Paper · 2024 · Akari Asai, Jacqueline He, Rulin Shao et al.

Summary

Large language models often fail researchers who need to digest scientific literature. They invent references, rely on outdated pre-training knowledge, and seldom show the sources behind their claims. OpenScholar addresses this with a fully open retrieval-augmented system built on OpenScholar-DataStore (OSDS), an index of 45 million open-access Semantic Scholar papers split into 237 million embedded passages. A trained retriever and reranker select evidence. An LM then drafts a cited answer, critiques it, and revises it through repeated self-feedback rounds that can trigger further retrieval, before a final citation check. The same pipeline generated synthetic training data for a compact open model, OpenScholar-8B. To evaluate it, the authors built ScholarQABench, an expert-curated benchmark spanning computer science, physics, neuroscience and biomedicine, scored with both automatic metrics and human judgments. OpenScholar-8B beats much larger proprietary systems on correctness, and its citation accuracy is close to that of human experts. Because everything is released, the work gives a reproducible, non-proprietary foundation for literature-synthesis assistants.

Key points

  • ScholarQABench: 2,967 expert-written questions and 208 long-form expert answers. The answers were written by PhD students and postdocs, and each took roughly an hour. The figure caption cites about 2.2k new questions.
  • Automatic results: OpenScholar-8B beats GPT-4o by about 5% and PaperQA2 by about 7% in correctness. The introduction gives 6% over PaperQA2 and 10% over Perplexity Pro.
  • Citation reliability: GPT-4o fabricated citations in 78–90% of cases. Plugging GPT-4o into the OpenScholar pipeline (OpenScholar-GPT4o) raised its correctness by 12%.
  • Human evaluation: 16 scientists compared system outputs with expert answers. They preferred OpenScholar-GPT4o 70% of the time and OpenScholar-8B 51% of the time, versus roughly 31–32% for GPT-4o without retrieval.
  • Released resources: code, the OpenScholar-8B checkpoint, trained retrievers and reranker, the OSDS index, ScholarQABench, expert-evaluation tools, and a public demo at openscholar.allen.ai.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓University of WashingtonSeattle · United States402 works · 212,763 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,137 citations🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations🎓University of North Carolina at Chapel HillChapel Hill · United States74 works · 23,278 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh and Hannaneh Hajishirzi. 2024. OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs. arXiv:2411.14199.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.14199 · arXiv 2411.14199

@misc{asai2024openscholar,
  title = {{OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs}},
  author = {Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh and Hannaneh Hajishirzi},
  year = {2024},
  doi = {10.48550/arxiv.2411.14199},
  eprint = {2411.14199},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.14199}
}

References

73 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788, 2024. URL https://arxiv.org/abs/2402.01788. ↗
  2. Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023. URL https://arxiv.org/abs/2311.07361. — [arXiv:2311.07361](https://arxiv.org/abs/2311.07361) ↗
  3. Akari Asai and Eunsol Choi. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In ACL, 2021. URL https://aclanthology.org/2021.acl-long. 118. ↗
  4. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics, 2023. URL https://aclanthology.org/2023. findings-acl.225. ↗
  5. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2024. URL https://openreview.net/forum?id=hSyW5go0v8. ↗
  6. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In ICLR, 2024. URL https://openreview.net/forum?id= 4WnqRR915j. ↗
  7. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024. URL https://arxiv.org/abs/2404.07738. ↗
  8. Neal R. Brodnik, Samuel Carton, Caelin Muir, Satanu Ghosh, Doug Downey, McLean P. Echlin, Tresa M. Pollock, and Samantha Daly. Perspective: Large Language Models in Applied Mechanics. Journal of Applied Mechanics, 2023. URL https://doi.org/10.1115/1.4062773. — [doi:10.1115/1.4062773](https://doi.org/10.1115/1.4062773) ↗
  9. Chenglei Chi, Qiaozi Cheng, Zheng Wen, Rongzhe Lin, Chunyang Wen, Zhaowei Wang, Cuiling Gao, Jian Zhang, Xu Jiang, Jian Yin, et al. Uni-SMART: Universal science multimodal analysis and research transformer. arXiv preprint arXiv:2403.10301, 2024. URL https://arxiv.org/ abs/2403.10301. — [arXiv:2403.10301](https://arxiv.org/abs/2403.10301) ↗
  10. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In EMNLP, Brussels, Belgium, 2018. Association for Computational Linguistics. URL https://aclanthology.org/ D18-1241. ↗
  11. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In NAACL, 2021. URL https://aclanthology.org/2021.naacl-main.365. ↗
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  13. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. In COLM, 2024. URL https://openreview.net/forum?id=CybBmzWBX0. ↗
  14. Ziheng Fang, Guangxu Wang, Jinsung Xu, Yifeng Cai, Jingya Wang, Qicheng Qiu, Ruixuan Zhang, Xiaofeng Chen, Jinna Wang, Jiayi Dong, et al. Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and knowledge reasoning tasks. arXiv preprint arXiv:2403.1… — BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
  15. Qiang Feng, Yuxi Li, Jintao Zou, Zhiwei Li, Zhiqiang Ding, Chao Zhang, Qinyan Zhang, Xueqi Hu, Weihao Peng, Xiangyu Meng, et al. K2: A foundation language model for geoscience knowledge understanding and generation. arXiv preprint arXiv:2306.05064, 2023. — [arXiv:2306.05064](https://arxiv.org/abs/2306.05064) find
  16. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627, 2023. URL https://arxiv.org/abs/ 2305.14627. ↗
  17. Xiaodan Gu, Zhen Wang, Zhengliang Shi, Hongyan Li, Xiaoye Chen, and Dehong Cheng. Me-llama: Foundation model for medical language understanding and generation. arXiv preprint arXiv:2402.12749, 2024. URL https://arxiv.org/abs/2402.12749. ↗
  18. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, 2020. URL — REALM: Retrieval-Augmented Language Model Pre-Training https://dl.acm.org/doi/pdf/10.5555/3524938.3525306. ↗
  19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, 2023. URL https://api.semanticscholar.org/CorpusID:263671541. ↗
  20. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702, 2023. URL https://arxiv.org/abs/2311.10702. ↗
  21. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. TMLR, 2022. URL https://openreview.net/forum?id=jKN1pXi7b0. ↗
  22. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In ACL, 2023. URL https://aclanthology.org/2023.emnlp-main.495/. ↗
  23. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In EMNLP-IJCNLP, 2019. URL https://aclanthology.org/D19-1259. ↗
  24. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020. URL https://aclanthology.org/2020.emnlp-main.550/. ↗
  25. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. RealTime QA: What’s the answer right now? In NeurIPS (Datasets and Benchmarks), 2023. URL — RealTime QA: What's the Answer Right Now? https://openreview.net/forum? id=HfKOIPCvsv&noteId=YNFU7iQmxA. ↗
  26. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In ICLR, 2024a. URL https://openreview.net/forum?id=8euJaTveKw. ↗
  27. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761, 2024b. URL h… find
  28. Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David W. Graham,… find
  29. Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. QASA: advanced question answering on scientific articles. In ICML, 2023. URL https://proceedings.mlr.press/v202/lee23n.html. ↗
  30. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS, 2020. URL https://proceedings.… ↗
  31. Junfeng Li, Junjie Gao, Siru Zhang, Yiwen Wang, Xinhang Yan, Hongyan Liu, Shiping Yang, Jie Qiao, and Qian Zhan. BioMistral: A collection of open-source pretrained large language models for biomedicine. In Findings of ACL, 2024a. URL https://aclanthology.org/2024. findings-acl.348/. ↗
  32. Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Associatio…
  33. Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In ACL, 2024c. URL https://aclanthology.org/2024.acl-long.769. ↗
  34. Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Findings of EMNLP, 2023. URL https://aclanthology.org/2023. findings-emnlp.467/. ↗
  35. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL, 2024. URL https://aclanthology.org/2024.tacl-1.9/. ↗
  36. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In ACL, 2020. URL https://aclanthology.org/2020. acl-main.447. ↗
  37. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  38. Yao Lu, Yue Dong, and Laurent Charlin. Multi-xscience: A large-scale dataset for extreme multi-document summarization of scientific articles. In EMNLP, 2020. URL https://aclanthology.org/2020.emnlp-main.648/. ↗
  39. Renqian Luo, Liai Sun, Yingce Xie, Zhiting Jiang, Yangbin Gu, Kun Shi, Dejia Xiong, Sheng He, Zhen Xu, and Tao Qin. Biogpt: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 2022. URL https://academic.oup.com/bib/ article/23/6/bbac409/6713511. ↗
  40. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self. Feedback, 2023. URL https://arxiv.org/abs/2303.17651. ↗
  41. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Expertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852, 2023. URL https://arxiv.org/abs/2309.07852. ↗
  42. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL, 2023. URL https://aclanthology.org/2023.acl-long.546. ↗
  43. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. In COLM, 2024. URL https://openreview.net/forum?id=dJMTn3QOWO. ↗
  44. Tuan Dung Nguyen, Yuan-Sen Ting, Ioana Ciuca, Charles O’Neill, Ze-Chang Sun, Maja Jabłońska, Sandor Kruk, Ernest Perkowski, Jack Miller, Jason Jason Jingsh Li, Josh Peek, Kartheik Iyer, Tomasz Rozanski, Pranav Khetarpal, Sharaf Zaman, David Brodrick, Sergio J. Rodriguez Mendez, Thang Bui, Alyssa Go… find
  45. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019. URL https://arxiv.org/abs/1901.04085. ↗
  46. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In NeurIPS, 2024. URL https://openreview.net/forum?id= 4NJBV6Wp0h. ↗
  47. Long N Phan, James T Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. Scifive: a text-to-text model for biomedical literature. arXiv preprint arXiv:2106.03598, 2021. URL https://arxiv.org/abs/2106.03598. — [arXiv:2106.03598](https://arxiv.org/abs/2106.03598) ↗
  48. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. TACL, 2023. URL https://aclanthology.org/2023.tacl-1.75/. ↗
  49. Shishir G Shaikh, Jaideep Ramachandran, Varnith Nanda, Benjamin Lunt, Miltiadis Allamanis, Daman Sharma, Sebastien Bubeck, and Prateek Jain. Darwin: Data analytics and reasoning with large language models for science. arXiv preprint arXiv:2308.13565, 2023. URL https://arxiv.org/abs/2308.13565. ↗
  50. Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval-based language models with a trillion-token datastore. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/f… ↗
  51. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024. URL https://arxiv.org/abs/2409.04109. ↗
  52. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 2023a. URL https://www.nature.com/articles/ s41586-023-06291-2. ↗
  53. Zeming Singhal, Charles Sutton, Adam Mottram, Owain Lavelle, Iz Beltagy, Leonardo Neves, Kyle Lo, Stephanie Hyland, Michael Wainwright, Alexander Wettig, et al. MEDITRON-70B: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079, 2023b. URL https://arxiv.org/abs/2311… ↗
  54. Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. Language agents achieve superhuman synthesis of scientific knowledge. preprint, 2024. URL https://paper.wikicrow.ai. ↗
  55. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. In ACL, 2024. URL https://aclanthology.org/2024.acl-l… ↗
  56. Cheng Tan, Miao Huang, Xianxin Huang, Qian Fu, and Bo Wu. PMC-LLaMA: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454, 2023. URL https://arxiv.org/ abs/2304.14454. ↗
  57. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS (Datasets and Benchmarks), 2021. URL https://openreview.net/forum? id=wCu6T5xFjeJ. ↗
  58. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. arXiv preprint arXiv:2407.13168, 2024. URL https://arxiv.org/abs/2407. 13168. ↗
  59. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In EMNLP, 2020. URL https://aclanthology.org/2020.emnlp-main.609. ↗
  60. David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. Sciriff: A resource to enhance language model instruction-following over scientific literature. arXiv preprint arXiv:2406.07835, 2024. URL https://a…
  61. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. AutoSurvey: Large language models can automatically write surveys. In NeurIPS, 2024. URL https://openreview.net/forum?id=FExX8pMrdT. ↗
  62. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packaged resources to advance general chinese embedding, 2023. URL — C-Pack: Packed Resources For General Chinese Embeddings https://arxiv.org/ abs/2309.07597. ↗
  63. Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented lms with compression and selective augmentation, 2023a. URL https://arxiv.org/abs/2310. 04408. ↗
  64. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. In ACL, 2023b. URL https://aclanthology.org/2023. acl-long.181. ↗
  65. Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, and David Wadden. KIWI: A dataset of knowledge-intensive writing instructions for answering research questions. In Findings of ACL, 2024. URL https://aclanthology.org/2024.findings-acl.770. ↗
  66. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and E. Cambria. Large language models for automated open-domain scientific hypotheses discovery. In ACL, 2023. URL https://api.semanticscholar.org/CorpusID:261557055. ↗
  67. Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. BioBART: Pretraining and evaluation of a biomedical generative language model. In The 21st Workshop on Biomedical Language Processing (BioNLP), May 2022. URL https://aclanthology.org/2022. bionlp-1.9. ↗
  68. Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. In Findings of EMNLP, 2023. URL https://aclanthology.org/2023.findings-emnlp.307. ↗
  69. Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models. In NeurIPS (Datasets and Benchmarks Track), 2024a. URL https://openreview.net/foru… ↗
  70. Yuqi Zhang, Zihao Zhao, Lanqing Hu, Shuai Wang, Penghui Jiao, Min Leng, Yuzhi Liu, Guotong Li, Chengming Xu, Chenhui Lin, et al. BioMedGPT: Open multimodal generative pre-trained transformer for biomedicine. Nature Medicine, 2024b. URL https://www.nature.com/ articles/s41591-024-03185-2. ↗
  71. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS (Datasets and Benchmarks Track), 2023a. URL https://open…
  72. Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, et al. Openresearcher: Unleashing ai for accelerated scientific research. arXiv preprint arXiv:2408.06941, 2024. URL https://arxiv.org/abs/2408.06941. ↗
  73. Zhiling Zheng, Zichao Rong, Nakul Rampal, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. A gpt-4 reticular chemist for guiding mof discovery. Angewandte Chemie International Edition, 2023b. URL https://arxiv.org/abs/2306.14915. — [arXiv:2306.14915](https://arxiv.org/abs/2306.14915) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asai-2024-openscholar-synthesizing-scientific-literature/