Elamur Research⌕
☰
home / papers / 2025 / PaSa: An LLM Agent for Comprehensive Academic Paper Search

PaSa: An LLM Agent for Comprehensive Academic Paper Search

Paper · 2025 · Yichen He, Guanhua Huang, Peiyuan Feng et al.

Summary

Search tools like Google Scholar handle narrow, detailed research questions poorly, so researchers spend long hours on manual literature surveys. PaSa addresses this with an agent made of two LLM components. The Crawler calls a search tool, follows the citations of papers it has already found, and decides when to stop, adding results to a growing paper queue. The Selector reads each queued paper and judges whether it fits the query. Both components run on a 7B model and are trained with reinforcement learning inside the AGILE framework. The authors use a new session-level PPO variant built to cope with sparse rewards and very long trajectories. Training data come from AutoScholarQuery, a synthetic set in which GPT-4o turned the Related Work sections of ICLR, ICML and NeurIPS 2023 and ACL and CVPR 2024 papers into queries. The answers to each query are the cited arXiv papers. A second benchmark, RealScholarQuery, holds 50 genuine researcher queries with annotated answers. Despite training only on synthetic data, PaSa-7B beats Google, Google Scholar, ChatGPT, GPT-o1 and a GPT-4o-prompted version of itself. This shows that compact RL-trained agents can make literature search much more complete.

Key points

  • AutoScholarQuery contains about 35k queries: roughly 33.5k for training, plus 1,000 for development and 1,000 for testing. A manual check of 100 pairs found 94.0% of queries well-formed, and 93.7% of their answer papers relevant.
  • RealScholarQuery has 50 fine-grained queries from AI researchers using the PaSa demo. On it, PaSa-7B exceeds Google with GPT-4o by 37.78% in recall@20 and 39.90% in recall@50. Compared with PaSa-GPT-4o, it also achieves 30.36% higher recall and 4.25% higher precision.
  • On the AutoScholarQuery test set, PaSa-7B gains 34.05% in Recall@20 and 39.36% in Recall@50 over Google with GPT-4o.
  • The Selector reaches 85% F1 and 95% precision on 200 annotated query-paper pairs. This is 5 points above GPT-4o and 30 points above Qwen-2.5-7b.
  • Ablations show the value of each component:
    • Dropping the citation-expansion action cuts recall by 22.98% on AutoScholarQuery and 32.21% on RealScholarQuery.
    • RL training adds 6.24% and 19.96% recall on the same two benchmarks.
  • The model, data and code are released at github.com/bytedance/pasa. The work covers machine learning papers only and 7B models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢ByteDanceBeijing · China41 works · 4,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li and Weinan E. 2025. PaSa: An LLM Agent for Comprehensive Academic Paper Search. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.572

@inproceedings{he2025pasa,
  title = {{PaSa: An LLM Agent for Comprehensive Academic Paper Search}},
  author = {Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li and Weinan E},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.572},
  url = {https://doi.org/10.18653/v1/2025.acl-long.572}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. 2024. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788. find
  2. Marwah Alaofi, Luke Gallagher, Mark Sanderson, Falk Scholer, and Paul Thomas. 2023. Can generative llms create query variants for test collections? an exploratory study. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pages 1869– 18… find
  3. A Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku; 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7 bbc618857627/Model_Card_Claude_3.pdf. ↗
  4. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738.
  5. Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. find
  6. Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. 2023. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288. find
  7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  8. Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. 2024. Agile: A novel reinforcement learning framework of llm agents. Advances in Neural Information Processing Systems, 37:5244–5284. find
  9. Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. 2024. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arXiv:2405.20625. — [arXiv:2405.20625](https://arxiv.org/abs/2405.20625) find
  10. Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. Research synthesis methods, 11(2):181–217. find
  11. Michael Gusenbauer and Neal R Haddaway. 2021. What every researcher should know about searching– clarified concepts, search advice, and an agenda to improve finding in academia. Research synthesis methods, 12(2):136–147. find
  12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  13. Karl Kingsley, Gillian M Galbraith, Matthew Herring, Eva Stowers, Tanis Stewart, and Karla V Kingsley. 2011. Why not just google it? an assessment of information literacy skills in a biomedical science curriculum. BMC medical education, 11:1–8. find
  14. Minghan Li, Honglei Zhuang, Kai Hui, Zhen Qin, Jimmy Lin, Rolf Jagerman, Xuanhui Wang, and Michael Bendersky. 2023. Generate, filter, and fuse: Query expansion via multi-step keyword generation for zero-shot neural rankers. arXiv preprint arXiv:2311.09175. — [arXiv:2311.09175](https://arxiv.org/abs/2311.09175) find
  15. Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X Zhang. 2024. Llms as research tools: A large scale survey of researchers’ usage and perceptions. arXiv preprint arXiv:2411.05025. — [arXiv:2411.05025](https://arxiv.org/abs/2411.05025) find
  16. Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. 2023. Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency. arXiv preprint arXiv:2309.17382. find
  17. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  18. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, pages 1–11. find
  19. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283. — [arXiv:2305.14283](https://arxiv.org/abs/2305.14283) find
  20. Lisa Messeri and MJ Crockett. 2024. Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002):49–58. find
  21. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  22. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. find
  23. Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large language model based long-tail query rewriting in taobao search. In Companion Proceedings of the ACM on Web Conference 2024, pages 20–28. find
  24. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199. find
  25. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924.
  26. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  27. Richard Van Noorden and Jeffrey M Perkel. 2023. Ai and science: what 1,600 researchers think. Nature, 621(7980):672–675. find
  28. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024a. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299, Bangkok, Thailand. Association for Computational… find
  29. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, et al. 2024b. Autosurvey: Large language models can automatically write surveys. arXiv preprint arXiv:2406.10252. find
  30. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10. find
  31. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  32. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: synergizing reasoning and acting in language models (2022). arXiv preprint arXiv:2210.03629.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2025-pasa-llm-agent-comprehensive-academic/