Elamur Research⌕
☰
home / papers / 2026 / AutoBool: Reinforcement-Learned LLM for Effective Automatic Systematic Reviews Boolean Query Generation

AutoBool: Reinforcement-Learned LLM for Effective Automatic Systematic Reviews Boolean Query Generation

Paper · 2026 · Shuai Wang, Harrisen Scells, Bevan Koopman et al.

Summary

Medical systematic reviews depend on Boolean search queries that capture nearly all relevant studies while keeping screening workload manageable. Prompted LLMs tend to produce queries with poor recall, and supervised fine-tuning is hard because no single correct query exists for a topic. AutoBool addresses this with reinforcement learning: it trains LLMs with GRPO, scoring each generated query by running it against PubMed and comparing the results to the review's included studies. The reward combines three parts: correct output format, syntactic and executable validity, and a retrieval term that favours recall while giving precision more weight as recall rises.

The authors also mined the PMC Open Access subset to build a large new benchmark of systematic reviews, each paired with its cited studies. AutoBool beats zero-shot and few-shot prompting. With smaller backbones such as Qwen3, it matches or surpasses GPT-4o and O3. It also gets close to expert-written queries while returning far fewer documents. This makes it a practical, lower-cost route to automating literature search.

Key points

  • Dataset: 65,588 systematic review topics drawn from PMC Open Access, with gold-standard studies taken from PMIDs cited in each review's results section.
    • The data is split by publication date into 32,794 training topics (2000–2021) and 32,794 test topics (2021–2025).
    • A leakage-free PubTemp subset holds 1,000 topics published after November 2024, later than Qwen3's knowledge cutoff.
    • The data is released on Hugging Face.
  • Reward design: formatting and validity each give +10 or −10. A query counts as valid only if it parses and returns between 1 and 200,000 PubMed hits. The retrieval reward scales precision by recall raised to α and applies a log transform.
  • Evaluation: experiments use the new data plus CLEF TAR and the Seed Collection, with 118 and 40 topics respectively.
  • Efficiency: AutoBool approaches expert-query effectiveness while retrieving 10–16 times fewer documents.
  • Ablations and prompts: ablations cover backbone, model size, decoding temperature and prompt style. The prompt styles tested are no reasoning, free-text, conceptual (PICO-style) and objective reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany78 works · 41,280 citations🎓The University of QueenslandBrisbane · Australia14 works · 4,603 citations🏛️Commonwealth Scientific and Industrial Research OrganisationCanberra · Australia17 works · 2,064 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksDecision Sciences · 249 worksStatistics, Probability and Uncertainty · 65 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Shuai Wang, Harrisen Scells, Bevan Koopman and Guido Zuccon. 2026. AutoBool: Reinforcement-Learned LLM for Effective Automatic Systematic Reviews Boolean Query Generation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.68

@inproceedings{wang2026autobool,
  title = {{AutoBool: Reinforcement-Learned LLM for Effective Automatic Systematic Reviews Boolean Query Generation}},
  author = {Shuai Wang, Harrisen Scells, Bevan Koopman and Guido Zuccon},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.68},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.68}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Justin Clark. 2013. Systematic reviewing: Introduction, locating studies and data abstraction. In Methods of clinical epidemiology, pages 187–211. Springer. find
  2. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, H… find
  3. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  4. Elke Hausner, Siw Waffenschmidt, Thomas Kaiser, and Michael Simon. 2012. Routine development of objectively derived search strategies. Systematic reviews, 1(1):19. find
  5. Evangelos Kanoulas, Rene Spijker, Dan Li, and Leif Azzopardi. 2018. find
  6. Clef 2018 technology assisted reviews in empirical medicine overview. In CLEF 2018 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS. find
  7. Wojciech Kusa, Oscar E Mendoza, Matthias Samwald, Petr Knoth, and Allan Hanbury. 2023. Csmed: bridging the dataset gap in automated citation screening for systematic literature reviews. Advances in Neural Information Processing Systems, 36:23468–23484. find
  8. Grace E. Lee and Aixin Sun. 2018. Seed-driven document ranking for systematic reviews in evidence-based medicine. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 455–464. find
  9. Andrew MacFarlane, Tony Russell-Rose, and Farhad Shokraneh. 2022. Search strategy formulation for systematic reviews: Issues, challenges and opportunities. Intelligent Systems with Applications, page 200091. find
  10. Minh Nguyen, Toan Quoc Nguyen, Kishan KC, Zeyu Zhang, and Thuy Vu. 2024. Reinforcement learning from answer reranking feedback for retrieval-augmented answer generation. In Proceedings of INTERSPEECH. find
  11. Eric Sayers. 2010. A general introduction to the e-utilities. Entrez Programming Utilities Help [Internet]. Bethesda: National Center for Biotechnology Information. find
  12. Harrisen Scells and Guido Zuccon. 2018. Generating better queries for systematic reviews. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 475–484, New York, NY, USA. ACM. find
  13. Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2020a. A computational approach for objectively derived systematic review search strategies. In Proceedings of the 42nd European Conference on Information Retrieval, pages 385–398. find
  14. Harrisen Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020b. Automatic boolean query formulation for systematic review literature search. In Proceedings of the 29th World Wide Web Conference, pages 1071–1081. find
  15. Moritz Staudinger, Wojciech Kusa, Florina Piroi, Aldo Lipani, and Allan Hanbury. 2024. A reproducibility and generalizability study of large language models for query generation. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval… find
  16. Mark Stevenson and Reem Bin-Hezam. 2023. Stopping methods for technology-assisted reviews based on point processes. ACM Transactions on Information Systems, 42(3):1–37. find
  17. Sebastian Straube, Judith Heinz, Patrick Landsvogt, and Tim Friede. 2021. Recall, precision, and coverage of literature searches in systematic reviews in occupational medicine: an overview of cochrane reviews. find
  18. Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2025. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning. arXiv preprint arXiv:2503.06034. — [arXiv:2503.06034](https://arxiv.org/abs/2503.06034) find
  19. U.S. National Library of Medicine. 2003. Pmc open access subset. https://pmc.ncbi.nlm.nih.gov/ tools/openftlist/. Last accessed: 2025-07-18. ↗
  20. Shuai Wang, Harrisen Scells, Justin Clark, Bevan Koopman, and Guido Zuccon. 2022a. From little things big things grow: A collection with seed studies for medical systematic review literature search. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Informat… find
  21. Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2022b. Neural rankers for effective screening prioritisation in medical systematic review literature search. In Proceedings of the 26th Australasian Document Computing Symposium, pages 1–10. find
  22. Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can chatgpt write a good boolean query for systematic review literature search? In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pages 1426–1436, New York… find
  23. Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2025. Reassessing large language model boolean query generation for systematic reviews. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, pages 3296–3305, New Yo… find
  24. Shuai Wang, Harrisen Scells, Ahmed Mourad, and Guido Zuccon. 2022c. Seed-driven document ranking for systematic reviews: A reproducibility study. In European Conference on Information Retrieval, pages 686–700. Springer. find
  25. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  26. Weinan Zhang, Xiangyu Zhao, Li Zhao, Dawei Yin, Grace Hui Yang, and Alex Beutel. 2020. Deep reinforcement learning for information retrieval: Fundamentals and advances. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2468–2471… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-autobool-reinforcement-learned-llm-effective/