Elamur Research⌕
☰
home / papers / 2026 / Automatic Paper Analysis and Categorisation for Systematic Reviews with Combined Reasoning-Augmented SFT and DAPO RL

Automatic Paper Analysis and Categorisation for Systematic Reviews with Combined Reasoning-Augmented SFT and DAPO RL

Paper · 2026 · Michela Lorandi, Anya Belz, Simon Mille et al.

Summary

Categorising studies along multiple dimensions is one of the most labour-intensive steps of a systematic review, and current LLM-based automation still performs poorly on it. Lorandi et al. treat paper categorisation as a reasoning-based structured-generation problem that does not depend on a particular annotation scheme. Given a cleaned paper, a question and a task description, a model first writes an evidence-grounded reasoning trace and then outputs JSON labels for every field. The team built a manually annotated corpus of controlled text generation (CTG) papers from the ACL Anthology. They used about 100 training examples per task. Open models (Qwen3, LLaMA3) were fine-tuned with LoRA in two stages: reasoning-augmented SFT on synthetic traces, then DAPO reinforcement learning. The RL reward covered both label correctness and output format. The CTG task served for method development. A second task, categorising human-evaluation properties (including hierarchical QCET nodes), was kept as a held-out test of generalisation. Combining SFT and RL gave gains of up to +53.8 points, beat much larger baselines, and outperformed RL alone. In a human-in-the-loop setting it reduced annotation time by roughly 80%.

Key points

  • CTG dataset: 140 ACL Anthology papers proposing new CTG systems, published up to December 2023. Of these, 110 papers (to August 2023) form the training set and 30 (September–December 2023) the test set.
  • CTG annotation scheme: each paper is labelled on four fields, describing the kind of control (Control Type), how it is realised (Control Implementation), how it is integrated (Control Integration) and what is being controlled (Control Attributes). The last of these is open-set, and most fields allow multiple labels.
  • Preprocessing and resources: papers were converted with Docling and stripped of appendices, references and similar sections, averaging about 9k tokens (max 14k). Synthetic reasoning traces came from LLaMA3.3 70B, and RL training needed three A100 80GB GPUs.
  • Results on the hierarchical QCET node task: SFT+DAPO reached 33.3% exact match at Level 3 and 40.8% F1 on complete annotations. A single multi-field model ran about 3x faster than separate per-field models.
  • Limitations: only open-source models were tested, and the method was evaluated only on NLP papers. Code and results are released at github.com/michelalorandi/sft_dapo_paper_annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations🔬ADAPT - Centre for Digital Content TechnologyDublin · Ireland10 works · 976 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Michela Lorandi, Anya Belz, Simon Mille and Craig Thomson. 2026. Automatic Paper Analysis and Categorisation for Systematic Reviews with Combined Reasoning-Augmented SFT and DAPO RL. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2022

@inproceedings{lorandi2026automatic,
  title = {{Automatic Paper Analysis and Categorisation for Systematic Reviews with Combined Reasoning-Augmented SFT and DAPO RL}},
  author = {Michela Lorandi, Anya Belz, Simon Mille and Craig Thomson},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2022},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2022}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  2. Anya Belz, Simon Mille, and Craig Thomson. 2025. The qcet taxonomy of standard quality criterion names and definitions for the evaluation of nlp systems. arXiv preprint arXiv:2509.22064. — [arXiv:2509.22064](https://arxiv.org/abs/2509.22064) find
  3. Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforcement learning. In International conference on machine learning, pages 1371–1382. PMLR. find
  4. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. CoRR.
  5. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164.
  6. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for…
  7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, and Timo Spinde. 2025. The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 137… find
  9. David Howcroft, Anya Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid Hasan, Saad Mahamood, Simon Mille, Sashank Santhanam, Emiel van Miltenburg, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th I… find
  10. Zixia Jia, Junpeng Li, Shichuan Zhang, Anji Liu, and Zilong Zheng. 2024. Combining supervised learning and reinforcement learning for multi-label classification tasks with partial labels. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… find
  11. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  12. Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xiaochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, and Hengxing Cai. 2024. ScilitLLM: How to adapt LLMs for scientific literature understanding. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenge… find
  13. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 3… find
  14. Trans. Mach. Learn. Res., 2023. find
  15. Yuning Mao, Jingjing Tian, Jiawei Han, and Xiang Ren. 2019. Hierarchical text classification with reinforced label assignment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJ… find
  16. Massimiliano Pronesti, Michela Lorandi, Paul Flanagan, Oisín Redmond, Anya Belz, and Yufang Hou. 2025. Enhancing study-level inference from clinical trial papers via reinforcement learning-based numeric reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process… find
  17. Qian Ruan, Ilia Kuznetsov, and Iryna Gurevych. 2024. Are large language models good classifiers? a study on edit intent classification in scientific document revisions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15049–15067, Miami, Florida, USA.… find
  18. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  19. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin…
  20. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  21. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Lingu…
  22. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Mach… find
  23. Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang, Jun Zhang, and Huajun Chen. 2020. Zero-shot text classification via reinforced self-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3014–3024, Online.… find
  24. Edward Yeo, Yuxuan Tong, Xinyao Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying long chain-of-thought reasoning in llms. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy.
  25. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. CoRR.
  26. Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Md. Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, and Yada Zhu. 2025. Evaluating large language models with enterprise benchmarks. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comp… find
  27. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57.
  28. Yiqing Zhao, Xiaohui Shen, and Lanfeng Pan. 2025. Trustworthiness-as-reward: Improving llm performance on text classification through reinforcement learning. find
  29. Control Acc F1 Qwen3 8B 30.3 30.3 + SFT only 71.7 71.7 + RL only EM 69.7 69.7 + RL only EM+F 69.7 69.7 + RL only EM+F+JSON (Ours) 67.6 67.6 66.9 66.9 + RL only J + RL only J+F 66.9 66.9 + RL only J+F+JSON 66.2 66.2 + SFT+RL EM 81.4 81.4 + SFT+RL EM+F 84.1 84.1 + SFT+RL EM+F+JSON (Ours) 84.1 84.1 82.… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lorandi-2026-automatic-paper-analysis-categorisation-systematic/