Elamur Research⌕
☰
home / papers / 2026 / R ESEARCH QA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

R ESEARCH QA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

Paper · 2026 · Li S. Yifei, Allen Chang, Chaitanya Malaviya et al.

Summary

Judging long-form answers to research questions usually requires domain experts. As a result, existing benchmarks are small and focus mostly on AI and engineering. The authors use academic survey articles as a stand-in for expert knowledge. A multi-stage pipeline built on gpt-4.1-mini collects about 54K surveys from top venues across 75 fields. It then turns well-cited survey sections into research queries, each paired with a query-specific rubric. Rubric criteria include citing key papers, explaining mechanisms, comparing approaches and noting limitations. The result, RESEARCHQA, has 21.4K queries and 160K rubric items. Ph.D.-level annotators confirmed that most queries and rubric items are meaningful. Combining rubric coverage scores with direct preference judgments made LLM judges agree more closely with experts. The authors benchmarked 18 systems, from parametric-only LLMs to deep research APIs. Every system left substantial gaps. The best, Perplexity's deep research, covered about 75% of rubric items. The resource makes affordable, multi-field evaluation of research-synthesis agents possible. It can also grow as new surveys are published.

Key points

  • Construction: Surveys were drawn from 660 top venues using Crossref, Semantic Scholar and S2ORC. Of 886K sections, 319K were kept and filtered down to 21.4K queries. The queries span 7 domains, with Health Sciences & Medicine the largest at 7.5K.
  • Expert validation: 31 Ph.D. annotators in 8 fields reviewed the data. They judged that 90% of queries reflect Ph.D.-level information needs. They also found that 87% of rubric items deserve at least a sentence in a good answer.
  • Judge accuracy: Human agreement with the majority vote was 84%. Direct LLM judges reached 71% (proprietary) and 64% (open). An ensemble that adds rubric coverage scores improved these to 74% and 69%.
  • System results: 18 systems were compared in 7.6K head-to-heads. No parametric or retrieval-augmented system exceeded 70% coverage. Production retrieval beat naive Google Scholar retrieval by about 7.6% coverage. Perplexity's deep research scored 75.29% and won 82% of comparisons against the runner-up.
  • Error analysis: Even the leading system fully met fewer than 11% of citation-related rubric items, and its full-satisfaction rates for items on limitations and comparisons were only 48% and 49%, respectively. Data is available at huggingface.co/datasets/realliyifei/ResearchQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States233 works · 65,583 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Li S. Yifei, Allen Chang, Chaitanya Malaviya and Mark Yatskar. 2026. R ESEARCH QA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.732

@article{yifei2026r,
  title = {{R ESEARCH QA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics}},
  author = {Li S. Yifei, Allen Chang, Chaitanya Malaviya and Mark Yatskar},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.732},
  url = {https://doi.org/10.1162/tacl.a.732}
}

References

64 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint, cs.CL/2303.08774.
  2. Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal. 2024. LLMs for literature review: Are we there yet? Transactions on Machine Learning Research. find
  3. Anthropic. 2024. Introducing Claude 4. find
  4. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph C. Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen tau Yi… — OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
  5. Sören Auer, Dante A. C. Barone, Cassiano Bartz, Eduardo G. Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, Ivan Shilin, Markus Stocker, and Eleni Tsalapati. 2023. The SciQA scientific question answering benchmark for scholarly kn… find
  6. Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence to discourse. Discourse Processes. find
  7. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. " O’Reilly Media, Inc.". find
  8. Ralph A. Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika. find
  9. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An open platform for evaluating LLMs by human preference. In International Conference on Machine Learning (IC…
  10. Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics.
  11. Janara Christensen, Stephen Soderland, Gagan Bansal, and Mausam. 2014. Hierarchical summarization: Scaling up multi-document summarization. In Association for Computational Linguistics (ACL). find
  12. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  13. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In North American Chapter of the Association for Computational Linguistics (NAACL). language model specialized in evaluating ot…
  14. Google DeepMind. 2025. Gemini Deep Research. find
  15. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. Conference on Language Modeling (CoLM). find
  16. Arpad E. Elo. 1966. The USCF Rating System: Its Development, Theory, and Applications. United States Chess Federation. find
  17. Seraphina Goldfarb-Tarrant, Alexander Robertson, Jasmina Lazic, Theodora Tsouloufi, Louise Donnison, and Karen Smyth. 2020. Scaling systematic literature reviews with machine learning pipelines. In ACL Workshop on Scholarly Document Processing. find
  18. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. arXiv preprint, cs.AI/2407.21783.
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR).
  20. Peter Jansen, Samiah Hassan, and Ruoyao Wang. 2025. Matter-of-fact: A benchmark for verifying the feasibility of literature-supported claims in materials science. In Empirical Methods in Natural Language Processing (EMNLP). find
  21. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pub-MedQA: A dataset for biomedical research question answering. In Empirical Methods in Natural Language Processing (EMNLP). find
  22. Tetsu Kasanishi, Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. 2023. SciReviewGen: A large-scale dataset for automatic literature review generation. In Association for Computational Linguistics (ACL). find
  23. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Y. Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In North American Ch…
  24. Peder Larsen and Markus Von Ins. 2010. The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics. find
  25. Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023. QASA: Advanced question answering on scientific articles. In International Conference on Machine Learning (ICML). find
  26. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning (ICML). find
  27. Bill Y. Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2025. WildBench: Benchmarking LLMs with challenging tasks from real users in the wild. In International Conference on Learning Representations (ICLR).
  28. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Empirical Methods in Natural Language Processing (EMNLP).
  30. Kyle Lo, Lucy L. Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In Association for Computational Linguistics (ACL).
  31. Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, and Chris Alberti. 2025. Dolomites: Domain-specific long-form methodical tasks. Transactions of the Association for Computational Linguistics.
  32. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. ExpertQA: Expert-curated questions and attributed answers. In North American Chapter of the Association for Computational Linguistics (NAACL).
  33. OpenAI. 2025. Introducing Deep Research. find
  34. Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. Neural Information Processing Systems (NeurIPS). find
  35. Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin. 2025. DeepScholar-Bench: A live benchmark and automated evaluation for generative research synthesis. find
  36. NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. find
  37. Perplexity. 2025. Research. find
  38. Introducing Perplexity Deep Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. 2025. Humanity’s last exam. arXiv preprint, cs.LG/2501.14249.
  39. Derek J. Price. 1963. Little Science, Big Science. Columbia University Press. find
  40. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. InFoBench: Evaluating instruction following ability in large language models. In Association for Computational Linguistics (ACL).
  41. Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Empirical Methods in Natural Language Processing (EMNLP).
  42. David Rein, Betty L. Hou, Asa C. Stickland, Jackson Petty, Richard Y. Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling (CoLM).
  43. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. 2025. ARB: Advanced reasoning benchmark for large language models. NeurIPS MATH-AI Workshop.
  44. Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by LLMs. Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natur… find
  45. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2025. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. In International Conference on Learning Representations (ICLR).
  46. Amanpreet Singh, Joseph C. Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka, Angele Zamarron, Cecile Nguyen, Jena D. Hwang, Jason Dunkleberger, Matt Latzke, Smita Rao, Jaron Lochner, Rob Evans, Rodney Kinney, Daniel S. Weld, Doug Downey, and Sergey Feldman. 2025. AI2 Scholar QA:… find
  47. Shruti Singh, Nandan Sarkar, and Arman Cohan. 2024. SciDQA: A deep reading comprehension dataset over scientific papers. In Empirical Methods in Natural Language Processing (EMNLP). find
  48. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. A long way to go: Investigating length correlations in RLHF. In Conference on Language Modeling (CoLM).
  49. Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. find
  50. Rodriques, and Andrew D. White. 2024. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint, cs.CL/2409.13740.
  51. More natural and complex questions for reasoning across dozens of documents. Transactions of the Association for Computational Linguistics. find
  52. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. Paper-Bench: Evaluating AI’s ability to replicate AI research. In International Conference on… find
  53. Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. 2023. QUDeval: The evaluation of questions under discussion discourse parsing. In Empirical Methods in Natural Language Processing (EMNLP).
  54. Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, and Greg Durrett. 2025. EvalAgent: Discovering implicit evaluation criteria from the web. Conference on Language Modeling (CoLM). find
  55. Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and Ian Foster. 2024. SciQAG: A framework for auto-generated science question answering dataset with fine-grained evaluation. arXiv preprint, cs.CL/2405.09939. find
  56. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. AutoSurvey: Large language models can automatically write surveys. Neural Information Processing Systems (NeurIPS).
  57. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2025. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark…
  58. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in LLM-as-a-judge. In NeurIPS Safe Generative AI Workshop.
  59. Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, and Reut Tsarfaty. 2025. find
  60. MoNaCo: Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, and David Wadden. 2024. KIWI: A dataset of knowledge-intensive writing instructions for answering research questions. In Association for Computational Linguistics (ACL). find
  61. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  62. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. In Association for Computational Linguistics (ACL).
  63. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Charles McGrady, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025. SciArena: An open evaluation platf… find
  64. Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. 2024. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Empirical Methods in Natural Language Processing (EMNLP). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yifei-2026-r-esearch-qa-evaluating-scholarly/