Elamur Research⌕
☰
home / papers / 2026 / RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

Paper · 2026 · Weicong Liu, Zixuan Yang, Yibo Zhao et al.

Summary

As AI/NLP submission volumes grow and research topics shift quickly in the LLM era, matching manuscripts to reviewers with real expertise has become harder. The authors note that most evaluation sets predate 2023 or depend on proxy or third-party labels. They also note that typical methods are trained to retrieve similar papers and then pool per-paper scores, so one off-topic publication can distort a reviewer's ranking. To address evaluation, they release LR-bench, built from recent arXiv papers. Candidate reviewers rated their own familiarity with each manuscript on a five-point scale through an email survey. To address modelling, they propose RATE. An LLM condenses each reviewer's recent publications into a keyword-based natural-language profile. An embedding model is then fine-tuned contrastively on positive and hard-negative pairs selected with BM25 scores, so no human annotation is needed. RATE achieves state-of-the-art results on LR-bench and the CMU gold-standard dataset and clearly outperforms strong embedding baselines. This offers a cheaper and more current basis for building and evaluating reviewer-assignment systems.

Key points

  • LR-bench contains 1,055 paper–reviewer–score annotations based on self-assessed five-level Likert familiarity. It is available on Hugging Face (Gnociew/LR-bench), and the code is on GitHub (Gnociew/RATE-Reviewer-Assignment).
  • The underlying unlabeled corpus has 161,228 arXiv papers from cs.AI, cs.CL, cs.CV, cs.IR and cs.LG, last revised between Oct 2023 and Oct 2025. Metadata was extracted with GROBID, and LLM-assisted author disambiguation produced 513,877 unique authors.
  • For annotation, 4,000 query papers (800 per sub-area) were sampled. Candidate reviewers were recalled using Sentence-BERT embeddings indexed in faiss.
  • In a human evaluation, RATE won 42% of comparisons against TPMS and 44% against SPECTER2 PRX, with lose rates of only 8–17%. In disagreement cases with SPECTER2 PRX, RATE matched expert preference 29 times versus 23 for the baseline.
  • The stated limitations are that author order is ignored, LLM-extracted keywords can be noisy, and reliability is weaker for sparsely published or cold-start reviewers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓East China Normal UniversityShanghai · China50 works · 2,174 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Weicong Liu, Zixuan Yang, Yibo Zhao and Xiang Li. 2026. RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2035

@inproceedings{liu2026rate,
  title = {{RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems}},
  author = {Weicong Liu, Zixuan Yang, Yibo Zhao and Xiang Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2035},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2035}
}

References

49 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nurmukhammed Aitymbetov and Dimitrios Zorbas. 2025. Autonomous machine learning-based peer reviewer selection system. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, pages 199–207, Abu Dhabi, UAE. Association for Computational Linguistics. find
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620,… find
  3. Salem Benferhat and Jérôme Lang. 2001. Conference paper assignment. International Journal of Intelligent Systems, pages 1183–1192. find
  4. Federico Bianchi and Flaminio Squazzoni. 2015. Is three better than one? simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review. In 2015 Winter Simulation Conference (WSC), pages 4081–4089. find
  5. N. Black, S. van Rooyen, F. Godlee, R. Smith, and S. Evans. 1998. What makes a good reviewer and a good review for a general medical journal? JAMA, pages 231–233. find
  6. Laurent Charlin and Richard S. Zemel. 2013. The toronto paper matching system: An automated paper-reviewer assignment system. find
  7. Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & deep learning for recommender systems. In Proceedings of the… find
  8. Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282, Online. Association for Comp…
  9. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the frontier of open…
  10. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The faiss library. Preprint, arXiv:2401.08281.
  11. Susan T. Dumais and Jakob Nielsen. 1992. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’92, page 233–244, New York, NY, USA. Association for Computing M… find
  12. Chase Fensore, Kaustubh Dhole, Joyce C Ho, and Eugene Agichtein. 2025. Evaluating hybrid retrieval augmented generation using dynamic test sets: Liv-erag challenge. Preprint, arXiv:2506.22644. — [arXiv:2506.22644](https://arxiv.org/abs/2506.22644) find
  13. Tanner Fiez, Nihar Shah, and Lillian Ratliff. 2020. A super* algorithm to optimize paper bidding in peer review. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), pages 580–589. find
  14. Naveen Garg, Telikepalli Kavitha, Amit Kumar, Kurt Mehlhorn, and Julián Mestre. 2010. Assigning papers to referees. Algorithmica, pages 119–136. find
  15. Armando Garrido Filipe Garine. 2014. The comprehensive assessment of team member effectiveness (catme): personality predicting teamwork competencies. find
  16. Judy Goldsmith and {Robert H.} Sloan. 2007. The ai conference paper assignment problem. In Preference Handling for Artificial Intelligence - Papers from the 2007 AAAI Workshop, Technical Report, AAAI Workshop - Technical Report, pages 53–57. 2007 AAAI Workshop ; Conference date: 22-07-2007 Through 2… find
  17. Arya McCarthy Amanda Stent Natalie Schluter Graham Neubig, John Wieting and Trevor Cohn. 2021. Acl reviewer matching. https://github.com/acl-org/reviewer-paper-matching. Accessed: 2025-12-23. ↗
  18. Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: a factorization-machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 1725–1731. AAAI Press. find
  19. Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 639–6… find
  20. Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar B. Shah. 2024. Vulnerability of text-matching in ml/ai conference reviewer assignments to collusions. ArXiv. find
  21. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  22. Maryam Karimzadehgan and ChengXiang Zhai. 2009. Constrained multi-aspect expertise matching for committee review assignment. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM ’09, page 1697–1700, New York, NY, USA. Association for Computing Machinery. find
  23. Maryam Karimzadehgan, ChengXiang Zhai, and Geneva Belford. 2008. Multi-aspect expertise matching for review assignment. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM ’08, page 1113–1122, New York, NY, USA. Association for Computing Machinery. find
  24. Harrison J. Kell, Michelle P. Martin-Raugh, Lauren M. Carney, Patricia A. Inglese, Lei Chen, and Gary Feng. 2017. Exploring methods for developing behaviorally anchored rating scales for evaluating structured interview performance. pages 1–26. find
  25. Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, and Dinesh Raghu. 2024. Matching papers and reviewers at large conferences. Artif. Intell., 331(C). find
  26. Yizhi Li, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2021. More robust dense retrieval with contrastive dual learning. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’21, page 287–296, New York, NY, USA. Association for Computing Machinery. find
  27. Weibin Liao, Yifan Zhu, Yanyan Li, Qi Zhang, Zhonghong Ou, and Xuesong Li. 2024. Revgnn: Negative sampling enhanced contrastive graph learning for academic reviewer recommendation. ACM Trans. Inf. Syst., 43(1). find
  28. Cheng Liu, Chenhuan Yu, Ning Gui, Zhiwu Yu, and Songgaojun Deng. 2023. Simgcl: graph contrastive learning by finding homophily in heterophily. Knowl. Inf. Syst., 66(3):2089–2114. find
  29. Patrice Lopez. 2009. GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In Proceedings of the 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), pages 473–474. find
  30. David Mimno and Andrew McCallum. 2007. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’07, page 500–509, New York, NY, USA. Association for Computing Machinery. find
  31. OpenReview. 2022. Paper-reviewer affinity modeling for openreview. https://github.com/ openreview/openreview-expertise. Accessed: 2025-12-23. ↗
  32. Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. 2022. Neighborhood contrastive learning for scientific document representations with citation embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11670–11688. find
  33. Qwen. 2025. Qwen3-max. https://qwen.ai/ apiplatform. Accessed: 2025-12-23. ↗
  34. Ana Carolina Ribeiro, Amanda Sizo, and Luís Paulo Reis. 2023. Investigating the reviewer assignment problem: A systematic literature review. Journal of Information Science. find
  35. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  36. Marko A. Rodriguez and Johan Bollen. 2008. An algorithm to determine peer-reviewers. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM ’08, page 319–328, New York, NY, USA. Association for Computing Machinery. find
  37. Nihar B. Shah. 2022. An overview of challenges, experiments, and computational solutions in peer review (extended version). find
  38. Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2023. SciRepEval: A multi-format benchmark for scientific document representations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566, Singapore. Association for Compu… find
  39. Wenbin Tang, Jie Tang, and Chenhao Tan. 2010. Expertise matching via constraint-based optimization. In 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, volume 1, pages 34–41. find
  40. Devendra Kumar Tayal, P. C. Saxena, Ankita Sharma, Garima Khanna, and Shubhangi Gupta. 2014. New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions. Applied Intelligence, pages 54–73. find
  41. S. Thurner and R. Hanel. 2011. Peer-review in a world with rational scientists: Toward selection of the average. The European Physical Journal B, 84(4):707–711. find
  42. Siyuan Wu, Leong Hou U., Sourav S. Bhowmick, and Wolfgang Gatterbauer. 2018. Pistis: A conflict of interest declaration and detection system for peer review management. In Proceedings of the 2018 International Conference on Management of Data, SIG-MOD ’18, page 1713–1716, New York, NY, USA. Associat… find
  43. Yunhong Xu and Yuanwei Du. 2013. A three-layer network model for reviewer recommendation. In 2013 Sixth International Conference on Business Intelligence and Financial Engineering, pages 552–556. find
  44. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  45. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025a. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  46. Yu Zhang, Yanzhen Shen, SeongKu Kang, Xiusi Chen, Bowen Jin, and Jiawei Han. 2025b. Chain-of-factors paper-reviewer matching. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1901–1910, New York, NY, USA. Association for Computing Machinery. find
  47. Ivan Stelmakh, Nihar Shah, and Aarti Singh. 2021. Peer-review4all: fair and accurate reviewer assignment in peer review. 22(1). find
  48. Ivan Stelmakh, John Wieting, Sarina Xi, Graham Neubig, and Nihar B. Shah. 2025. A gold standard dataset for the reviewer assignment problem. Preprint, arXiv:2303.16750. find
  49. Shicheng Tan, Zhen Duan, Shu Zhao, Jie Chen, and Yanping Zhang. 2021. Improved reviewer assignment based on both word and semantic features. Information Retrieval Journal, pages 175–204. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-rate-reviewer-profiling-annotation-free/