Elamur Research⌕
☰
home / papers / 2026 / SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams

SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams

Paper · 2026 · Chenglong Wang, Canjia Li, Xingzhao Zhu et al.

Summary

Search relevance models trained on fixed datasets lose accuracy as live query traffic changes, for example when users adopt new slang or cultural references. Self-evolution, where a model learns from its own predictions on unlabeled queries, could help. At industrial scale it has two weaknesses: useful examples are rare among enormous volumes of traffic, and self-generated labels may be wrong and compound errors. The authors propose SERM, built on a generative LLM relevance model that outputs a graded score together with a rationale. SERM adds two multi-agent modules. The first is a sample miner that finds query–document pairs exposing the model's weaknesses. The second is an annotator that labels those pairs using a two-level agreement scheme. The approach was tested on a large industrial search platform (the authors are from Northeastern University, ByteDance and Peking University) with multilingual offline evaluation and online experiments. Over repeated iterations, SERM improved steadily. Plain self-training, by contrast, suffered from error propagation. This makes SERM a practical way to keep relevance models current with real query streams.

Key points

  • Base model: given a query and document, the LLM relevance model produces a relevance score (e.g., 0–4) plus an explanatory rationale.
  • Multi-agent sample miner (MSM):
    • A User Feedback Agent flags pairs where clicks or dwell time show strong user interest but model confidence is low.
    • A Click Model Feedback Agent uses a pre-trained click model to offset position bias and sparse clicks.
    • An intrinsic agent relies on model disagreement (K sampled judgments) and model uncertainty.
  • Multi-agent relevance annotator (MRA): combines retrieval of external knowledge, rationale aggregation, and Inner-Agreement and Inter-Agreement agents to produce reliable labels.
  • Main result: after three self-evolution iterations, SERM gained +2.99 points in NDCG@1 over the baseline model.
  • Scope: evaluated on a platform handling billions of requests per day, limited to document search. Image and video search were left untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢ByteDanceBeijing · China41 works · 4,180 citations🎓Northeastern UniversityShenyang · China42 works · 1,940 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Chenglong Wang, Canjia Li, Xingzhao Zhu, Yifu Huo, Huiyu Wang, Weixiong Lin, Yun Yang, Qiaozhi He, Tian Hua Zhou, Changxiaojia, JingBo Zhu and Tong Xiao. 2026. SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.823

@inproceedings{wang2026serm,
  title = {{SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams}},
  author = {Chenglong Wang, Canjia Li, Xingzhao Zhu, Yifu Huo, Huiyu Wang, Weixiong Lin, Yun Yang, Qiaozhi He, Tian Hua Zhou, Changxiaojia, JingBo Zhu and Tong Xiao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.823},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.823}
}

References

51 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judit Bar-Ilan, Kevin Keenoy, Mark Levene, and Eti Yaari. 2009. Presentation bias is significant in determining user preference for search results—a user study. Journal of the American Society for Information Science and Technology, 60(1):135–149. find
  2. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  3. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. 2024. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first Internatio…
  4. Zeyuan Chen, Haiyan Wu, Kaixin Wu, Wei Chen, Mingjie Zhong, Jia Xu, Zhongyi Liu, and Wei Zhang. 2024. Towards boosting llms-driven relevance modeling with progressive retrieved behavior-augmented prompting. ArXiv preprint, abs/2408.09439. find
  5. Aleksandr Chuklin, Ilya Markov, and Maarten De Rijke. 2022. Click models for web search. Springer Nature. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Understanding bert rankers under distillation. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, pages 149–152. find
  8. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, and 1 others. Critic: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Kon… — Reinforced Self-Training (ReST) for Language Modeling
  9. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI…
  10. Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, and Xia Song. 2024. Scaling laws for multilingual language models. ArXiv preprint, abs/2410.12883. find
  11. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
  12. Yuelyu Ji, Zhuochun Li, Rui Meng, and Daqing He. 2025. Reason-to-rank: Distilling direct and comparative reasoning from large language models for document reranking. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2320– 2329. find
  13. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327.
  14. Canjia Li, Xiaoyang Wang, Dongdong Li, Yiding Liu, Yu Lu, Shuaiqiang Wang, Zhicong Cheng, Simiu Gu, and Dawei Yin. 2023a. Pretrained language model based web search ranking: From relevance to satisfaction. arXiv preprint arXiv:2306.01599. — [arXiv:2306.01599](https://arxiv.org/abs/2306.01599) find
  15. Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2023b. Parade: Passage representation aggregation fordocument reranking. ACM Transactions on Information Systems, 42(2):1–26. find
  16. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023c. CAMEL: communicative agents for "mind" exploration of large language model society. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2…
  17. Lihong Li, Jinyoung Kim, and Imed Zitouni. 2015. Toward predicting the outcome of an A/B experiment for search relevance. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pages 37–46. ACM. find
  18. Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Weichao Wang, Xingshan Zeng, Lifeng Shang, and 1 others. 2023. Self: Self-evolution with language feedback. ArXiv preprint, abs/2310.00533. — [arXiv:2310.00533](https://arxiv.org/abs/2310.00533) find
  19. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing. find
  20. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 2421– 2425.…
  21. Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: contextualized embeddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1101–1104.… find
  22. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  23. Arif Mahmood, Wenxiong Kang, and 1 others. 2024. Pseudo-label refinement for improving self-supervised learning systems. ArXiv preprint, abs/2410.14242. — [arXiv:2410.14242](https://arxiv.org/abs/2410.14242) find
  24. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with bert. ArXiv preprint, abs/1910.14424. find
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  26. Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and 1 others. 2024. Scaling large language model-based multi-agent collaboration. ArXiv preprint, abs/2406.07155. find
  27. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Lingu…
  28. Tian Tang, Zhixing Tian, Zhenyu Zhu, Chenyang Wang, Haiqing Hu, Guoyu Tang, Lin Liu, and Sulong Xu. 2025. Lref: A novel llm-based relevance framework for e-commerce search. In Companion Proceedings of the ACM on Web Conference 2025, pages 468–475. find
  29. Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024,… find
  30. Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322. find
  31. Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Tong Xiao, Chun-liang Zhang, Tongran Liu, and 1 others. 2025. Gram: A generative foundation reward model for reward generalization. arXiv preprint arXiv:2506.14175. — [arXiv:2506.14175](https://arxiv.org/abs/2506.14175) find
  32. Chenglong Wang, Yifu Huo, Yang Gan, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Chunliang Zhang, Tongran Liu, Anxiang Ma, and 1 others. 2026. Probing preference representations: A multi-dimensional evaluation and analysis method for reward models. In Proceedings of the AAAI Conference on Artificial I… find
  33. Chenglong Wang, Hang Zhou, Kaiyan Chang, Bei Li, Yongyu Mu, Tong Xiao, Tongran Liu, and Jingbo Zhu. 2024a. Hybrid alignment training for large language models. ArXiv preprint, abs/2406.15178. find
  34. Chenglong Wang, Hang Zhou, Yimin Hu, Yifu Huo, Bei Li, Tongran Liu, Tong Xiao, and Jingbo Zhu. 2024b. Esrl: Efficient sampling-based reinforcement learning for sequence generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19107–19115. find
  35. Qianlong Wang, Zhiyuan Wen, Qin Zhao, Min Yang, and Ruifeng Xu. 2021. Progressive self-training with discriminator for aspect term extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 257–268, Online and Punta Cana, Dominican Republic. Associat… find
  36. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  37. Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Jia Xu, Zhongyi Liu, Jinjie Gu, and 1 others. 2024a. Cprm: A llm-based continual pre-training framework for relevance modeling in commercial search. ArXiv preprint, abs/2412.01269. find
  38. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2024b. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling. find
  39. Shaowei Yao, Jiwei Tan, Xi Chen, Juhao Zhang, Xiaoyi Zeng, and Keping Yang. 2022. Reprbert: Distilling BERT to an efficient representation-based relevance model for e-commerce. In KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 202… find
  40. Dezhi Ye, Jie Liu, Junwei Hu, Jiabin Fan, Bowen Tian, Haijin Liang, and Jin Ma. 2025. Applying large language model for relevance search in tencent. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5171–5181. find
  41. Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly Jr., Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, Jean-Marc Langlois, and Yi Chang. 2016. Ranking relevance in yahoo search. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery an… find
  42. Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2023. A two-stage adaptation of large language models for text ranking. ArXiv preprint, abs/2311.16720. find
  43. Guorui Zhou, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, Qiang Luo, Qianqian Wang, Qigen Hu, Rui Huang, Shiyao Wang, and 1 others. 2025. Onerec technical report. arXiv preprint arXiv:2506.13695. find
  44. Hang Zhou, Chenglong Wang, Yimin Hu, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2024a. Prior constraints-based reward model training for aligning large language models. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), pages 1395– 1407. find
  45. Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. 2024b. Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2081–2088. IEEE. find
  46. Dawei Zhu, Xiaoyu Shen, Michael Hedderich, and Dietrich Klakow. 2023. Meta self-refinement for robust learning with weak supervision. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1043–1058, Dubrovnik, Croatia. Association for C… find
  47. Ziming Zhu, Chenglong Wang, Shunjie Xing, Yifu Huo, Fengning Tian, Quan Du, Di Yang, Chunliang Zhang, Tong Xiao, and Jingbo Zhu. 2025. Latextrans: Structured latex translation with multi-agent coordination. arXiv preprint arXiv:2508.18791. — [arXiv:2508.18791](https://arxiv.org/abs/2508.18791) find
  48. Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2025. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning. ArXiv preprint, abs/2503.06034. — [arXiv:2503.06034](https://arxiv.org/abs/2503.06034) find
  49. Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024,… find
  50. Lixin Zou, Weixue Lu, Yiding Liu, Hengyi Cai, Xiaokai Chu, Dehong Ma, Daiting Shi, Yu Sun, Zhicong Cheng, Simiu Gu, and 1 others. 2022. Pre-trained language model-based retrieval and ranking for web search. ACM Transactions on the Web, 17(1):1–36. find
  51. Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. 2021. Pre-trained language model based ranking in baidu search. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, Aug… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-serm-self-evolving-relevance-model/