Elamur Research⌕
☰
home / papers / 2026 / CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

Paper · 2026 · Ruiyao Xu, Mihir Parmar, Tiankai Yang et al.

Summary

Preference-based alignment of LLMs depends on human-labelled response pairs, which are costly to collect. Self-rewarding pipelines produce such pairs cheaply, but they can amplify the model's own mistakes. Active preference learning keeps labels trustworthy, yet it leaves most of the unlabeled pool unused. CoAct combines the two approaches.

In each round, the model samples several answers per instruction. The response whose final answer occurs most often becomes the preferred one, and the rarest becomes the dispreferred one. High-agreement items are kept as self-labeled pairs, except those that k-NN distance flags as probable confident errors. These flagged items, together with low-agreement items, are sent to an oracle under a fixed budget. The oracle is GPT-5 given reference answers. The oracle-checked examples also serve as in-context demonstrations for generating new instructions the model can solve. Training uses DPO plus an NLL regularization term.

On Llama3-8B and Qwen3-4B, CoAct beats the strongest baseline by 4–8 points after four iterations and transfers best to GPQA and MMLU-Pro. This makes it a practical option when annotation is scarce.

Key points

  • Average improvements: after four iterations across both model families, CoAct gains +13.25% on GSM8K, +8.19% on MATH and +13.16% on WebInstruct.
  • k-NN error detection: k-NN selection of high-consistency samples finds items the oracle judges wrong at rates of 82.56% (GSM8K), 87.32% (MATH) and 84.17% (WebInstruct). Picking the lowest-consistency items instead gives 70.42%, 74.23% and 72.56% (Llama3-8B, iteration 4).
  • Ablation on Llama3-8B (GSM8K / MATH / WebInstruct):
    • Oracle-only data reaches 36.38 / 11.95 / 13.53.
    • The full system with augmentation and self-labeling reaches 43.58 / 14.46 / 15.97.
    • On MATH, instruction augmentation alone (13.34) beats self-labeling alone (12.12).
  • Robustness findings:
    • The consistency threshold τ works best around 4/8–5/8.
    • GPT-5 judging without reference answers performs about as well as with them.
    • Self-consistency correlates positively with accuracy (Pearson), and the correlation strengthens across iterations.
  • Limitations and code: the authors note extra compute from multi-sampling, a focus on objective-answer reasoning tasks only, and the need for per-dataset threshold tuning. Code is available at github.com/rux001/CoAct.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations🎓Northwestern UniversityLaoag · Philippines4 works · 26 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ruiyao Xu, Mihir Parmar, Tiankai Yang, Zhengyu Hu, Yue Zhao and Kaize Ding. 2026. CoAct: Co-Active LLM Preference Learning with Human-AI Synergy. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.751

@inproceedings{xu2026coact,
  title = {{CoAct: Co-Active LLM Preference Learning with Human-AI Synergy}},
  author = {Ruiyao Xu, Mihir Parmar, Tiankai Yang, Zhengyu Hu, Yue Zhao and Kaize Ding},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.751},
  url = {https://doi.org/10.18653/v1/2026.acl-long.751}
}

References

77 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Amini, Tim Vieira, and Ryan Cotterell. 2024. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571. find
  2. Anthropic. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. In arXiv preprint arXiv:2204.05862. find
  3. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, and 1 others. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. find
  4. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In AISTATS.
  5. Parikshit Bansal and Amit Sharma. 2023. Large language models as annotators: Enhancing generalization of nlp models at minimal cost. arXiv preprint arXiv:2306.15766. find
  6. Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, and Douwe Kiela. 2022. Models in the loop: Aiding crowdworkers with generative annotation assistants. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find
  7. Erdem Biyik and Dorsa Sadigh. 2018. Batch active preference-based learning of reward functions. In CoRL. find
  8. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika. find
  9. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. In ICML. find
  10. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In NeurIPS. find
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  12. Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. 2025. Active preference optimization for sample efficient rlhf. In ECML PKDD. find
  13. Kaize Ding, Xiaoxiao Ma, Yixin Liu, and Shirui Pan. 2024. Divide and denoise: Empowering simple models for robust semi-supervised node classification against label noise. In KDD. find
  14. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In ACL. find
  15. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. In ICML. find
  16. Adam Gleave and Geoffrey Irving. 2022. Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472. — [arXiv:2203.07472](https://arxiv.org/abs/2203.07472) find
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  18. Jianfeng He, Linlin Yu, Changbin Li, Runing Yang, Fanglan Chen, Kangshuo Li, Min Zhang, Shuo Lei, Xuchao Zhang, Mohammad Beigi, and 1 others. 2025. Survey of uncertainty estimation in llms-sources, methods, applications, and challenges. Information Fusion. find
  19. Dan Hendrycks, Collin Burns, Saurav Kadavath, Arjun Agarwal, Steven Levi, Kevin Ellis, and 1 others. 2021. Measuring mathematical problem solving with the math dataset. In NeurIPS.
  20. Yuxuan Lei, Tianfu Wang, Jianxun Lian, Zhengyu Hu, Defu Lian, and Xing Xie. 2026. Humanllm: Towards personalized understanding and simulation of human nature. arXiv preprint arXiv:2601.15793. — [arXiv:2601.15793](https://arxiv.org/abs/2601.15793) find
  21. Zhengyu Hu, Yichuan Li, Zhengyu Chen, Jingang Wang, Han Liu, Kyumin Lee, and Kaize Ding. 2024a. Let’s ask gnn: Empowering large language model for graph in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1396–1409. find
  22. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. find
  23. Zhengyu Hu, Jianxun Lian, Zheyuan Xiao, Max Xiong, Yuxuan Lei, Tianfu Wang, Kaize Ding, Ziang Xiao, Nicholas Jing Yuan, and Xing Xie. 2025a. Population-aligned persona generation for llm-based social simulation. arXiv preprint arXiv:2509.10127. — [arXiv:2509.10127](https://arxiv.org/abs/2509.10127) find
  24. Zhengyu Hu, Jianxun Lian, Zheyuan Xiao, Seraphina Zhang, Tianfu Wang, Nicholas Jing Yuan, Xing Xie, and Hui Xiong. 2025b. Unveiling the learning mind of language models: A cognitive framework and empirical study. arXiv preprint arXiv:2506.13464. — [arXiv:2506.13464](https://arxiv.org/abs/2506.13464) find
  25. Zhengyu Hu, Linxin Song, Jieyu Zhang, Zheyuan Xiao, Tianfu Wang, Zhengyu Chen, Nicholas Jing Yuan, Jianxun Lian, Kaize Ding, and Hui Xiong. 2024b. Explaining length bias in llm-based preference evaluations. arXiv preprint arXiv:2407.01085. find
  26. Zhengyu Hu, Jieyu Zhang, Zhihan Xiong, Alexander Ratner, Kaize Ding, and Ranjay Krishna. 2026. Towards acyclic preference evaluation of language models via multiple evaluators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 21903–21911. find
  27. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023a. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
  28. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023b. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798.
  29. Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, and Furu Wei. 2025. Preference optimization for reasoning with pseudo feedback. In ICLR.
  30. Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. 2025. Correlated proxies: A new definition and improved mitigation for reward hacking. In ICLR. find
  31. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In ICML. find
  32. Dongyuan Li, Zhen Wang, Yankai Chen, Renhe Jiang, Weiping Ding, and Manabu Okumura. 2024a. A survey on deep active learning: Recent advances and new frontiers. IEEE Transactions on Neural Networks and Learning Systems. find
  33. Yichuan Li, Kaize Ding, Jianling Wang, and Kyumin Lee. 2024b. Empowering large language models for textual data augmentation. In Findings of ACL. find
  34. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science.
  35. Xiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus, See-Kiong Ng, and Bryan Kian Hsiang Low. 2026. Activedpo: Active direct preference optimization for sample-efficient alignment. In ICLR. find
  36. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computat…
  37. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, and 1 others. 2025a. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352.
  38. Yue Liu, Hongcheng Gao, Shengfang Zhai, Yufei He, Jun Xia, Zhengyu Hu, Yulin Chen, Xihong Yang, Jiaheng Zhang, Stan Z Li, and 1 others. 2025b. Guardreasoner: Towards reasoning-based llm safeguards. arXiv preprint arXiv:2501.18492. — [arXiv:2501.18492](https://arxiv.org/abs/2501.18492) find
  39. Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, and Ming Zhang. 2025. A survey on efficient large language model training: From data-centric perspectives. In Proceedings of the 63rd Annual Meeting of the Association for Computationa…
  40. Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. 2025. General-reasoner: Advancing llm reasoning across all domains. In NeurIPS.
  41. Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. 2023. Sample efficient reinforcement learning from human feedback via active exploration. arXiv preprint arXiv:2312.00267. find
  42. Luckeciano C Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. 2024. Deep bayesian active learning for preference modeling in large language models. In NeurIPS. find
  43. William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. 2024. Active preference learning for large language models. In ICML. find
  44. Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. 2013. Learning with noisy labels. In NeurIPS. find
  45. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
  46. Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. In NeurIPS.
  47. Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. 2025. Self-consistency preference optimization. In ICML. find
  48. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS.
  49. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  50. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715. find
  51. Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. 2017. Active preference-based learning of reward functions. In Robotics: Science and Systems. find
  52. Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. 2025. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444. find
  53. Yunyi Shen, Hao Sun, and Jean-Francois Ton. 2025. Active reward modeling: Adaptive preference labeling for large language model alignment. In ICML. find
  54. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546, Abu Dhabi, United Arab Emirates. Association for C… find
  55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. find
  56. Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distribution detection with deep nearest neighbors. In ICML. find
  57. Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, and Xueqi Cheng. 2025. Too consistent to detect: A study of self-consistent errors in LLMs. In EMNLP.
  58. Hemish Veeraboina. 2023. Aime problem set 1983-2024. find
  59. Tianfu Wang, Yi Zhan, Jianxun Lian, Zhengyu Hu, Nicholas Jing Yuan, Qi Zhang, Xing Xie, and Hui Xiong. 2025a. Llm-powered multi-agent framework for goal-oriented learning in intelligent tutoring system. In Companion Proceedings of the ACM on Web Conference 2025, pages 510–519. find
  60. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024a. Human-llm collaborative annotation through effective verification of llm labels. In CHI. find
  61. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR.
  62. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS.
  63. Ziqing Wang, Yibo Wen, Abhishek Pandy, Han Liu, and Kaize Ding. 2026. Molmem: Memory-augmented agentic reinforcement learning for sample-efficient molecular optimization. Preprint, arXiv:2604.12237. find
  64. Ziqing Wang, Yibo Wen, William Pattie, Xiao Luo, Weimin Wu, Jerry Yao-Chieh Hu, Abhishek Pandey, Han Liu, and Kaize Ding. 2025b. Polo: Preference-guided multi-turn reinforcement learning for lead optimization. arXiv preprint arXiv:2509.21737. — [arXiv:2509.21737](https://arxiv.org/abs/2509.21737) find
  65. Ziqing Wang, Kexin Zhang, Zihan Zhao, Yibo Wen, Abhishek Pandey, Han Liu, and Kaize Ding. 2025c. A survey of large language models for text-guided molecular discovery: from molecule generation to optimization. arXiv preprint arXiv:2505.16094.
  66. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, and Sainbayar Sukhbaatar. 2025. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. In EMNLP.
  67. Ruiyao Xu and Kaize Ding. 2025. Large language models for anomaly and out-of-distribution detection: A survey. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5992–6012, Albuquerque, New Mexico. Association for Computational Linguistics. find
  68. Ruiyao Xu and Kaize Ding. 2026. GNN-as-judge: Unleashing the power of LLMs for graph learning with GNN feedback. In ICLR. find
  69. Ruiyao Xu, Noelle I. Samia, and Han Liu. 2026. DS2-instruct: Domain-specific data synthesis for large language models instruction tuning. In Findings of the Association for Computational Linguistics: EACL 2026.
  70. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  71. Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, and 1 others. 2022. Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS. find
  72. Tiankai Yang, Yi Nian, Li Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan A. Rossi, Kaize Ding, Xia Hu, and Yue Zhao. 2025b. AD-LLM: Benchmarking large language models for anomaly detection. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1524–1547, Vie… find
  73. Yuqing Yang, Yan Ma, and Pengfei Liu. 2024. Weak-to-strong reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8350–8367, Miami, Florida, USA. Association for Computational Linguistics. find
  74. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR.
  75. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In ICML. find
  76. Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. 2025. No free lunch: Rethinking internal feedback for llm reasoning. arXiv preprint arXiv:2506.17219. — [arXiv:2506.17219](https://arxiv.org/abs/2506.17219) find
  77. Siyan Zhao, John Dang, and Aditya Grover. 2024. Group preference optimization: Few-shot alignment of large language models. In ICLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-coact-co-active-llm-preference/