Elamur Research⌕
☰
home / papers / 2026 / CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

Paper · 2026 · Hang Su, Zequn Liu, Chen Hu et al.

Summary

Evaluating whether LLM answers are actually personalized is difficult. Common metrics such as BLEU and ROUGE only measure word overlap. History-matching heuristics stay at the surface level, and LLM-as-a-Judge setups usually depend on generic, hand-written criteria. The authors take a data-driven route instead. They use StackExchange cases where the asker accepted a different answer than the one the community voted highest, which they call Community-Individual Preference Divergence (CIPD). For each case, an LLM reads the user's question history and the competing answers and infers why that user made the choice. These rationales are then aggregated into six cognitive factors, which were checked by experts. On top of these factors they build CoPA, a benchmark of 1,985 user profiles. A model's response is rated against each profile on every factor using a 3-point Likert scale. The scores correlate well with human judgments. Experiments show that giving models user profiles improves both answer quality and preference alignment. This gives personalized QA a more fine-grained, empirically grounded evaluation standard.

Key points

  • CIPD occurs in all four StackExchange category groups, including technical ones like Engineering & Tools. In 72.70% of CIPD questions, the accepted answer was posted no earlier than the top-voted one, so answer timing does not explain the divergence.
  • Filtering began with about 6.8 million questions from 173 communities. Keeping only users with at least three CIPD questions left 626,786 questions from 15,963 users, which produced roughly 52,199 extracted rationales.
  • During rationale extraction, the LLM sees up to 50 of the user's recent questions. It returns structured JSON with a reason, a supporting theory and an explanation.
  • In human validation, two annotators scored 50 sampled pairs. Their agreement was a weighted Cohen's Kappa of 0.643, and their ratings correlated with the LLM judge at Spearman 0.754.
  • Stated limitations:
    • The factors come from a single, learning-oriented source.
    • The factors may overlap.
    • The pipeline relies on LLMs at several stages, which risks circularity and bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓East China Normal UniversityShanghai · China50 works · 2,174 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia and Liu Zhen. 2026. CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.737

@inproceedings{su2026copa,
  title = {{CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors}},
  author = {Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia and Liu Zhen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.737},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.737}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Lada A Adamic, Jun Zhang, Eytan Bakshy, and Mark S Ackerman. 2008. Knowledge sharing and yahoo answers: everyone knows something. In Proceedings of the 17th international conference on World Wide Web, pages 665–674. find
  3. Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen Herring, and Sujay Kumar Jauhar. 2024. Knowledge-augmented large language models for personalized contextual query suggestion. In Proceedings of the ACM Web Conference 2024, pages 3355–3366. find
  4. Raymond B Cattell. 1963. Theory of fluid and crystallized intelligence: A critical experiment. Journal of educational psychology, 54(1):1. find
  5. Aili Chen, Chengyu Du, Jiangjie Chen, Jinghan Xu, Yikai Zhang, Siyu Yuan, Zulong Chen, Liangyue Li, and Yanghua Xiao. 2025. DEEPER insight into your user: Directed persona refinement for dynamic persona modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist… find
  6. Zhenlong Dai, Chang Yao, Wenkang Han, Ying Yuan, Zhipeng Gao, and Jingyuan Chen. 2024. MPCoder: Multi-user personalized code generator with explicit and implicit style representation learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  7. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can LLM be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126–10141, Miami, Florida, USA. Association for Computational Linguistics. personalization at inference time. In The Thirteent… find
  8. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  9. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. 2024. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
  10. Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Complex knowledge base question answering: A survey. IEEE Transactions on Knowledge and Data Engineering, 35(11):11196–11215. find
  11. Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. 2024. Learning to rewrite prompts for personalized text generation. In Proceedings of the ACM Web Conference 2024, pages 3367–3378. find
  12. Ben Liu, Jihai Zhang, Fangquan Lin, Xu Jia, and Min Peng. 2025a. One size doesn’t fit all: A personalized conversational tutoring agent for mathematics instruction. In Companion Proceedings of the ACM on Web Conference 2025, pages 2401–2410. find
  13. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2025b. LLMs + persona-plug = personalized LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9373– 9385, Vi…
  14. Langming Liu, Shilei Liu, Yujin Yuan, Yizhen Zhang, Bencheng Yan, Zhiyuan Zeng, Zihao Wang, Jiaqi Liu, Di Wang, Wenbo Su, et al. 2025c. Uqabench: Evaluating user embedding for prompting llms in personalized question answering. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery a… find
  15. Kevin Kyung Nam, Mark S Ackerman, and Lada A Adamic. 2009. Questions in, knowledge in? a study of naver’s question answering community. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 779–788. find
  16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  18. Jerry Zhi-Yang He, Sashrika Pandey, Mariah L Schrum, and Anca Dragan. Context steering: Controllable Alireza Salemi, Surya Kallumadi, and Hamed Zamani. 2024a. Optimization methods for personalizing large language models through retrieval augmentation. In Proceedings of the 47th International ACM SIG… find
  19. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024b. Lamp: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7370–7392.
  20. Alireza Salemi and Hamed Zamani. 2025. LaMP-QA: A benchmark for personalized long-form question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1139–1159, Suzhou, China. Association for Computational Linguistics.
  21. Nikhil Sharma, Q Vera Liao, and Ziang Xiao. 2024. Generative echo chamber? effect of llm-powered search systems on diverse information seeking. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–17. find
  22. Teng Shi, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, and Han Li. 2025. Retrieval augmented generation with collaborative filtering for personalized text generation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages… find
  23. Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj, Homa Fashandi, Kevin Ferreira, and Chul Lee. 2024. Personal large language model agents: A case study on tailored travel planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track… find
  24. Charles Spearman. 1961. The proof and measurement of association between two things. find
  25. Hang Su, Yun Yang, Tianyang Liu, Xin Liu, Peng Pu, and Xuesong Lu. 2025. Personalized question answering with user profile generation and compression. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 4744–4763, Suzhou, China. Association for Computational Linguistics. find
  26. Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge base completion via search-based question answering. In Proceedings of the 23rd international conference on World wide web, pages 515– 526. find
  27. Junda Wu, Cheng-Chun Chang, Tong Yu, Zhankui He, Jianing Wang, Yupeng Hou, and Julian McAuley. 2024. Coral: collaborative retrieval-augmented large language models improve long-tail recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3391–34… find
  28. Derong Xu, Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, and Enhong Chen. 2025. Harnessing large language models for knowledge graph question answering via adaptive multi-aspect retrieval-augmentation. In Proceedings of the AAAI Conference on Artificia… find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  30. Jinghao Zhang, Yuting Liu, Wenjie Wang, Qiang Liu, Shu Wu, Liang Wang, and Tat-Seng Chua. 2025a. Personalized text generation with contrastive activation steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7128–7141, Vie…
  31. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025b. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  32. Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. 2024. Hydra: Model factorization framework for black-box llm personalization. Advances in Neural Information Processing Systems, 37:100783–100815. find
  33. Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2024. Democratizing large language models via personalized parameter-efficient fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6476–6491, Miami, Florida, USA.…
  34. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2026-copa-benchmarking-personalized-question-answering/