Elamur Research⌕
☰
home / papers / 2026 / ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues

ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues

Paper · 2026 · Yeji Park, Jiwon Tark and Taesik Gong

Summary

LLM personalization usually relies on fixed user profiles or text-only signals. These miss the fact that a person's expertise can change from one query to the next, even within a single subject. ExPerT addresses this with a two-stage framework. First, an LLM estimates the user's expertise for each query using few-shot in-context prompting. It combines the query text with word-level keystroke statistics, such as timing and fluency, that are derived from raw key-down and key-up events. Second, the predicted level, on a five-point scale from Novice to Expert, is inserted into a generation prompt that adjusts detail, terminology and conceptual depth. The authors collected 1270 annotated queries from 40 participants in chemistry, computer science and business through a custom chatbot. ExPerT cut expertise-inference MAE by roughly two thirds relative to the best baseline and raised satisfaction ratings noticeably. The work shows that typing behaviour is a useful, profile-free signal for adapting answers to users, though it also raises privacy and fairness concerns.

Key points

  • Dataset: 40 participants across three domains, 1270 queries with keystroke logs, self-reported expertise and satisfaction scores. The preprocessing pipeline and an anonymized dataset are released at github.com/UbiquitousAILab/ExPerT.
  • Inference accuracy: MAE of 0.398 versus 1.162 for IDL (Cheng et al., 2024), the best-performing comparison method, a 65.7% reduction.
  • Satisfaction: on a 5-point Likert scale, expertise-conditioned responses rose from 3.71 to 4.36 (+17.52%, p < 0.001). Gains held at every self-reported expertise level.
  • Follow-up study: 16 returning participants gave 522 ratings of responses generated with gpt-5.3-chat-latest.
    • Predicted expertise (M = 3.975) performed about as well as self-reported expertise (M = 4.010), with no significant difference (p = .241).
    • Both outperformed responses with no expertise conditioning (M = 3.586).
  • Limitations:
    • The method is sensitive to prompt and few-shot exemplar design.
    • Keystroke signals are noisy and may reflect device, fatigue or motor differences rather than expertise.
    • The participant pool was small and consisted mostly of students.
    • Expertise labels are self-reported rather than measured objectively.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea UniversitySeoul · South Korea26 works · 9,490 citations🎓Ulsan National Institute of Science and TechnologyUlsan · South Korea2 works

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yeji Park, Jiwon Tark and Taesik Gong. 2026. ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.959

@inproceedings{park2026expert,
  title = {{ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues}},
  author = {Yeji Park, Jiwon Tark and Taesik Gong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.959},
  url = {https://doi.org/10.18653/v1/2026.acl-long.959}
}

References

53 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alejandro Acien, Aythami Morales, John V. Monaco, Ruben Vera-Rodriguez, and Julian Fierrez. 2022. Typenet: Deep learning keystroke biometrics. IEEE Transactions on Biometrics, Behavior, and Identity Science, 4(1):57–70. find
  2. Anthropic. 2025. Introducing claude 4. https://www.anthropic.com/news/claude-4. Accessed: 2026-01-01. ↗
  3. Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen Herring, and Sujay Kumar Jauhar. 2024. Knowledge-augmented large language models for personalized contextual query suggestion. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 3355–3366, New York, NY, USA. Association for Comp… find
  4. Szeyi Chan, Jiachen Li, Bingsheng Yao, Amama Mahmood, Chien-Ming Huang, Holly Jimison, Elizabeth D. Mynatt, and Dakuo Wang. 2025. "mango mango, how to let the lettuce dry without a spinner?": Exploring user perceptions of using an llm-based conversational assistant toward cooking partner. Proc. ACM… find
  5. Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11149–11159, Singapore. Association for Computational Linguistics. find
  6. Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. 2025. Pad: Personalized alignment at decoding-time. In The Thirteenth International Conference on Learning Representations. find
  7. Chuanqi Cheng, Quan Tu, Wei Wu, Shuo Shang, Cunli Mao, Zhengtao Yu, and Rui Yan. 2024. “in-dialogues we learn”: Towards personalized dialogue without pre-defined profiles through in-dialogue learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10… find
  8. Chun-Wei Chiang, Zhuoran Lu, Zhuoyan Li, and Ming Yin. 2024. Enhancing ai-assisted group decision making through llm-powered devil’s advocate. In Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI ’24, page 103–119, New York, NY, USA. Association for Computing Machi… find
  9. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  10. Hai Dang, Ben Lafreniere, Tovi Grossman, Kashyap Todi, and Michelle Li. 2025. Authoring llm-based assistance for real-world contexts and tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI ’25, page 211–230, New York, NY, USA. Association for Computing Mach… find
  11. Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018. Observations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–12, New York, NY, USA. Association for Computing Machinery. find
  12. Eva Eigner and Thorsten Händler. 2024. Determinants of llm-assisted decision-making. Preprint, arXiv:2402.17385. — [arXiv:2402.17385](https://arxiv.org/abs/2402.17385) find
  13. Clayton Epp, Michael Lippold, and Regan L. Mandryk. 2011. Identifying emotional states using keystroke dynamics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, page 715–724, New York, NY, USA. Association for Computing Machinery. find
  14. Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, and Hongxin Wei. 2024. On the noise robustness of in-context learning for text generation. the Association for Computational Linguistics (Volume 1: Long Papers), pages 2247–2256, Melbourne, Australia. Association for Computational Linguist… find
  15. Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2021. Augmenting scientific papers with justin-time, position-sensitive definitions of terms and symbols. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, N… find
  16. Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, and Maartje ter Hoeve. 2024. On the way to llm personalization: Learning to remember user conversations. Preprint, arXiv:2411.13405. find
  17. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in LLM simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10289–10307, Bangkok, Thailand. Association for Computational Linguistics.
  18. Slava Kalyuga. 2021. The Expertise Reversal Principle in Multimedia Learning, page 171–182. Cambridge Handbooks in Psychology. Cambridge University Press. find
  19. Kevin S. Killourhy and Roy A. Maxion. 2009. Comparing anomaly-detection algorithms for keystroke dynamics. In 2009 IEEE/IFIP International Conference on Dependable Systems & Networks, pages 125–134. find
  20. Julia Kiseleva, Alejandro Montes García, Jaap Kamps, and Nikita Spirin. 2015. The impact of technical domain expertise on search behavior and task outcome. Preprint, arXiv:1512.07051. — [arXiv:1512.07051](https://arxiv.org/abs/1512.07051) find
  21. Jiaxin Mao, Yiqun Liu, Noriko Kando, Min Zhang, and Shaoping Ma. 2018. How does domain expertise affect users’ search interaction and outcome in exploratory search? ACM Trans. Inf. Syst., 36(4). find
  22. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  23. A. F. M. Nazmul Haque Nahin, Jawad Mohammad Alam, Hasan Mahmud, and Kamrul Hasan. 2014. Identifying emotion by keystroke dynamics and text pattern analysis. Behav. Inf. Technol., 33(9):987–996. find
  24. OpenAI. 2024a. Chatgpt-4o model documentation. https://platform.openai.com/docs/ models/chatgpt-4o-latest. Accessed: 2025-07-20. ↗
  25. OpenAI. 2024b. Gpt-4.1 model documentation. https://platform.openai.com/docs/models/gpt-4.1. Accessed: 2025-07-20. ↗
  26. Agata Kołakowska. 2015. Recognizing emotions on the basis of keystroke dynamics. In 2015 8th International Conference on Human System Interaction (HSI), pages 291–297. find
  27. OpenAI. 2026. Gpt-5.3 chat model documentation. https://developers.openai.com/api/ docs/models/gpt-5.3-chat-latest. Accessed: 2026-04-18. ↗
  28. William H. Kruskal and W. Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583– 621. find
  29. Shramay Palta, Nirupama Chandrasekaran, Rachel Rudinger, and Scott Counts. 2025. Speaking the right language: The impact of expertise (mis)alignment in user-AI interactions. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pa… find
  30. Xinrui Lin, Heyan Huang, Kaihuang Huang, Xin Shu, and John Vines. 2025. Seeking inspiration through human-llm interaction. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. Association for Computing Machinery. find
  31. Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. 2024. Once: Boosting content-based recommendation with both open- and closed-source large language models. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, page 452–461, New York, NY, USA. Associa… find
  32. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of Barbara Plank. 2016. Keystroke dynamics as signal for shal…
  33. The COLING 2016 Organizing Committee. find
  34. Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2025. Personalizing reinforcement learning from human feedback with variational preference learning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, U… find
  35. Juan A. Rodriguez, Nicholas Botzer, David Vazquez, Christopher Pal, Marco Pedersoli, and Issam Laradji. 2024. Intentgpt: Few-shot intent discovery with large language models. Preprint, arXiv:2411.10670. — [arXiv:2411.10670](https://arxiv.org/abs/2411.10670) find
  36. Hayeong Song, Jennifer Healey, Alexa F Siu, Curtis Wigington, and John Stasko. 2023. Experts prefer text but videos help novices: an analysis of the utility of multi-media content. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, New York, NY, USA.… find
  37. Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji. 2025. Persona-DB: Efficient large language model personalization for response prediction with collaborative data refinement. In Proceedings of the 31st International Conference on Computatio…
  38. Xinye Tang, Haijun Zhai, Chaitanya Belwal, Vineeth Thayanithi, Philip Baumann, and Yogesh K Roy. 2025. Dynamic context-aware prompt recommendation for domain-specific ai applications. Preprint, arXiv:2506.20815. — [arXiv:2506.20815](https://arxiv.org/abs/2506.20815) find
  39. Edmund Tong, Amir Zadeh, Cara Jones, and Louis-Philippe Morency. 2017. Combating human trafficking with multimodal deep models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1547–1556, Vancouver, Canada. Association for Comp… find
  40. Boshi Wang, Xiang Yue, and Huan Sun. 2023a. Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11865–11881, Singapore. Association for Computational Linguistics. find
  41. Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. 2023b. Cue-CoT: Chain-of-thought prompting for responding to in-depth dialogue questions with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12047–12064, Singapore.… find
  42. Linghe Wang, Minhwa Lee, Ross Volkov, Luan Tuyen Chau, and Dongyeop Kang. 2025. Scholawrite: A dataset of end-to-end scholarly writing process. Preprint, arXiv:2502.02904. — [arXiv:2502.02904](https://arxiv.org/abs/2502.02904) find
  43. Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2024. Ai persona: Towards life-long personalization of llms. Preprint, arXiv:2412.13103. — [arXiv:2412.13103](https://arxiv.org/abs/2412.13103) find
  44. Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  45. Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023. Scattershot: Interactive in-context example curation for text transformation. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, page 353–367, New York, NY, USA. Association for… find
  46. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2025. Expertprompting: Instructing large language models to be distinguished experts. Preprint, arXiv:2305.14688. find
  47. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, Copenhagen, Denmark. Association for Computat…
  48. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. find
  49. Shiquan Zhang, Ying Ma, Le Fang, Hong Jia, Simon D’Alfonso, and Vassilis Kostakos. 2024a. Enabling on-device llms personalization with smartphone sensing. In Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’24, page 186–190, New York, NY, US… find
  50. Yichi Zhang, Zhuo Chen, Yin Fang, Yanxi Lu, Li Fangming, Wen Zhang, and Huajun Chen. 2024b. Knowledgeable preference alignment for LLMs in domain-specific question answering. In Findings of the Association for Computational Linguistics: ACL 2024, pages 891–904, Bangkok, Thailand. Association for Com… find
  51. Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134– 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  52. Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. 2025. Do llms recognize your preferences? evaluating personalized preference following in llms. In The Thirteenth International Conference on Learning Representations. find
  53. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-expert-personalizing-llm-responses-users/