Elamur Research⌕
☰
home / papers / 2026 / Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

Paper · 2026 · Parker Seegmiller and Sarah Masud Preum

Summary

Deployed LLMs face user prompts that change over time as new tasks and new users appear. Earlier robustness studies usually assumed out-of-distribution shift rather than measuring it. This paper asks how much prompt drift occurs in practice and how it affects instruction-tuned models. The authors present LENS, a data-centric framework built on real user requests from WildChat. They split English first-turn prompt/response pairs into in-distribution (ID) training buckets and out-of-distribution (OOD) evaluation buckets along three axes: time, latent user group, and geography. Models fine-tuned on ID data are compared with oracle models trained on data from the OOD bucket. Unsupervised measures such as MAUVE quantify the size of each shift. Even moderate drift causes steep quality losses, and degradation grows as the time gap widens. The study argues that instruction-following skills learned from past users do not carry over reliably to new ones, so deployed systems need continuous, data-driven monitoring.

Key points

  • Scale: The study covers 192 ID/OOD settings built from 27 distinct ID training sets. It trains 81 models on 4.68M prompts and evaluates them on 57.6k prompts. The data comes from WildChat, which holds about 1M queries to OpenAI models collected between April 2023 and May 2024.
  • Overall loss: Across settings, ID models lose on average 73% against the oracles. Along the user-group axis, the average loss rate reaches 88%.
  • Size of shift (MAUVE): The average MAUVE score is 0.54 for time-separated prompts and only 0.05 for geography-separated prompts.
  • Qualitative patterns:
    • Instructions become more specific over time.
    • Heavy users write more templated prompts.
    • Regional prompts differ culturally and linguistically, including second-language English and code-mixing.
  • Limitations:
    • Responses are scored with LLM-as-Judge.
    • The experiments took roughly 1,200 A100 GPU hours.
    • Intersections of axes, such as specific user groups over time, are not explored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dartmouth CollegeHanover · United States37 works · 42,625 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Parker Seegmiller and Sarah Masud Preum. 2026. Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1508

@inproceedings{seegmiller2026measuring,
  title = {{Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance}},
  author = {Parker Seegmiller and Sarah Masud Preum},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1508},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1508}
}

References

82 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdulaal, Chen Jin, Nina Montaña-Brown, Aryo Pradipta Gema, Daniel C Castro, Daniel C Alexander, Philip Alexander Teare, Tom Diethe, Dino Oglic, and Amrutha Saseendran. Balancing act: Diversity and consistency in large language model ensembles. In The Thirteenth International Conference on Lea… find
  2. Oshin Agarwal and Ani Nenkova. 2022. Temporal effects on pre-trained models for language processing tasks. Transactions of the Association for Computational Linguistics, 10:904–921.
  3. AI@Meta. 2024. Llama 3 model card. find
  4. Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-jussà. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541. find
  5. A Binu Jose and Pranesh Das. 2022. A multi-objective approach for inter-cluster and intra-cluster distance analysis for numeric data. In Soft Computing: Theories and Applications: Proceedings of SoCTA 2021, pages 319–332. Springer. find
  6. Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, and 1 others. 2024. The art of saying no: Contextual noncompliance in language models. Advances in Neural Information Processing Sy…
  7. Natalie Grace Brigham, Chongjiu Gao, Tadayoshi Kohno, Franziska Roesner, and Niloofar Mireshghallah. Developing story: Case studies of generative ai’s use in journalism. In Workshop on Socially Responsible Language Modelling Research. find
  8. Yuheng Bu, Shaofeng Zou, Yingbin Liang, and Venugopal V Veeravalli. 2016. Estimation of kl divergence between large-alphabet distributions. In 2016 IEEE International Symposium on Information Theory (ISIT), pages 1118–1122. IEEE. find
  9. Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. 2023. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv preprint arXiv:2305.09246. — [arXiv:2305.09246](https://arxiv.org/abs/2305.09246) find
  10. Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustra-tiadis, Christof Monz, Arianna Bisazza, and Maarten Rijke. 2024. The sifo benchmark: Investigating the sequential instruction following ability of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pag… find
  11. Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. 2024. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o. find
  12. Hannah Devinney, Jenny Björklund, and Henrik Björklund. 2024. We don’t talk about that: Case studies on intersectional analysis of social bias in large language models. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 33–44. find
  13. Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2025. The sharelm collection and plugin: contributing human-model chats for the benefit of the community. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 167–177. find
  14. Guanting Dong, Xiaoshuai Song, Yutao Zhu, Runqi Qiao, Zhicheng Dou, and Ji-Rong Wen. 2025. Toward verifiable instruction-following alignment for retrieval augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23796–23804. find
  15. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring causal effects of data statistics on language model’sfactual’predictions. arXiv preprint arXiv:2207.14251. — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) find
  16. Iker García-Ferrero, Rodrigo Agerri, Aitziber Atutxa, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana 32694 10 Ramirez-Romero, German Rigau, and 1 others. 2024. Medical mt5: An open-source multilingual text-to-text llm for the medical domain. In LREC-COL… find
  17. Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. 2023. Robustness of learning from task instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13935–13948. find
  18. Ashim Gupta, Rishanth Rajendhran, Nathan Stringham, Vivek Srikumar, and Ana Marasović. 2024. Whispers of doubt amidst echoes of triumph in nlp robustness. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  19. Xuan He, Da Yin, and Nanyun Peng. 2024. Guiding through complexity: What makes good supervision for hard reasoning tasks? arXiv preprint arXiv:2410.20533. find
  20. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, and 1 others. 2021. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference… find
  21. Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744–2751.
  22. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
  23. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  24. Xiaolei Huang and Michael Paul. 2018. Examining temporality in document classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694–699. find
  25. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella J. Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Fries… find
  26. Evgenia Ilia and Wilker Aziz. 2024. Predict the next word. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 234–255. find
  27. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, and 1 others. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. CoRR.
  28. Myeongjun Jang, Antonios Georgiadis, Yiyun Zhao, and Fran Silavong. 2024. Driftwatch: A tool that automatically detects data drift and extracts representative examples affected by drift. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist… find
  29. Robin Jia. 2024. Does distribution shift matter in the era of pre-trained language models? Domain Adaptation and Related Areas Workshop, Simons Institute. Workshop presentation. find
  30. Thinking Machines John Schulman. 2025. Lora without regret. find
  31. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15696–15707. PMLR.
  32. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  33. Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, and 1 others. 2023. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Sys…
  34. Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. 2025. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv:2503.05061. — [arXiv:2503.05061](https://arxiv.org/abs/2503.05061) find
  35. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, and 1 others. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Proces… find
  36. Benjamin LeBrun, Alessandro Sordoni, and Timothy J O’Donnell. 2022. Evaluating distributional distortion in neural language modeling. In International Conference on Learning Representations. find
  37. Yoonsang Lee, Pranav Atreya, Xi Ye, and Eunsol Choi. 2024. Crafting in-context examples according to lms’ parametric knowledge. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2069–2085. find
  38. Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. 2024. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations. find
  39. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024a. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations.
  40. Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, and 1 others. 2024b. Interpretable user satisfaction estimation for conversational systems with large language models. arXiv preprint arXiv:2403.12388. find
  41. Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, pages 31131–31146. find
  42. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. Timelms: Diachronic language models from twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 251–260.
  43. Jinghui Lu, Maeve Henchion, and Brian Mac Namee. 2020. Diverging divergences: Examining variants of jensen shannon divergence for corpus comparison tasks. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6740– 6744. find
  44. Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. 2022. Time waits for no one! analysis and challenges of temporal misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech…
  45. Yihan Ma, Xinyue Shen, Yixin Wu, Boyang Zhang, Michael Backes, and Yang Zhang. 2024. The death and life of great prompts: analyzing the evolution of llm prompts from the structural perspective. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21990– 22… find
  46. Rhodes Massenon, Ishaya Gambo, Javed Ali Khan, Christopher Agbonkhese, and Ayed Alwadain. 2025. ” my ai is lying to me”: User-reported llm hallucinations in ai mobile apps reviews. Scientific Reports, 15(1):30397. find
  47. R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. 2023. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638. find
  48. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. find
  49. Amanda Myntti, Erik Henriksson, Veronika Laippala, and Sampo Pyysalo. 2025. Register always matters: Analysis of llm pretraining data through the lens of language variation. arXiv preprint arXiv:2504.01542. — [arXiv:2504.01542](https://arxiv.org/abs/2504.01542) find
  50. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, and 1 others. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. find
  51. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  52. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828.
  53. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  54. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. Findings of the Association for Computational Linguistics: EMNLP 2022.
  55. Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. 2025. Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27617–27627. 32696 12
  56. Parker Seegmiller, Joseph Gatto, Omar Sharif, Madhusudan Basak, and Sarah Masud Preum. 2024. Do llms find human answers to fact-driven questions perplexing? a case study on reddit. In 1st Workshop on Reliable Evaluation of Large Language Models for Factual Information (REALInfo-2024), Co-located wit… find
  57. Parker Seegmiller and Sarah Preum. 2023. Statistical depth for ranking and characterizing transformer-based text embeddings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9600–9611. find
  58. Kaisong Song, Lidong Bing, Wei Gao, Jun Lin, Lujun Zhao, Jiancheng Wang, Changlong Sun, Xiaozhong Liu, and Qiong Zhang. 2019. Using customer service dialogues for satisfaction analysis with context-assisted multiple instance learning. In Proceedings of the 2019 conference on empirical methods in nat… find
  59. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38.
  60. Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2024. Evaluating the zero-shot robustness of instruction-tuned language models. In International Conference on Learning Representations. ICLR. find
  61. Mahan Tafreshipour, Aaron Imani, Eric Huang, Eduardo Santana de Almeida, Thomas Zimmermann, and Iftekhar Ahmed. 2025. Prompting in the wild: An empirical study of prompt evolution in software repositories. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pages 68… find
  62. Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. 2023. Id and ood performance are sometimes inversely correlated on real-world datasets. Advances in Neural Information Processing Systems, 36:71703–71722. find
  63. Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024a. Understanding user experience in large language model interactions. arXiv preprint arXiv:2401.08329. — [arXiv:2401.08329](https://arxiv.org/abs/2401.08329) find
  64. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916.
  65. Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Ziet-low, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, and 1 others. 2022. Assaying out-of-distribution generalization in transfer learning. Advances in Neural Information Processing Systems, 35:7… find
  66. Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2025a. Natural context drift undermines the natural language understanding of large language models. arXiv preprint arXiv:2509.01093. find
  67. Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2025b. Pay attention to real world perturbations! natural robustness evaluation in machine reading comprehension. arXiv preprint arXiv:2502.16523. — [arXiv:2502.16523](https://arxiv.org/abs/2502.16523) find
  68. Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-augmented generation with knowledge graphs for customer service question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Inf… find
  69. Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. In Findings of the Association for Computational Linguistics: ACL 2023… find
  70. Yilun Yang and Yekun Chai. 2025. Codemixbench: Evaluating code-mixing capabilities of llms across 18 languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2139–2169. find
  71. Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Du Yaxin, Yang Liu, Yanfeng Wang, and Siheng Chen. 2024. Fedllm-bench: Realistic benchmarks for federated learning of large language models. Advances in Neural Information Processing Systems, 37:111106– 111130. find
  72. Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations. Advances in Neural Information Processing Systems, 36:58478–58507. find
  73. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations. find
  74. Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. 2024a. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.00732. find
  75. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: 32697 13 find
  76. Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Lingui… find
  77. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024b. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations.
  78. Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, and Bing Qin. 2024c. Deciphering the impact of pretraining data on large language models through machine unlearning. In Findings of the Association for Computational Linguistics ACL 2024, pages 9386–9406. find
  79. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, and 1 others. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations. Lianmin Zheng, Wei-Lin…
  80. Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410.
  81. Ruiqi Zhong, Heng Wang, Dan Klein, and Jacob Steinhardt. 2024. Explaining datasets in words: Statistical models with natural language parameters. Advances in Neural Information Processing Systems, 37:79350– 79380. find
  82. Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Jiantao Qiu, Chi Zhang, Ying Qian, and Conghui He. 2025. Meta-rater: A multi-dimensional data selection method for pre-training language models. arXiv preprint arXiv:2504.14194. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seegmiller-2026-measuring-distribution-shift-user-prompts/