📄 Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization

Summary

LLMs often guess who they are talking to from indirect hints, a behavior called implicit personalization. This can lower answer quality for some groups. The authors test how much stereotypes drive these guesses during multi-turn chats. They built template-based synthetic English conversations that separately control two things: whether the user states their demographic identity, and whether they ask about items stereotypically linked to a group. These items cover food, drinks, hobbies and character traits. They examined three LLMs (OLMo, Gemma, Llama) in two ways. First, through model internals, using linear probes and surprisal. Second, through generated answers to direct and indirect questions about the user. All models kept explicitly stated demographics in their internal representations. Without such statements, they inferred identity from stereotypical cues, especially for gender and race. For gender, stereotypes could even override a user's explicit self-description in every model. Steering activations with the probe weights largely restored the stated identity. This matters because hidden, stereotype-based user modeling affects fairness and trust, and it calls for more transparency and control.

Key points

• The setup spans 3 LLMs and 13 demographic groups across 4 attributes: age, gender, race and SES. Conversations last 6 rounds and draw on 404 items. Stereotype associations come from social science literature and from BBQ (for character traits).
• Some models avoid naming the user's demographics when asked directly. Internal signals (probes, surprisal) still show that the information is retained.
• After probe-based steering toward the stated group, that group had the lowest surprisal in over 77% of conversations, and in 100% for many groups. Steering toward a neutral "no information" class worked poorly and sometimes made stereotyped mentions in indirect answers more frequent.
• Steering barely changed MMLU and IFEval scores. The only significant drops (p < 0.01) were OLMo's IFEval score and Llama's MMLU score.
• Code and data: github.com/Veranep/implicit-personalization-stereotypes. Limitations include a U.S.-centered selection of groups and stereotypes, and no analysis of intersectional identities.