📄 Apprentissage par renforcement pour l'alignement des agents LLMs avec des environnements interactifs : quantification et réduction du surapprentissage aux prompts

Summary

The authors study how reinforcement learning (RL) affects LLM-based agents that are fine-tuned to make sequential decisions in a text-based interactive environment. They want to know how much such an agent depends on the exact wording of the prompt it was trained on. To find out, they propose a new analytical approach: after RL fine-tuning, the model is tested on other, paraphrased versions of the instructions. On unfamiliar wordings, the agent's results drop noticeably. In other words, the model overfits to the prompt and does not learn the task itself. To find the cause, the authors examine the model's internal representations and the tokens that most strongly influence its decisions. To reduce this dependence, they add a contrastive loss to training, which makes the agent more robust and improves generalization. The work matters for anyone who fine-tunes language models for interactive tasks: an agent that works with only one instruction template is unreliable in real-world use.

Key points

• Object of study: LLM agents fine-tuned with RL in a text-based environment for sequential decision-making.
• Main finding: after RL training, the agent is sensitive to the wording of the prompt, and its performance drops on variants that were not seen during training.
• To find the source of this sensitivity, the authors analyze the model's internal representations and its most salient (saliency) tokens.
• As a remedy, the authors propose a contrastive loss that increases robustness and the ability to generalize.
• The original was published in NAACL 2025 Findings (aclanthology.org/2025.findings-naacl.390). A French translation appeared in TALN 2025 (vol. 2, pp. 6–7) under the CC BY 4.0 license. The authors are from Sorbonne Université/ISIR, Inria (Flowers), Hugging Face and Univ Angers.