Agent Learning via Early Experience
Paper · 2025 · Kai Zhang, Xiangchao Chen, Bo Liu et al.
Summary
Reinforcement learning is hard to apply to language agents when environments give no verifiable reward, as on websites, or demand long, costly multi-turn rollouts. As a result, most agents are still fine-tuned on expert demonstrations, which are expensive to scale and cover few situations. The authors propose "early experience" as a middle path between imitation and RL. Starting from states in expert trajectories, the agent tries its own alternative actions and records the resulting next states. Those observed outcomes then serve as reward-free training signals.
Two ways of using this data are studied. Implicit world modeling (IWM) trains the policy to predict future states, which grounds it in environment dynamics. Self-reflection (SR) has the model contrast the outcomes of its own actions with the expert's choice and learn from rationales it generates.
Both methods are tested with several model families across eight environments, which span navigation in embodied and web settings, tool calling over many turns, and planning across extended horizons. Both beat pure imitation learning, generalize better out of domain, need less expert data, and give stronger starting checkpoints for later RL.
Key points
- With Llama-3.1-8B, imitation learning reaches 47.3 on WebShop and 80.5 on ALFWorld. IWM raises these to 58.6 and 85.9, and SR to 58.2 and 85.2.
- The paper compares early experience against three baselines: Long CoT prompting, STaR-style rationales, and DPO on expert vs. non-expert actions.
- All three do worse than early experience.
- DPO collapses within tens of training steps.
- The method is data-efficient compared with imitation learning trained on the full demonstration set:
- On WebShop, early experience seeded with only 1/8 of the demonstrations already beats it.
- On ALFWorld, half of the demonstrations is enough to beat it.
- The branching factor K is the number of alternative actions tried per expert state.
- IWM keeps improving as K grows.
- SR works best with a modest K of about 2–4.
- The gains persist across model scale on WebArena (Llama-3.2-3B, Llama-3.1-8B, Llama-3.3-70B; the 70B model was tuned with LoRA). Checkpoints trained with early experience also make better RL warm starts than imitation-only checkpoints. The paper was published at ICML 2026.
Institutions
United StatesFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works
Cite
Cite as Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, Dat Huynh, Hengduo Li, Zi Yang, Sara Cao, Lawrence Jang, Shuyan Zhou, Jiacheng Zhu, Huan Sun, Jason Weston, Yu Su and Yifan Wu. 2025. Agent Learning via Early Experience. arXiv:2510.08558.
@misc{zhang2025agent,
title = {{Agent Learning via Early Experience}},
author = {Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, Dat Huynh, Hengduo Li, Zi Yang, Sara Cao, Lawrence Jang, Shuyan Zhou, Jiacheng Zhu, Huan Sun, Jason Weston, Yu Su and Yifan Wu},
year = {2025},
doi = {10.48550/arxiv.2510.08558},
eprint = {2510.08558},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2510.08558}
}References
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. In NeurIPS, 2017.
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-agent-learning-early-experience/