Elamur Research⌕
☰
home / papers / 2025 / Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

Paper · 2025 · Shuo Tang, Xianghe Pang, Zexi Liu et al.

Summary

Post-training turns base LLMs into instruction followers, but it needs high-quality instruction data. Such data is expensive to annotate, scarce, and often limited by privacy. Existing synthetic approaches such as Self-Instruct, WizardLM and Magpie usually produce instructions that are not tied to any realistic user situation. The authors address this with MATRIX, a multi-agent simulator. Agents are built from anonymized real social-media profiles, given life goals, and act inside a virtual society. Communication is organized by a homophily principle: agents with similar profile embeddings are grouped together. The resulting text scenarios feed MATRIX-Gen, which retrieves relevant scenarios for a target requirement and prompts an aligned LLM to write instructions and responses grounded in each agent's persona and actions. This produces SFT, DPO, reasoning and domain-specific datasets. With only 20K synthetic pairs, Llama-3-8B-Base beats Meta's Llama-3-8B-Instruct, which used over 10M pairs, on AlpacaEval 2 and Arena-Hard. The result suggests that grounding instructions in realistic scenarios yields very data-efficient post-training.

Key points

  • Agent profiles: 1,000 real human profiles were collected from X, each with a name, description and past tweets. They were found through tag searches seeded by Alpaca instructions and anonymized with an LLM. Historical tweets initialize each agent's memory bank.
  • Released datasets: MATRIX-Gen-SFT, MATRIX-Gen-DPO and MATRIX-Gen-Reason, plus domain sets MATRIX-Gen-Code, MATRIX-Gen-MT (multi-turn) and MATRIX-Gen-Safe. The final model is called MATRIX-Tuned-Model and is trained with SFT followed by DPO.
  • Evaluation scope: the method was compared against 20 baselines on 12 benchmarks. These cover general tasks, math, coding, multi-turn dialogue and safety.
  • Ablations:
    • Scaling the simulation up to roughly 10³ agents and 10⁴ scenarios improves AlpacaEval 2 and Arena-Hard scores.
    • Homophily-guided communication beats both random communication and no communication.
  • Data properties: Llama-3-8B-Instruct was used as a judge, following the Magpie protocol. MATRIX-Gen-DPO instructions were rated higher in quality and harder than the SFT set. The main stated limitation is the LLM inference cost of running large simulations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang and Siheng Chen. 2025. Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1136

@inproceedings{tang2025synthesizing,
  title = {{Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation}},
  author = {Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang and Siheng Chen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1136},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1136}
}

References

4 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. argilla. 2024. Argilla dpo dataset.
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Associatio…
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2025-synthesizing-post-training-data-llms/