Elamur Research⌕
☰
home / papers / 2024 / ResearchTown: Simulator of Human Research Community

ResearchTown: Simulator of Human Research Community

Paper · 2024 · Haofei Yu, Zhaochen Hong, Zirui Cheng et al.

Summary

Can a community of LLM agents reproduce how human scientists produce and assess research? The authors address this with ResearchTown, a multi-agent simulator that models a research community as an agent-data graph. Researchers become agent nodes, each an LLM with a profile that operates on data. Papers become data nodes, and edges encode authorship, reviewing and citation. The inference engine, TextGNN, adapts message passing from Graph Neural Networks but keeps every hidden state in text form. This lets reading, paper writing and review writing all be treated as variants of a single aggregation step. For evaluation, they build ResearchBench, which hides a paper node and measures how closely the simulator's reconstruction matches the original. This gives a scalable, embedding-based alternative to costly and subjective human ratings. Simulated papers and reviews resemble real ones reasonably well. Quality improves as more agents are added and holds up when unrelated papers are included. Pairing distant fields produces unusual cross-disciplinary ideas. For library users, the work offers a reusable benchmark and a graph-based view of scientific collaboration.

Key points

  • ResearchBench uses node-masking prediction and contains 1,000 paper-writing tasks and 200 review-writing tasks that require multi-agent collaboration. Code is at github.com/ulab-uiuc/research-town and data at huggingface.co/datasets/ulab-ai/research-bench.
  • The community graph has two node types (researchers and papers) and three edge types (author, review, cite). Richer node types such as blogs and codebases are left as possible extensions.
  • Generated papers reach an average similarity of 0.68 to the real texts, while generated reviews reach 0.49, as measured with a state-of-the-art text embedding model.
  • In a human check, similarity judgments correlated with LLM scores at Pearson 0.61. Human and LLM ratings of intrinsic quality agreed poorly. Both rated simulated papers as slightly less novel than real ones (5.50 vs 5.90 from humans, 7.39 vs 7.85 from the LLM).
  • Mixing NLP with astronomy or criminology produced novel research directions. Combining too many unrelated domains (e.g., LLM, biology, criminology, astronomy) led to incoherent, jargon-stitched research questions. The paper appeared at ICML 2025.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng and Jiaxuan You. 2024. ResearchTown: Simulator of Human Research Community. arXiv:2412.17767.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.17767 · arXiv 2412.17767

@misc{yu2024researchtown,
  title = {{ResearchTown: Simulator of Human Research Community}},
  author = {Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng and Jiaxuan You},
  year = {2024},
  doi = {10.48550/arxiv.2412.17767},
  eprint = {2412.17767},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.17767}
}

References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI4Science, M. R. and Quantum, M. A. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023. — [arXiv:2311.07361](https://arxiv.org/abs/2311.07361) find
  2. Ba, J. L. Layer normalization. arXiv:1607.06450, 2016.
  3. Holm, A. N., Plank, B., Wright, D., and Augenstein, I. Longitudinal citation prediction using temporal graph neural networks. arXiv preprint arXiv:2012.05742, 2020. — [arXiv:2012.05742](https://arxiv.org/abs/2012.05742) find
  4. Hu, S., Shen, L., Zhang, Y., and Tao, D. Learning multi-agent communication from graph modeling perspective. In The Twelfth International Conference on Learning Representations, 2024a. arXiv preprint Baek, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea gener…
  5. Hu, X., Fu, H., Wang, J., Wang, Y., Li, Z., Xu, R., Lu, Y., Jin, Y., Pan, L., and Lan, Z. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas. arXiv preprint arXiv:2410.14255, 2024b. find
  6. Cao, B., Cai, D., Zhang, Z., Zou, Y., and Lam, W. On the worst prompt performance of large language models. arXiv preprint arXiv:2406.10248, 2024. — [arXiv:2406.10248](https://arxiv.org/abs/2406.10248) find
  7. Hua, W., Fan, L., Li, L., Mei, K., Ji, J., Ge, Y., Hemphill, L., and Zhang, Y. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227, 2023. find
  8. Chen, M., Li, T., Sun, H., Zhou, Y., Zhu, C., Wang, H., Pan, J. Z., Zhang, W., Chen, H., Yang, F., et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025.
  9. Huang, Q., Vora, J., Liang, P., and Leskovec, J. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
  10. Chen, Z., Mao, H., Wen, H., Han, H., Jin, W., Zhang, H., Liu, H., and Tang, J. Label-free node classification on graphs with large language models (llms). In The Twelfth Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage… — Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
  11. Tang, J., Zhang, J., Yao, L., Li, J.-Z., Zhang, L., and Su, Z. Arnetminer: extraction and mining of academic social networks. In Knowledge Discovery and Data Mining, 2008. find
  12. Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, 2020.
  13. Tasse, G. N., Jarvis, D., James, S., and Rosman, B. Skill machines: Temporal logic skill composition in reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. find
  14. Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  15. Tu, Z., Aranguri, S., and Jacot, A. Mixed dynamics in linear networks: Unifying the lazy and active regimes. arXiv preprint arXiv:2405.17580, 2024. — [arXiv:2405.17580](https://arxiv.org/abs/2405.17580) find
  16. Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  17. Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023.
  18. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. find
  19. West, J. D., Wesley-Smith, I., and Bergstrom, C. T. A recommendation system based on hierarchical clustering of an article-level citation network. IEEE Transactions on Big Data, 2016. find
  20. Lee, S., Sim, W., Shin, D., Kim, S., and Kim, S. Reasoning abilities of large language models through the lens of abstraction and reasoning. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, 2024. find
  21. Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023. — [arXiv:2310.18940](https://arxiv.org/abs/2310.18940) find
  22. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  23. Martinkus, K., Papp, P. A., Schesch, B., and Wattenhofer, R. Agent-based graph neural networks. In The Eleventh International Conference on Learning Representations, 2023. find
  24. Yan, H., Li, C., Long, R., Yan, C., Zhao, J., Zhuang, W., Yin, J., Zhang, P., Han, W., Sun, H., Deng, W., Zhang, Q., Sun, L., Xie, X., and Wang, S. A comprehensive study on text-attributed graphs: Benchmarking and rethinking. In Thirty-seventh Conference on Neural Information Processing Systems Data… find
  25. Newman, M. E. The structure of scientific collaboration networks. Proceedings of the national academy of sciences, 2001. find
  26. Yang, J. and Leskovec, J. Defining and evaluating network communities based on ground-truth. Knowledge and Information Systems, 2012. find
  27. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020.
  28. Yang, J., Liu, Z., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., and Xie, X. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 2021. find
  29. Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.
  30. Yu, H., Wang, C., Zhuang, P., Menapace, W., Siarohin, A., Cao, J., Jeni, L. A., Tulyakov, S., and Lee, H.-Y. 4real: Towards photorealistic 4d scene generation via video diffusion models. arXiv preprint arXiv:2406.07472, 2024. — [arXiv:2406.07472](https://arxiv.org/abs/2406.07472) find
  31. Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP. In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Lin…
  32. Zhao, J., Qu, M., Li, C., Yan, H., Liu, Q., Li, R., Xie, X., and Tang, J. Learning on large-scale text-attributed graphs via variational inference. In The Eleventh International Conference on Learning Representations, 2023. find
  33. Zhou, X., Zhu, H., Mathur, L., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., et al. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2023.
  34. Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. GPTSwarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2024-researchtown-simulator-human-research-community/