Elamur Research⌕
☰
home / papers / 2026 / LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

Paper · 2026 · Viet Thanh Pham, Lizhen Qu, Thuy-Trang Vu et al.

Summary

LLM-based agents are usually judged by whether they finish tasks. Static benchmarks such as CDEval and NormAd test cultural knowledge, but they cannot show how cultural misalignment appears when an agent pursues goals over time. LiveCultureBench fills this gap. It places a target LLM agent in a simulated small city, represented as a graph of homes, offices, shops and public spaces. The city is populated by synthetic residents whose profiles follow real census distributions. Over one simulated day, the target agent works through a goal split into subtasks. Supporting agents help, distract or pressure it, and each location carries cultural norms taken from CultureBank. An LLM verifier outside the simulation scores every step for task progress, norm violations and social appropriateness. The authors also study the verifier itself, using conformal-style sampling and repeated-evaluation consistency to flag judgments that may be unreliable. The experiments show three patterns:

  • models from the same family share similar cultural biases;
  • agents give up cultural appropriateness to complete tasks;
  • performance falls as scenarios become more multicultural.

This matters for anyone deploying agents across cultures, and it shows that LLM judges still need human oversight.

Key points

  • Time and actions: the simulation runs from 07:00 to 22:00 in 30-minute steps, shortened to 5 minutes during conversations. Agents choose from five action families: move, talk, location-specific actions, phone/message, and wait.
  • Population: the city has 1000 agents sampled from the 2021 Australian Census for Melbourne, covering age, gender, occupation, nationality and family composition. Agents are linked by family, work, education and social relationship ties.
  • Norms: cultural norms are human-annotated entries from the CultureBank dataset, attached to specific locations and checked online as the agent acts.
  • Verifier reliability: risk in the verifier is estimated with uncertainty signals and conformal sampling rejection, building on CLM. Human-annotated data is used for calibration and testing.
  • Resources and limits: the dataset and code are available at github.com/thanhpv2102/LiveCultureBench. Cultural coverage is limited by the norm resources available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Viet Thanh Pham, Lizhen Qu, Thuy-Trang Vu, Gholamreza Haffari and Dinh Phung. 2026. LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1247

@inproceedings{pham2026liveculturebench,
  title = {{LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations}},
  author = {Viet Thanh Pham, Lizhen Qu, Thuy-Trang Vu, Gholamreza Haffari and Dinh Phung},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1247},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1247}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastasios N. Angelopoulos and Stephen Bates. 2022. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Preprint, arXiv:2107.07511. find
  2. Nicolas Bougie and Narimawa Watanabe. 2025. CitySim: Modeling urban behaviors and city dynamics with large-scale LLM-driven agent simulation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 215–229, Suzhou (China). Association for Comp… find
  3. Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. 2025. Neither valid nor reliable? investigating the use of llms as judges. Preprint, arXiv:2508.18076. — [arXiv:2508.18076](https://arxiv.org/abs/2508.18076) find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.
  5. Joshua M. Epstein. 1999. Agent-based computational models and generative social science. Complex., 4(5):41–60. find
  6. Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. 2023. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Preprint, arXiv:2312.11970. find
  7. Aaron Grattafiori and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  9. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. find
  10. CAMEL: Communicative agents for ”mind” exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems. find
  11. Charles M. Macal and Michael J. North. 2013. Agent-based modeling and simulation: introductory tutorial. In Proceedings of the 2013 Winter Simulation Conference: Simulation: Making Decisions in a Complex World, WSC ’13, page 362–376. IEEE Press. find
  12. Mistral AI. 2025. Introducing mistral 3. Mistral AI blog post. find
  13. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. Ass… find
  14. Viet Thanh Pham, Zhuang Li, Lizhen Qu, and Gholamreza Haffari. 2025a. CultureInstruct: Curating multi-cultural instructions at scale. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… find
  15. Viet Thanh Pham, Lizhen Qu, Zhuang Li, Suraj Sharma, and Gholamreza Haffari. 2025b. SurveyPilot: an agentic framework for automated human opinion collection from social media. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 43… find
  16. Viet Thanh Pham, Shilin Qu, Farhad Moghimifar, Suraj Sharma, Yuan-Fang Li, Weiqing Wang, and Reza Haf. 2024. Multi-cultural norm base: Frame-based norm discovery in multi-cultural settings. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 24–35, Miami, FL, USA.… find
  17. Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. 2025. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors… — [arXiv:2502.08691](https://arxiv.org/abs/2502.08691) find
  18. Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. 2024. Conformal language modeling. Preprint, arXiv:2306.10193. find
  19. Abhinav Sukumar Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2025. NormAd: A framework for measuring the cultural adaptability of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingui…
  20. Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computation… find
  21. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. 2024. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4…
  22. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. find
  23. Minghan Wang, Viet Thanh Pham, Farhad Moghimifar, and Thuy-Trang Vu. 2025. Proverbs run in pairs: Evaluating proverb translation capability of large language model. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1646–1662, Vienna, Austria. Association for Computational… find
  24. Yuhang Wang, Yanxu Zhu, Chao Kong, Shuyu Wei, Xiaoyuan Yi, Xing Xie, and Jitao Sang. 2024. CDEval: A benchmark for measuring the cultural dimensions of large language models. In Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP, pages 1–16, Bangkok, Thailand. Association for Co… find
  25. An Yang, Anfeng Li, Baosong Yang, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  26. Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. Sotopia: Interactive evaluation for social intelligence in language agents. Preprint, arXiv:2310.11667.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2026-liveculturebench-multi-agent-multi-cultural/