Elamur Research⌕
☰
home / papers / 2026 / Scaling Collaborative Effort with Agents

Scaling Collaborative Effort with Agents

Paper · 2026 · Shannon Zejiang Shen, Valerie Chen, Ken Gu et al.

Summary

This position paper argues that LLM agents are mostly built and judged as one-shot task finishers. That suits poorly the many real tasks where user goals are vague and change as work proceeds. The authors present five case studies (data analysis, travel planning, financial advising, education, math discovery). In each, current agents deliver polished but opaque outputs, handle feedback badly, or lock in early assumptions. To evaluate collaboration directly, they propose collaborative effort scaling, which tracks how joint human–agent utility changes as user effort grows. Good agents should show two properties. First, value should keep rising with more user input. Second, they should keep users engaged long enough when a task needs it. Applying the framework to a simulated travel-planning setup in Collaborative-Gym, they find that extra user effort often adds little or nothing over a fully autonomous baseline. The main reason is that agents react to each request locally and lack a coherent global plan. The framework offers a diagnostic lens for building agents that support users through the whole problem-solving process, not just at the final answer.

Key points

  • The framework has two axes: User Effort (cognitive and investigative work by the user) and Utility of Joint Actions (what the human–agent team achieves together). It is inspired by ML scaling laws.
  • The desired agent properties are interaction sustainability and maximum usability, the latter meaning the agent avoids early abandonment of the interaction.
  • The empirical study uses simulated users in Collaborative-Gym (Shao et al., 2024) on travel planning (Xie et al., 2024). There, state-of-the-art agents proved mediocre collaborators.
  • The authors acknowledge limitations: a single domain and simulated rather than real users. Human evaluation is named as a priority for future work.
  • Code is available at github.com/clinicalml/collaborative-effort-scaling. The paper appeared in Findings of ACL 2026 and comes from MIT, CMU, UW, and Stanford.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of WashingtonSeattle · United States353 works · 200,043 citations🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu and David Sontag. 2026. Scaling Collaborative Effort with Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.595

@inproceedings{shen2026scaling,
  title = {{Scaling Collaborative Effort with Agents}},
  author = {Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu and David Sontag},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.595},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.595}
}

References

67 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Babak Abedin, Christian Meske, Iris Junglas, Fethi Rabhi, and Hamid R Motahari-Nezhad. 2022. Designing and managing human-ai interactions. Information Systems Frontiers, 24(3):691–697. find
  2. All Hands. 2025. All hands switch your job. build what matters. Accessed: 2025-05-19. find
  3. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for human-ai interaction. In Proceedings of the 2019 CHI Conference on Human Factors in… find
  4. Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, and Daniel S Weld. 2024. Challenges in human-agent communication. arXiv preprint arXiv:2412.10380. — [arXiv:2412.10380](https://arxiv.org/abs/2412.10380) find
  5. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S. Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. Preprint, arXiv:2006.14779. — [arXiv:2006.14779](https://arxiv.org/abs/2006.14779) find
  6. Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakc, and Rei mariman. 2024. Generative ai can harm learning. find
  7. Erik Brynjolfsson. 2022. The turing trap: The promise & peril of human-like artificial intelligence. Daedalus, 151:272–287. find
  8. Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig. 2025a. Code with me or for me? how increasing ai automation transforms developer workflows. arXiv preprint arXiv:2507.08149. find
  9. Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2025b. Need help? designing proactive ai assistants for programming. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–18. 12263 10 find
  10. Cursor. 2025. Cursor - the ai code editor. Accessed: 2025-05-19. find
  11. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069.
  12. Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, and 1 others. 2024. An interactive agent foundation model. arXiv preprint arXiv:2402.05929. — [arXiv:2402.05929](https://arxiv.org/abs/2402.05929) find
  13. Christian Fieberg, Lars Hornuf, and David Streich. 2024. Using large language models for financial advice. Available at SSRN 4850039. find
  14. Jane Fine, Mahi Kolla, and Ilai Soloducho. 2025. Data science agent in colab: The future of data analysis with gemini. Accessed: 2025-03-09. find
  15. R for Data Science Community. 2024. The great american coffee taste test. Accessed: 2025-03-11. find
  16. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, and 1 others. 2024. Magentic-one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468. find
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  18. Andreas Haupt and Erik Brynjolfsson. Ai should not be an imitation game: Centaur evaluations. ICML 2025 Position Paper Track, forthcoming. find
  19. Katharine E Henry, Rachel Kornfield, Anirudh Sridharan, Robert C Linton, Catherine Groh, Tony Wang, Albert Wu, Bilge Mutlu, and Suchi Saria. 2022. Human–machine teaming is key to ai adoption: clinicians experiences with a deployed machine learning system. NPJ digital medicine, 5(1):97. find
  20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and 1 others. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  21. E. Horvitz. 1999. Principles of mixed-initiative user interfaces. pages 159–166. find
  22. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? abs/2310.06770.
  23. ArXiv, Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. ArXiv, abs/2503.09516.
  24. L. Kaelbling, M. Littman, and A. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artif. Intell., 101:99–134. find
  25. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  26. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024a. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. Preprint, arXiv:2401.13649.
  27. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024b. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.
  28. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. Llms get lost in multi-turn conversation.
  29. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. 2023. Evaluating human-langu… find
  30. Belinda Z Li, Been Kim, and Zi Wang. 2025a. Questbench: Can llms ask the right question to acquire information in reasoning tasks? arXiv preprint arXiv:2503.22674. — [arXiv:2503.22674](https://arxiv.org/abs/2503.22674) find
  31. Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. 2023. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589. find
  32. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024a. Mediq: Question-asking LLMs and a benchmark for reliable interactive clinical reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  33. Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S Ilgen, Yulia Tsvetkov, and Maarten Sap. 2025b. Aligning llms to ask good questions a case study in clinical reasoning. arXiv preprint arXiv:2502.14860. 12264 11 — [arXiv:2502.14860](https://arxiv.org/abs/2502.14860) find
  34. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025c. Search-o1: Agentic search-enhanced large reasoning models. ArXiv, abs/2501.05366.
  35. You Li, Yi Li, Qian Chen, and Yaping Chang. 2024b. Humans as teammates: The signal of human–ai teaming enhances consumer acceptance of chatbots. International Journal of Information Management, 76:102771. find
  36. Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD workshop on human computation, pages 68–76. find
  37. Andrew W Lo and Jillian Ross. 2024. Can chatgpt plan your retirement?: Generative ai and financial advice. Generative AI and Financial Advice (February 11, 2024). find
  38. Megan Ma. 2025. Rethinking human-ai agent collaboration for the knowledge worker. https://law.stanford.edu/2025/04/01/ rethinking-human-ai-agent-collaboration-for-the-knowledge-worker/. Accessed: 2025-05-18. ↗
  39. Fritz Machlup. 1962. The production and distribution of knowledge in the United States, volume 278. Princeton university press. find
  40. ManusAI. 2025. Manus agent. Accessed: 2025-05-19. find
  41. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations.
  42. Margaret Mitchell, Avijit Ghosh, Alexandra Sasha Luccioni, and Giada Pistilli. 2025. Fully autonomous ai agents should not be developed. Preprint, arXiv:2502.02649. — [arXiv:2502.02649](https://arxiv.org/abs/2502.02649) find
  43. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hanna Hajishirzi, Luke S. Zettlemoyer, Percy Liang, Emmanuel J. Candes, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. ArXiv, abs/2501.19393.
  44. Michael Norman, Vince Kellen, Shava Smallen, Brian DeMeulle, Shawn Strande, Ed Lazowska, Naomi Alterman, Rob Fatland, Sarah Stone, Amanda Tan, and 1 others. 2021. Cloudbank: Managed services to simplify cloud access for computer science research and education. In Practice and Experience in Advanced… find
  45. OpenAI. Openai o3 and o4-mini system card. find
  46. OpenAI. 2025. Introducing operator. Accessed: 2025-05-19. find
  47. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  48. Alexis Ross and Jacob Andreas. 2024. Toward in-context teaching: Adapting examples to students’ misconceptions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13283–13310, Bangkok, Thailand. Association for Computational Ling… find
  49. D. Russell, M. Stefik, P. Pirolli, and S. Card. 1993. The cost structure of sensemaking. Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. find
  50. Omar Shaikh, Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2025. Navigating rifts in human-llm grounding: Study and benchmark. arXiv preprint arXiv:2503.13975.
  51. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. 2024. Collaborative gym: A framework for enabling and evaluating human-agent collaboration. Preprint, arXiv:2412.15701. find
  52. Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. 2025. The leaderboard illusion. Preprint, arXiv:2504.20879. find
  53. Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. 2025. Coding agents with multimodal browsing are generalist problem solvers. arXiv preprint arXiv:2506.03011.
  54. T. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2023. Cognitive architectures for language agents. ArXiv, abs/2309.02427. find
  55. Takehiro Takayanagi, Kiyoshi Izumi, Javier Sanz-Cruzado, Richard McCreadie, and Iadh Ounis. 2025. Are generative ai agents effective personalized financial advisors? In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 286–295. find
  56. Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. 2025. Interactive agents to overcome ambiguity in software engineering. arXiv preprint arXiv:2502.13069. find
  57. Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From human-human collaboration to human-ai collaboration: Designing ai systems that can work together with people. In Extended Abstracts of the 2020 CHI Conference on Human Factors in… find
  58. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations.
  59. Lilian Weng. 2023. Llm-powered autonomous agents. lilianweng.github.io. find
  60. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023a. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.
  61. Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, and 1 others. 2023b. Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms. arXiv preprint arXiv:2307.10168. — [arXiv:2307.10168](https://arxiv.org/abs/2307.10168) find
  62. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents. In arXiv.org. find
  63. John Yang, Carlos E. Jimenez, Alexander Wettig, K. Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. ArXiv, abs/2405.15793.
  64. John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. 2025. SWE-bench multimodal: Do AI systems generalize to visual software domains? In The Thirteenth Internationa… find
  65. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models, 2023. URL https://arxiv.org/abs/2210.03629. ↗
  66. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023a. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
  67. Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and 1 others. 2023b. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667. 12266 13

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-scaling-collaborative-effort-agents/