Elamur Research⌕
☰
home / papers / 2026 / Agent-Ops: A Multi-Agent Orchestration Framework for End-to-End SOP Automation in E-Commerce Operations

Agent-Ops: A Multi-Agent Orchestration Framework for End-to-End SOP Automation in E-Commerce Operations

Paper · 2026 · Apoorva Singh, Sanjay Agrawal, Sayanta Adhikari et al.

Summary

LLMs reason well but are unreliable on long workflows that demand strict adherence to procedure. One such workflow is resolving product suppressions in e-commerce, where listings are pulled for quality, authenticity or IP reasons. Account Managers spend about half an hour per case, moving between many internal tools and checking seller documents in several languages. The authors, from Amazon India, built Agent-Ops, a coordinated set of specialised agents:

  • SOP Groomer turns vague human-written procedures into executable specifications. It splits steps, asks RAG-based clarification questions, pulls answers from multimodal sources and merges the results. Experts are consulted only on difficult policy points.
  • WebAgent learns from demonstrations and finds page elements in three stages, so it keeps working when interfaces change.
  • Document Verification Agent checks invoices, certificates and supply-chain papers in multiple languages.

The system runs on a proprietary LLM. It was used by 100 Account Managers over a three-month deployment covering seven SOP categories in three regions. It reached 85–97% end-to-end accuracy and cut handling time from roughly 30 to 5 minutes. This suggests that specialised agents combined with systematic procedure refinement can make LLM automation production-ready.

Key points

  • WebAgent vs. baselines: WebAgent reaches 91.3% task completion and 86.5% consistency. GPT-4 scores 65.48% zero-shot and 71.82% few-shot. Rule-based Selenium scores 78.34%; it is more consistent (91.2%) but breaks when interfaces change. Without three-stage element identification, WebAgent's consistency falls to 58.3%.
  • Document verification: Validation accuracy is 94.2%, compared with 76.3% for standalone OCR (Tesseract or AWS Textract) and 82.7% for GPT-4V. Component scores are 96.3% OCR character accuracy, 93.7% field extraction, 89.4% fuzzy matching and 87.2% on multi-level invoice chains. Supported languages include English, Hindi, Arabic, Portuguese and Turkish.
  • SOP grooming: Groomed SOPs raise accuracy from 74.9% to 88.1%, a gain of 13.2 points. With raw SOPs, 42% of failures involve navigation, while 31% stem from validation criteria and the remaining 27% from decision rules.
  • Single vs. multiple agents: Replacing the specialised agents with one general-purpose agent lowers accuracy from 88.1% to 78.6%.
  • Accuracy by category: Intellectual-property SOPs score highest (97%), and product-quality SOPs score 85–92%. Differences across regions come mainly from how varied the documents are and from language complexity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksEngineering · 133 worksIndustrial and Manufacturing Engineering · 10 works

Cite

Cite as Apoorva Singh, Sanjay Agrawal, Sayanta Adhikari, Vinayak S Puranik, Shivam Tiwari and Dheeraj Assudani. 2026. Agent-Ops: A Multi-Agent Orchestration Framework for End-to-End SOP Automation in E-Commerce Operations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.29

@inproceedings{singh2026agent,
  title = {{Agent-Ops: A Multi-Agent Orchestration Framework for End-to-End SOP Automation in E-Commerce Operations}},
  author = {Apoorva Singh, Sanjay Agrawal, Sayanta Adhikari, Vinayak S Puranik, Shivam Tiwari and Dheeraj Assudani},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.29},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.29}
}

References

36 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anonymous. 2025. Sop-bench: Evaluating llm agents on complex industrial sops. find
  2. Amazon Machine Learning Conference. AMLC 2025. find
  3. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021. Docformer: End-to-end transformer for document understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 993–1003. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  6. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for "mind" exploration of large language model society. Advances in Neural Information Processing Systems.
  7. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2024. Self-improving web agents. arXiv preprint arXiv:2404.07887. — [arXiv:2404.07887](https://arxiv.org/abs/2404.07887) find
  8. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  9. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.
  10. Microsoft. 2020. Playwright: Fast and reliable end-to-end testing for modern web apps. GitHub Repository. find
  11. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, and 1 others. 2023. Metagpt: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352.
  12. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091.
  13. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. European Conference on Computer Vision, pages 498–517.
  14. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. Advances in Neural Information Processing Systems.
  15. Mandar Kulkarni. 2025. Agent-s: Llm agentic workflow to automate standard operating procedures. arXiv preprint arXiv:2503.15520. — [arXiv:2503.15520](https://arxiv.org/abs/2503.15520) find
  16. LangChain. 2024. Langgraph: Building stateful, multi-actor applications with llms. LangChain Documentation. find
  17. Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. International Conference on Machine Learning, pages 18893–1… find
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–…
  19. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2022. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  20. Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Christoph Auer. 2022. Tableformer: Table structure understanding with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4614–4623. find
  21. OpenAI. 2023. Gpt-4v(ision) system card. OpenAI Technical Report. find
  22. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems.
  23. Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
  24. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. International Conference on Machine Learning, pages 3135–3144. find
  25. Anubhav Shrimal, Stanley Kanagaraj, Kriti Biswas, Swarnalatha Raghuraman, Anish Nediyanchath, Yi Zhang, and Promod Yenigalla. 2024. Marco: Multi-agent real-time chat orchestration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1381–1… find
  26. Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2023. Unifying vision, text, and layout for universal document processing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19254– 19264. find
  27. Shuai Wang, Zhe Li, Jing Zhao, and 1 others. 2024. A survey of multi-ai agent collaboration: Theories, frameworks, and applications. ACM Computing Surveys. find
  28. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems.
  29. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155.
  30. Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. Human-in-the-loop machine learning: A survey. arXiv preprint arXiv:2108.00941. — [arXiv:2108.00941](https://arxiv.org/abs/2108.00941) find
  31. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, and 1 others. 2021. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. Proceedings of the 59th Annual Meeting of the Association for Computational Linguis…
  32. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1192–1200.
  33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations.
  34. Anbang Ye, Qianran Ma, Jia Chen, Muqi Li, Tong Li, Fujiao Liu, Siqi Mai, Meichen Lu, Haitao Bao, and Yang You. 2025. Sop-agent: Empower general purpose ai agent with domain-specific sops. arXiv preprint arXiv:2501.09316. — [arXiv:2501.09316](https://arxiv.org/abs/2501.09316) find
  35. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Seeact: Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.
  36. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-agent-ops-multi-agent-orchestration/