Elamur Research
☰
home / papers / 2022 / WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2207.01206

Summary

WebShop is a benchmark for agents that must read natural-language goals and act on web pages. Earlier interactive benchmarks either used toy language or needed heavy human effort to collect data and judge outcomes. WebShop simulates an online store with 1.18 million products scraped from amazon.com and 12,087 crowd-sourced shopping instructions. To buy a matching item, an agent searches, browses result and product pages, picks options and then purchases. The reward is computed automatically by comparing the bought item's attributes, type, options and price with the instruction. The environment is packaged for OpenAI Gym and has an HTML view and a simplified text view. The authors trained agents with imitation learning on more than 1,600 human demonstrations, then fine-tuned them with reinforcement learning, using BART, BERT and ResNet components. The best agent clearly beats a rule-based baseline but remains well below skilled humans. The trained agents also worked reasonably well on the real Amazon and eBay sites without any extra training. This shows that a scalable simulator can produce practical web agents.

Key points

  • Results on WebShop: The best IL+RL agent scores 62.4 with a 28.7% success rate. The rule baseline gets 45.6 and 9.6%, and human experts get 82.1 and 59.6%.
  • Choice oracle: An oracle that exhaustively picks the best item and options raises success from 9.6% to 85.4% using the rule baseline's queries. With human queries it goes from 59.6% to 87.8%. This means choosing the right actions, not search, is the main bottleneck. Better queries still help somewhat.
  • Zero-shot transfer: On amazon.com, IL+RL reaches 65.9 / 25% against 45.8 / 19% for the rule baseline. On ebay.com it reaches 62.3 / 21% against 31.7 / 7%.
  • Speed: Agents finish an episode in under 8 seconds. Humans on Amazon take about 815 seconds on average.
  • Open problems: The analysis points to more robust query generation, explicit memory, and better handling of noisy web text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

287 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 266 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2207.01206 · arXiv 2207.01206

@misc{yao2022webshop,
  title = {{WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents}},
  author = {Yao, Shunyu and Chen, Howard and Yang, John and Narasimhan, Karthik},
  year = {2022},
  doi = {10.48550/arxiv.2207.01206},
  eprint = {2207.01206},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.01206}
}

References

59 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Adolphs, B. Boerschinger, C. Buck, M. C. Huebscher, M. Ciaramita, L. Espeholt, T. Hofmann, and Y. Kilcher. Boosting Search Engines with Interactive Agents. arXiv preprint arXiv:2109.00527, 2021. — [arXiv:2109.00527](https://arxiv.org/abs/2109.00527) find
  2. A. Aghajanyan, D. Okhonko, M. Lewis, M. Joshi, H. Xu, G. Ghosh, and L. Zettlemoyer. Htlm: Hyper-text pre-training and prompting of language models. ArXiv, abs/2107.06955, 2021. — [arXiv:2107.06955](https://arxiv.org/abs/2107.06955) find
  3. J. Andreas, J. Bufe, D. Burkett, C. Chen, J. Clausman, J. Crawford, K. Crim, J. DeLoach, L. Dorner, J. Eisner, et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556–571, 2020. find
  4. E. M. Bender and A. Koller. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, 2020. find our card
  5. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016. find
  6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  7. P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gašić. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278, 2018. find
  8. A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer. Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands. arXiv preprint arXiv:2202.02312, 2022. — [arXiv:2202.02312](https://arxiv.org/abs/2202.02312) find
  9. J. Chung, Çaglar Gülçehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv, abs/1412.3555, 2014. find
  10. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv, abs/1810.04805, 2019. find
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019. find
  12. A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019. find
  13. M. Fortunato, M. Tan, R. Faulkner, S. Hansen, A. Puigdomènech Badia, G. Buttimore, C. Deck, J. Z. Leibo, and C. Blundell. Generalization of reinforcement learners with working and episodic memory. Advances in neural information processing systems, 32, 2019. find
  14. X. Guo, M. Yu, Y. Gao, C. Gan, M. Campbell, and S. Chang. Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02386, 2020. find
  15. I. Gur, U. Rueckert, A. Faust, and D. Hakkani-Tur. Learning to Navigate the Web. arXiv preprint arXiv:1812.09195, 2018. — [arXiv:1812.09195](https://arxiv.org/abs/1812.09195) find
  16. I. Gur, N. Jaques, K. Malta, M. Tiwari, H. Lee, and A. Faust. Adversarial Environment Generation for Learning to Navigate the Web. arXiv preprint arXiv:2103.01991, 2021. — [arXiv:2103.01991](https://arxiv.org/abs/2103.01991) find
  17. M. Hausknecht, P. Ammanabrolu, M.-A. Côté, and X. Yuan. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903–7910, 2020. find
  18. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. find
  19. A. Hotti, R. S. Risuleo, S. Magureanu, A. Moradi, and J. Lagergren. The Klarna Product Page Dataset: A RealisticBenchmark for Web Representation Learning. arXiv preprint arXiv:2111.02168, 2021. — [arXiv:2111.02168](https://arxiv.org/abs/2111.02168) find
  20. P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, P. Georgiev, A. Goldin, A. Santoro, et al. A data-driven approach for learning to control computers. arXiv preprint arXiv:2202.08137, 2022. — [arXiv:2202.08137](https://arxiv.org/abs/2202.08137) find
  21. S. Jia, J. Kiros, and J. Ba. Dom-q-net: Grounded RL on Structured Language. arXiv preprint arXiv:1902.07257, 2019. — [arXiv:1902.07257](https://arxiv.org/abs/1902.07257) find
  22. M. Komeili, K. Shuster, and J. Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566, 2021. find our card
  23. A. Lampinen, S. Chan, A. Banino, and F. Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. Advances in Neural Information Processing Systems, 34:28182–28195, 2021. find
  24. A. Lazaridou, A. Potapenko, and O. Tieleman. Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning. In ACL, 2020. find
  25. A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. ArXiv, abs/2203.05115, 2022. find our card
  26. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv preprint arXiv:1910.13461, 2019. find our card
  27. X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121–137. Springer, 2020. find our card
  28. J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, and R. Nogueira. Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationss. arXiv preprint arXiv:2102.10073, 2021. — [arXiv:2102.10073](https://arxiv.org/abs/2102.10073) find
  29. E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and P. Liang. Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration. arXiv preprint arXiv:1802.08802, 2018. find
  30. J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI, 2019. find
  31. S. Mazumder and O. Riva. FLIN: A Flexible Natural Language Interface for Web Navigation. arXiv preprint arXiv:2010.12844, 2020. find
  32. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016. find
  33. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. WebGPT: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332, 2021. find our card
  34. K. Narasimhan, A. Yala, and R. Barzilay. Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2355–2365, 2016. find
  35. D. Ni. ScraperAPI, 2015. URL https://www.scraperapi.com/. ↗
  36. R. Nogueira and K. Cho. End-to-End Goal-Driven Web Navigation. Advances in Neural Information Processing Systems, 29, 2016. find
  37. R. Nogueira and K. Cho. Task-Oriented Query Reformulation with Reinforcement Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 574–583, 2017. find
  38. P. Pasupat, T.-S. Jiang, E. Z. Liu, K. Guu, and P. Liang. Mapping natural language commands to web elements. In Empirical Methods in Natural Language Processing (EMNLP), 2018. find
  39. P. Pasupat, T.-S. Jiang, E. Z. Liu, K. Guu, and P. Liang. Mapping natural language commands to web elements. In EMNLP, 2018. find
  40. D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017. find
  41. A. Ronacher. Flask API, 2010. URL https://flask.palletsprojects.com/en/2.1.x/. ↗
  42. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603, 2016. find our card
  43. T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of Bits: An Open-Domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017. find
  44. M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740–10749, 2020. find our card
  45. K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. D. Szlam, and J. Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. ArXiv, abs/2203.13224, 2022. find
  46. Y. Su, A. H. Awadallah, M. Khabsa, P. Pantel, M. Gamon, and M. Encarnacion. Building Natural Language Interfaces to Web APIs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 177–186, 2017. find
  47. Y. Su, A. Hassan Awadallah, M. Wang, and R. W. White. Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 855–864, 2018. find
  48. D. Toyama, P. Hamel, A. Gergely, G. Comanici, A. Glaese, Z. Ahmed, T. Jackson, S. Mourad, and D. Precup. AndroidEnv: A Reinforcement Learning Platform for Android. arXiv preprint arXiv:2105.13231, 2021. — [arXiv:2105.13231](https://arxiv.org/abs/2105.13231) find
  49. J. Tuyls, S. Yao, S. Kakade, and K. Narasimhan. Multi-stage episodic control for strategic exploration in text games. arXiv preprint arXiv:2201.01251, 2022. — [arXiv:2201.01251](https://arxiv.org/abs/2201.01251) find
  50. V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, and S. Wermter. Survey on reinforcement learning for language processing. arXiv preprint arXiv:2104.05565, 2021. — [arXiv:2104.05565](https://arxiv.org/abs/2104.05565) find
  51. X. Wang, C. Macdonald, and I. Ounis. Deep reinforced query reformulation for information retrieval. arXiv preprint arXiv:2007.07987, 2020. — [arXiv:2007.07987](https://arxiv.org/abs/2007.07987) find
  52. Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021. find our card
  53. G. Wayne, C.-C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska-Barwinska, J. Rae, P. Mirowski, J. Z. Leibo, A. Santoro, et al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint arXiv:1803.10760, 2018. — [arXiv:1803.10760](https://arxiv.org/abs/1803.10760) find
  54. K. Williams, S. H. Hashemi, and I. Zitouni. Automatic Task Completion Flows from Web APIs. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1009–1012, 2019. find
  55. S. Yao, R. Rao, M. J. Hausknecht, and K. Narasimhan. Keep CALM and Explore: Language Models for Action Generation in Text-based Games. ArXiv, abs/2010.02903, 2020. find
  56. S. Yao, K. Narasimhan, and M. Hausknecht. Reading and acting while blindfolded: The need for semantics in text game agents. arXiv preprint arXiv:2103.13552, 2021. find
  57. X. Yuan, J. Fu, M.-A. Côté, Y. Tay, C. J. Pal, and A. Trischler. Interactive machine comprehension with information seeking agents. In ACL, 2020. find
  58. V. Zhong, A. W. Hanjie, S. Wang, K. Narasimhan, and L. Zettlemoyer. Silg: The multi-domain symbolic interactive language grounding benchmark. Advances in Neural Information Processing Systems, 34:21505–21519, 2021. find
  59. S. Zhuang, H. Ren, L. Shou, J. Pei, M. Gong, G. Zuccon, and D. Jiang. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206.10128, 2022. — [arXiv:2206.10128](https://arxiv.org/abs/2206.10128) find