Elamur Research⌕
☰
home / papers / 2024 / WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Paper · 2024 · Xing Han Lu, Zden\vek Kasner and Siva Reddy

Summary

The paper introduces conversational web navigation, a task in which an agent operates a real browser to complete a user's goal while exchanging messages with that user over many turns, rather than acting on one self-contained command. To support it, the authors release WebLINX, a benchmark of 2337 expert-recorded demonstrations on 155 real websites. It contains over 100K actions and utterances, each linked to a DOM tree, screenshots and video frames. Full pages are too large for LLM contexts, so the authors propose Dense Markup Ranking (DMR), a compact retrieval-style ranker that scores HTML elements against the action history and discards irrelevant ones. They also design evaluation metrics tailored to each action type. Across 19 models from 8 architectures, small finetuned text decoders outperform the strongest zero-shot system (GPT-4V) and larger finetuned multimodal models pretrained on screenshots. However, all finetuned models degrade on unseen websites, categories, regions, or instructors who cannot see the screen. WebLINX matters as the first large-scale testbed for dialogue-driven agents on real websites, and it exposes a clear generalization gap.

Key points

  • WebLINX scale: 2337 demonstrations on 155 real-world websites, with more than 100K recorded interactions and about 43 turns per demonstration on average. The core actions are click, load, say, submit and textinput.
  • Dense Markup Ranking (DMR): a similarity-based element ranker built on text-retrieval architectures. It prunes the DOM into a short input, which is combined with the instruction, the action history and, for multimodal models, a screenshot.
  • Evaluation splits: in-domain TEST_IID plus out-of-domain splits.
    • TEST_WEB covers unseen sites.
    • TEST_CAT covers new subcategories.
    • TEST_GEO covers new geographies.
    • TEST_VIS covers an instructor with no view of the browser.
  • Model comparison: 19 models were tested, 5 of them zero-shot. Finetuned models beat GPT-4V, and Sheared-LLaMA outperforms the larger, screenshot-pretrained Fuyu.
  • Remaining weaknesses: qualitative analysis shows that GPT-4V lacks situational awareness. Finetuned models still fail on simple cases, such as clicking links that do not exist. Code, data and models are released for research at mcgill-nlp.github.io/weblinx.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Xing Han Lu, Zden\vek Kasner and Siva Reddy. 2024. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. arXiv:2402.05930.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05930 · arXiv 2402.05930

@misc{lu2024weblinx,
  title = {{WebLINX: Real-World Website Navigation with Multi-Turn Dialogue}},
  author = {Xing Han Lu, Zdenvek Kasner and Siva Reddy},
  year = {2024},
  doi = {10.48550/arxiv.2402.05930},
  eprint = {2402.05930},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05930}
}

References

104 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. Adept. Adept ACT-1 – ”A machine learning model that can interact with everything on your computer.”. https://www.adept.ai/blog/act-1, 2023. Accessed: 2023/08/31. ↗
  2. Bunian, S., Li, K., Jemmali, C., Harteveld, C., Fu, Y., and El-Nasr, M. S. VINS: Visual Search for Mobile User Interface Design. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021, pp. 423:1–423:14, 2021. URL https://doi.org/10.1145/3411… — [doi:10.1145/3411764.3445762](https://doi.org/10.1145/3411764.3445762) ↗
  3. Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., and Reddy, S. TopiOCQA: Open-domain Conversational Question Answering with Topic Switching. Trans. Assoc. Comput. Linguistics, 10:468–483, 2022. URL https://doi.org/10.1162/tacl a 00471. ↗
  4. Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., and Plummer, B. A. A Dataset for Interactive Vision-language Navigation with Unknown Command Feasibility. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, volume 13668 o… — [doi:10.1007/978-3-031-20074-8](https://doi.org/10.1007/978-3-031-20074-8) find
  5. Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., and Zettlemoyer, L. HTLM: Hyper-text Pre-training and Prompting of Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL https://openreview.net/f… ↗
  6. Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… — Flamingo: a Visual Language Model for Few-Shot Learning
  7. Carroll, J. M. and Rosson, M. B. Usability Engineering. In Topi, H. and Tucker, A. (eds.), Computing Handbook, Third Edition: Information Systems and Information Technology, pp. 32: 1–22. CRC Press, 2014. find
  8. Chang, C., Kayed, M., Girgis, M. R., and Shaalan, K. F. A Survey of Web Information Extraction Systems. IEEE Trans. Knowl. Data Eng., 18(10):1411–1428, 2006. URL https://doi.org/10.1109/TKDE.2006.152. — [doi:10.1109/tkde.2006.152](https://doi.org/10.1109/tkde.2006.152) ↗
  9. Chen, H., Liu, X., Yin, D., and Tang, J. A Survey on Dialogue Systems: Recent Advances and New Frontiers. SIGKDD Explor., 19(2):25–35, 2017. URL https://doi.org/10.1145/3166054.3166058. ↗
  10. Assouel, R., Marty, T., Caccia, M., Laradji, I., Drouin, A., Rajeswar, S., Palacios, H., Cappart, Q., Vazquez, D., Chapados, N., Gasse, M., and Lacoste, A. The unsolved challenges of LLMs in open-ended web tasks: A case study. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL… find
  11. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… — Evaluating Large Language Models Trained on Code
  12. Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S. Fuyu-8B: A Multimodal Architecture for AI Agents, October 2023. URL https://www.adept.ai/blog/fuyu-8b/. ↗
  13. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — Language Models are Few-Shot Learners
  14. Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep Reinforcement Learning from Human Preferences. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing System… — [arXiv:2201.10608](https://arxiv.org/abs/2201.10608) find
  15. Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299–4307, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ d5e2c0adad503c91f91df240d0cd4e49-Abstract.html. ↗
  16. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2Web: Towards a Generalist Agent for the Web. CoRR, abs/2306.06070, 2023. URL https://doi.org/10.48550/arXiv.2306.06070. ↗
  17. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J.,… find
  18. Dettmers, T. and Zettlemoyer, L. The case for 4-bit precision: k-bit Inference Scaling Laws. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 7750– 7774, 2023. URL https://proceedings.mlr.p… ↗
  19. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J.,… find
  20. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Represent…
  21. Cormer, M., Mann, R., Moffatt, K., and Cohen, R. Towards an Improved Vision-based Web Page Segmentation Algorithm. In 2017 14th Conference on Computer and Robot Vision (CRV), pp. 345–352, 2017. find
  22. Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., Chapados, N., and Lacoste, A. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. find
  23. Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. CoRR, abs/2307.08691, 2023. URL https://doi.org/10.48550/arXiv.2307.08691. ↗
  24. Furuta, H., Nachum, O., Lee, K., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal Web Navigation with Instruction-finetuned Foundation Models. CoRR, abs/2305.11854, 2023. URL https://doi.org/10.48550/arXiv.2305.11854. — [doi:10.48550/arxiv.2305.11854](https://doi.org/10.48550/arxiv.2305.11854) ↗
  25. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and Memory-efficient Exact Attention with IO-awareness. In NeurIPS, 2022. URL http://papers.nips.cc/paper files/paper/2022/ hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html. ↗
  26. Google. The bfloat16 numerical format — Cloud TPU, December 2023. URL https://cloud.google.com/tpu/docs/ bfloat16. ↗
  27. Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Int. J. Comput. Vis., 127(4):398–414, 2019. URL https://doi.org/10.1007/s11263-018-1116-0. — [doi:10.1007/s11263-018-1116-0](https://doi.org/10.1007/s11263-018-1116-0) ↗
  28. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A. W., Tucker, P. A., Yang, K., and Ng, A. Y. Large Scale Distributed Deep Networks. In Bartlett, P. L., Pereira, F. C. N., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neura… find
  29. Gur, I. and Yan, X. Learning conversational web interfaces. 2019. URL https://api.semanticscholar.org/CorpusID: 211481529. ↗
  30. Gur, I., Jaques, N., Miao, Y., Choi, J., Tiwari, M., Lee, H., and Faust, A. Environment Generation for Zero-shot Compositional Reinforcement Learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14… find
  31. Deng, X., Shiralkar, P., Lockard, C., Huang, B., and Sun, H. DOM-LM: Learning Generalizable Representations for URL https://proceedings.neurips.cc/paper/2021/hash/ 218344619d8fb95d504ccfa11804073f-Abstract.html. Kann, K., Ebrahimi, A., Koh, J. J., Dudy, S., and Roncone, A. Open-domain Dialogue Gener… ↗
  32. Gur, I., Furuta, H., Huang, A. V., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, 2024. find
  33. Karpukhin, V., Oguz, B., Min, S., Lewis, P. S. H., Wu, L., Edunov, S., Chen, D., and Yih, W. Dense Passage Retrieval for Open-domain Question Answering. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP…
  34. He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. WebVoyager: Building an End-to-end Web Agent with Large Multimodal Models, 2024.
  35. He, P., Liu, X., Gao, J., and Chen, W. Deberta: decoding-enhanced Bert with Disentangled Attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL https://openreview.net/forum?id=XPZIaotutsD. ↗
  36. Kembhavi, A., Salvato, M., Kolve, E., Seo, M. J., Hajishirzi, H., and Farhadi, A. A Diagram is Worth a Dozen Images. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I…
  37. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL https://openreview.net/forum?id=d7KBj… ↗
  38. Kiesel, J., Kneist, F., Meyer, L., Komlossy, K., Stein, B., and Potthast, M. Web Page Segmentation Revisited: Evaluation Framework and Dataset. In CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pp. 3047–3054, 2020… — [doi:10.1145/3340531.3412782](https://doi.org/10.1145/3340531.3412782) find
  39. Hudeček, V. and Dušek, O. Are LLMs All You Need for Task-oriented Dialogue? CoRR, abs/2304.06556, 2023. URL https://doi.org/10.48550/arXiv.2304.06556. — [doi:10.48550/arxiv.2304.06556](https://doi.org/10.48550/arxiv.2304.06556) ↗
  40. Humphreys, P. C., Raposo, D., Pohlen, T., Thornton, G., Chhaparia, R., Muldal, A., Abramson, J., Georgiev, P., Santoro, A., and Lillicrap, T. P. A data-driven approach for learning to control computers. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Interna… find
  41. Kim, G., Baldi, P., and McAleer, S. Language Models can Solve Computer Tasks. CoRR, abs/2303.17491, 2023. URL https://doi.org/10.48550/arXiv.2303.17491. ↗
  42. Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. ↗
  43. HyperWrite. HyperWrite AI Personal Assistant – ”The first publicly available AI agent that can operate a browser like a human.”. https://www.hyperwriteai.com/personal-assistant, 2023. Accessed: 2023/08/31. ↗
  44. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks, 2024.
  45. Jaccard, P. The distribution of the flora in the alpine zone. 1. New phytologist, 11(2):37–50, 1912. find
  46. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on… find
  47. Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https://openreview.net/ forum?id=Bkg6RiCqY7. ↗
  48. Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., Cord, M., and Sanh, V. OBELICS: An Open Web-scale Filtered Dataset of Interleaved Image-text Documents, 2023.
  49. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3195–3204, 2019. URL http://openaccess.thecvf.com/… ↗
  50. Lee, K., Joshi, M., Turc, I. R., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M., and Toutanova, K. Pix2Struct: Screenshot Parsing as Pre-training for Visual Language Understanding. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii… find
  51. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. In Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 2006–…
  52. Li, J., Xu, Y., Cui, L., and Wei, F. MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 6078–6… find
  53. Multi-On. Multi-on – ”The world’s first Personal AI Agent & Life Copilot”. https://multion.ai, 2023. Accessed: 2023/08/31. ↗
  54. Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. Mapping Natural Language Instructions to Mobile UI Action Sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8198–8210, 2020. URL https://doi.org/10.18653…
  55. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. WebGPT: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332…
  56. Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards General Text Embeddings with Multi-stage Contrastive Learning. CoRR, abs/2308.03281, 2023a. URL https://doi.org/10.48550/arXiv.2308.03281. ↗
  57. Nogueira, R. F. and Cho, K. End-to-end Goal-driven Web Navigation. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1903–1911, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/… ↗
  58. Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards General Text Embeddings with Multi-stage Contrastive Learning. CoRR, abs/2308.03281, 2023b. URL https://doi.org/10.48550/arXiv.2308.03281. ↗
  59. OpenAI. Introducing Triton: Open-source GPU programming for neural networks, July 2021. URL https://openai.com/research/triton. ↗
  60. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Reinforcement Learning on Web Interfaces using Workflow-guided Exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018. URL https://… find
  61. OpenAI. Introducing ChatGPT, November 2022. URL https://openai.com/blog/chatgpt. ↗
  62. OpenAI. GPT-4V(ision) System Card. Technical Report, 2023a. URL https://cdn.openai.com/papers/GPTV System Card.pdf. ↗
  63. Liu, W., Meng, X., and Meng, W. ViDE: A Vision-based Approach for Deep Web Data Extraction. IEEE Trans. Knowl. Data Eng., 22(3):447–460, 2010. URL https://doi.org/10.1109/TKDE.2009.109. — [doi:10.1109/tkde.2009.109](https://doi.org/10.1109/tkde.2009.109) ↗
  64. OpenAI. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023b. URL https://doi.org/10.48550/arXiv.2303.08774. ↗
  65. OpenAI. New models and developer products announced at DevDay, November 2023c. URL https://openai.com/blog/new-models-and-developer-products-announced-at-devday. ↗
  66. Reimers, N. and Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992, Hong Kong, C…
  67. OpenAI. ChatGPT Plugins. https://openai.com/blog/ chatgpt-plugins, 2023d. Accessed: 2023/09/03. ↗
  68. Pasupat, P., Jiang, T., Liu, E. Z., Guu, K., and Liang, P. Mapping natural language commands to web elements. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - Nov… find
  69. Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces. CoRR, abs/2306.00245, 2023. URL https://doi.org/10.48550/arXiv.2306.00245. — [doi:10.48550/arxiv.2306.00245](https://doi.org/10.48550/arxiv.2306.00245) ↗
  70. Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of Bits: An Open-domain Platform for Web-based Agents. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Researc… find
  71. Peng, A., Wu, M., Kilpatrick, L., and Heidel, S. GPT-3.5 Turbo fine-tuning and API updates, August 2023. URL https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates. ↗
  72. Pinsky, Y. Bard can now connect to your google apps and services, Sep 2023. URL https://blog.google/products/ bard/google-bard-new-features-update-sept-2023/. ↗
  73. Shu, R., Mansimov, E., Alkhouli, T., Pappas, N., Romeo, S., Gupta, A., Mansour, S., Zhang, Y., and Roth, D. Dialog2API: Task-oriented Dialogue with API Description and Example Programs. CoRR, abs/2212.09946, 2022. URL https://doi.org/10.48550/arXiv.2212.09946. ↗
  74. Popovic, M. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT at EMNLP 2015, 17-18 September 2015, Lisbon, Portugal, pp. 392–395, 2015. URL https://doi.org/10.18653/v1/w15-3049. ↗
  75. Sun, L., Chen, X., Chen, L., Dai, T., Zhu, Z., and Yu, K. META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6699–6712, 202… find
  76. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. CoRR, abs/2305.18290, 2023. URL https://doi.org/10.48550/ arXiv.2305.18290.
  77. Together. Redpajama: an open dataset for training large language models, october 2023. URL https://github.com/ togethercomputer/RedPajama-Data. ↗
  78. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL http://jmlr.org/papers/v21/20-074.html. ↗
  79. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302.13971, 2023a. URL https://doi.org/10.48550/arXiv.230… ↗
  80. Rastogi, A., Zang, X., Sunkara, S., Gupta, R., and Khaitan, P. Towards Scalable Multi-domain Conversational Agents: The Schema-guided Dialogue Dataset. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con…
  81. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — [doi:10.48550/arxiv.2307.10088](https://doi.org/10.48550/arxiv.2307.10088) find
  82. Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open Foundation and… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  83. Wu, J., Wang, S., Shen, S., Peng, Y., Nichols, J., and Bigham, J. P. WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 286:1–286:14, 2023. URL… — [doi:10.1145/3544548.3581158](https://doi.org/10.1145/3544548.3581158) find
  84. Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. CoRR, abs/2310.06694, 2023. URL https://doi.org/10.48550/arXiv.2310.06694. ↗
  85. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, p…
  86. Xiao, S., Liu, Z., Zhang, P., and Muennighof, N. C-Pack: Packaged Resources To Advance General Chinese Embedding. CoRR, abs/2309.07597, 2023a. URL https://doi.org/10.48550/arXiv.2309.07597. ↗
  87. Xiao, S., Liu, Z., Zhang, P., and Muennighof, N. C-Pack: Packaged Resources To Advance General Chinese Embedding. CoRR, abs/2309.07597, 2023b. URL https://doi.org/10.48550/arXiv.2309.07597. ↗
  88. Wang, Q., Fang, Y., Ravula, A., Feng, F., Quan, X., and Liu, D. WebFormer: The Web-page Transformer for Structure Information Extraction. In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, pp. 3124–3133, 2022. URL https://doi.org/10.1145/3485447.3512032. — [doi:10.1145/3485447.3512032](https://doi.org/10.1145/3485447.3512032) ↗
  89. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
  90. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. MiniLM: Deep Self-attention Distillation for Task-agnostic Compression of Pre-trained Transformers. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual C… find
  91. Xu, K., Kordi, Y., Sanders, K., Wang, Y., Byerly, A., Zhang, J., Durme, B. V., and Khashabi, D. Tur[k]ingbench: A challenge benchmark for web agents, 2024. find
  92. Xu, N., Masling, S., Du, M., Campagna, G., Heck, L., Landay, J. A., and Lam, M. Grounding Open-domain Instructions to Automate Web Support Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-H… find
  93. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned Language Models are Zero-shot Learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL https://openreview.net/forum?… ↗
  94. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web-Shop: Towards Scalable Real-world Web Interaction with Grounded Language Agents. In NeurIPS, 2022. URL https://arxiv.org/abs/2207.01206. ↗
  95. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. CoRR, abs/1910.03771, 2019. URL http://arxiv.org/abs/1910.03771. ↗
  96. Zhang, Y., Sun, S., Galley, M., Chen, Y., Brockett, C., Gao, X., Gao, J., Liu, J., and Dolan, B. DIALOGPT : Large-scale Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, A… — DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
  97. Wu, J., Zhang, X., Nichols, J., and Bigham, J. P. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. In UIST ’21: The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA, October 10-14, 2021, pp. 470–483, 2021. URL https://doi.org/10.1145/3… — [doi:10.1145/3472749.3474763](https://doi.org/10.1145/3472749.3474763) find
  98. Zhang, Z., Takanobu, R., Zhu, Q., Huang, M., and Zhu, X. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10): 2011–2027, 2020b. find
  99. Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proc. VLDB Endow., 16(12)… find
  100. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. GPT-4V(ision) is a Generalist Web Agent, if Grounded. CoRR, abs/2401.01614, 2024. URL https://doi.org/10.48550/ arXiv.2401.01614.
  101. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., and Neubig, G. WebArena: A Realistic Web Environment for Building Autonomous Agents. CoRR, abs/2307.13854, 2023. URL https://doi.org/10.48550/arXiv.2307.13854. ↗
  102. Zhou, Y., Sheng, Y., Vo, N., Edmonds, N., and Tata, S. Simplified DOM Trees for Transferable Attribute Extraction from the Web. CoRR, abs/2101.02415, 2021. URL https://arxiv.org/abs/2101.02415. — [arXiv:2101.02415](https://arxiv.org/abs/2101.02415) ↗
  103. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models. CoRR, abs/2304.10592, 2023. URL https://doi.org/10.48550/ arXiv.2304.10592.
  104. Contents \#\# Introduction \#\# Related Work 2.1 Web Navigation Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Website Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Conversational Interfaces .… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2024-weblinx-real-world-website-navigation/