Back

WebArena: A Realistic Web Environment for Building Autonomous Agents

Shuyan ZhouFrank F. XuHao ZhuXuhui ZhouRobert LoAbishek SridharXianyi ChengYonatan BiskDaniel FriedUri Alon

article2023ICLR2,020 citations

Introduces WebArena, a realistic multi-domain web environment that benchmarks autonomous language agents on long-horizon tasks, revealing that top models like GPT-4 achieve only a 14.4% success rate compared to 78.2% for humans.

Listen

Autonomous artificial intelligence agents powered by large language models have shown significant potential to execute everyday digital tasks from natural language instructions. However, existing testing environments often oversimplify real-world web complexity or use static, non-interactive data, creating a disconnect between laboratory evaluations and actual performance. To address this issue, the article introduces WebArena, a realistic, standalone, and reproducible web simulation platform alongside a comprehensive benchmark designed to measure how well autonomous agents perform authentic, multi-step web tasks.

The article set out to evaluate the functional correctness of state-of-the-art autonomous agents executing complex, natural language-guided web tasks within an authentic, multi-domain digital environment.

To conduct this evaluation, the researchers built an environment hosting fully operational web platforms across four standard domains: e-commerce, social discussion forums, collaborative software development, and content management, supplemented by digital tools such as maps, calculators, and documentation. The platform was packaged into self-contained Docker containers to ensure reproducible, isolated testing. The researchers constructed a benchmark of 812 long-horizon tasks across three categories: information seeking, site navigation, and content or configuration management. Instead of merely comparing an agent's text output against reference scripts, the evaluation measured functional correctness by programmatically verifying underlying database changes, application states, or accurate answers.

The findings show a substantial gap between human capabilities and automated agents. While human evaluators achieved an end-to-end task completion rate of 78.24%, the top-performing agent based on GPT-4 achieved a success rate of only 14.41%. Chain-of-thought reasoning improved model performance modestly, yielding an increase of roughly two percentage points over direct action prediction. Additionally, prompting instructions heavily affected behavior: instructing agents to detect unachievable tasks caused GPT-4 to mistakenly abandon more than half of feasible tasks due to premature stopping. Detailed error analysis revealed that leading models frequently suffered from observation bias by selecting the first visible answer, repeating redundant actions, and failing to actively explore complex interfaces.

These results indicate that current large language models are not yet reliable enough to operate autonomously across complex, multi-step enterprise workflows without human oversight. For organizations exploring automated web agents, deployment in mission-critical operations carries significant risk of incomplete execution, incorrect state modifications, and unhandled errors. Closing the performance gap will require substantial advancements in hierarchical planning, error recovery, and dynamic memory rather than relying solely on base model scale.

Decision-makers should treat autonomous web agents as experimental tools suitable primarily for assisted, human-in-the-loop pilot projects rather than fully autonomous deployment. Future technical development must focus on improving long-horizon reasoning, self-correction, and robust state tracking. Organizations evaluating agent capabilities should adopt functional, outcome-based verification environments like WebArena to benchmark performance realistically before authorizing automated actions in production systems.

Confidence in these findings is supported by the deterministic and reproducible structure of the containerized environment, though limitations remain. The benchmark focuses primarily on text and accessibility tree inputs rather than complete visual perception, and human baseline scores were derived from a sample of technically proficient annotators rather than a broader demographic. Nevertheless, the evidence strongly confirms that automated web task completion remains an open challenge requiring continued architectural innovation.

Carnegie Mellon Universityautonomous agentscollaborative software developmentcontent managementWebArena environmentautonomous agentsweb agents
arXiv: 2307.13854
Cover for WebArena: A Realistic Web Environment for Building Autonomous Agents

Abstract

With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress.

Topics

autonomous agentscollaborative software developmentcontent managementcontext managemente-commerceknowledge baselanguage modellarge language modelsreasoningsoftware development

Keywords

WebArena environmentautonomous agentsweb agentsinteractive environmentsnatural language commandsweb-based tasksexecution trajectoriesinformation-seeking tasksevaluation environmentnavigation actions

Table of Contents

  • 1 Introduction
  • 2 WebArena: Websites as an Environment for Autonomous Agents
  • 2.1 Controlling Agents through High-level Natural Language
  • 2.2 Website Selection
  • 2.3 Observation Space
  • 2.4 Action Space
  • 3 Benchmark Suite of Web-based Tasks
  • 3.1 Intent Collection
  • 3.2 Evaluation Annotation
  • 4 Baseline Web Agents
  • 5 Results
  • 5.1 Analysis
  • 6 Related Work
  • 7 Conclusion
  • References
  • A Appendix
  • A.1 Website Implementation
  • A.2 Environment Delivery and Reset
  • A.3 User Roles Simulation
  • A.4 Intent Distribution
  • A.5 Human Performance
  • A.6 Experiment Configurations
  • A.7 Prompt for fuzzy_match
  • A.8 The Accuracy of Fuzzy Match Function
  • A.9 The Prompts of the Baseline Web Agents
  • A.10 Additional Error Analysis

Knowls

  1. Knowl 1 — WebArena Environment Formulation and Architecture

    model/method

    The WebArena environment models autonomous web interaction as a tuple E=⟨S,A,O,T⟩\mathcal{E} = \langle \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T} \rangle, where S\mathcal{S} is the state space of the web environment, A\mathcal{A} is the action space, O\mathcal{O} is the observation space, and T:S×A→S\mathcal{T}: \mathcal{S} \times \mathcal{A} \to \mathcal{S} is a deterministic transition function governed by the underlying implementation of the web applications.

    Given a natural language intent ii, at discrete time step tt, the autonomous agent emits an action at∈Aa_t \in \mathcal{A} based on the intent ii, the current observation ot∈Oo_t \in \mathcal{O}, the historical action trace a1t−1a_1^{t-1}, and the observation history o1t−1o_1^{t-1}. This produces a state transition to st+1=T(st,at)s_{t+1} = \mathcal{T}(s_t, a_t) and yields an observation ot+1∈Oo_{t+1} \in \mathcal{O}. Task success is measured by a reward function r(a1T,s1T)∈{0,1}r(a_1^T, s_1^T) \in \{0, 1\} evaluated across the action sequence a1Ta_1^T and intermediate states s1Ts_1^T over TT steps.

    To ensure full reproducibility without relying on live web platforms, WebArena hosts four complete web applications and auxiliary utilities as self-contained Docker containers without external volume dependencies:

    1. E-Commerce: An Adobe Magento deployment populated with approximately 90,000 products across 300 categories with product reviews, options, and descriptions.
    2. Social Forum: A Postmill platform (open-source counterpart of Reddit) containing 95 subreddits, 127,390 posts, and 661,781 users.
    3. Collaborative Software Development: A GitLab instance hosting 300 repositories and over 1,000 user accounts with stratified commit, pull request, and issue histories.
    4. Content Management System (CMS): Adobe Magento's merchant administration portal.
    5. Utility Tools and Knowledge Bases: An OpenStreetMap instance covering the Northeast US, an offline English Wikipedia dump hosted via Kiwix (May 2023 cutoff), official documentation for GitLab and Magento, an on-platform calculator, and a scratchpad.

    To reflect real-world permission dynamics, user profiles are initialized with pre-cached session cookies representing distinct user permissions (e.g., standard customer, open-source maintainer, forum contributor, or merchant administrator).

  2. Knowl 2 — WebArena Action Space

    definition

    The action space A\mathcal{A} in WebArena is a compound action space emulating user keyboard, mouse, and browser navigation operations. Elements can be targeted either by continuous screen coordinates (x,y)(x, y) or via unique integer element IDs generated by traversing the page's Document Object Model (DOM) or accessibility tree. The available actions comprise:

    1. Page Operation Actions:

      • click [id]: Clicks the UI element identified by numerical [id].
      • hover [id]: Moves the cursor over the element [id].
      • type [id] [content] [press_enter_after=0|1]: Types text [content] into element [id], optionally submitting with the Enter key (press_enter_after=1 by default).
      • press [key_comb]: Simulates a keyboard key press or combination (e.g., Ctrl+v, Enter).
      • scroll [direction=down|up]: Scrolls the viewport in the specified vertical direction.
      • noop: Executes no operation.

    2. Tab Management Actions:

      • new_tab: Opens a new blank browser tab.
      • tab_focus [tab_index]: Switches browser focus to the tab at integer [tab_index].
      • close_tab: Closes the currently active browser tab.

    3. URL Navigation Actions:

      • goto [url]: Navigates directly to the specified URL.
      • go_back: Navigates to the previously visited URL in history.
      • go_forward: Re-navigates forward in browsing history.

    4. Task Completion Action:

      • stop [answer]: Signals execution termination. If the task requires an answer string, [answer] contains the textual output; if the agent assesses that the task cannot be fulfilled due to missing information or functionality, it issues stop [N/A].

  3. Knowl 3 — WebArena Observation Space Representations

    definition

    The observation space O\mathcal{O} in WebArena exposes the current web page URL, the list of currently opened browser tabs, and the content of the currently focused tab. WebArena provides three distinct representation modes for rendering web page contents:

    1. Accessibility Tree: A simplified, structured subset of the Document Object Model (DOM) displaying only semantically functional and visible elements. Each element is annotated with its accessible role (e.g., button, link, textbox), textual label or static text, interactive properties (e.g., focused: True, required: False, focusable: True), and a unique prepended integer element ID (e.g., [1582] button 'Add to Cart'). This converts element targeting into an nn-way discrete classification problem while maintaining compact context size.
    2. HTML DOM Tree: The raw Document Object Model tree structure of the web page including hierarchical HTML tags, classes, and attributes.
    3. Screenshot Image: A pixel-level RGB array capturing the visual rendering of the active browser viewport.

    For all observation modes, WebArena supports bounding the observation to the current viewport area to accommodate models with context length or resolution limitations.

  4. Knowl 4 — WebArena Benchmark Suite and Intent Taxonomy

    definition

    The WebArena benchmark consists of 812 grounded natural language web tasks generated from 241 parameterized intent templates (averaging 3.3 instantiations per template). Tasks represent multi-step, realistic web interactions requiring planning across three primary task categories and cross-site workflows:

    1. Information Seeking (Expects Textual Output): Tasks requiring multi-page navigation or inspection of user-specific state to extract factual values (e.g., locating past purchase dates or comparing driving and walking durations across coordinates).
    2. Site Navigation: Tasks requiring interactive traversal via search interfaces, links, and menus to arrive at a target web resource or specific subpage.
    3. Content and Configuration Operation: Tasks requiring modifications to the web environment's underlying databases or state, including updating user profile settings, managing software repositories and issues on GitLab, posting content or comments on Reddit, and managing product listings on CMS.

    Cross-Site Tasks: 5.9% of benchmark tasks necessitate navigating between multiple distinct web applications (e.g., finding museum listings on Wikipedia, computing travel routes on OpenStreetMap, and committing the itinerary to a GitLab repository).

    Unachievable Tasks: The benchmark explicitly incorporates impossible tasks resulting from absent functional support, insufficient data, or permission boundaries. Agents are evaluated on their ability to recognize infeasibility and output "N/A" rather than hallucinating answers or performing invalid actions.

  5. Knowl 5 — Programmatic Functional Correctness Evaluation Framework

    model/method

    WebArena evaluates task success by verifying functional outcomes on the environment state and extracted answers rather than computing lexical overlap against reference action sequences.

    1. Information-Seeking Evaluation (rinfo(a^,a∗)r_{\text{info}}(\hat{a}, a^*)): Compares the agent's predicted answer a^\hat{a} to reference answer a∗a^* using one of three metrics:

      • exact_match: Assigns 1 if a^==a∗\hat{a} == a^* (case-normalized), 0 otherwise.
      • must_include: Assigns 1 if all requisite key substrings in a∗a^* appear in a^\hat{a}.
      • fuzzy_match: Uses an LLM (gpt-4-0613) instructed via a grading prompt to evaluate whether a^\hat{a} is semantically equivalent to a∗a^* (e.g., normalizing date and time formatting such as "2h58min" vs "2 hours 58 minutes").

    2. State and Navigation Evaluation (rprog(s)r_{\text{prog}}(s)): Programmatically verifies environment state modifications and page locations along execution trajectory ss using:

      • Locators: Functions extracting critical target data via direct database queries, application backend APIs, or in-browser JavaScript DOM execution (e.g., document.querySelector('.submission__inner').outerText).
      • Assertions: Validates that retrieved URLs or database records satisfy expected constraints using exact_match and must_include checks on the extracted state properties.

  6. Knowl 6 — End-to-End Task Success Rates on WebArena

    data/table

    Baseline web agents were evaluated across the 812 WebArena tasks using few-shot (2-shot) in-context learning with accessibility tree observations and element IDs. Evaluations compare direct action emission against Chain-of-Thought (CoT) reasoning, with and without an Unachievable (UA) hint in the prompt instructing the model to stop if a task cannot be performed.

    Model CoT UA Hint SR (%) SRAC_{\text{AC}} (%) SRUA_{\text{UA}} (%)
    TEXT-BISON-001 ✓ ✓ 5.05 4.00 27.78
    GPT-3.5 ✓ 6.41 4.90 38.89
    GPT-3.5 ✓ ✓ 8.75 6.44 58.33
    GPT-4 ✓ ✓ 11.70 8.63 77.78
    GPT-3.5 5.10 4.90 8.33
    GPT-3.5 ✓ 6.16 6.06 8.33
    GPT-4 ✓ 14.41 13.02 44.44
    Human - ✓ 78.24 77.30 100.00
    • SR\text{SR}: Overall task success rate across all 812 benchmark tasks.
    • SRAC\text{SR}_{\text{AC}}: Success rate on achievable tasks.
    • SRUA\text{SR}_{\text{UA}}: Success rate on unachievable tasks.

    Human performance was evaluated on a sample of 170 tasks (one per template) by five computer science graduate students, achieving an overall success rate of 78.24% (74.68% on information seeking, 81.32% on other categories) with an average task completion time of 110 seconds.

  7. Knowl 7 — Ablation of Unachievable Task Prompting in Autonomous Web Agents

    empirical result

    Providing an explicit Unachievable Hint (UA hint) in the agent system prompt—instructing the model to output stop [N/A] if it considers the task impossible—creates a trade-off between identifying impossible tasks and prematurely abandoning achievable tasks.

    1. False Negative Halting: When given the UA hint, GPT-4 with Chain-of-Thought falsely classified 54.9% of achievable tasks as impossible, lowering achievable task success rate (SRAC\text{SR}_{\text{AC}}) to 8.63% and overall success rate (SR\text{SR}) to 11.70%.
    2. Performance Gain upon Removal: Removing the UA hint improved GPT-4's achievable success rate to 13.02% and its overall success rate to 14.41%.
    3. Autonomous Infeasibility Reasoning: Even without the UA hint, GPT-4 maintained a 44.44% success rate on unachievable tasks (SRUA\text{SR}_{\text{UA}}) by generating zero-shot non-achievability rationales before issuing stop [N/A].
    4. Model Disparity: In contrast to GPT-4, GPT-3.5 without the UA hint rarely recognized unachievable tasks, dropping from SRUA=58.33%\text{SR}_{\text{UA}} = 58.33\% to 8.33%8.33\%, and instead generated hallucinated answers, executed invalid actions repeatedly, or timed out.

  8. Knowl 8 — Baseline Agent Execution Configuration and Termination Protocol

    experimental setup

    The baseline autonomous agents were evaluated under the following operational configuration:

    • Base Models: gpt-3.5-turbo-16k-0613, gpt-4-0613, and text-bison-001.
    • Sampling Hyperparameters: Temperature T=1.0T = 1.0 and top-p=0.9p = 0.9 to promote exploratory action selection. (A deterministic baseline for gpt-3.5-turbo-16k-0613 at T=0.0T = 0.0 achieved an SR\text{SR} of 6.28% with CoT and no UA hint).
    • Step Budget: Maximum limit of 30 environment state transitions per task.
    • Early Termination Rules:
      1. Execution is aborted if the agent emits the identical action more than 3 times consecutively on the same observation.
      2. Execution is aborted if the agent generates 3 consecutive syntactically or contextually invalid actions.
    • Retries: For text-bison-001, up to 10 generation retries per step were permitted until a valid action syntax was produced.
    • Prompt Context: Prompts include environment rules, available action grammar, URL navigation instructions, credentials page pointers, and 2 full demonstration trajectories.

  9. Knowl 9 — Observation Biases and Error Patterns in LLM Web Agents

    limitation

    Qualitative and quantitative trajectory analyses of GPT-4 and GPT-3.5 web agents revealed three systematic failure mechanisms in complex web environments:

    1. Observation Bias (Superficial Latching): Agents frequently extract the first visually or textually related data encountered on a landing page rather than navigating to dedicated report generators or filtering interfaces. For example, when tasked with finding the top-selling product of 2022 on the CMS, the agent grabbed current bestsellers displayed on the dashboard rather than configuring the historical date filter in the sales report interface.
    2. Failure to Ground Granular Observation Updates: Agents frequently disregard updated input field states in the accessibility tree and ignore historical action records. For instance, after typing a search string into an input box, the model repeatedly re-issued type commands despite the accessibility tree indicating the textbox already contained the query text, triggering loop termination limits.
    3. Sub-Task Generalization Failure Across Templates: Across 61 intent templates where GPT models succeeded on at least one instance, GPT-4 achieved 100% success on only 4 templates, and GPT-3.5 achieved 100% on none. Variations in target complexity within the same template (e.g., forking a single repository vs. forking all repositories belonging to an organization) led to sharp degradation due to extended horizon lengths and error accumulation.

Coverage note — None was omitted; all contributed aspects of WebArena—environment design, benchmark creation, evaluation methodology, empirical results, and error analyses—are represented.

References

  1. 1.Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3674–3683. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00387. URL http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html.
  2. 2.Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
  3. 3.Yonatan Bisk, Jan Buys, Karl Pichotta, and Yejin Choi. Benchmarking hierarchical script knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4077–4085, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1412. URL https://aclanthology.org/N19-1412.
  4. 4.S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 82–90, Suntec, Singapore, 2009. Association for Computational Linguistics. URL https://aclanthology.org/P09-1010.
  5. 5.Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. URL https://arxiv.org/abs/1606.01540.
  6. 6.Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
  7. 7.Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.
  8. 8.Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=rc8o_j8I8PX.
  9. 9.Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023.
  10. 10.Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interactive environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4089–4098. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00430. URL http://openaccess.thecvf.com/content_cvpr_2018/html/Gordon_IQA_Visual_Question_CVPR_2018_paper.html.
  11. 11.Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023.
  12. 12.Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118–9147. PMLR, 2022. URL https://proceedings.mlr.press/v162/huang22a.html.
  13. 13.Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant. ArXiv preprint, abs/1905.01978, 2019. URL https://arxiv.org/abs/1905.01978.
  14. 14.Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. ArXiv preprint, abs/2303.17491, 2023. URL https://arxiv.org/abs/2303.17491.
  15. 15.Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
  16. 16.Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/569ff987c643b4bedf504efda8f786c2-Abstract.html.
  17. 17.Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026.
  18. 18.Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.
  19. 19.Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893–18912. PMLR, 2023.
  20. 20.Xinze Li, Yixin Cao, Muhao Chen, and Aixin Sun. Take a break in the middle: Investigating subgoals towards hierarchical script generation. ArXiv preprint, abs/2305.10907, 2023. URL https://arxiv.org/abs/2305.10907.
  21. 21.Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198–8210, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.729. URL https://aclanthology.org/2020.acl-main.729.
  22. 22.Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493–9500. IEEE, 2023.
  23. 23.Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=ryTp3f-0-.
  24. 24.Jieyi Long. Large language model guided tree-of-thought. ArXiv preprint, abs/2305.08291, 2023. URL https://arxiv.org/abs/2305.08291.
  25. 25.Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384–1403, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.90.
  26. 26.Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281–300, 2016.
  27. 27.Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  28. 28.OpenAI. Chatgpt: Optimizing language models for dialogue. 2022.
  29. 29.OpenAI. Gpt-4 technical report. arXiv, pp. 2303–08774, 2023.
  30. 30.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
  31. 31.Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8494–8502. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00886. URL http://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html.
  32. 32.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264.
  33. 33.Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL https://aclanthology.org/P18-2124.
  34. 34.Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00245, 2023.
  35. 35.Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3135–3144. PMLR, 2017. URL http://proceedings.mlr.press/v70/shi17a.html.
  36. 36.Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv preprint, abs/2303.11366, 2023. URL https://arxiv.org/abs/2303.11366.
  37. 37.Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10737–10746. IEEE, 2020. doi: 10.1109/CVPR42600.2020.01075. URL https://doi.org/10.1109/CVPR42600.2020.01075.
  38. 38.Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=0IOX0YcCdTn.
  39. 39.Abishek Sridhar, Robert Lo, Frank F Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation. arXiv preprint arXiv:2305.14257, 2023.
  40. 40.Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. ArXiv preprint, abs/2105.13231, 2021. URL https://arxiv.org/abs/2105.13231.
  41. 41.Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291, 2023. URL https://arxiv.org/abs/2305.16291.
  42. 42.Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. ArXiv preprint, abs/2212.10481, 2022. URL https://arxiv.org/abs/2212.10481.
  43. 43.Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  44. 44.Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1022–1032, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.80. URL https://aclanthology.org/2021.naacl-main.80.
  45. 45.Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259.
  46. 46.Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. volume abs/2207.01206, 2022a. URL https://arxiv.org/abs/2207.01206.
  47. 47.Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022b. URL https://arxiv.org/abs/2210.03629.
  48. 48.Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, abs/2305.10601, 2023. URL https://arxiv.org/abs/2305.10601.
  49. 49.Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arxiv 2017. ArXiv preprint, abs/1709.00103, 2017. URL https://arxiv.org/abs/1709.00103.
  50. 50.Shuyan Zhou, Pengcheng Yin, and Graham Neubig. Hierarchical control of situated agents through natural language. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pp. 67–84, Seattle, USA, 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.suki-1.8. URL https://aclanthology.org/2022.suki-1.8.
  51. 51.Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, and Graham Neubig. Show me more details: Discovering hierarchies of procedures from semi-structured web data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2998–3012, Dublin, Ireland, 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.214. URL https://aclanthology.org/2022.acl-long.214.

Citation

MLA
Zhou, S., et al. “WebArena: A Realistic Web Environment for Building Autonomous Agents”. arXiv, 2023, http://arxiv.org/abs/2307.13854v4.
APA
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv. http://arxiv.org/abs/2307.13854v4
Chicago
Zhou, S., F. F. Xu, H. Zhu, et al. 2023. “WebArena: A Realistic Web Environment for Building Autonomous Agents”. arXiv. http://arxiv.org/abs/2307.13854v4.
Harvard
Zhou, S. et al. (2023) “WebArena: A Realistic Web Environment for Building Autonomous Agents”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2307.13854v4.
Vancouver
1. Zhou S, Xu FF, Zhu H, et al (2023) WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv

BibTeX

@article{zhou2023webarena,
  title = {WebArena: A Realistic Web Environment for Building Autonomous Agents},
  author = {Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2307.13854v4},
  eprint = {2307.13854}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors