Large Language Model-Brained GUI Agents: A Survey

Modern digital workflows increasingly rely on graphical user interfaces (GUIs), which are inherently designed for human visual interaction rather than machine automation. Historically, automating tasks across software required rigid, script-based or rule-based methods that easily broke when layouts shifted or workflows changed. Meanwhile, application programming interface (API) automation remains constrained because many applications lack exposed, unified, or accessible APIs. The rapid rise of multimodal Large Language Models (LLMs) and Visual Language Models (VLMs) offers a solution by enabling "LLM-brained" GUI agents that interpret visual interfaces, follow natural language instructions, and interact with software just like human users.

This article provides a comprehensive survey that traces the evolution, architectures, training pipelines, benchmarks, and real-world applications of LLM-powered GUI agents. It systematically evaluates how these agents function across web, mobile, and desktop environments, and outlines the major technical challenges and strategic steps required to achieve robust, autonomous computer interaction.

To establish a holistic overview, the article synthesizes findings from over 500 research studies and industry milestones. It analyzes the end-to-end agent architecture, which comprises environmental perception (via screenshots, accessibility trees, and computer vision), prompt engineering, model inference, action execution (mouse clicks, keystrokes, gestures, and API calls), and short- and long-term memory systems. The analysis highlights key technical enhancements, such as multi-agent collaboration, vision-based UI grounding, self-reflection frameworks, self-evolution mechanisms, and reinforcement learning.

Several key findings emerge from the synthesis. First, multimodal visual perception significantly outperforms traditional text-only and accessibility-tree methods, especially in dynamic environments where underlying code hierarchies are missing or non-standard. Second, incorporating search and simulation techniques—such as Monte Carlo Tree Search, world models, and best-first search—substantially improves multi-step decision-making, boosting task success rates by up to 39% to 50% compared to standard reactive models. Third, hybrid interaction frameworks that dynamically combine UI operations with native system APIs maximize versatility while drastically reducing execution latency. Finally, advanced capabilities like self-reflection and experience-based memory retrieval allow agents to autonomously recover from navigation errors, turning single-trial failures into successful multi-step completions.

These findings indicate a major paradigm shift toward intuitive, conversational computing where non-technical users can automate complex cross-application workflows without specialized code. For organizations, this significantly lowers the barrier to workflow automation and robotic process automation (RPA), but it also introduces latency costs and reliability risks inherent in sequential, multi-step LLM reasoning.

Leaders and developers should pursue a hybrid architecture that prioritizes fast API execution where available while retaining visual GUI automation as a universal fallback. Organizations looking to deploy GUI agents should implement iterative reasoning frameworks, maintain structured memory repositories for continuous learning, and conduct controlled pilot programs before deployment. Further research is recommended to optimize real-time inference latency and develop standardized, high-fidelity safety and verification protocols.

While the article provides high confidence in the architectural foundations and rapid performance trajectory of GUI agents, leaders should exercise caution regarding real-world edge cases. The underlying models still face challenges with latency, execution errors on subtle visual shifts, and potential security risks when interacting with sensitive software environments without human oversight.