EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Developing autonomous, vision-driven embodied agents—such as robotic systems that navigate physical environments and manipulate objects based on human instructions—has become a central focus in artificial intelligence. While large foundation models show strong general reasoning capabilities, systematic evaluation of Multi-modal Large Language Models acting directly as vision-driven physical agents remains limited. The article addresses this gap by presenting EMBODIEDBENCH, a comprehensive evaluation suite designed to assess off-the-shelf multimodal models across hierarchical action tiers and diverse agent competencies.

The main objective of the article is to establish a standardized, multifaceted benchmark that measures how effectively 24 leading proprietary and open-source multimodal models perform both high-level semantic planning and low-level physical control in simulated embodied environments. The benchmark spans 1,128 test instances across four distinct virtual domains: two high-level household planning environments and two low-level environments requiring continuous or discretized navigation and robotic arm manipulation. The evaluation systematically tests models across six core agent capabilities: base execution, commonsense reasoning, complex instruction comprehension, spatial awareness, visual appearance identification, and long-horizon planning.

The investigation yields several critical findings. First, while current multimodal models perform relatively well on high-level semantic planning tasks (such as household organization), they struggle significantly with low-level physical control; even the top-performing model, GPT-4o, achieved an average task success rate of only 28.9% on manipulation tasks. Second, visual input is indispensable for low-level control—removing vision leads to a 40% to 70% drop in navigation success—whereas high-level planning tasks rely almost entirely on text-based semantic cues. Third, long-horizon planning emerges as the single greatest bottleneck across all models, with performance degrading steeply as action sequences lengthen. In addition, proprietary models generally outperform open-source alternatives, though open-source models demonstrate consistent improvements as model scale increases.

These findings have direct operational and strategic implications for organizations investing in robotics and embodied automation. High-level reasoning models cannot be deployed directly into real-world robotic control loops without specialized grounding, as they frequently commit planning errors, miss execution steps, and fail to estimate physical 3D spatial coordinates accurately. Organizations facing safety-critical and cost-sensitive deployment timelines should decouple high-level task planners from low-level execution controllers rather than relying on end-to-end foundation models. Furthermore, ablation experiments show that moderate visual resolutions and visual in-context demonstrations significantly enhance performance, whereas multi-step or multi-view image feeds tend to confuse current models.

To advance toward viable embodied systems, the article recommends prioritizing research in 3D spatial reasoning, visual in-context learning, and hierarchical planning architectures that can reliably manage long-horizon execution. Decision-makers should validate models within tailored simulation pipelines before deploying them into physical hardware. A primary limitation of this study is that all experiments were conducted within virtual simulators rather than physical real-world environments. While this ensures reproducible, safe, and cost-effective benchmarking, stakeholders should exercise caution and conduct physical pilot testing before translating these simulation findings into real-world operational deployments.