AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

Deploying autonomous robots in dynamic, real-world environments requires systems that can not only execute tasks but also recognize and learn from their own mistakes. While modern vision-language models have significantly improved robot perception and planning, they frequently struggle with recognizing failures and typically reduce error detection to binary success-or-failure checks. Without detailed, language-based explanations of why an action failed, robotic systems cannot autonomously diagnose errors, adapt policies, or recover effectively.

The article introduces AHA, an open-source vision-language model designed to detect manipulation failures and provide descriptive, natural-language explanations of why those errors occurred. To train AHA, the authors created FailGen, an automated pipeline that procedurally perturbs successful simulated demonstrations across a taxonomy of seven common robotic failure modes, producing the 49,000-example AHA dataset. AHA was instruction-tuned on this dataset alongside standard visual question-answering data and subsequently evaluated across novel simulated tasks, different physics simulators, real-world robotic setups, and multiple downstream robotic manipulation frameworks.

The evaluation produced several key findings. First, AHA demonstrated superior failure reasoning and generalization, outperforming leading proprietary models such as GPT-4o with five-shot in-context learning by 10.3% overall and outperforming its base model by over 43% across evaluation benchmarks. Second, AHA generalized effectively across embodiments and domains, achieving a 4.9% improvement over GPT-4o in-context learning on real-world UR5 robot failures despite being trained exclusively on simulated data. Third, integrating AHA into downstream frameworks—such as automated reward design for reinforcement learning, task and motion planning, and zero-shot trajectory verification—boosted downstream task success rates by an average of 21.4% compared to GPT-4 baselines, including a 36.7% improvement in planning task success. Finally, testing confirmed that domain-specific tuning did not degrade general visual question-answering capabilities, which remained within 1.5% of the baseline.

These results demonstrate that providing structured, free-form failure reasoning dramatically improves robotic decision-making, policy refinement, and autonomous error recovery without requiring costly real-world failure collection. Organizations developing autonomous robotic systems can lower deployment risks and operational downtime by incorporating specialized failure-reasoning models into planning and verification pipelines.

Based on these findings, teams using foundation models in robotics should adopt automated failure generation frameworks to train diagnostic models and integrate descriptive natural-language feedback into planning loops rather than relying on binary verification. Future work should focus on expanding the failure taxonomy beyond the seven predefined modes to encompass more open-ended real-world failures and distilling policy failures directly from large-scale autonomous rollouts.

Confidence in these findings is supported by consistent cross-simulator, real-world, and multi-metric evaluations. However, stakeholders should note that AHA's reasoning remains primarily aligned with the specific structural failure modes present in its fine-tuning data, meaning performance may vary when encountering unexpected or unmodeled real-world failure types.