Back

π0: A Vision-Language-Action Flow Model for General Robot Control

Kevin BlackNoah BrownDanny DriessAdnan EsmailMichael EquiChelsea FinnNiccolo FusaiLachy GroomKarol HausmanBrian Ichter

article2024arXiv2,588 citations

Presents π₀, a vision-language-action model that combines flow matching with pre-trained vision-language representations to enable generalist, highly dexterous control across single-arm, dual-arm, and mobile manipulation platforms.

Listen

The article addresses the challenge of creating flexible, general-purpose robot systems capable of handling diverse physical tasks in real-world settings. Current robot learning struggles with limited data availability, poor generalization across environments, and insufficient robustness to variations or errors, limiting practical deployment despite advances in specialized AI.

The article set out to evaluate whether a generalist robot policy, pre-trained on large-scale diverse data and refined through targeted fine-tuning, could achieve broad capabilities and high dexterity across multiple robot platforms and complex tasks.

The approach involved developing π0, a model built on a pre-trained vision-language model backbone augmented with an action expert using flow matching to generate continuous actions. It was pre-trained on roughly 10,000 hours of robot data spanning seven robot configurations and 68 tasks, combined with open-source datasets, then either used directly or fine-tuned on smaller high-quality datasets for specific applications.

Key findings indicate that the pre-trained model substantially outperformed prior vision-language-action models and diffusion-based baselines on out-of-box tasks such as shirt folding and table bussing, with near-perfect success on easier variants and large margins on harder ones. It demonstrated strong language command following, benefiting from both human and high-level model guidance, and achieved effective fine-tuning on new dexterous tasks with as little as one to five hours of data. On complex multi-stage tasks including laundry folding, box assembly, and egg packing, the full pre-training and fine-tuning recipe enabled reliable performance exceeding 50 percent of maximum scores where other methods failed.

These results suggest that combining Internet-scale semantic knowledge from vision-language models with large-scale robot pre-training and a post-training phase can produce policies that balance broad generalization with precise, fluent execution, potentially reducing the data and engineering effort needed for new robot applications.

The article indicates that next steps should focus on refining data composition and weighting strategies during pre-training, determining the minimum data volumes required for near-perfect reliability on specific tasks, and testing transfer to additional domains such as navigation or locomotion.

Limitations include incomplete reliability on some evaluated tasks, uncertainty about optimal data mixtures, and the absence of evidence for universality across highly distinct robot domains; confidence in the core performance gains is high based on consistent outperformance across multiple experiments, but caution is warranted when extrapolating to untested settings or larger scales.

internetlanguage modelmobile manipulatorπ0Vision-Language ModelFlow Matching
arXiv: 2410.24164Physical-Intelligence/openpi
Cover for π0: A Vision-Language-Action Flow Model for General Robot Control

Abstract

Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence. However, bringing robot learning to the level of generality required for effective real-world systems faces major obstacles in terms of data, generalization, and robustness. In this paper, we discuss how generalist robot policies (i.e., robot foundation models) can address these challenges, and how we can design effective generalist robot policies for complex and highly dexterous tasks. We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We then discuss how this model can be trained on a large and diverse dataset from multiple dexterous robot platforms, including single-arm robots, dual-arm robots, and mobile manipulators. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people and from a high-level VLM policy, and its ability to acquire new skills via fine-tuning. Our results cover a wide variety of tasks, such as laundry folding, table cleaning, and assembling boxes.

Topics

internetlanguage modelmobile manipulatorrobot learningrobot systemsemanticsrobot armsrobot handrobotic handsrobotic systems

Keywords

π0Vision-Language ModelFlow MatchingGeneralist Robot PolicyCross-Embodiment TrainingDexterous ManipulationPre-Training MixtureHigh-Frequency Action ChunkLanguage Command FollowingMulti-Stage Task

Table of Contents

  • I Introduction
  • II Related Work
  • III Overview
  • IV The π0\pi_{0} Model
  • V Data Collection and Training Recipe
  • V-A Pre-training and post-training
  • V-B Language and high-level policies
  • V-C Robot system details
  • VI Experimental Evaluation
  • VI-A Evaluating the base model
  • VI-B Following language commands
  • VI-C Learning new dexterous tasks
  • VI-D Mastering complex multi-stage tasks
  • VII Discussion, Limitations, and Future Work
  • References
  • A
  • A-A Contributions
  • A-B Model Architecture Details
  • A-C Non-VLM Baseline Architecture
  • A-D Inference
  • A-E Evaluation Details

Knowls

  1. Knowl 1 — Dual-Expert Vision-Language-Action Architecture

    model/method

    The π0\pi_0 model is a 3.3-billion-parameter vision-language-action (VLA) foundation model designed for real-time, high-frequency continuous robot control. Its architecture is structured as a two-expert mixture-of-experts (MoE) transformer that combines a large pre-trained vision-language model (VLM) backbone with a smaller, dedicated action expert initialized from scratch.

    1. Vision-Language Backbone (3.0B parameters): Built upon the open-source PaliGemma model, which pairs a 400M-parameter SigLIP vision transformer with a 2.6B-parameter Gemma 2B language model. The language model configuration comprises 18 layers, embedding width wVLM=2048w_{\text{VLM}} = 2048, MLP hidden dimension mlp_dim=16384\text{mlp\_dim} = 16384, 18 attention heads, and 1 key-value head (multi-query attention) with head dimension 256. Observation inputs comprising nn RGB images [It1,…,Itn][I_t^1, \dots, I_t^n] (encoded via the SigLIP vision transformer) and tokenized natural language commands ℓt\ell_t are routed through this backbone.

    2. Action Expert (300M parameters): A transformer expert initialized from scratch that processes robotics-specific tokens: the robot's proprioceptive joint configuration vector qtq_t and noisy continuous action chunks Atτ=[atτ,…,at+H−1τ]A_t^\tau = [a_t^\tau, \dots, a_{t+H-1}^\tau] for an action chunk horizon of H=50H = 50. To enable fast multi-step flow matching inference, the action expert is downscaled to embedding width w=1024w = 1024 and MLP hidden dimension mlp_dim=4096\text{mlp\_dim} = 4096, while retaining 18 layers.

    3. Inter-Expert Interaction: The two experts interact exclusively across their shared self-attention layers at each of the 18 transformer depths. Tokens routed to different experts share the attention mechanism but use expert-specific weights for feed-forward and projection transformations.

  2. Knowl 2 — Conditional Flow Matching for High-Frequency Robot Action Chunking

    model/method

    To represent complex, multi-modal, and continuous high-frequency action distributions, π0\pi_0 models an action chunk At=[at,at+1,…,at+H−1]∈RH×dA_t = [a_t, a_{t+1}, \dots, a_{t+H-1}] \in \mathbb{R}^{H \times d} (with horizon H=50H = 50 and action dimension d≤18d \le 18) conditioned on the multimodal observation prefix ot=[It1,…,Itn,ℓt,qt]o_t = [I_t^1, \dots, I_t^n, \ell_t, q_t] using conditional flow matching.

    During training, a continuous flow matching timestep τ∈[0,1]\tau \in [0, 1] is sampled, and Gaussian noise ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I) is added along a linear probability path: q(Atτ∣At)=N(τAt,(1−τ)I)q(A_t^\tau \mid A_t) = \mathcal{N}(\tau A_t, (1 - \tau)I) Atτ=τAt+(1−τ)ϵA_t^\tau = \tau A_t + (1 - \tau)\epsilon

    The neural network predicts the time-dependent denoising vector field vθ(Atτ,ot)v_\theta(A_t^\tau, o_t) that matches the target marginal vector field u(Atτ∣At)=At−ϵu(A_t^\tau \mid A_t) = A_t - \epsilon. The model is supervised using the mean squared error loss: Lτ(θ)=Ep(At∣ot),q(Atτ∣At)[∥vθ(Atτ,ot)−(At−ϵ)∥2]\mathcal{L}^\tau(\theta) = \mathbb{E}_{p(A_t \mid o_t), q(A_t^\tau \mid A_t)} \left[ \| v_\theta(A_t^\tau, o_t) - (A_t - \epsilon) \|^2 \right]

    Each noisy action vector at′τ∈Rda_{t'}^\tau \in \mathbb{R}^d (for t≤t′<t+Ht \le t' < t+H) is projected to the action expert embedding width w=1024w = 1024 by an MLP that conditions on the flow timestep τ\tau: et′τ=W3⋅swish(W2⋅concat(W1at′τ,ϕ(τ)))e_{t'}^\tau = W_3 \cdot \text{swish}\left( W_2 \cdot \text{concat}(W_1 a_{t'}^\tau, \phi(\tau)) \right) where ϕ(τ)∈Rw\phi(\tau) \in \mathbb{R}^w is a sinusoidal positional encoding function, W1∈Rw×dW_1 \in \mathbb{R}^{w \times d}, W2∈Rw×2wW_2 \in \mathbb{R}^{w \times 2w}, and W3∈Rw×wW_3 \in \mathbb{R}^{w \times w}. The decoded transformer outputs for these HH tokens are mapped to predicted action velocities via a linear projection layer.

  3. Knowl 3 — Blockwise Causal Attention Masking and Prefix Key-Value Caching

    model/method

    The full input token sequence of π0\pi_0 is partitioned into three sequential blocks to preserve pre-trained VLM representations and enable prefix key-value (KV) caching during sampling:

    1. Block 1 (Vision-Language Prefix): Token embeddings from camera images and language instructions [It1,…,Itn,ℓt][I_t^1, \dots, I_t^n, \ell_t].
    2. Block 2 (Proprioceptive State): Projected robot joint configuration token [qt][q_t].
    3. Block 3 (Action Chunk): Projected noisy action tokens [atτ,…,at+H−1τ][a_t^\tau, \dots, a_{t+H-1}^\tau].

    The attention mechanism is governed by two structural rules:

    • Intra-block attention: Tokens within each individual block attend to all other tokens in the same block bidirectionally.
    • Inter-block causal masking: Tokens in earlier blocks cannot attend to tokens in subsequent blocks.

    This masking structure ensures that:

    • Block 1 tokens do not attend to robotics-specific tokens (qtq_t or AtτA_t^\tau), preventing distribution shift relative to the original Internet-scale VLM pre-training.
    • The observation prefix [It1,…,Itn,ℓt,qt][I_t^1, \dots, I_t^n, \ell_t, q_t] (Blocks 1 and 2) is independent of the flow matching timestep τ\tau and noisy action inputs AtτA_t^\tau. Consequently, the attention keys and values for Blocks 1 and 2 are computed once per control step and cached across all iterative integration passes of the flow matching solver.

  4. Knowl 4 — Shifted Beta Timestep Sampling Distribution for Action Flow Matching

    model/method

    Standard generative flow matching for computer vision typically samples timesteps τ∼U(0,1)\tau \sim \mathcal{U}(0, 1) or from logit-normal distributions centered on intermediate noise levels. In robot action modeling, the observation oto_t strongly constrains the action distribution; predicting the conditional mean action E[At∣ot]\mathbb{E}[A_t \mid o_t] under high noise levels (low τ\tau) is informative and difficult, whereas low noise levels (high τ\tau) primarily require learning an identity mapping.

    To allocate network capacity to high-noise regimes, π0\pi_0 samples training timesteps τ\tau from a shifted Beta distribution truncated at cutoff s=0.999s = 0.999: p(τ)=Beta(s−τs;1.5,1),τ∈[0,s]p(\tau) = \text{Beta}\left(\frac{s - \tau}{s}; 1.5, 1\right), \quad \tau \in [0, s]

    Timesteps τ>s\tau > s are not sampled during training. This truncation introduces no approximation error at inference time as long as the numerical integration step size δ≥1−s=0.001\delta \ge 1 - s = 0.001 (allowing up to 1,000 integration steps, while δ=0.1\delta = 0.1 with 10 steps is used in practice).

  5. Knowl 5 — Cross-Embodiment Dataset Aggregation and Action Space Unification

    model/method

    The pre-training mixture for π0\pi_0 aggregates over 10,000 hours of robot demonstrations across 7 distinct physical configurations and 68 broad multi-stage tasks:

    1. Data Composition: 903 million timesteps from proprietary dexterous datasets (106M single-arm steps, 797M dual-arm steps) combined with 9.1% open-source robot data (OXE Magic Soup, BridgeData v2, and DROID).

    2. Embodiment Coverage:

    • Single-arm UR5e (7-DoF action space, 2 cameras)
    • Bimanual UR5e (14-DoF action space, 3 cameras)
    • Franka Panda (8-DoF action space, 2 cameras)
    • Bimanual Trossen ViperX / ALOHA (14-DoF action space, 3 cameras)
    • Bimanual ARX & AgileX (14-DoF action space, 3 cameras)
    • Mobile ALOHA / Trossen & ARX nonholonomic mobile base (16-DoF action space, 3 cameras)
    • Mobile Fibocom holonomic mobile base with dual ARX arms (17-DoF action space, 3 cameras)

    1. Dimensionality Unification: Configuration vectors qtq_t and action vectors ata_t are zero-padded to 18 dimensions, the maximum dimensionality in the mixture (accommodating two 6-DoF arms, two grippers, a mobile base with planar translation and rotation, and a vertically actuated torso). Missing camera slots are masked up to 3 image inputs.

    2. Task-Embodiment Balancing: To prevent high-frequency or long tasks (e.g., laundry folding) from dominating training, each (task, embodiment) combination with nn demonstrations is reweighted proportional to n0.43n^{0.43}.

  6. Knowl 6 — Action Generation and Open-Loop Execution Algorithm

    algorithm

    At deployment, π0\pi_0 samples action chunks via forward Euler numerical integration over the flow matching vector field, executing action chunks open-loop without temporal ensembling.

    Input: RGB images It1,…,ItnI_t^1, \dots, I_t^n, language command ℓt\ell_t, proprioceptive state qtq_t, integration steps K=10K = 10, step size δ=1.0/K=0.1\delta = 1.0 / K = 0.1, execution horizon KexecK_{\text{exec}}
    Output: Executed robot actions
    1. Construct observation prefix ot=[It1,…,Itn,ℓt,qt]o_t = [I_t^1, \dots, I_t^n, \ell_t, q_t]
    2. Encode images It1,…,ItnI_t^1, \dots, I_t^n with SigLIP vision transformer
    3. Compute and cache key-value activations KVprefix\text{KV}_{\text{prefix}} for Blocks 1 and 2 across all 18 transformer layers
    4. Sample initial Gaussian noise chunk At0∼N(0,I)∈RH×dA_t^0 \sim \mathcal{N}(0, I) \in \mathbb{R}^{H \times d}, where H=50H = 50
    5. for k=0,1,…,K−1k = 0, 1, \dots, K - 1 do
    6. τ=k⋅δ\tau = k \cdot \delta
    7. Embed noisy actions AtτA_t^\tau with flow timestep embedding MLP conditioned on ϕ(τ)\phi(\tau)
    8. Evaluate action expert with full attention over AtτA_t^\tau and cached KVprefix\text{KV}_{\text{prefix}} to obtain vθ(Atτ,ot)v_\theta(A_t^\tau, o_t)
    9. Atτ+δ=Atτ+δ⋅vθ(Atτ,ot)A_t^{\tau + \delta} = A_t^\tau + \delta \cdot v_\theta(A_t^\tau, o_t)
    10. end for
    11. Unpad predicted action chunk At=At1=[at,at+1,…,at+H−1]A_t = A_t^1 = [a_t, a_{t+1}, \dots, a_{t+H-1}]
    12. for i=0,1,…,Kexec−1i = 0, 1, \dots, K_{\text{exec}} - 1 do
    13. Execute action at+ia_{t+i} on robot hardware
    14. end for

    Inference Timings and Control Rates (measured on an NVIDIA GeForce RTX 4090 GPU):

    • Vision encoding: 14 ms; Observation forward pass: 32 ms; 10 action expert integration steps: 27 ms; Network latency (off-board mobile robot): 13 ms. Total latency: 73 ms (on-board) and 86 ms (off-board).
    • For 20 Hz robots (UR5e, Franka), inference is invoked every 0.8 s (Kexec=16K_{\text{exec}} = 16 actions).
    • For 50 Hz robots (ALOHA, ARX, Mobile Fibocom), inference is invoked every 0.5 s (Kexec=25K_{\text{exec}} = 25 actions).

  7. Knowl 7 — Architecture of the Non-VLM Baseline Model ($\pi_0$-small)

    model/method

    To isolate the empirical benefits of Internet-scale VLM pre-training from model architecture, an ablation baseline named π0\pi_0-small (470M parameters) is trained without VLM initialization using the following design:

    1. Language Encoder: Uses DistilBERT to encode natural language instructions ℓt\ell_t, replacing the Gemma 2B language backbone.
    2. Vision Encoders: Uses pre-trained R26-S-32 ResNet-ViT hybrid encoders with separate, unshared weights for each camera viewpoint.
    3. Observation Backbone: A transformer encoder trained from scratch on the robot pre-training mixture that processes concatenated image and text embeddings.
    4. Action Expert: Implemented using the Diffusion Transformer (DiT) architecture with AdaLN-Zero layers for conditioning on the flow matching timestep τ\tau, replacing Gemma-style multi-query attention.
    5. Inter-Module Attention: The action expert attends to the outputs of the observation encoder via encoder-decoder cross-attention layers, rather than using a unified decoder-only MoE self-attention mechanism.

  8. Knowl 8 — Out-of-Box Zero-Shot Generalization and Model Comparisons

    empirical result

    The base pre-trained π0\pi_0 model (trained for 700k steps) was evaluated out of the box via direct natural language prompting across five manipulation tasks without downstream fine-tuning (scores normalized from 0.0 to 1.0, averaged over 10 trials):

    1. Shirt Folding (Bimanual ARX): π0\pi_0 achieves an average score of ∼0.95\sim 0.95, whereas OpenVLA (7B) scores 0.0, Octo (93M) scores ∼0.05\sim 0.05, and π0\pi_0-small scores ∼0.50\sim 0.50.
    2. Bussing Easy (UR5e): π0\pi_0 achieves ∼0.95\sim 0.95, outperforming Octo (∼0.20\sim 0.20), OpenVLA (∼0.15\sim 0.15), and π0\pi_0-small (∼0.40\sim 0.40).
    3. Bussing Hard (UR5e): π0\pi_0 attains ∼0.65\sim 0.65, compared to ∼0.10\sim 0.10 for π0\pi_0-small and <0.05<0.05 for OpenVLA and Octo.
    4. Grocery Bagging (UR5e): π0\pi_0 reaches ∼0.80\sim 0.80, while π0\pi_0-small reaches ∼0.45\sim 0.45, and OpenVLA/Octo achieve <0.15<0.15.
    5. Toast from Toaster (Bimanual Trossen): π0\pi_0 achieves ∼0.75\sim 0.75, while OpenVLA, Octo, and π0\pi_0-small achieve 0.0.

    A compute-parity version of π0\pi_0 trained for only 160k steps matches or exceeds all baseline models (OpenVLA trained for 160k steps, Octo trained for 320k steps, and OpenVLA fine-tuned solely on UR5e data). Autoregressive token-discretization VLAs (OpenVLA) struggle primarily because they cannot output continuous high-frequency action chunks.

  9. Knowl 9 — Language Following and Hierarchical Policy Guidance Gains from VLM Pre-Training

    empirical result

    The role of VLM pre-training in natural language grounding was evaluated by comparing π0\pi_0 to the non-VLM baseline π0\pi_0-small on three language-conditioned tasks (Bussing, Table Setting, Grocery Bagging) across three operational conditions:

    1. Flat Prompt (π0\pi_0-flat / π0\pi_0-small-flat): Commanded with an overall task string (e.g., "bag the groceries").
    2. Human Guidance (π0\pi_0-human / π0\pi_0-small-human): Commanded with step-by-step intermediate instructions (~2-second subtask segments) provided by a human expert.
    3. Hierarchical VLM Policy (π0\pi_0-HL): Commanded autonomously with intermediate subtask instructions generated by a high-level VLM planner (SayCan style).

    Empirical Findings (10 trials per task):

    • Instruction Following Accuracy: π0\pi_0 achieves 80% to 90% instruction following accuracy across tasks, compared to 30% to 40% for π0\pi_0-small.
    • Task Progress: Supplying human step-by-step commands increases π0\pi_0 task progress from ∼0.65\sim 0.65 to ∼0.95\sim 0.95 on Bussing, ∼0.45\sim 0.45 to ∼0.80\sim 0.80 on Grocery Bagging, and ∼0.20\sim 0.20 to ∼0.75\sim 0.75 on Table Setting. Autonomous high-level VLM guidance (π0\pi_0-HL) yields comparable gains (∼0.90\sim 0.90 on Bussing, ∼0.70\sim 0.70 on Grocery Bagging, ∼0.60\sim 0.60 on Table Setting).
    • In contrast, π0\pi_0-small does not improve significantly when provided with high-level VLM guidance, demonstrating that large-scale VLM pre-training is necessary for policies to effectively interpret and execute high-level semantic plans.

  10. Knowl 10 — Fine-Tuning Sample Efficiency of Pre-Trained $\pi_0$ on Downstream Tasks

    empirical result

    The sample efficiency of pre-trained π0\pi_0 was evaluated across five downstream tasks spanning different degrees of divergence from pre-training: Stack Bowls (UR5e, easy tier), Towel Folding (Bimanual ARX, easy tier), Tupperware in Microwave (Bimanual ARX, unseen microwave object), Paper Towel Replacement (Bimanual UR5e, hard tier), and Franka Items in Drawer (Franka, hard tier).

    Models were fine-tuned with 1, 5, and 10 hours of task demonstration data and compared against Action Chunking Transformer (ACT) and Diffusion Policy (DP) trained from scratch, OpenVLA and Octo fine-tuned from OXE checkpoints, and π0\pi_0 trained from scratch without pre-training (averaged over 10 trials per task):

    1. Performance across Data Scales: Pre-trained π0\pi_0 consistently outperforms ACT, DP, Octo, and OpenVLA across all data regimes, reaching an average task progress of ∼0.50\sim 0.50 at 1 hour, ∼0.80\sim 0.80 at 5 hours, and ∼0.85\sim 0.85 at 10 hours.
    2. Pre-Training Transfer: Pre-trained π0\pi_0 outperforms π0\pi_0 trained from scratch by up to 2×2\times, with the largest relative improvements occurring in the low-data regime (1 hour) and on tasks structurally related to pre-training skills (e.g., Towel Folding and Stack Bowls).
    3. Baseline Comparison: Baselines trained from scratch (ACT and Diffusion Policy) often outperform pre-trained baselines (OpenVLA and Octo), indicating that prior generalist architectures struggled to transfer pre-training benefits to dexterous fine-grained manipulation.

  11. Knowl 11 — Post-Training Mastery of Long-Horizon Multi-Stage Physical Tasks

    empirical result

    To evaluate autonomous execution of extended multi-stage manipulation behaviors lasting between 5 and 20 minutes, π0\pi_0 was post-trained on curated demonstration datasets (5 to 100+ hours per task) and evaluated across seven challenging tasks:

    1. Laundry Folding (Bimanual ARX): Picking crumpled shirts/shorts from a bin, flattening, folding, and stacking onto a pile (Score: ∼0.80\sim 0.80).
    2. Table Bussing (UR5e): Clearing 12 cluttered, occluded dish and trash items into respective bins (Score: ∼0.88\sim 0.88).
    3. Mobile Laundry Folding (Mobile Fibocom): Coordinated whole-body navigation and bimanual laundry folding (Score: ∼0.92\sim 0.92).
    4. Dryer Unloading (Mobile Fibocom): Approaching a dryer, placing a hamper on a stool, opening the door, extracting 5 garments into the hamper, and closing the door (Score: ∼0.50\sim 0.50).
    5. Cardboard Box Building (Bimanual Trossen): Unfolding a flattened box, folding flaps with dual-arm table-bracing, and interlocking tabs (Score: ∼0.60\sim 0.60).
    6. To-Go Food Packing (Bimanual AgileX): Grasping multiple food items from a plate into a takeout box and latching the lid (Score: ∼0.60\sim 0.60).
    7. Egg Carton Packing (Bimanual Trossen): Grasping 6 slippery eggs from a bowl into specific carton slots and closing the lid (Score: ∼0.82\sim 0.82).

    Across all seven tasks, the pre-trained and fine-tuned π0\pi_0 model achieves >50%>50\% average task progress, substantially exceeding both out-of-box base π0\pi_0 (which achieves <0.20<0.20 progress) and π0\pi_0 trained from scratch on task data (which scores ∼0.20\sim 0.20 on Laundry Folding and ∼0.30\sim 0.30 on Box Building). Pre-training provides an underlying repertoire of error recoveries, while post-training instills execution consistency and fluency.

  12. Knowl 12 — Limitations and Open Challenges in Robot Foundation Models

    limitation

    The development and evaluation of π0\pi_0 reveal several open challenges:

    1. Empirical Dataset Composition: The weighting scheme (n0.43n^{0.43}) and proportional blending of datasets were determined heuristically; a principled framework for predicting positive or negative transfer across disparate robot platforms and tasks remains an open problem.
    2. Performance Predictability: While π0\pi_0 achieves high success rates across diverse tasks, it does not achieve 100% reliability on all multi-stage benchmarks, and predicting the requisite volume of post-training data for deterministic execution remains difficult.
    3. Domain Transfer Scope: The empirical validation is restricted to stationary and mobile manipulation; whether cross-embodiment flow matching foundation models exhibit positive transfer to fundamentally different domains—such as legged locomotion, autonomous driving, and aerial navigation—has not yet been demonstrated.

Coverage note — None was omitted; all primary architectural components, flow matching objectives, dataset compositions, execution algorithms, and quantitative experimental evaluations have been fully covered.

References

  1. 1.Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. 2.Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  3. 3.Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716–23736, 2022.
  4. 4.Jorge Aldaco, Travis Armstrong, Robert Baruch, Jeff Bingham, Sanky Chan, Kenneth Draper, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, et al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.02292, 2024.
  5. 5.Lucas Beyer, Andreas Steiner, Andre Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
  6. 6.Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. RoboAgent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788–4795. IEEE, 2024.
  7. 7.Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
  8. 8.Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909.12200, 2019.
  9. 9.Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023.
  10. 10.OX-Embodiment Collaboration, A Padalkar, A Pooley, A Jain, A Bewley, A Herzog, A Irpan, A Khazatsky, A Rai, A Singh, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864, 1(2), 2023.
  11. 11.Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  12. 12.Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR, 2022.
  13. 13.Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021.
  14. 14.Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
  15. 15.Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, and Nur Muhammad Mahi Shafiullah. Robot utility models: General policies for zero-shot deployment in new environments. arXiv preprint arXiv:2409.05865, 2024.
  16. 16.William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
  17. 17.Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRL), 2024.
  18. 18.Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018.
  19. 19.Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine-grained text-to-image synthesis. arXiv preprint arXiv:2407.07614, 2024.
  20. 20.Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  21. 21.John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žı́dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  22. 22.Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pages 651–673. PMLR, 2018.
  23. 23.Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.
  24. 24.Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.
  25. 25.Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.
  26. 26.Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421–436, 2018.
  27. 27.Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  28. 28.Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
  29. 29.Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695, 2024.
  30. 30.Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
  31. 31.Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401.12202, 2024.
  32. 32.Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577, 2022.
  33. 33.Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. RoboTurk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018.
  34. 34.Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023.
  35. 35.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
  36. 36.William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  37. 37.Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406–3413. IEEE, 2016.
  38. 38.Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.
  39. 39.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  40. 40.Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  41. 41.Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.
  42. 42.V Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  43. 43.Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint arXiv:2311.16098, 2023.
  44. 44.Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
  45. 45.Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
  46. 46.Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  47. 47.Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021.
  48. 48.Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  49. 49.Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
  50. 50.Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024.
  51. 51.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  52. 52.Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. BridgeData v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723–1736. PMLR, 2023.
  53. 53.Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  54. 54.Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  55. 55.Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, and Jian Tang. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. arXiv preprint arXiv:2409.12514, 2024.
  56. 56.Kuan-Ting Yu, Maria Bauza, Nima Fazeli, and Alberto Rodriguez. More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 30–37. IEEE, 2016.
  57. 57.Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
  58. 58.Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. arXiv preprint arXiv:2410.13126, 2024.
  59. 59.Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024.
  60. 60.Minjie Zhu, Yichen Zhu, Jinming Li, Junjie Wen, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, et al. Scaling diffusion policy in transformer to 1 billion parameters for robotic manipulation. arXiv preprint arXiv:2409.14411, 2024.

Citation

MLA
Black, K., et al. “$π_0$: A Vision-Language-Action Flow Model for General Robot Control”. arXiv, 2024, https://doi.org/10.48550/arXiv.2410.24164.
APA
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X., … Zhilinsky, U. (2024). $π_0$: A Vision-Language-Action Flow Model for General Robot Control. arXiv. https://doi.org/10.48550/arXiv.2410.24164
Chicago
Black, K., N. Brown, D. Driess, et al. 2024. “$π_0$: A Vision-Language-Action Flow Model for General Robot Control”. Preprint, ArXiv. https://doi.org/10.48550/arXiv.2410.24164.
Harvard
Black, K. et al. (2024) “$π_0$: A Vision-Language-Action Flow Model for General Robot Control”. arXiv. Available at: https://doi.org/10.48550/arXiv.2410.24164.
Vancouver
1. Black K, Brown N, Driess D, et al (2024) $π_0$: A Vision-Language-Action Flow Model for General Robot Control. https://doi.org/10.48550/arXiv.2410.24164

BibTeX

@misc{https://doi.org/10.48550/arxiv.2410.24164,
  doi = {10.48550/ARXIV.2410.24164},
  url = {https://arxiv.org/abs/2410.24164},
  author = {Black, Kevin and Brown, Noah and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and Fusai, Niccolo and Groom, Lachy and Hausman, Karol and Ichter, Brian and Jakubczak, Szymon and Jones, Tim and Ke, Liyiming and Levine, Sergey and Li-Bell, Adrian and Mothukuri, Mohith and Nair, Suraj and Pertsch, Karl and Shi, Lucy Xiaoyang and Tanner, James and Vuong, Quan and Walling, Anna and Wang, Haohuan and Zhilinsky, Ury},
  keywords = {Machine Learning (cs.LG), Robotics (cs.RO), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {$π_0$: A Vision-Language-Action Flow Model for General Robot Control},
  publisher = {arXiv},
  year = {2024},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors