Back

RT-1: Robotics Transformer for Real-World Control at Scale

Anthony BrohanNoah BrownJustice CarbajalYevgen ChebotarJoseph DabisChelsea FinnKeerthana GopalakrishnanKarol HausmanAlex HerzogJasmine Hsu

article2023RSS2,822 citations

Presents RT-1, a transformer architecture trained on extensive multi-task real-world data to scale robotic control and generalize language-conditioned manipulation skills across novel objects and environments.

Listen

The article addresses the challenge of training robots to handle many tasks in varied real-world settings. Current robotic learning often relies on narrow, task-specific data sets, which limits flexibility and requires expensive new collections for each new goal. This matters because real-world robotic data is hard to gather at scale, unlike the large general data sets that have transformed vision and language models.

The article set out to evaluate whether a single high-capacity model trained on broad robotic demonstrations could achieve strong performance on known tasks while generalizing to new instructions, objects, and environments without further task-specific training.

The work collected roughly 130,000 successful demonstrations across more than 700 instructions using a fleet of 13 mobile manipulators over 17 months. Researchers trained and compared several model architectures on this data, then tested them in over 3,000 real-world trials that measured success on familiar tasks, novel combinations, added distractors, changed backgrounds, and long sequences of actions executed through an existing planning system.

RT-1 reached 97 percent success on training instructions, 25 percent above the next best baseline. It completed 76 percent of unseen instructions, 24 percent above the next best result. It handled added distractors at 83 percent success, 36 percent above the next baseline, and new backgrounds at 59 percent, 18 percent above the next result. Adding simulation or data from a different robot improved generalization on new objects and scenes with little loss on original tasks. The model also supported sequences of up to 50 steps in realistic kitchens.

These outcomes show that a suitably designed transformer can absorb diverse robotic experience and deliver usable generalization, lowering the cost of adapting robots to new settings. Performance gains were largest when data covered many tasks rather than simply more examples of fewer tasks.

Further progress requires larger and more varied environments to strengthen background robustness, faster non-expert methods to collect additional skills, and extensions that handle entirely novel motions. The main limitations are that the approach inherits the ceiling of its human demonstrations, cannot yet invent new motion patterns, and has been tested only on relatively simple manipulation behaviors.

The results rest on extensive real-robot trials and controlled ablations, giving high confidence in the reported gains within the tested kitchens and object sets.

Everyday RobotsGooglecomputer visionencoder-decoderhigh capacityRT‑1Robotics Transformerlarge‑scale dataset
arXiv: 2212.06817
Cover for RT-1: Robotics Transformer for Real-World Control at Scale

Abstract

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at this http URL

Topics

computer visionencoder-decoderhigh capacityhttpnatural language processingreal robotrobotic armsspeech recognitionrobotic systems

Keywords

RT‑1Robotics Transformerlarge‑scale datasetFiLM EfficientNet tokenizationTokenLearner tokenizationTransformer architectureimitation learningzero‑shot generalizationabsorb heterogeneous datareal‑time control

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Preliminaries
  • 4 System Overview
  • 5 RT-1: Robotics Transformer
  • 5.1 Model
  • 5.2 Data
  • 6 Experiments
  • 6.1 Experimental Setup
  • 6.2 Can RT-1 learn to perform a large number of instructions, and to generalize to new tasks, objects and environments?
  • 6.3 Can we push the resulting model further by incorporating heterogeneous data sources such as simulation or data from different robots?
  • 6.4 How do various methods generalize long-horizon robotic scenarios?
  • 6.5 How do generalization metrics change with varying amounts of data quantity and data diversity?
  • 7 Conclusions, Limitations and Future Work
  • References
  • A Author Contributions
  • B Model Card
  • C Model and Data
  • C.1 Model inference
  • C.2 Data collection at scale.
  • C.3 Model Selection at Scale
  • C.4 Data collection process
  • D Experiments
  • D.1 Evaluation Details
  • D.2 Heterogeneous Data
  • D.3 Long-horizon Evaluation Details
  • D.4 Model Ablations
  • D.5 Summary and Analysis

Knowls

  1. Knowl 1 — RT-1 Architecture and Vision-Language Tokenization

    model/method

    Robotics Transformer 1 (RT-1) is a 35M parameter multi-task vision-language-action model designed for real-time robotic control. The architecture operates on a short history of 6 RGB image observations (300×300×3300 \times 300 \times 3) and a natural language instruction to predict discretized robot actions.

    The instruction is embedded into a 512-dimensional vector using a pretrained Universal Sentence Encoder (USE). Visual inputs are processed through an EfficientNet-B3 backbone pretrained on ImageNet, comprising 26 MBConv blocks interleaved with Feature-wise Linear Modulation (FiLM) layers. To preserve the pretrained representations, the FiLM affine transformation parameters (dense projection layers fcf_c and hch_c) are identity-initialized with weights set to zero. The FiLM-conditioned EfficientNet produces a 9×9×5129 \times 9 \times 512 feature map per image (81 visual tokens).

    To reduce sequence length for the Transformer backbone, a TokenLearner module applies element-wise spatial attention over the 81 tokens of each frame to soft-select and compress them down to 8 tokens (8×5128 \times 512). Across the 6-frame history, the resulting 48 vision-language tokens are augmented with learned positional encodings and fed into a decoder-only Transformer sequence model consisting of 8 self-attention layers (19M parameters) that outputs discretized action tokens.

  2. Knowl 2 — Action Representation and Training Objective of RT-1

    model/method

    RT-1 represents robotic actions as a vector at∈R11a_t \in \mathbb{R}^{11} across 11 discrete action dimensions at timestep tt:

    1. Arm movement (7 dimensions): end-effector translational displacements (Δx,Δy,Δz)(\Delta x, \Delta y, \Delta z), rotational displacements (Δroll,Δpitch,Δyaw)(\Delta \text{roll}, \Delta \text{pitch}, \Delta \text{yaw}), and continuous gripper openness.
    2. Base movement (3 dimensions): planar base translational displacements (Δx,Δy)(\Delta x, \Delta y) and rotational displacement Δyaw\Delta \text{yaw}.
    3. Control mode (1 discrete dimension): a 3-way categorical selector switching between controlling the arm, controlling the base, or terminating the episode.

    Each continuous variable is discretized into 256 uniform categorical bins bounded within its minimum and maximum values. The model is trained via behavioral cloning using a categorical cross-entropy loss over all 11 action dimensions with causal masking. Action dimensions are generated non-autoregressively in parallel at each step rather than autoregressively dimension-by-dimension, which reduces inference latency without sacrificing task success.

  3. Knowl 3 — Real-Time Inference Acceleration and Delay Compensation in RT-1

    model/method

    To achieve closed-loop real-time control at a target frequency of 3 Hz, the RT-1 neural network inference budget is constrained to under 100 ms per step to account for camera acquisition and communication latencies. RT-1 achieves a 15 ms inference latency on standard robot hardware via two complementary optimizations:

    1. TokenLearner token compression: Subsampling the visual feature tokens from 81 to 8 per frame yields a 2.4x inference speedup.
    2. Sliding window token caching: Because RT-1 operates on a sliding history of 6 image frames, vision-language tokens computed for previous frames are cached and reused across overlapping future inference windows, resulting in a 1.7x inference speedup.

    In addition, parallel (non-autoregressive) action token generation reduces latency by approximately 2x compared to autoregressive generation (15 ms vs. 36 ms). To prevent robot jitter caused by execution timing fluctuations, a fixed-time waiting mechanism delays action execution until exactly 280 ms (the maximum observed pipeline latency across all hardware components) after the observation timestamp.

  4. Knowl 4 — Zero-Shot Generalization and Robustness of RT-1 Against Baselines

    empirical result

    RT-1 was evaluated across over 3000 real-world trials on Everyday Robots mobile manipulators against three baseline architectures trained on the exact same dataset of 130k demonstrations: Gato (37M parameter Transformer), BC-Z (ResNet backbone with continuous action regression), and BC-Z XL (scaled ResNet matching RT-1 parameter count).

    Model Seen Tasks Unseen Tasks Distractors Backgrounds
    Gato 65% 52% 43% 35%
    BC-Z 72% 19% 47% 41%
    BC-Z XL 56% 43% 23% 35%
    RT-1 (ours) 97% 76% 83% 59%
    • Seen Tasks: Evaluated over 200 distinct training tasks under varying physical object and robot positions. RT-1 achieved 97% success, outperforming BC-Z by 25% and Gato by 32%.
    • Unseen Tasks: Evaluated on 21 novel held-out skill-object combinations. RT-1 achieved 76% success, outperforming the next best baseline (Gato) by 24%.
    • Distractor Robustness: Evaluated over 30 tasks with 0 to 9 distractor objects and heavy occlusions. RT-1 achieved 83% success (100% on easy 0–5 distractors, 100% on medium 9 distractors, 64% on hard occluded distractors), compared to 43% for Gato and 47% for BC-Z.
    • Background Robustness: Evaluated over 22 tasks across different counter textures (patterned tablecloths) and completely new office kitchen environments. RT-1 achieved 59% success, exceeding the next best baseline by 18%.

  5. Knowl 5 — Scaling Laws of Task Diversity Versus Data Quantity in Multi-Task Policies

    empirical result

    Ablation experiments evaluating RT-1 on seen tasks, unseen tasks, distractor robustness, and background robustness revealed that dataset diversity (the number of distinct tasks) has a significantly stronger impact on policy generalization than raw data volume (number of demonstrations per task):

    Dataset Variant % Tasks % Data Seen Tasks Unseen Tasks Distractors Backgrounds
    RT-1 (Full) 100% 100% 97% 76% 83% 59%
    RT-1 (Cap 200 demos/task) 100% 51% 71% 52% 39% 59%
    RT-1 (Cap 100 demos/task) 100% 37% 55% 57% 35% 47%
    RT-1 (Cap 50 demos/task) 100% 22% 59% 14% 31% 41%
    RT-1 (Narrowed diversity) 75% 97% 86% 67% 42% 53%

    Removing 25% of the task diversity while retaining 97% of the total demonstration data volume caused unseen task performance to drop from 76% to 67% and distractor robustness to drop from 83% to 42%. This drop matches or exceeds the degradation observed when halving the dataset size (51% data) while preserving 100% of the task diversity.

  6. Knowl 6 — Cross-Domain Absorption of Synthetic Simulation Data in RT-1

    empirical result

    To evaluate data absorption across domains, RT-1 was trained on a combination of real robot demonstrations and 518k simulated picking trajectories involving objects never encountered in the real world (generated via reinforcement learning bootstrapped from real demonstrations).

    When evaluated on physical robots in the real world:

    • Real Tasks / Objects: Success rate remained stable at 90% for Real + Sim training compared to 92% for Real-Only training (-2% difference).
    • Seen Skills on Simulation-Only Objects: Grasping real instances of objects seen only in simulation increased from 23% (Real-Only policy) to 87% (Real + Sim policy), a +64% absolute improvement.
    • Unseen Skills on Simulation-Only Objects: Executing skills never trained on those objects in simulation or reality (e.g., executing "move X near Y" when object X was only seen in simulated "pick X" tasks) improved from 7% (Real-Only policy) to 33% (Real + Sim policy), a +26% absolute improvement.

  7. Knowl 7 — Cross-Morphology Transfer of Bin-Picking Data Across Robot Platforms

    empirical result

    RT-1 was evaluated for cross-morphology knowledge transfer by combining human demonstrations on Everyday Robots (EDR) mobile manipulators with 209k reinforcement learning grasp trajectories collected on a stationary Kuka IIWA robot from the QT-Opt dataset.

    The Kuka 4-DoF actions were transformed into RT-1's action space by setting Δroll=0\Delta \text{roll} = 0 and Δpitch=0\Delta \text{pitch} = 0, preserving Δyaw\Delta \text{yaw}, converting binary gripper commands to continuous openness, and assigning the language prompt "pick anything". The model was trained on a 2:1 mixture ratio (EDR data : Kuka data).

    Training Data Classroom Eval (EDR Tasks) Bin-Picking Eval (EDR on Kuka setup)
    EDR only data 92% 22%
    Kuka bin-picking only data 0% 0%
    Kuka bin-picking + EDR data 90% (-2%) 39% (+17%)

    While training on Kuka data alone yielded 0% success on the EDR robot, co-training with both datasets improved bin-picking performance on EDR from 22% to 39% (a 1.8x gain) while retaining 90% performance on native EDR classroom tasks.

  8. Knowl 8 — SayCan Long-Horizon Task Execution Across Realistic Kitchens

    empirical result

    RT-1 was integrated as the low-level manipulation policy within the SayCan natural language planning framework to execute long-horizon instructions (averaging 9.6 total steps and 2.4 manipulation skills per instruction) across two physical office kitchens:

    • Kitchen1: An office kitchen sharing layout similarities with the training classroom.
    • Kitchen2: A novel office kitchen with unseen countertops, lighting, background, drawer styles, and an exposed sink.
    SayCan in Kitchen1 SayCan in Kitchen2
    Model Planning Execution Planning Execution
    SayCan w/ Gato 87% 33% 87% 0%
    SayCan w/ BC-Z 87% 53% 87% 13%
    SayCan w/ RT-1 (ours) 87% 67% 87% 67%

    In Kitchen1, SayCan + RT-1 achieved a 67% execution success rate (vs. 53% for BC-Z and 33% for Gato). In Kitchen2, SayCan + RT-1 maintained a 67% execution success rate without degradation, whereas SayCan + BC-Z dropped to 13% and SayCan + Gato failed completely (0%). RT-1's robustness to base placement variations and background visual shifts enabled execution of multi-step chained plans of up to 50 steps.

  9. Knowl 9 — Ablation Study of Architectural Decisions in RT-1

    empirical result

    Controlled ablations of RT-1 design choices yielded the following performance metrics across Seen Tasks, Unseen Tasks, Distractor Robustness, Background Robustness, and Inference Latency (ms):

    Model Configuration Seen Tasks Unseen Tasks Distractors Backgrounds Inference (ms)
    RT-1 (Full) 97% 76% 83% 59% 15
    RT-1 w/ continuous actions 68% (-29%) 43% (-33%) 37% (-46%) 35% (-24%) 16
    RT-1 w/o ImageNet pre-training 84% (-13%) 43% (-33%) 60% (-23%) 41% (-18%) 15
    RT-1 w/o history (1 frame) 82% (-15%) 62% (-14%) 50% (-33%) 59% (+0%) 15
    RT-1 w/ autoregressive actions 85% (-12%) 71% (-5%) 67% (-16%) 65% (+6%) 36
    RT-1 w/o Transformer 86% (-13%) 62% (-14%) 67% (-16%) 59% (+0%) 26
    RT-1 w/o big model (25M params) 89% (-8%) 62% (-14%) 77% (-6%) 53% (-6%) 13.5

    Key takeaways:

    1. Discretizing action dimensions into 256 bins vs. continuous Gaussian regression (MSE loss) provided the largest gains (+29% on seen tasks, +33% on unseen tasks, +46% on distractors), reflecting the necessity of capturing multimodal action distributions in demonstration data.
    2. ImageNet pre-training of the visual backbone is vital for out-of-distribution transfer (+33% on unseen tasks, +18% on backgrounds).
    3. Temporal history (6 frames) is crucial for distractor robustness (+33%).
    4. Autoregressive action generation slowed inference by >2x (from 15 ms to 36 ms) without improving task performance.

  10. Knowl 10 — Everyday Robots Large-Scale Multi-Task Manipulation Dataset

    experimental setup

    The primary dataset used to train RT-1 comprises 130k tele-operated robot demonstrations collected over 17 months across 13 Everyday Robots mobile manipulators equipped with a 7-DoF arm, a two-fingered parallel gripper, and a mobile base.

    The dataset covers 744 natural language instructions grouped under 12 skill categories:

    • Pick Object: 130 tasks
    • Move Object Near Object: 337 tasks
    • Place Object into Receptacle: 84 tasks
    • Pick Object from Receptacle and Place on Counter: 162 tasks
    • Place Object Upright: 8 tasks
    • Knock Object Over: 8 tasks
    • Open Drawer: 3 tasks
    • Close Drawer: 3 tasks
    • Long-Horizon Kitchen Skills (open glass jar, pull napkin from dispenser, grab scooper): 9 tasks

    Demonstrations were collected by operators using two virtual reality remotes mapping 3D Cartesian position and rotational displacement to 6D tool displacements and joystick inputs to mobile base driving distances and turning angles. Data collection was automated using dispatch software that randomized scene configurations, background environments, and task assignments for each robot station.

  11. Knowl 11 — Real-to-Sim Checkpoint Selection Using RetinaGAN

    model/method

    To evaluate policies and select training checkpoints across hundreds of tasks without requiring expensive physical rollouts, RT-1 utilizes a real-to-sim simulation pipeline over 551 distinct tasks:

    1. A physics-based simulator was constructed supporting 551 manipulation instructions with randomized initial object arrangements, base poses, and automated success metrics.
    2. A RetinaGAN image-to-image translation network was trained to translate synthetic simulator frames into photorealistic visual observations matching the camera characteristics and appearance of the real physical environment.
    3. Policies trained purely on real-world demonstration data were evaluated zero-shot in simulation by conditioning action predictions on RetinaGAN-transformed visual inputs at each timestep.

    While real-trained policies achieve lower absolute success rates in simulation than on physical hardware, the relative rank-order of simulation success rates across policy checkpoints correlates strongly with their real-world performance ranking, making simulation an effective off-policy checkpoint selection tool.

  12. Knowl 12 — Multi-Level Realistic Kitchen Generalization Benchmark (L1–L3)

    empirical result

    To evaluate simultaneous distribution shifts in realistic environments (restocking snacks in drawers, cleaning knocked-over bottles, preparing snack platters, retrieving lost items), tasks were structured into three cumulative generalization levels:

    • Level 1 (L1): Novel kitchen counter layout, new background visual scene, and altered lighting conditions.
    • Level 2 (L2): L1 distribution shifts plus previously unseen distractor objects (e.g., kitchen containers and jars).
    • Level 3 (L3): L2 distribution shifts plus novel task objects, drastic task setting shifts, or unseen manipulation locations (e.g., counters adjacent to kitchen sinks).
    Model All Levels L1 L2 L3
    Gato 30% 63% 25% 0%
    BC-Z 45% 38% 50% 50%
    BC-Z XL 55% 63% 75% 38%
    RT-1 (ours) 70% 88% 75% 50%

    RT-1 achieved 70% overall success across all levels (88% on L1, 75% on L2, and 50% on L3), outperforming Gato (30%), BC-Z (45%), and BC-Z XL (55%).

  13. Knowl 13 — Limitations of RT-1

    limitation

    The RT-1 framework exhibits several key limitations:

    1. Imitation Learning Bottleneck: As a behavioral cloning approach trained via supervised imitation learning on demonstrations, RT-1 cannot discover novel behaviors or surpass the execution capabilities of human demonstrators.
    2. Generalization Boundaries: Generalization to unseen language instructions is strictly compositional, relying on novel recombinations of previously observed objects, verbs, and visual attributes; RT-1 cannot execute entirely novel physical motion trajectories or manipulation primitives that were never demonstrated.
    3. Dexterity Scope: The demonstrated capabilities focus on coarse pick-and-place, sliding, and drawer opening behaviors, and do not extend to fine-grained, highly dexterous, or force-controlled bi-manual manipulation.

Coverage note — None was omitted; all primary architectural components, action space design, inference acceleration methods, real-world benchmarks, cross-domain/cross-morphology transfer results, SayCan evaluations, scaling law analyses, ablations, and stated limitations are fully covered.

References

  1. 1.Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. 2.Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018.
  3. 3.Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.
  4. 4.Michael Jae-Yoon Chung, Abram L Friesen, Dieter Fox, Andrew N Meltzoff, and Rajesh PN Rao. A bayesian developmental approach to robotic goal-based imitation learning. PloS one, 10(11): e0141965, 2015.
  5. 5.Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
  6. 6.Marc Peter Deisenroth, Peter Englert, Jan Peters, and Dieter Fox. Multi-task policy search for robotics. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 3876–3881. IEEE, 2014.
  7. 7.Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2169–2176. IEEE, 2017.
  8. 8.Miroslav Dud1ık, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601, 2011.
  9. 9.Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021.
  10. 10.Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 538–547, 2019.
  11. 11.Roy Fox, Ron Berenstein, Ion Stoica, and Ken Goldberg. Multi-task hierarchical imitation learning for home automation. In 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pp. 1–8. IEEE, 2019.
  12. 12.Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018.
  13. 13.Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint arXiv:2203.11931, 2022.
  14. 14.Josiah P Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
  15. 15.Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai. RetinaGAN: An object-aware approach to sim-to-real transfer, 2020. URL https://arxiv.org/abs/2011.03148.
  16. 16.De-An Huang, Yu-Wei Chao, Chris Paxton, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, and Dieter Fox. Motion reasoning for goal-based imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4878–4884. IEEE, 2020.
  17. 17.Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. Advances in Neural Information Processing Systems, 32, 2019.
  18. 18.Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. RLBench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020.
  19. 19.Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991–1002. PMLR, 2021.
  20. 20.Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021.
  21. 21.Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2022.
  22. 22.Tom Jurgenson, Or Avner, Edward Groshev, and Aviv Tamar. Sub-goal trees a framework for goal-based reinforcement learning. In International Conference on Machine Learning, pp. 5020–5030. PMLR, 2020.
  23. 23.Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651–673. PMLR, 2018.
  24. 24.Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021a.
  25. 25.Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. MT-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021b.
  26. 26.Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 259–266. IEEE, 2010.
  27. 27.Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205.15241, 2022a.
  28. 28.Kuang-Huei Lee, Ted Xiao, Adrian Li, Paul Wohlhart, Ian Fischer, and Yao Lu. PI-QT-Opt: Predictive information improves multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2210.08217, 2022b.
  29. 29.Ian Lenz, Honglak Lee, and Ashutosh Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705–724, 2015.
  30. 30.Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020.
  31. 31.Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. Def, 2(6):4, 2006.
  32. 32.Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  33. 33.Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pp. 1303–1315. PMLR, 2022.
  34. 34.Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 4055–4064. PMLR, 2018.
  35. 35.Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15942–15952, 2021.
  36. 36.Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11671. URL https://ojs.aaai.org/index.php/AAAI/article/view/11671.
  37. 37.Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406–3413. IEEE, 2016.
  38. 38.Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
  39. 39.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
  40. 40.Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia D1ı́az-Rodrı́guez, and David Filliat. Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics. arXiv preprint arXiv:1901.08651, 2019.
  41. 41.Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821–8831. PMLR, 2021.
  42. 42.Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
  43. 43.Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 34:12786–12797, 2021.
  44. 44.Ashutosh Saxena, Justin Driemeyer, Justin Kearns, and Andrew Ng. Robotic grasping of novel objects. Advances in neural information processing systems, 19, 2006.
  45. 45.Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. arXiv preprint arXiv:2206.11251, 2022.
  46. 46.Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pp. 906–915. PMLR, 2018.
  47. 47.Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
  48. 48.Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209.05451, 2022.
  49. 49.Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan, and Matthew Gombolay. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 7(2):1635–1642, 2021.
  50. 50.Avi Singh, Eric Jang, Alexander Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levinev, Mohi Khansari, and Chelsea Finn. Scalable multi-task imitation learning with autonomous improvement. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2167–2173. IEEE, 2020.
  51. 51.Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139–13150, 2020.
  52. 52.Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105–6114. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/tan19a.html.
  53. 53.Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507–1514, 2011.
  54. 54.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  55. 55.Ulrich Viereck, Andreas Pas, Kate Saenko, and Robert Platt. Learning a visuomotor controller for real world robotic grasping using simulated depth images. In Conference on robot learning, pp. 291–300. PMLR, 2017.
  56. 56.Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, and Alexander Herzog. Thinking while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv:2004.06089, 2020.
  57. 57.Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094–1100. PMLR, 2020.
  58. 58.Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5628–5635. IEEE, 2018.
  59. 59.Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. arXiv preprint arXiv:2106.03427, 2021.

Citation

MLA
Brohan, A., et al. “RT-1: Robotics Transformer for Real-World Control at Scale”. arXiv, 2022, http://arxiv.org/abs/2212.06817v2.
APA
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., … Zitkovich, B. (2022). RT-1: Robotics Transformer for Real-World Control at Scale. arXiv. http://arxiv.org/abs/2212.06817v2
Chicago
Brohan, A., N. Brown, J. Carbajal, et al. 2022. “RT-1: Robotics Transformer for Real-World Control at Scale”. arXiv. http://arxiv.org/abs/2212.06817v2.
Harvard
Brohan, A. et al. (2022) “RT-1: Robotics Transformer for Real-World Control at Scale”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2212.06817v2.
Vancouver
1. Brohan A, Brown N, Carbajal J, et al (2022) RT-1: Robotics Transformer for Real-World Control at Scale. arXiv

BibTeX

@article{brohan2022robotics,
  title = {RT-1: Robotics Transformer for Real-World Control at Scale},
  author = {Brohan, Anthony and Brown, Noah and Carbajal, Justice and Chebotar, Yevgen and Dabis, Joseph and Finn, Chelsea and Gopalakrishnan, Keerthana and Hausman, Karol and Herzog, Alex and Hsu, Jasmine and Ibarz, Julian and Ichter, Brian and Irpan, Alex and Jackson, Tomas and Jesmonth, Sally and Joshi, Nikhil J and Julian, Ryan and Kalashnikov, Dmitry and Kuang, Yuheng and Leal, Isabel and Lee, Kuang-Huei and Levine, Sergey and Lu, Yao and Malla, Utsav and Manjunath, Deeksha and Mordatch, Igor and Nachum, Ofir and Parada, Carolina and Peralta, Jodilyn and Perez, Emily and Pertsch, Karl and Quiambao, Jornell and Rao, Kanishka and Ryoo, Michael and Salazar, Grecia and Sanketi, Pannag and Sayed, Kevin and Singh, Jaspiar and Sontakke, Sumedh and Stone, Austin and Tan, Clayton and Tran, Huong and Vanhoucke, Vincent and Vega, Steve and Vuong, Quan and Xia, Fei and Xiao, Ted and Xu, Peng and Xu, Sichun and Yu, Tianhe and Zitkovich, Brianna},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2212.06817v2},
  eprint = {2212.06817}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF