Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Tony Z. ZhaoVikash KumarSergey LevineChelsea Finn

article2023Robotics: Science and Systems Conference2,279 citations

Develops Action Chunking with Transformers (ACT), an imitation learning algorithm paired with low-cost hardware that allows dual-arm robots to master high-precision manipulation tasks with high success rates from only ten minutes of demonstrations.

Listen

Precise robotic manipulation tasks—such as inserting a battery, opening a condiment cup, or threading a cable tie—traditionally require expensive industrial robots, high-precision sensors, and complex calibration. The high cost and operational complexity of these platforms present major barriers to deploying automation in real-world settings. The article evaluates whether an end-to-end imitation learning framework combined with low-cost, off-the-shelf hardware can successfully perform delicate, contact-rich bimanual manipulation.

To address this challenge, the authors developed ALOHA, an open-source bimanual teleoperation platform built for under $20,000 using accessible robot arms, custom 3D-printed grippers, and commodity webcams. Alongside the hardware, they introduced Action Chunking with Transformers (ACT), a novel imitation learning algorithm that formulates control as a generative sequence-prediction model. ACT maps standard camera images and robot joint positions directly to future action sequences while using temporal ensembling to smooth trajectories. The system was evaluated on two simulated tasks and six real-world tasks requiring delicate coordination, using datasets of only 50 to 100 human demonstrations per task (representing roughly 10 to 20 minutes of operational data).

Across the evaluations, the combined system demonstrated strong performance. ACT achieved high success rates on challenging real-world tasks, including 96% on battery insertion, 92% on putting a shoe on a foot, 88% on sliding open a ziploc bag, and 84% on prying open a translucent condiment cup. In comparative benchmarks, ACT substantially outperformed prior imitation learning baselines, which suffered from compounding execution errors and largely achieved 0% final success on the real-world tasks. Ablation experiments showed that predicting chunks of actions rather than single steps was the primary driver of success, and that high-frequency control at 50 Hz reduced task completion time by 38% compared to a 5 Hz baseline (or conversely, a 5 Hz baseline caused a 62% slowdown).

These findings indicate that algorithmic innovation can effectively compensate for the mechanical imprecisions of low-cost hardware. Organizations can achieve high-dexterity automation without investing in specialized six-figure robotics platforms or manually engineering complex physics models for deformable and contact-rich objects. Because the learning pipeline requires only minutes of demonstration data and standard computing resources (training in about 5 hours on a single consumer GPU), it dramatically lowers the cost, time, and engineering overhead required to deploy robotic solutions.

To build upon these results, teams exploring fine manipulation should consider adopting action-chunked generative policies and evaluating low-cost leader-follower setups for data collection. However, decision-makers should recognize current operational boundaries: the hardware remains payload- and force-limited (supporting up to 750 grams) and lacks multi-fingered hands for complex tactile actions such as unbuttoning or twisting tightly sealed containers. Additionally, highly ambiguous visual scenarios—such as unwrapping candy wrappers or localizing thin, low-contrast items—still lead to degraded policy reliability. Future development should focus on integrating multi-task pretraining, expanded datasets, and improved visual perception before deploying in unstructured production environments.

Meta Stanford University University of California Berkeley action sequences demonstrations encoder-decoder Learning Action Manipulation

arXiv: 2304.13705 tonyzhaozh/act

Paper: Diffusion policy: Visuomotor policy learning via action diffusion, Cheng Chi et al. (2023). This paper introduces the action diffusion formulation that serves as the core policy architecture adopted by the source work for continuous control.
Paper: RT-1: Robotics Transformer for Real-World Control at Scale, Anthony Brohan et al. (2023). This work establishes large-scale real-world robotic data collection and transformer-based policy learning that directly inform the experimental approach of the source.

Paper: RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation, Songming Liu et al. (2025). This work directly builds upon the source's bimanual manipulation setup by scaling a diffusion transformer model to a multi-billion parameter foundation model across diverse robotic datasets.
Paper: OpenVLA: An Open-Source Vision-Language-Action Model, Moo Jin Kim et al. (2024). This paper extends the source's imitation learning paradigm into a large-scale open-source vision-language-action model trained on hundreds of thousands of bimanual and manipulation trajectories.
Paper: π0: A Vision-Language-Action Flow Model for General Robot Control, Kevin Black et al. (2024). This work continues the source's trajectory by combining a vision-language backbone with continuous action flow matching to achieve general robot control across multiple platforms.

Cover for Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Abstract

Fine manipulation tasks, such as threading cable ties or slotting a battery, are notoriously difficult for robots because they require precision, careful coordination of contact forces, and closed-loop visual feedback. Performing these tasks typically requires high-end robots, accurate sensors, or careful calibration, which can be expensive and difficult to set up. Can learning enable low-cost and imprecise hardware to perform these fine manipulation tasks? We present a low-cost system that performs end-to-end imitation learning directly from real demonstrations, collected with a custom teleoperation interface. Imitation learning, however, presents its own challenges, particularly in high-precision domains: errors in the policy can compound over time, and human demonstrations can be non-stationary. To address these challenges, we develop a simple yet novel algorithm, Action Chunking with Transformers (ACT), which learns a generative model over action sequences. ACT allows the robot to learn 6 difficult tasks in the real world, such as opening a translucent condiment cup and slotting a battery with 80-90% success, with only 10 minutes worth of demonstrations. Project website: this https URL

Topics

action sequences demonstrations encoder-decoder generative model imitation learning teleoperation

Keywords

Learning Action Manipulation Tasks Imitation Learning Action Chunking Teleoperation Bimanual Action Sequence Policy

I Introduction
II Related Work
III ALOHA: A Low-cost Open-source Hardware System for Bimanual Teleoperation
IV Action Chunking with Transformers
IV-A Action Chunking and Temporal Ensemble
IV-B Modeling human data
IV-C Implementing ACT
V Experiments
V-A Tasks
V-B Data Collection
V-C Experiment Results
VI Ablations
VI-A Action Chunking and Temporal Ensembling
VI-B Training with CVAE
VI-C Is High-Frequency Necessary?
VII Limitations and Conclusion
References
-A Comparing ALOHA with Prior Teleoperation Setups
-B Example Image Observations
-C Detailed Architecture Diagram
-D Experiment Details and Hyperparameters
-E User Study Details
-F Limitations

Knowls

Knowl 1 — Action Chunking with Transformers Policy Architecture and Generative Formulation
model/method
Action Chunking with Transformers (ACT) is an imitation learning algorithm designed for high-precision, contact-rich manipulation. Rather than predicting a single-step action $\pi_\theta(a_t \mid s_t)$ , ACT models a generative policy over action chunks of length $k$ , denoted $\pi_\theta(a_{t:t+k} \mid s_t)$ , which reduces the effective planning horizon by a factor of $k$ and mitigates compounding errors.

Model Formulation

ACT is formulated as a Conditional Variational Autoencoder (CVAE) to capture multi-modal and non-Markovian distributions inherent in human teleoperation demonstrations:
1. CVAE Encoder ( $q_\phi(z \mid a_{t:t+k}, \bar{o}_t)$ ): Implemented with a BERT-like transformer encoder containing 4 self-attention layers. It takes as input a learned [CLS] token (dimension 512), the embedded current robot joint positions $\bar{o}_t \in \mathbb{R}^{14}$ projected to 512 dimensions via a linear layer, and the future action chunk $a_{t:t+k} \in \mathbb{R}^{k \times 14}$ projected linearly to $k \times 512$ with added sinusoidal positional embeddings. The output feature corresponding to the [CLS] token is mapped by a linear layer to the mean and log-variance of a latent style variable $z \in \mathbb{R}^{32}$ . The encoder conditions only on proprioception $\bar{o}_t$ rather than images to accelerate training, and is discarded at inference time.
2. CVAE Decoder / Policy Network ( $\pi_\theta(\hat{a}_{t:t+k} \mid o_t, z)$ ): Takes the full observation $o_t$ (comprising 4 RGB images of dimension $480 \times 640 \times 3$ and 14-DoF joint positions) alongside $z$ . Each image is passed through a ResNet-18 backbone to produce a $15 \times 20 \times 512$ feature map, flattened to $300 \times 512$ , augmented with 2D sinusoidal position embeddings, and concatenated across the 4 cameras into a $1200 \times 512$ sequence. The current joint positions and $z$ are projected linearly to 512 dimensions each and appended, yielding a sequence of dimension $1202 \times 512$ . This sequence is processed by a 4-layer transformer encoder. A 7-layer transformer decoder queries the encoder outputs via cross-attention, using a fixed $k \times 512$ sinusoidal positional embedding as queries, and projects the outputs through an MLP to predict continuous target joint positions $\hat{a}_{t:t+k} \in \mathbb{R}^{k \times 14}$ .
Objective Function

The parameters $\theta$ and $\phi$ are trained end-to-end to minimize the variational objective: $\mathcal{L} = \mathcal{L}_{\text{reconst}} + \beta \mathcal{L}_{\text{reg}}$ where the reconstruction loss uses the $L_1$ norm: $\mathcal{L}_{\text{reconst}} = \frac{1}{k} \sum_{j=0}^{k-1} \| \hat{a}_{t+j} - a_{t+j} \|_1$ and the regularization term is the Kullback-Leibler divergence between the approximate posterior and a standard normal prior: $\mathcal{L}_{\text{reg}} = D_{\text{KL}}(q_\phi(z \mid a_{t:t+k}, \bar{o}_t) \parallel \mathcal{N}(0, I))$ At test time, the latent variable is deterministically fixed to $z = 0$ (the prior mean).
Knowl 2 — ACT Policy Training Procedure
algorithm
The training routine for Action Chunking with Transformers (ACT) optimizes both the CVAE recognition network and the generative transformer policy on demonstration episodes using the $L_1$ reconstruction and KL-divergence loss.
Input: Demonstration dataset $\mathcal{D}$ , action chunk size $k$ , KL weight parameter $\beta$ , total training iterations $N$
Initialize CVAE encoder $q_\phi(z \mid a_{t:t+k}, \bar{o}_t)$ with parameters $\phi$
Initialize CVAE decoder $\pi_\theta(\hat{a}_{t:t+k} \mid o_t, z)$ with parameters $\theta$
for iteration $n = 1, 2, \dots, N$ do
    Sample observation $o_t$ and ground-truth action trajectory $a_{t:t+k}$ from $\mathcal{D}$
    Extract proprioceptive joint observation $\bar{o}_t$ from $o_t$
    Sample style latent vector $z \sim q_\phi(z \mid a_{t:t+k}, \bar{o}_t)$ via reparameterization
    Predict action chunk $\hat{a}_{t:t+k} = \pi_\theta(o_t, z)$
    Compute reconstruction loss $\mathcal{L}_{\text{reconst}} = \frac{1}{k} \sum_{j=0}^{k-1} \|\hat{a}_{t+j} - a_{t+j}\|_1$
    Compute KL regularization loss $\mathcal{L}_{\text{reg}} = D_{\text{KL}}(q_\phi(z \mid a_{t:t+k}, \bar{o}_t) \parallel \mathcal{N}(0, I))$
    Compute total objective $\mathcal{L} = \mathcal{L}_{\text{reconst}} + \beta \mathcal{L}_{\text{reg}}$
    Update parameters $\theta, \phi$ using the Adam optimizer on $\nabla_{\theta, \phi} \mathcal{L}$
end for
Output: Trained decoder policy $\pi_\theta$
Implementation Hyperparameters
- Learning rate: $1 \times 10^{-5}$
- Batch size: 8
- Transformer encoder layers: 4; Transformer decoder layers: 7
- Hidden dimension: 512; Feedforward dimension: 3200; Attention heads: 8; Dropout: 0.1
- Chunk size $k$ : 100; KL weight $\beta$ : 10
- Total parameters: ~80M (trains in ~5 hours on a single 11GB NVIDIA RTX 2080 Ti GPU).
Knowl 3 — Temporal Ensembling for ACT Policy Inference
algorithm
Executing action chunks open-loop every $k$ timesteps can produce discrete, jerky transitions when switching between execution and observation. ACT resolves this by querying the policy at every environment timestep $t$ , resulting in overlapping predicted action chunks for each future timestep. Temporal ensembling computes an exponentially weighted average across all overlapping predictions targeted at the same timestep, producing smooth, continuous robot motion without introducing temporal lag.
Input: Trained policy $\pi_\theta$ , episode length $T$ , chunk size $k$ , exponential weighting parameter $m$
Initialize FIFO buffers $B[t] = []$ for each timestep $t \in \{1, 2, \dots, T\}$
for timestep $t = 1, 2, \dots, T$ do
    Receive environment observation $o_t$
    Set style variable $z = 0$ (the mean of the prior distribution $\mathcal{N}(0, I)$ )
    Predict action sequence chunk $\hat{a}_{t:t+k} = \pi_\theta(o_t, z)$
    for offset $j = 0, 1, \dots, k-1$ do
        if $t + j \le T$ then
            Append predicted action $\hat{a}_{t+j}$ to buffer $B[t+j]$
        end if
    end for
    Retrieve predictions for current step $A_t = B[t]$
    Compute weights $w_i = \exp(-m \cdot i)$ for prediction index $i \in \{0, 1, \dots, |A_t|-1\}$ , where $i=0$ corresponds to the oldest prediction
    Compute aggregated action $a_t = \frac{\sum_{i=0}^{|A_t|-1} w_i A_t[i]}{\sum_{i=0}^{|A_t|-1} w_i}$
    Execute action $a_t$ on the follower robot arms
end for
Unlike conventional action smoothing which blends actions across adjacent time steps and introduces systematic tracking delay, temporal ensembling aggregates multiple distinct policy inferences that target the exact same absolute timestep $t$ . Inference requires approximately 0.01 seconds per timestep on an RTX 2080 Ti GPU.
Knowl 4 — ALOHA Low-Cost Open-Source Bimanual Teleoperation System
experimental setup
ALOHA (A Low-cost Open-source Hardware System for Bimanual Teleoperation) is an accessible dual-arm robot platform designed for contact-rich, fine manipulation under a $20,000 budget.

Kinematic and Mechanical Design
- Follower Arms: Two 6-DoF ViperX 300 robot arms with parallel-jaw grippers ( $750\,\text{g}$ working payload, $750\,\text{mm}$ reach, $1500\,\text{mm}$ total span, $1\,\text{mm}$ repeatability, $5\text{--}8\,\text{mm}$ positional accuracy). Grippers are fitted with custom 3D-printed see-through fingers lined with high-friction grip tape to manipulate thin plastic films and maintain visual line-of-sight.
- Leader Arms: Two 6-DoF WidowX 250 robot arms, which are smaller geometrically scaled versions of the follower arms. Teleoperation operates via direct joint-space mapping (follower joints mimic leader joints directly), avoiding inverse kinematics (IK) singularities near joint limits, reducing computational latency, and utilizing the leader arm's inertia to dampen high-frequency human hand tremor.
- Ergonomic Mechanisms: Leader robots feature a retrofitted 3D-printed handle-and-scissor mechanism for continuous gripper control and a rubber band gravity compensation system to enable operator sessions longer than 30 minutes without fatigue.
Perception and Control Infrastructure
- Sensing: Four Logitech C922x RGB cameras streaming at $480 \times 640$ resolution: one top camera, one front camera (rotated $90^\circ$ vertically), and two wrist-mounted cameras positioned on the follower end-effectors for close-up views.
- Frequency & Action Representation: Teleoperation and data recording operate at $50\,\text{Hz}$ . Actions record the leader robot joint positions ( $14$ -DoF total absolute target angles), implicitly embedding force control through the low-level follower PID position errors.

Knowl 5 — Success Rates Across Simulated and Real-World Bimanual Manipulation Tasks

data/table

The performance of ACT was benchmarked against four imitation learning baselines: BC-ConvMLP (standard behavior cloning with CNN feature extraction), BeT (Behavior Transformer with k-means action discretization and continuous offset prediction), RT-1 (Robotics Transformer 1 with discretized action bins), and VINN (Visual Nearest Neighbors retrieval). Simulated tasks were tested on scripted demonstrations and stochastic human teleoperation demonstrations (3 seeds, 50 trials each). Real-world tasks were evaluated with human demonstrations on ALOHA (1 seed, 25 trials each).

Cube Transfer (sim)

Bimanual Insertion (sim)

Slide Ziploc (real)

Slot Battery (real)

Method

Touched

Lifted

Transfer

Grasp

Contact

Insert

Grasp

Pinch

Open

Grasp

Place

Insert

BC-ConvMLP

34 | 3

17 | 1

1 | 0

5 | 0

1 | 0

BeT

60 | 16

51 | 13

27 | 1

21 | 0

4 | 0

3 | 0

RT-1

44 | 4

33 | 2

2 | 0

0 | 0

1 | 0

VINN

13 | 17

9 | 11

3 | 0

6 | 0

1 | 0

ACT (Ours)

97 | 82

90 | 60

86 | 50

93 | 76

90 | 66

32 | 20

100

Note for simulation columns: Entries indicate [scripted data success | human data success] in %.

	Open Cup (real)			Thread Velcro (real)			Prep Tape (real)				Put On Shoe (real)
Method	Tip	Grasp	Open	Lift	Grasp	Insert	Grasp	Cut	Handover	Hang	Lift	Insert	Support	Secure
BeT	12	0	0	24	0	0	8	0	0	0	12	0	0	0
ACT (Ours)	100	96	84	92	40	20	96	92	72	64	100	92	92	92

ACT achieved $88\%$ on Slide Ziploc, $96\%$ on Slot Battery, $84\%$ on Open Cup, and $92\%$ on Put On Shoe using only 50 demonstrations per task (~10 minutes of teleoperation data). Baselines consistently failed at later subtasks ( $0\%$ final success) due to compounding errors and non-Markovian pauses.

Knowl 6 — Ablation Studies on Action Chunk Size, Temporal Ensembling, and CVAE Objective
empirical result
Systematic ablations across four simulated environments (Transfer Cube and Bimanual Insertion using scripted and human data) isolate the influence of ACT's core components:
1. Chunk Size ( $k$ ): Evaluating ACT without temporal ensembling across varying chunk horizons yields an average task success rate of:
  - $k = 1$ (single-step Markovian BC): $1\%$
  - $k = 10$ : $21\%$
  - $k = 100$ : $44\%$
  - $k = 200$ : $42\%$
  - $k = 400$ (near open-loop control): $41\%$ Augmenting baseline algorithms with action chunking also produces performance improvements: BC-ConvMLP improves from $1\%$ ( $k=1$ ) to $25\%$ ( $k=200$ ), and VINN improves from $0\%$ ( $k=1$ ) to $37\%$ ( $k=400$ ).
2. Temporal Ensembling (TE): Incorporating temporal ensembling across overlapping predictions improves ACT's average success rate from $44\%$ to $47.3\%$ ( $+3.3\%$ ) and BC-ConvMLP from $25\%$ to $29\%$ ( $+4\%$ ). Conversely, temporal ensembling degrades the non-parametric baseline VINN from $37\%$ to $17\%$ ( $-20\%$ ) because VINN retrieves exact ground-truth demonstrations where ensembling disparate trajectory modes introduces averaging artifacts.
3. CVAE Generative Objective: On deterministic scripted demonstrations, removing the CVAE objective (training ACT with deterministic L1 regression) causes negligible difference ( $59\%$ with CVAE vs. $58\%$ without CVAE). On stochastic human demonstrations, removing the CVAE objective causes performance to collapse from $35.3\%$ to $2.0\%$ , demonstrating that CVAE latent modeling is essential for handling multi-modal, non-stationary human data.
Knowl 7 — Impact of Teleoperation Control Frequency on Fine Manipulation
empirical result
A user study evaluating 6 human participants (4 male, 2 female, aged 22–25) measured completion times for two fine-manipulation tasks under $50\,\text{Hz}$ vs. $5\,\text{Hz}$ teleoperation on the ALOHA system:
- Threading a Zip Cable Tie ( $4\,\text{mm} \times 1.5\,\text{mm}$ $4 mm \times 1.5 mm$ hole, $0.8\,\text{mm} \times 3.5\,\text{mm}$ $0.8 mm \times 3.5 mm$ tie):
  - Average completion time at $50\,\text{Hz}$ : $20\,\text{seconds}$
  - Average completion time at $5\,\text{Hz}$ : $33\,\text{seconds}$
- Unstacking Two Plastic Cups ( $2.5\,\text{mm}$ $2.5 mm$ stacking clearance):
  - Average completion time at $50\,\text{Hz}$ : $10\,\text{seconds}$
  - Average completion time at $5\,\text{Hz}$ : $16\,\text{seconds}$
Reducing the control frequency from $50\,\text{Hz}$ to $5\,\text{Hz}$ increased average task execution duration by $62\%$ . A Repeated Measures Design analysis verified that $50\,\text{Hz}$ control significantly outperforms $5\,\text{Hz}$ ( $p < 0.001$ ), demonstrating that high-frequency closed-loop feedback is critical for fine-grained bimanual manipulation.
Knowl 8 — Hardware and Algorithmic Limitations in ALOHA and ACT
limitation
The ALOHA hardware and ACT policy exhibit specific physical and algorithmic limitations:

Hardware Limitations
- Multi-finger dexterity: Two-jaw parallel grippers cannot perform operations requiring independent fingers on a single hand (e.g., holding a child-proof pill bottle while simultaneously depressing a safety tab and twisting the cap).
- Actuator torque limits: Low-cost Dynamixel motors lack sufficient torque to lift heavy objects, twist factory-sealed water bottle caps, or pop tight marker caps.
- Fingernail manipulation: The absence of thin fingernails prevents lifting adhesive tape edges stuck flat to surfaces or prying aluminum soda can pull-tabs.
Policy Learning Limitations
- Candy Unwrapping: In a 10-trial test using 50 demonstrations, the policy achieved 10/10 picks and 8/10 pulling actions, but 0/10 unwrapping completions (rising to 3/5 successful candies when allowed 10 attempts per candy). The failure stemmed from visual ambiguity in identifying the printed wrapper seam orientation.
- Flat Ziploc Bag Opening: When the bag was initialized flat on the table, the policy picked the bag reliably but failed during subsequent mid-air multi-step unzipping because minor grasp variations caused large deictic and deformable state shifts.

Coverage note — None was omitted; all key architectural components, algorithms, hardware designs, experimental comparisons, ablation studies, user studies, and limitations are fully covered.

References

1.Viperx 300 robot arm 6dof. URL https://www.trossenrobotics.com/viperx-300-robot-arm-6dof.aspx.
2.Widowx 250 robot arm 6dof. URL https://www.trossenrobotics.com/widowx-250-robot-arm-6dof.aspx.
3.Highly dexterous manipulation system - capabilities - part 1, Nov 2014. URL https://www.youtube.com/watch?v=TearcKVj0iY.
4.Assembly performance metrics and test methods, Apr 2022. URL https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manipulation-assembly/assembly.
5.Teleoperated robots - shadow teleoperation system, Nov 2022. URL https://www.shadowrobot.com/teleoperation/.
6.Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, and Lerrel Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. arXiv preprint arXiv:2210.06463, 2022.
7.Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Anand Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Ho Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. ArXiv, abs/2212.06817, 2022.
8.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, abs/2005.12872, 2020.
9.Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Stephen McAleer, Hao Dong, Zongqing Lu, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. ArXiv, abs/2206.08686, 2022.
10.Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Kumar Gupta. Efficient bimanual manipulation using learned task schemas. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1149–1155, 2019.
11.Sudeep Dasari and Abhinav Kumar Gupta. Transformers for one-shot visual imitation. In Conference on Robot Learning, 2020.
12.Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Neural Information Processing Systems, 2019.
13.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
14.Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, P. Abbeel, and Wojciech Zaremba. One-shot imitation learning. ArXiv, abs/1703.07326, 2017.
15.Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. ArXiv, abs/2109.13396, 2021.
16.Peter R. Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5:492–499, 2019.
17.Peter R. Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian S. Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. ArXiv, abs/2109.00137, 2021.
18.Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph Gonzalez, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. Learning dense visual correspondences in simulation to smooth and fold real fabrics. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515–11522, 2020.
19.Jennifer Grannen, Priya Sundaresan, Brijen Thananjeyan, Jeffrey Ichnowski, Ashwin Balakrishna, Minho Hwang, Vainavi Viswanath, Michael Laskey, Joseph Gonzalez, and Ken Goldberg. Untangling dense knots by learning task-relevant keypoints. In Conference on Robot Learning, 2020.
20.Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. ArXiv, abs/2105.03655, 2021.
21.Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan D. Ratliff, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170, 2019.
22.Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
23.Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016.
24.Ryan Hoque, Ashwin Balakrishna, Ellen R. Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning, 2021.
25.Stephen James, Michael Bloesch, and Andrew J. Davison. Task-embedded control networks for few-shot imitation learning. ArXiv, abs/1810.03237, 2018.
26.Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, 2022.
27.R G Jenness and C D Wicker. Master–slave manipulators and remote maintenance at the oak ridge national laboratory, Jan 1975. URL https://www.osti.gov/biblio/4179544.
28.Edward Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4613–4619, 2021.
29.Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In International Conference on Robotics and Automation (ICRA), 2021.
30.Michael Kelly, Chelsea Sidrane, K. Driggs-Campbell, and Mykel J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pages 8077–8083, 2018.
31.Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation. IEEE Robotics and Automation Letters, 6:1630–1637, 2021.
32.Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Robot peels banana with goal-conditioned dual-action deep imitation learning. ArXiv, abs/2203.09749, 2022.
33.Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
34.Oliver Kroemer, Christian Daniel, Gerhard Neumann, Herke van Hoof, and Jan Peters. Towards learning hierarchical skills for multi-phase manipulation tasks. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1503–1510, 2015.
35.Lucy Lai, Ann Z Huang, and Samuel J Gershman. Action chunking as policy compression, Sep 2022. URL psyarxiv.com/z8yrv.
36.Michael Laskey, Jonathan Lee, Roy Fox, Anca D. Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, 2017.
37.Alex X. Lee, Henry Lu, Abhishek Gupta, Sergey Levine, and P. Abbeel. Learning force-based manipulation of deformable objects from multiple demonstrations. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 177–184, 2015.
38.Weiwei Li. Optimal control for biological movement systems. 2006.
39.Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart'in-Mart'in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021.
40.Kunal Menda, K. Driggs-Campbell, and Mykel J. Kochenderfer. Ensembledagger: A bayesian approach to safe imitation learning. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5041–5048, 2018.
41.Samuel Paradis, Minho Hwang, Brijen Thananjeyan, Jeffrey Ichnowski, Daniel Seita, Danyal Fer, Thomas Low, Joseph Gonzalez, and Ken Goldberg. Intermittent visual servoing: Efficiently learning policies robust to instrument changes for high-precision surgical manipulation. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7166–7173, 2020.
42.Jyothish Pari, Nur Muhammad, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:2112.01511, 2021.
43.Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. 2009 IEEE International Conference on Robotics and Automation, pages 763–768, 2009.
44.Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NIPS, 1988.
45.Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 7:10873–10881, 2022.
46.Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758–3765, 2017.
47.Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2010.
48.Seyed Sina Mirrazavi Salehian, Nadia Figueroa, and Aude Billard. A unified framework for coordinated multi-arm motion planning. The International Journal of Robotics Research, 37:1205 – 1232, 2018.
49.Nur Muhammad (Mahi) Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. ArXiv, abs/2206.11251, 2022.
50.Kaushik Shivakumar, Vainavi Viswanath, Anrui Gu, Yahav Avigal, Justin Kerr, Jeffrey Ichnowski, Richard Cheng, Thomas Kollar, and Ken Goldberg. Sgtm 2.0: Autonomously untangling long cables using interactive perception. ArXiv, abs/2209.13706, 2022.
51.Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. ArXiv, abs/2109.12098, 2021.
52.Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. ArXiv, abs/2209.05451, 2022.
53.Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube. RSS, 2022.
54.Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi, Dimos V. Dimarogonas, and Danica Kragic. Dual arm manipulation - a survey. Robotics Auton. Syst., 60:1340–1353, 2012.
55.Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS, 2015.
56.srcteam. Shadow teleoperation system plays jenga, Mar 2021. URL https://www.youtube.com/watch?v=7K9brH27jvM.
57.srcteam. How researchers are using shadow robot's technology, Jun 2022. URL https://www.youtube.com/watch?v=p36fYIoTD8M.
58.srcteam. Shadow teleoperation system, Jun 2022. URL https://www.youtube.com/watch?v=cx8eznfDUJA.
59.Simon Stepputtis, Maryam Bandari, Stefan Schaal, and Heni Ben Amor. A system for imitation learning of contact-rich bimanual manipulation policies. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11810–11817, 2022.
60.Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Jeffrey Ichnowski, Ellen R. Novoseller, Minho Hwang, Michael Laskey, Joseph Gonzalez, and Ken Goldberg. Untangling dense non-planar knots by learning manipulation features and recovery policies. ArXiv, abs/2107.08942, 2021.
61.Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Causal imitation learning under temporally correlated noise. In International Conference on Machine Learning, 2022.
62.Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1–5, 2015.
63.Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
64.Stephen Tu, Alexander Robey, Tingnan Zhang, and N. Matni. On the sample complexity of stability constrained imitation learning. In Conference on Learning for Dynamics & Control, 2021.
65.Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
66.Solomon Wiznitzer, Luke Schmitt, and Matt Trossen. interbotix_ros_manipulators. URL https://github.com/Interbotix/interbotix_ros_manipulators.
67.Fan Xie, A. M. Masum Bulbul Chowdhury, M. Clara De Paolis Kaluza, Linfeng Zhao, Lawson L. S. Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. ArXiv, abs/2010.05134, 2020.
68.Andy Zeng, Peter R. Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
69.Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Ken Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8, 2017.
70.Allan Zhou, Moo Jin Kim, Lirui Wang, Peter R. Florence, and Chelsea Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. ArXiv, abs/2301.08556, 2023.
71.Áron Horváth, Eszter Ferentzi, Kristóf Schwartz, Nina Jacobs, Pieter Meyns, and Ferenc Köteles. The measurement of proprioceptive accuracy: A systematic literature review. Journal of Sport and Health Science, 2022. ISSN 2095-2546. doi: https://doi.org/10.1016/j.jshs.2022.04.001. URL https://www.sciencedirect.com/science/article/pii/S2095254622000473.

Citation

MLA

Zhao, T. Z., et al. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware”. arXiv, 2023, http://arxiv.org/abs/2304.13705v1.

APA

Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv. http://arxiv.org/abs/2304.13705v1

Chicago

Zhao, T. Z., V. Kumar, S. Levine, and C. Finn. 2023. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware”. arXiv. http://arxiv.org/abs/2304.13705v1.

Harvard

Zhao, T.Z. et al. (2023) “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2304.13705v1.

Vancouver

1. Zhao TZ, Kumar V, Levine S, Finn C (2023) Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv

BibTeX

@article{zhao2023learning,
  title = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
  author = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2304.13705v1},
  eprint = {2304.13705}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Action Chunking with Transformers Policy Architecture and Generative Formulation

Model Formulation

Objective Function

Knowl 2 — ACT Policy Training Procedure

Implementation Hyperparameters

Knowl 3 — Temporal Ensembling for ACT Policy Inference

Knowl 4 — ALOHA Low-Cost Open-Source Bimanual Teleoperation System

Kinematic and Mechanical Design

Perception and Control Infrastructure

Knowl 5 — Success Rates Across Simulated and Real-World Bimanual Manipulation Tasks

Knowl 6 — Ablation Studies on Action Chunk Size, Temporal Ensembling, and CVAE Objective

Knowl 7 — Impact of Teleoperation Control Frequency on Fine Manipulation

Knowl 8 — Hardware and Algorithmic Limitations in ALOHA and ACT

Hardware Limitations

Policy Learning Limitations

References

Citation

BibTeX

Source Code

Access the Paper