RoboDreamer: Learning Compositional World Models for Robot Imagination

Siyuan ZhouYilun DuJiaben ChenYandong LiDit-Yan YeungChuang Gan

article2024ICML131 citations

Proposes a compositional video diffusion framework that breaks complex instructions and multimodal goals into primitive components, enabling robots to imagine and execute plans for novel object-action combinations they never saw during training.

Massachusetts Institute of Technology The Hong Kong University of Science and Technology University of California, San Diego University of Central Florida University of Massachusetts Amherst factorization natural languages robotic arms

arXiv: 2404.12377

Cover for RoboDreamer: Learning Compositional World Models for Robot Imagination

Abstract

Text-to-video models have demonstrated substantial potential in robotic decision-making, enabling the imagination of realistic plans of future actions as well as accurate environment simulation. However, one major issue in such models is generalization – models are limited to synthesizing videos subject to language instructions similar to those seen at training time. This is heavily limiting in decision-making, where we seek a powerful world model to synthesize plans of unseen combinations of objects and actions in order to solve previously unseen tasks in new environments. To resolve this issue, we introduce RoboDreamer, an innovative approach for learning a compositional world model by factorizing the video generation. We leverage the natural compositionality of language to parse instructions into a set of lower-level primitives, which we condition a set of models on to generate videos. We illustrate how this factorization naturally enables compositional generalization, by allowing us to formulate a new natural language instruction as a combination of previously seen components. We further show how such a factorization enables us to add additional multimodal goals, allowing us to specify a video we wish to generate given both natural language instructions and a goal image. Our approach can successfully synthesize video plans on unseen goals in the RT-X, enables successful robot execution in simulation, and substantially outperforms monolithic baseline approaches to video generation.

Topics

factorization natural languages robotic arms video streams digital videos video contents robot system

1. Introduction
2. Background
2.1. Planning with Text-Conditioned Video Generation
2.2. Executing Videos Plans
3. RoboDreamer
3.1. Text Parser
3.2. Compositional Generation
3.3. Multi-modal Composition
4. Experiments
4.1. Evaluation on Video Generation
4.2. Evaluation on Robotic Planning
5. Related Work
6. Conclusion
Impact Statement
Acknowledgements
References
A. Experimental Details
A.1. Video Diffusion
A.2. Other Details
B. Additional Results
B.1. Visualization on RLBench
B.2. More Results on Video Generation
B.3. IMO Metrics

Knowls

Knowl 1 — RoboDreamer Compositional Video Generation Formulation
model/method
RoboDreamer models the video generation probability distribution $p_\theta(\tau | L)$ for a video trajectory $\tau = [x_1, \dots, x_H]$ conditioned on a natural language instruction $L$ by factorizing it over a set of $N$ parsed semantic components $\{l_i\}_{i=1}^N$ :

$p_\theta(\tau | L) \propto \prod_{i=1}^N p_\theta(\tau | l_i)^{\frac{1}{N}}$

where each $l_i$ represents an isolated sub-component of the command (such as a verb phrase representing an action or a prepositional phrase representing a spatial relationship).

By leveraging the equivalence between diffusion models and energy-based models (EBMs), each individual density $p_\theta(\tau | l_i)$ corresponds to a learned score function $\epsilon_\theta(\tau_t, t | l_i)$ at diffusion timestep $t$ . The score function of the product distribution is the arithmetic mean of the individual score functions:

$\nabla_{\tau} \log p_\theta(\tau_t, t | L) = \frac{1}{N} \sum_{i=1}^N \epsilon_\theta(\tau_t, t | l_i)$

This compositional factorization allows the generative world model to generalize zero-shot to novel combinations of language instructions at inference time, provided the constituent primitive components $l_i$ have been encountered in training distribution.
Knowl 2 — Language Instruction Parsing into Action and Spatial Primitives
model/method
To structure unstructured language goals into factorized components for video synthesis, RoboDreamer utilizes a constituency parser combined with rule-based syntactic decomposition.

Given a natural language task description $L$ , the parser decomposes the instruction into:
1. Verb Phrases (VP): Representing the physical actions of the task (e.g., "place water bottle" or "pick orange").
2. Prepositional Phrases (PP): Representing target spatial arrangements, source locations, or relative object configurations (e.g., "from bottom drawer" or "into top drawer").
The resulting set of language primitives $\{l_i\}_{i=1}^N$ is passed as individual conditional inputs to the diffusion model network during training and compositional inference.
Knowl 3 — Multimodal Instruction Composition in RoboDreamer
model/method
RoboDreamer extends compositional factorization beyond text by jointly conditioning video generation on both a set of language sub-components $L = \{l_i\}_{i=1}^N$ and a set of multimodal goal specifications $M = \{m_j\}_{j=1}^K$ , such as goal images or goal sketches. The joint generative distribution is defined as the normalized product of all component distributions:

$p_\theta(\tau | L, M) \propto \prod_{i=1}^N p_\theta(\tau | l_i)^{\frac{1}{N+K}} \prod_{j=1}^K p_\theta(\tau | m_j)^{\frac{1}{N+K}}$

To train the model to denoise under subsets of multimodal inputs, RoboDreamer minimizes a mean squared error objective over randomly selected subsets $S$ of language primitives $l_{S_i}$ and multimodal conditions $M_{S_j}$ :

$\mathcal{L}_{\text{MSE}} = \left\| \frac{1}{2M} \sum_{i=1}^M \epsilon_\theta(\tau_t, t | l_{S_i}) + \frac{1}{2M} \sum_{j=1}^M \epsilon_\theta(\tau_t, t | M_{S_j}) - \epsilon \right\|^2$

where $\tau_t$ is the noisy video at timestep $t$ corrupted by Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$ , and $M$ is the number of sampled components in the subset.
Knowl 4 — RoboDreamer Compositional Video Diffusion Training
algorithm
RoboDreamer trains a conditional score-based diffusion network $\epsilon_\theta$ to model both individual and composite language conditions using a hybrid subset-denoising objective.
Input: Video diffusion score network $\epsilon_\theta$ , training step count $N_\text{steps}$ , noise schedule $\bar{\alpha}_t$
for step $= 1$ to $N_\text{steps}$ do
    Sample clean video $\tau_0$ and parsed language instruction primitives $L = \{l_i\}_{i=1}^N$
    Sample diffusion timestep $t \sim \text{Uniform}(\{1, \dots, T\})$
    Sample Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$
    Compute noisy video latent $\tau_t = \sqrt{\bar{\alpha}_t} \tau_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$
    Sample a subset $S' \subseteq L$ of $M$ components $\{l_{S'_i}\}_{i=1}^M$
    Compute composite denoising loss: $\mathcal{L}_\text{MSE} = \|\frac{1}{M} \sum_{i=1}^M \epsilon_\theta(\tau_t, t | l_{S'_i}) - \epsilon\|^2$
    Take gradient descent step on $\mathcal{L}_\text{MSE}$ with respect to $\theta$
end for
Knowl 5 — RoboDreamer Compositional Diffusion Inference Algorithm
algorithm
At inference time, RoboDreamer synthesizes video plans for unseen task combinations by aggregating score predictions conditioned on each parsed instruction primitive and applying classifier-free guidance.
Input: Diffusion model $\epsilon_\theta$ , parsed language instructions $L = \{l_i\}_{i=1}^N$ , guidance weight $w$ , noise variance parameters $\alpha_t, \bar{\alpha}_t, \sigma_t$
Sample initial noise video $\tau_T \sim \mathcal{N}(0, I)$
for $t = T$ down to $1$ do
    Compute unconditional score $\epsilon_\text{uncond} \leftarrow \epsilon_\theta(\tau_t, t)$
    Compute composite guided score $\tilde{\epsilon} \leftarrow \epsilon_\text{uncond} + \sum_{i=1}^N w (\epsilon_\theta(\tau_t, t | l_i) - \epsilon_\text{uncond})$
    Sample random noise $z \sim \mathcal{N}(0, I)$ if $t > 1$ , else $z \leftarrow 0$
    Update latent video: $\tau_{t-1} \leftarrow \frac{1}{\sqrt{\alpha_t}} \left( \tau_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \tilde{\epsilon} \right) + \sigma_t z$
end for
Output: Synthesized video plan $\tau_0$
Knowl 6 — Closed-Loop Robotic Planning via Video Plans and Inverse Dynamics
model/method
RoboDreamer converts robotic task execution into a two-stage decision process using the Unified Predictive Decision Process (UPDP) framework $G = \langle \mathcal{X}, \mathcal{C}, H, \rho \rangle$ , where $\mathcal{X}$ is the image observation space, $\mathcal{C}$ is the text task instruction space, $H$ is the horizon length, and $\rho(\cdot | x_0, c) : \mathcal{X} \times \mathcal{C} \to \Delta(\mathcal{X}^H)$ is the conditional video generator.
1. Plan Generation: Given the current observation $x_0$ and language command $c$ , the video diffusion model generates a predicted future video rollout $\tau = [x_1, x_2, \dots, x_H]$ .
2. Action Inference: An inverse dynamics model $\pi(a_t | x_t, x_{t+1}, s_t)$ —implemented as a ResNet-18 backbone followed by an MLP—takes consecutive generated video frames $x_t, x_{t+1}$ and the current robot state $s_t$ to infer low-level robot actions $a_t$ .
3. Closed-Loop Execution: To prevent cascading compounding errors during execution, the robot periodically regenerates fresh video plans from its current observation $x_t$ and resumes action execution.
Knowl 7 — RoboDreamer Architecture and Cascaded Video Diffusion Setup
model/method
The video generation architecture of RoboDreamer is built upon AVDC and Imagen video diffusion models with the following specifications:
- Cascaded Diffusion Pipeline: Video generation utilizes a three-stage cascaded diffusion process that generates base frames at $8 \times 64 \times 64$ resolution and sequentially super-resolves them to $8 \times 128 \times 128$ and $8 \times 256 \times 256$ .
- U-Net Backbone: Consists of 4 ResNet blocks with base channel dimension 128 and channel multipliers $[1, 2, 4, 8]$ . Each block contains spatial-temporal convolution layers and cross-attention layers for instruction conditioning. Temporal attention is added in the last encoder block and the first decoder block.
- Conditioning Mechanisms: Textual instructions are encoded using a frozen T5-XXL text encoder. Goal images and goal sketches are encoded using the pre-trained VQ-VAE encoder from Stable Diffusion. All condition embeddings are unified using PerceiverSampler before being injected via cross-attention into the U-Net ResNet blocks.
- Temporal Consistency: The initial conditioning frame $x_0$ is concatenated channel-wise to all noisy frames prior to input into the U-Net.

Knowl 8 — Zero-Shot Video Generation Evaluation on Seen and Unseen RT-1 Instructions

data/table

RoboDreamer was evaluated on video generation quality and task alignment on the RT-1 robotic manipulation dataset (trained on $\sim 70\text{k}$ demonstrations across 500 tasks). Human evaluators (at least 3 raters per sample over 128 samples across $>20$ text prompts) rated task completion on a binary scale ( $1 = \text{executable and solves task}$ , $0 = \text{unreasonable or fails}$ ).

RoboDreamer outperforms monolithic text-to-video baselines (AVDC and HiP) and an unparsed variant of RoboDreamer, particularly on unseen task instruction combinations where it achieves an $81.3\%$ success score compared to $46.9\%$ for AVDC.

Knowl 9 — Multimodal-Conditioned Video Generation Performance

data/table

Video synthesis performance was evaluated under multimodal conditioning settings on the RT-1 benchmark: text only ( $t$ ), text + sketch ( $t+s$ ), and text + goal image ( $t+i$ ). Metrics include human evaluation score (Human $\uparrow$ ), Fréchet Video Distance (FVD $\downarrow$ ), and target object bounding-box Intersection-over-Union (IMO $\uparrow$ ) detected using GroundingDINO.

Model	Human (%) $\uparrow$	FVD $\downarrow$	IMO $\uparrow$
AVDC	46.9	517.1	–
RoboDreamer (t)	81.3	487.8	63.5
RoboDreamer (t+s)	94.7	454.7	72.5
RoboDreamer (t+i)	95.8	444.3	78.1

Adding visual goal sketches ( $t+s$ ) or goal images ( $t+i$ ) substantially improves video visual quality (lower FVD), human execution plausibility (up to $95.8\%$ ), and precise object spatial alignment (IMO increasing from $63.5$ to $78.1$ ).

Knowl 10 — Robotic Planning Task Success Rates on RLBench

data/table

RoboDreamer was deployed for closed-loop robotic planning on RLBench using a 7-DoF Franka Panda manipulator receiving single front-camera RGB observations without goal images.

Model	lamp off	lamp on	stack blocks	lift block	take shoes	close box	Average (%)
Image-BC	60.1	47.0	0.0	0.0	0.0	82.4	31.6
Hiveformer	81.2	53.2	10.6	28.2	1.0	90.8	44.2
UniPi	70.6	47.1	7.1	23.3	3.8	94.1	41.0
RoboDreamer	96.3	51.9	18.5	22.2	10.5	96.3	49.3

RoboDreamer achieves an average success rate of $49.3\%$ , outperforming Image-BC ( $31.6\%$ ), Hiveformer ( $44.2\%$ ), and UniPi ( $41.0\%$ ). It achieves notable gains on long-horizon multi-step manipulation tasks such as stack blocks ( $18.5\%$ ) and take shoes ( $10.5\%$ ).

Knowl 11 — Limitations of RoboDreamer
limitation
RoboDreamer has three primary documented limitations:
1. Single-Camera Observation Constraint: The model operates only on single-camera RGB inputs and does not integrate multi-camera observations or explicit 3D inductive biases, restricting performance on tasks requiring detailed 3D spatial geometry.
2. Visual Generalization to Out-of-Distribution Real Environments: The model exhibits reduced generalization when deployed on diverse real-world images that deviate substantially from robotics training datasets.
3. Moving-Camera Scenarios: Video diffusion generation fidelity deteriorates significantly under moving camera settings, necessitating additional camera stabilization or motion conditioning mechanisms.

Coverage note — None was omitted; the knowls comprehensively cover the theoretical formulations, training and inference algorithms, architecture and setup details, multimodal conditioning extensions, empirical evaluations on RT-1 and RLBench, and documented limitations.

References

1.Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
2.Ajay, A., Han, S., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L., Srivastava, A., and Agrawal, P. Compositional foundation models for hierarchical planning. arXiv preprint arXiv:2309.08587, 2023.
3.Bar, A., Herzig, R., Wang, X., Rohrbach, A., Chechik, G., Darrell, T., and Globerson, A. Compositional video synthesis with action graphs. arXiv preprint arXiv:2006.15327, 2020.
4.Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
5.Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
6.Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato, M. Residual energy-based models for text generation. arXiv preprint arXiv:2004.11714, 2020.
7.Du, Y., Li, S., Sharma, Y., Tenenbaum, J., and Mordatch, I. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34: 15608–15620, 2021.
8.Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International Conference on Machine Learning, pp. 8489–8510. PMLR, 2023a.
9.Du, Y., Yang, M., Dai, B., Dai, H., Nachum, O., Tenenbaum, J. B., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111, 2023b.
10.Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023.
11.Gkanatsios, N., Jain, A., Xian, Z., Zhang, Y., Atkeson, C., and Fragkiadaki, K. Energy-based models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304.14391, 2023.
12.Guhur, P.-L., Chen, S., Pinel, R. G., Tapaswi, M., Laptev, I., and Schmid, C. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning, pp. 175–187. PMLR, 2023.
13.Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
14.Hu, H., Chan, K. C., Su, Y.-C., Chen, W., Li, Y., Sohn, K., Zhao, Y., Ben, X., Gong, B., Cohen, W., et al. Instruct-imagen: Image generation with multi-modal instruction. arXiv preprint arXiv:2401.01952, 2024.
15.Huang, S., Wang, Z., Li, P., Jia, B., Liu, T., Zhu, Y., Liang, W., and Zhu, S.-C. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750–16761, 2023.
16.Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
17.James, S., Ma, Z., Arrojo, D. R., and Davison, A. J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020.
18.Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022.
19.Kitaev, N., Cao, S., and Klein, D. Multilingual constituency parsing with self-attention and pre-training. arXiv preprint arXiv:1812.11760, 2018.
20.Ko, P.-C., Mao, J., Du, Y., Sun, S.-H., and Tenenbaum, J. B. Learning to act from actionless videos through dense correspondences, 2023.
21.Liang, Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., and Luo, P. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, 2023.
22.Liu, G., Feng, Z., Gao, Y., Yang, Z., Liang, X., Bao, J., He, X., Cui, S., Li, Z., and Hu, Z. Composable text controls in latent space with ODEs. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 16543–16570, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.1030. URL https://aclanthology.org/2023.emnlp-main.1030.
23.Liu, N., Li, S., Du, Y., Tenenbaum, J., and Torralba, A. Learning to compose visual relations. Advances in Neural Information Processing Systems, 34:23166–23178, 2021.
24.Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423–439. Springer, 2022.
25.Liu, W., Du, Y., Hermans, T., Chernova, S., and Paxton, C. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects. In RSS 2023, 2023b.
26.Nie, W., Vahdat, A., and Anandkumar, A. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34:13497–13510, 2021.
27.Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S. V., Tan, S. Z., Momennejad, I., Hofmann, K., et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677, 2023.
28.Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
29.Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022.
30.Shi, C., Ni, H., Li, K., Han, S., Liang, M., and Min, M. R. Exploring compositional visual generation with latent classifier guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 853–862, 2023.
31.Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
32.Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022.
33.Yang, M., Du, Y., Dai, B., Schuurmans, D., Tenenbaum, J. B., and Abbeel, P. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872, 2023a.
34.Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuurmans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023b.
35.Yu, W., Chen, W., Yin, S., Easterbrook, S., and Garg, A. Modular action concept grounding in semantic video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3605–3614, 2022.
36.Zhang, E., Lu, Y., Wang, W. Y., and Zhang, A. Lad: Language augmented diffusion for reinforcement learning. In Second Workshop on Language and Reinforcement Learning, 2022.
37.Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847, 2023.
38.Zhou, S., Du, Y., Zhang, S., Xu, M., Shen, Y., Xiao, W., Yeung, D.-Y., and Gan, C. Adaptive online replanning with diffusion models. arXiv preprint arXiv:2310.09629, 2023.

Citation

MLA

Zhou, S., et al. “RoboDreamer: Learning Compositional World Models for Robot Imagination”. arXiv, 2024, http://arxiv.org/abs/2404.12377v1.

APA

Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., & Gan, C. (2024). RoboDreamer: Learning Compositional World Models for Robot Imagination. arXiv. http://arxiv.org/abs/2404.12377v1

Chicago

Zhou, S., Y. Du, J. Chen, Y. Li, D.-Y. Yeung, and C. Gan. 2024. “RoboDreamer: Learning Compositional World Models for Robot Imagination”. arXiv. http://arxiv.org/abs/2404.12377v1.

Harvard

Zhou, S. et al. (2024) “RoboDreamer: Learning Compositional World Models for Robot Imagination”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2404.12377v1.

Vancouver

1. Zhou S, Du Y, Chen J, Li Y, Yeung D-Y, Gan C (2024) RoboDreamer: Learning Compositional World Models for Robot Imagination. arXiv

BibTeX

@article{zhou2024robodreamer,
  title = {RoboDreamer: Learning Compositional World Models for Robot Imagination},
  author = {Zhou, Siyuan and Du, Yilun and Chen, Jiaben and Li, Yandong and Yeung, Dit-Yan and Gan, Chuang},
  year = {2024},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2404.12377v1},
  eprint = {2404.12377}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Model	Seen (%)	Unseen (%)
AVDC	63.1	46.9
HiP	70.3	50.1
RoboDreamer w/o parsing	85.5	68.8
RoboDreamer	90.1	81.3

RoboDreamer: Learning Compositional World Models for Robot Imagination

Abstract

Topics

Table of Contents

Knowls

Knowl 1 — RoboDreamer Compositional Video Generation Formulation

Knowl 2 — Language Instruction Parsing into Action and Spatial Primitives

Knowl 3 — Multimodal Instruction Composition in RoboDreamer

Knowl 4 — RoboDreamer Compositional Video Diffusion Training

Knowl 5 — RoboDreamer Compositional Diffusion Inference Algorithm

Knowl 6 — Closed-Loop Robotic Planning via Video Plans and Inverse Dynamics

Knowl 7 — RoboDreamer Architecture and Cascaded Video Diffusion Setup

Knowl 8 — Zero-Shot Video Generation Evaluation on Seen and Unseen RT-1 Instructions

Knowl 9 — Multimodal-Conditioned Video Generation Performance

Knowl 10 — Robotic Planning Task Success Rates on RLBench

Knowl 11 — Limitations of RoboDreamer

References

Citation

BibTeX

Access the Paper