Back

MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Pengyang LingJiazi BuPan Zhang 0001Xiaoyi DongYuhang ZangTong WuHuaian ChenJiaqi Wang 0003Yi Jin 0002

article2025ICLR103 citations

Proposes MotionClone, a training-free framework that transfers camera and object motions from reference videos to text-to-video and image-to-video generation by extracting motion guidance from sparse temporal attention maps in a single denoising step.

Shanghai Artificial Intelligence LaboratoryShanghai Jiao Tong UniversityThe Chinese University of Hong KongUniversity of Science and Technology of Chinamotion synthesisvideo streams
arXiv: 2406.05338LPengYang/MotionClone
Cover for MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Abstract

Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

Topics

motion synthesisvideo streams

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 2.1 Text-to-video diffusion models
  • 2.2 Controllable video generation
  • 2.3 Attention feature control
  • 3 MotionClone
  • 3.1 Preliminaries
  • 3.2 Observation
  • 3.3 Motion Representation
  • 3.4 Motion Guidance
  • 4 Experiments
  • 4.1 Implementation details
  • 4.2 Experimental setup
  • 4.3 Qualitative Comparison
  • 4.4 Quantitative comparison
  • 4.5 Versatile application
  • 4.6 Ablation and Analysis
  • 4.7 Limitation
  • 5 Conclusion
  • References
  • A Appendix
  • A.1 Baseline description
  • A.2 More generated results
  • A.3 Broader Impact

Citation

MLA
Ling, P., et al. “MotionClone: Training-Free Motion Cloning for Controllable Video Generation”. arXiv, 2024, http://arxiv.org/abs/2406.05338v6.
APA
Ling, P., Bu, J., Zhang, P., Dong, X., Zang, Y., Wu, T., Chen, H., Wang, J., & Jin, Y. (2024). MotionClone: Training-Free Motion Cloning for Controllable Video Generation. arXiv. http://arxiv.org/abs/2406.05338v6
Chicago
Ling, P., J. Bu, P. Zhang, et al. 2024. “MotionClone: Training-Free Motion Cloning for Controllable Video Generation”. arXiv. http://arxiv.org/abs/2406.05338v6.
Harvard
Ling, P. et al. (2024) “MotionClone: Training-Free Motion Cloning for Controllable Video Generation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2406.05338v6.
Vancouver
1. Ling P, Bu J, Zhang P, Dong X, Zang Y, Wu T, Chen H, Wang J, Jin Y (2024) MotionClone: Training-Free Motion Cloning for Controllable Video Generation. arXiv

BibTeX

@article{ling2024motionclone,
  title = {MotionClone: Training-Free Motion Cloning for Controllable Video Generation},
  author = {Ling, Pengyang and Bu, Jiazi and Zhang, Pan and Dong, Xiaoyi and Zang, Yuhang and Wu, Tong and Chen, Huaian and Wang, Jiaqi and Jin, Yi},
  year = {2024},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2406.05338v6},
  eprint = {2406.05338}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors