Human Motion Diffusion Model

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or and Amit H. Bermano Tel Aviv University, Israel

[email protected]

Abstract

Natural and expressive human motion generation is the holy grail of computer animation. It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain. MDM is transformer-based, combining insights from motion generation literature. A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion step. This facilitates the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss. As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion and action-to-motion[^1]. https://guytevet.github.io/mdm-page/.

[^1]: Our code can be found at https://github.com/GuyTevet/motion-diffusion-model.

Executive Summary: Generating realistic, expressive, and diverse human animations from high-level descriptions is a fundamental challenge across gaming, visual effects, and robotics. Existing neural motion frameworks frequently struggle with physical artifacts, limited diversity, and an inability to handle ambiguous prompts where a single phrase can describe many distinct movements. Generative diffusion techniques handle this type of complex mapping well in image synthesis, but standard implementations require massive computing resources and remain difficult to adapt to structured motion data.

The article demonstrates and evaluates the Motion Diffusion Model, a diffusion-based framework designed to generate, edit, and complete three-dimensional human motion efficiently across diverse input modalities. The main objective is to establish a generic, lightweight generative pipeline that balances sample fidelity and motion diversity while enforcing physical plausibility.

The authors implemented a transformer encoder-only architecture that models motion sequences directly rather than relying on heavy image-oriented backbones. Unlike typical diffusion networks that predict additive noise, the framework directly predicts the clean motion signal at each step, enabling the seamless integration of physical and kinematic constraints such as foot-contact and velocity regularizers. The model was trained and evaluated on several standard benchmarks, including HumanML3D and KIT for text-to-motion generation, as well as HumanAct12 and UESTC for action-to-motion generation, supported by side-by-side human perceptual evaluations.

The findings establish that the framework outperforms existing state-of-the-art baselines across multiple benchmarks. In text-driven generation, it achieved superior distribution fidelity and diversity scores, with human study participants preferring its generated animations over real ground-truth motion captures 42.3% of the time and favoring it over competing generative models in 54.8% to 90.4% of pairwise comparisons. In action-driven tasks, the system established top-tier accuracy and distribution matching on both tested benchmarks, successfully eliminating common motion anomalies such as foot sliding and jitter. Furthermore, the framework operates without requiring specialized task retraining to perform spatial body-part editing and temporal motion completion, all while training in roughly three days on a single mid-range graphics processing unit.

These results demonstrate that high-quality, editable motion synthesis does not require prohibitive computational budgets or restrictive latent-space assumptions. For production studios and animation pipelines, this technology offers substantial reductions in manual cleanup and prototyping time. The architecture reliably supports diverse motion generation and controllable local edits without necessitating custom pipeline engineering for each distinct task.

Organizations evaluating generative animation tools should consider adopting transformer-based diffusion pipelines that natively enforce kinematic constraints for prototyping, in-betweening, and automated motion completion. When balancing diversity against prompt fidelity, teams should utilize classifier-free guidance tuning to achieve the optimal trade-off for their specific creative workflows.

The primary operational limitation is inference speed: because the diffusion process requires iterative sequential sampling across 1,000 steps, generating an animation requires approximately one minute rather than real-time execution. While this latency is well suited for pre-production, prototyping, and offline animation, real-time interactive applications will require further optimization or faster sampling algorithms before direct deployment.

1. Introduction

Section Summary: Human motion generation from text is challenging because of the enormous range of possible movements, the high cost of good data, and the many-to-many nature of mapping language descriptions to actions. Prior methods using auto-encoders or VAEs tend to restrict output variety, whereas diffusion models can capture richer distributions without such assumptions. The paper introduces a lightweight, transformer-based Motion Diffusion Model that incorporates geometric losses, supports multiple conditioning types, and delivers state-of-the-art quality and diversity across text-to-motion, action-to-motion, and editing tasks while training efficiently on a single GPU.

Human motion generation is a fundamental task in computer animation, with applications spanning from gaming to robotics. It is a challenging field, due to several reasons, including the vast span of possible motions, and the difficulty and cost of acquiring high quality data. For the recently emerging text-to-motion setting, where motion is generated from natural language, another inherent problem is data labeling. For example, the label "kick" could refer to a soccer kick, as well as a Karate one. At the same time, given a specific kick there are many ways to describe it, from how it is performed to the emotions it conveys, constituting a many-to-many problem. Current approaches have shown success in the field, demonstrating plausible mapping from text to motion ([1, 2, 3]). All these approaches, however, still limit the learned distribution since they mainly employ auto-encoders or VAEs ([4]) (implying a one-to-one mapping or a normal latent distribution respectively). In this aspect, diffusion models are a better candidate for human motion generation, as they are free from assumptions on the target distribution, and are known for expressing well the many-to-many distribution matching problem we have described.

Diffusion models ([5, 6, 7]) are a generative approach that is gaining significant attention in the computer vision and graphics community. When trained for conditioned generation, recent diffusion models ([8, 9]) have shown breakthroughs in terms of image quality and semantics. The competence of these models have also been shown for other domains, including videos ([10]), and 3D point clouds ([11]). The problem with such models, however, is that they are notoriously resource demanding and challenging to control.

In this paper, we introduce Motion Diffusion Model (MDM) — a carefully adapted diffusion based generative model for the human motion domain. Being diffusion-based, MDM gains from the native aforementioned many-to-many expression of the domain, as evidenced by the resulting motion quality and diversity (Figure 1). In addition, MDM combines insights already well established in the motion generation domain, helping it be significantly more lightweight and controllable.

First, instead of the ubiquitous U-net ([12]) backbone, MDM is transformer-based. As we demonstrate, our architecture (Figure 2) is lightweight and better fits the temporal and non-spatial nature of motion data (represented as a collection of joints). A large volume of motion generation research is devoted to learning using geometric losses ([13, 14, 15]). Some, for example, regulate the velocity of the motion ([16]) to prevent jitter, or specifically consider foot sliding using dedicated terms ([17]). Consistently with these works, we show that applying geometric losses in the diffusion setting improves generation.

The MDM framework has a generic design enabling different forms of conditioning. We showcase three tasks: text-to-motion, action-to-motion, and unconditioned generation. We train the model in a classifier-free manner ([18]), which enables trading-off diversity to fidelity, and sampling both conditionally and unconditionally from the same model. In the text-to-motion task, our model generates coherent motions (Figure 1) that achieve state-of-the-art results on the HumanML3D ([19]) and KIT ([20]) benchmarks. Moreover, our user study shows that human evaluators prefer our generated motions over real motions $42%$ of the time (Figure 4(a)). In action-to-motion, MDM outperforms the state-of-the-art ([21, 16]), even though they were specifically designed for this task, on the common HumanAct12 ([21]) and UESTC ([22]) benchmarks.

Lastly, we also demonstrate completion and editing. By adapting diffusion image-inpainting ([23, 24]), we set a motion prefix and suffix, and use our model to fill in the gap. Doing so under a textual condition guides MDM to fill the gap with a specific motion that still maintains the semantics of the original input. By performing inpainting in the joints space rather than temporally, we also demonstrate the semantic editing of specific body parts, without changing the others (Figure 3).

Overall, we introduce Motion Diffusion Model, a motion framework that achieves state-of-the-art quality in several motion generation tasks, while requiring only about three days of training on a single mid-range GPU. It supports geometric losses, which are non trivial to the diffusion setting, but are crucial to the motion domain, and offers the combination of state-of-the-art generative power with well thought-out domain knowledge.

2. Related Work

Section Summary: Recent research on human motion generation has explored neural networks that create realistic movements from motion capture data, often guided by inputs like partial poses, action labels, audio, or natural language descriptions. For text-to-motion tasks in particular, methods commonly learn a shared latent space linking language and motion using techniques such as autoencoders or variational autoencoders, with some leveraging pretrained models like CLIP to broaden capabilities beyond limited datasets. Separately, diffusion models, which generate samples by reversing a gradual noising process, have advanced image synthesis and are now being adapted for motion tasks through classifier-free conditioning on text embeddings, enabling both generation and localized editing while offering efficiency advantages over concurrent approaches.

2.1 Human Motion Generation

Neural motion generation, learned from motion capture data, can be conditioned by any signal that describes the motion. Many works use parts of the motion itself for guidance. Some predict motion from its prefix poses ([25, 26, 27, 28]). Others ([29, 30, 14, 31]) solve in-betweening and super-resolution tasks using bi-directional GRU ([32]) and Transformer ([33]) architectures. [34] use auto-encoder to learn motion latent representation, then utilize it to edit and control motion with spatial constraints such as root trajectory and bone lengths. Motion can be controlled with a high-level guidance given from action class ([21, 16, 35]), audio ([36, 37]) and natural language ([3, 1]). In most cases authors suggests a dedicated approach to map each conditioning domain into motion.

In recent years, the leading approach for the Text-to-Motion task is to learn a shared latent space for language and motion. JL2P ([3]) learns the KIT motion-language dataset ([20]) with an auto-encoder, limiting one-to-one mapping from text to motion. TEMOS ([1]) and T2M ([19]) suggest using a VAE ([4]) to map a text prompt into a normal distribution in latent space. Recently, MotionCLIP ([2]) leverages the shared text-image latent space learned by CLIP ([38]) to expand text-to-motion out of the data limitations and enabled latent space editing.

The human motion manifold can also be learned without labels, as shown by [34], V-Poser ([39]), and more recently the dedicated MoDi architecture ([40]). We show that our model is capable for such an unsupervised setting as well.

2.2 Diffusion Generative Models

Diffusion models ([5, 6]) are a class of neural generative models, based on the stochastic diffusion process as it is modeled in Thermodynamics. In this setting, a sample from the data distribution is gradually noised by the diffusion process. Then, a neural model learns the reverse process of gradually denoising the sample. Sampling the learned data distribution is done by denoising a pure initial noise. [7] and [41] further developed the practices for image generation applications. For conditioned generation, [42], introduced classifier-guided diffusion, which was later on adapted by GLIDE ([43]) to enable conditioning over CLIP textual representations. The Classifier-Free Guidance approach [18] enables conditioning while trading-off fidelity and diversity, and achieves better results ([43]). In this paper, we implement text-to-motion by conditioning on CLIP in a classifier-free manner, similarly to text-to-image ([8, 9]). Local editing of images is typically defined as an inpainting problem, where a part of the image is constant, and the inpainted part is denoised by the model, possibly under some condition ([23, 24]). We adapt this technique to edit motion's specific body parts or temporal intervals (in-betweening) according to an optional condition.

More recently, concurrent to this work, [44] and [45] have suggested diffusion models for motion generation. Our work requires significantly fewer GPU resources and makes design choices that enable geometric losses, which improve results.

3. Motion Diffusion Model

Section Summary: The Motion Diffusion Model generates realistic human motion sequences of any length by starting with random noise and gradually refining it into coherent movements, guided by inputs such as text descriptions, audio, or action labels. It employs a transformer network that directly predicts the clean motion at each step rather than noise, and is trained with a basic reconstruction loss plus extra geometric penalties that enforce natural joint positions, foot placement, and smooth velocities to reduce artifacts like foot sliding. Sampling proceeds iteratively from full noise back to a clean sequence, with optional classifier-free guidance to balance quality and variety, while also supporting targeted edits like filling in missing frames or modifying specific body parts during generation.

An overview of our method is described in Figure 2. Our goal is to synthesize a human motion $x^{1:N}$ of length $N$ given an arbitrary condition $c$. This condition can be any real-world signal that will dictate the synthesis, such as audio ([36, 37]), natural language (text-to-motion) ([2, 19]) or a discrete class (action-to-motion) ([21, 16]). In addition, unconditioned motion generation is also possible, which we denote as the null condition $c=\emptyset$. The generated motion $x^{1:N}={x^i}_{i=1}^{N}$ is a sequences of human poses represented by either joint rotations or positions $x^i\in \mathbb{R}^{J\times D}$, where $J$ is the number of joints and $D$ is the dimension of the joint representation. MDM can accept motion represented by either locations, rotations, or both (see Section 4).

Framework. Diffusion is modeled as a Markov noising process, ${x^{1:N}{t}}{t=0}^T$, where $x^{1:N}_{0}$ is drawn from the data distribution and

$ { q(x^{1:N}{t} | x^{1:N}{t-1}) = \mathcal{N}(\sqrt{\alpha_{t}}x^{1:N}{t-1},(1-\alpha{t})I), } $

where $\alpha_{t} \in (0,1)$ are constant hyper-parameters. When $\alpha_{t}$ is small enough, we can approximate $x^{1:N}{T} \sim \mathcal{N}(0,I)$. From here on we use $x{t}$ to denote the full sequence at noising step $t$.

In our context, conditioned motion synthesis models the distribution $p(x_{0}|c)$ as the reversed diffusion process of gradually cleaning $x_{T}$. Instead of predicting $\epsilon_t$ as formulated by [7], we follow [8] and predict the signal itself, i.e., $\hat{x}_{0} = G(x_t, t, c)$ with the simple objective ([7]),

$ { \mathcal{L}\text{simple} = E{x_0 \sim q(x_0|c), t \sim [1,T]}[| x_0 - G(x_t, t, c)|_2^2] } $

Geometric losses. In the motion domain, generative networks are standardly regularized using geometric losses [16, 17]. These losses enforce physical properties and prevent artifacts, encouraging natural and coherent motion. In this work we experiment with three common geometric losses that regulate (1) positions (in case we predict rotations), (2) foot contact, and (3) velocities.

$ { \mathcal{L}\text{pos} = \frac{1}{N} \sum{i =1}^{N} | FK(x^i_0) - FK(\hat{x}^i_0)|_{2}^{2}, } $

$ { \mathcal{L}\text{foot} = \frac{1}{N-1} \sum{i =1}^{N-1} | (FK(\hat{x}^{i+1}0) - FK(\hat{x}^i_0)) \cdot f_i|{2}^{2}, } $

$ { \mathcal{L}\text{vel} = \frac{1}{N-1} \sum{i =1}^{N-1} | (x^{i+1}_0 - x^i_0) - (\hat{x}^{i+1}0 - \hat{x}^i_0)|{2}^{2} } $

In case we predict joint rotations, $FK(\cdot)$ denotes the forward kinematic function converting joint rotations into joint positions (otherwise, it denotes the identity function). $f_i \in {0,1}^J$ is the binary foot contact mask for each frame $i$. Relevant only to feet, it indicates whether they touch the ground, and are set according to binary ground truth data ([17]). In essence, it mitigates the foot-sliding effect by nullifying velocities when touching the ground.

Overall, our training loss is

$ { \mathcal{L} = \mathcal{L}\text{simple} + \lambda\text{pos}\mathcal{L}\text{pos} + \lambda\text{vel}\mathcal{L}\text{vel} + \lambda\text{foot}\mathcal{L}_\text{foot}. } $

Model. Our model is illustrated in Figure 2. We implement $G$ with a straightforward transformer ([33]) encoder-only architecture. The transformer architecture is temporally aware, enabling learning arbitrary length motions, and is well-proven for the motion domain ([16, 31, 46]). The noise time-step $t$ and the condition code $c$ are each projected to the transformer dimension by separate feed-forward networks, then summed to yield the token $z_{tk}$. Each frame of the noised input $x_t$ is linearly projected into the transformer dimension and summed with a standard positional embedding. $z_{tk}$ and the projected frames are then fed to the encoder. Excluding the first output token (corresponding to $z_{tk}$), the encoder result is projected back to the original motion dimensions, and serves as the prediction $\hat{x}_{0}$. We implement text-to-motion by encoding the text prompt to $c$ with CLIP ([38]) text encoder, and action-to-motion with learned embeddings per class.

Sampling from $p(x_{0}|c)$ is done in an iterative manner, according to [7]. In every time step $t$ we predict the clean sample $\hat{x}{0} = G(x_t, t, c)$ and noise it back to $x{t-1}$. This is repeated from $t=T$ until $x_0$ is achieved (Figure 2 right). We train our model $G$ using classifier-free guidance ([18]). In practice, $G$ learns both the conditioned and the unconditioned distributions by randomly setting $c=\emptyset$ for $10%$ of the samples, such that $G(x_t, t, \emptyset)$ approximates $p(x_0)$. Then, when sampling $G$ we can trade-off diversity and fidelity by interpolating or even extrapolating the two variants using $s$:

$ { G_s(x_t, t, c) = G(x_t, t, \emptyset) + s \cdot (G(x_t, t, c) - G(x_t, t, \emptyset)) } $

Editing. We enable motion in-betweening in the temporal domain, and body part editing in the spatial domain, by adapting diffusion inpainting to motion data. Editing is done only during sampling, without any training involved. Given a subset of the motion sequence inputs, when sampling the model (Figure 2 right), at each iteration we overwrite $\hat{x}_{0}$ with the input part of the motion. This encourages the generation to remain coherent to original input, while completing the missing parts. In the temporal setting, the prefix and suffix frames of the motion sequence are the input, and we solve a motion in-betweening problem ([14]). Editing can be done either conditionally or unconditionally (by setting $c=\emptyset$). In the spatial setting, we show that body parts can be re-synthesized according to a condition $c$ while keeping the rest intact, through the use of the same completion technique.

4. Experiments

Section Summary: The experiments section describes testing the MDM diffusion model on core motion generation tasks including text-to-motion, action-to-motion, and unconditional sampling, along with extensions for motion in-betweening and text-guided body-part editing via inpainting. Evaluations used established benchmarks such as KIT and HumanML3D, with comparisons across different neural backbones, standard metrics for quality and diversity, and a user study. Models were trained for roughly three days each on a single consumer GPU, achieving strong results that matched or exceeded prior state-of-the-art approaches.

We implement MDM for three motion generation tasks: Text-to-Motion(Section 4.1), Action-to-Motion(Section 4.2) and unconditioned generation(Section 5.2. Each sub-section reviews the data and metrics of the used benchmarks, provides implementation details, and presents qualitative and quantitative results. Then, we show implementations of motion in-betweening (both conditioned and unconditioned) and body-part editing by adapting diffusion inpainting to motion (Section 5.1). Our models have been trained with $T=1000$ noising steps and a cosine noise schedule. All of them have been trained on a single NVIDIA GeForce RTX 2080 Ti GPU for a period of about $3$ days.

4.1 Text-to-Motion

Text-to-motion is the task of generating motion given an input text prompt. The output motion is expected to be both implementing the textual description, and a valid sample from the data distribution (i.e. adhering to general human abilities and the rules of physics). In addition, for each text prompt, we also expect a distribution of motions matching it, rather than just a single result. We evaluate our model using two leading benchmarks - KIT ([20]) and HumanML3D ([19]), over the set of metrics suggested by [19]: R-precision and Multimodal-Dist measure the relevancy of the generated motions to the input prompts, FID measures the dissimilarity between the generated and ground truth distributions (in latent space), Diversity measures the variability in the resulting motion distribution, and MultiModality is the average variance given a single text prompt. For the full implementation of the metrics, please refer to [19]. We use HumanML3D as a platform to compare different backbones of our model, discovering that the diffusion framework is relatively agnostic to this attribute. In addition, we conduct a user study comparing our model to current art and ground truth motions.

Data. HumanML3D is a recent dataset, textually re-annotating motion capture from the AMASS ([47]) and HumanAct12 ([21]) collections. It contains $14,616$ motions annotated by $44,970$ textual descriptions. In addition, it suggests a redundant data representation including a concatenation of root velocity, joint positions, joint velocities, joint rotations and the foot contact binary labels. We also use in this section the same representation for the KIT dataset, brought by the same publishers. Although limited in the number ($3,911$) and the diversity of samples, most of the text-to-motion research is based on KIT, hence we view it as important to evaluate using it as well.

Implementation. In addition to our Transformer encoder-only backbone (Section 3), we experiment MDM with three more backbones: (1) Transformer decoder injects $z_{tk}$ through the cross-attention layer, instead of as an input token. (2) Transformer decoder + input token, where $z_{tk}$ is injected both ways, and (3) GRU ([32]) concatenate $z_{tk}$ to each input frame (Table 1). Our models were trained with batch size $64$, $8$ layers (except GRU that was optimal at $2$), and latent dimension $512$. To encode the text we use a frozen CLIP-ViT-B/32 model. Each model was trained for $500K$ steps, afterwhich a checkpoint was chosen that minimizes the FID metric to be reported. Since foot contact and joint locations are explicitly represented in HumanML3D, we don't apply geometric losses in this section. We evaluate our models with guidance-scale $s=2.5$ which provides a diversity-fidelity sweet spot (Figure 4).

Quantitative evaluation. We evaluate and compare our models to current art (JL2P [3], Text2Gesture ([48]), and T2M ([19])) with the metrics suggested by [19]. As can be seen, MDM achieves state-of-the-art results in FID, Diversity, and MultiModality, indicating high diversity per input text prompt, and high-quality samples, as can also be seen qualitatively in Figure 1.

User study. We asked 31 users to choose between MDM and state-of-the-art works in a side-by-side view, with both samples generated from the same text prompt randomly sampled from the KIT test set. We repeated this process with 10 samples per model and 10 repetitions per sample. This user study enabled a comparison with the recent TEMOS model ([1]), which was not included in the HumanML3D benchmark. Figure 4 shows that most of the time, MDM was preferred over the compared models, and even preferred over ground truth samples in $42.3%$ of the cases.

::: {caption="Table 1: Quantitative results on the HumanML3D test set. All methods use the real motion length from the ground truth. ' $\rightarrow$ ' means results are better if the metric is closer to the real distribution. We run all the evaluation 20 times (except MultiModality runs 5 times) and ± indicates the 95% confidence interval. Bold indicates best result."}

:::

::: {caption="Table 2: Quantitative results on the KIT test set."}

:::

::: {caption="Table 3: Evaluation of action-to-motion on the HumanAct12 dataset. Our model leads the board in three out of four metrics. Ground-truth evaluation results are slightly different for each of the works, due to implementation differences, such as python package versions. It is important to assess the diversity and multimodality of each model using its own ground-truth results, as they are measured by their distance from GT. We show the GT metrics measured by our model and by the leading compared work, INR ([35]). Bold indicates best result, $\underline{\text{underline}}$ indicates second best, ± indicates 95% confidence interval, $\rightarrow$ indicates that closer to real is better."}

:::

::: {caption="Table 4: Evaluation of action-to-motion on the UESTC dataset. The performance improvement with our model shows a clear gap from state-of-the-art. Bold indicates best result, $\underline{\text{underline}}$ indicates second best, ± indicates 95% confidence interval, $\rightarrow$ indicates that closer to real is better."}

:::

4.2 Action-to-Motion

Action-to-motion is the task of generating motion given an input action class, represented by a scalar. The output motion should faithfully animate the input action, and at the same time be natural and reflect the distribution of the dataset on which the model is trained. Two dataset are commonly used to evaluate action-to-motion models: HumanAct12 ([21]) and UESTC ([22]). We evaluate our model using the set of metrics suggested by [21], namely Fréchet Inception Distance (FID), action recognition accuracy, diversity and multimodality. The combination of these metrics makes a good measure of the realism and diversity of generated motions.

Data. HumanAct12 ([21]) offers approximately 1200 motion clips, organized into 12 action categories, with 47 to 218 samples per label. UESTC ([22]) consists of 40 action classes, 40 subjects and 25K samples, and is split to train and test. We adhere to the cross-subject testing protocol used by current works, with 225-345 samples per action class. For both datasets we use the sequences provided by [16].

Implementation. The implementation presented in Figure 2 holds for all the variations of our work. In the case of action-to-motion, the only change would be the substitution of the text embedding by an action embedding. Since action is represented by a scalar, its embedding is fairly simple; each input action class scalar is converted into a learned embedding of the transformer dimension.

The experiments have been run with batch size $64$, a latent dimension of $512$, and an encoder-transformer architecture. Training on HumanAct12 and UESTC has been carried out for $750K$ and $2M$ steps respectively. In our tables we display the evaluation of the checkpoint that minimizes the FID metric.

Quantitative evaluation. Table 3 and Table 4 reflect MDM's performance on the HumanAct12 and UESTC datasets respectively. We conduct 20 evaluations, with 1000 samples in each, and report their average and a 95% confidence interval. We test two variations, with and without foot contact loss. Our model leads the board for both datasets. The variation with no foot contact loss attains slightly better results; nevertheless, as shown in our supplementary video, the contribution of foot contact loss to the quality of results is important, and without it we witness artifacts such as shakiness and unnatural gestures.

5. Additional Applications

Section Summary: The section describes two extra uses for the motion generation model. One involves editing motions by locking certain parts in place, such as the start and end frames or selected body joints, and letting the model fill in the rest to create smooth, consistent results. The other explores generating motions without any labels or conditions at all, showing that the approach can still produce plausible outputs on unlabeled data and remains competitive with specialized methods while also supporting optional controls and edits.

5.1 Motion Editing

In this section we implement two motion editing applications - in-betweening and body part editing, both using the same approach in the temporal and spatial domains correspondingly. For in-betweening, we fix the first and last $25%$ of the motion, leaving the model to generate the remaining $50%$ in the middle. For body part editing, we fix the joints we don't want to edit and leave the model to generate the rest. In particular, we experiment with editing the upper body joints only. In Figure 3 we show that in both cases, using the method described in Section 3 generates smooth motions that adhere both to the fixed part of the motion and the condition (if one was given).

\begin{tabular}{lcccccc}
  \toprule

  \multirow{2}{2cm}{Method} & 
  \multirow{2}{1.5cm}{FID$\downarrow$} &
  \multirow{2}{2cm}{KID$\downarrow$} &
  \multirow{2}{2cm}{Precision$\uparrow$ Recall$\uparrow$} &
  \multirow{2}{2cm}{Multimodality$\uparrow$} \\
  \\

  \midrule

  ACTOR [16] & 48.80 & 0.53 & 0.72, 0.74 & 14.10 \\ 

  MoDi [40] & \textbf{13.03} & \textbf{0.12} & \textbf{0.71}, \textbf{0.81} & \textbf{17.57}\\ 

  \midrule

  MDM (ours) & 31.92 & 0.36 & 0.66, 0.62 & 17.00\\  

  \bottomrule
  \end{tabular}

5.2 Unconstrained Synthesis

The challenging task of unconstrained synthesis has been studied by only a few ([34, 40]). In the presence of data labeling, e.g., action classes or text description, the labels work as a supervising factor, and facilitate a structured latent space for the training network. The lack of labeling make training more difficult. The human motion field possesses rich unlabeled datasets ([49]), and the ability to train on top of them is an advantage. Daring to test MDM in the challenging unconstrained setting, we follow MoDi([40]) for evaluation. We use the metrics they suggest (FID, KID, precision/recall and multimodality), and run on an unconstrained version of the HumanAct12 ([21]) dataset.

Data. Although annotated, we use HumanAct12 (see Section 4.2) in an unconstrained fashion, ignoring its labels. The choice of HumanAct12 rather than a dataset with no labels (e.g., Mixamo ([49])), is for compatibility with previous publications.

Implementation. Our model uses the same architecture for all forms of conditioning, as well as for the unconstrained setting. The only change to the structure shown in Figure 2, is the removal of the conditional input, such that $z_{tk}$ is composed of the projection of $t$ only. To simulate an unconstrained behavior, ACTOR [16] has been trained by ([40]) with a labeling of one class to all motions.

Quantitative evaluation. The results of our evaluation are shown in Table 5. We demonstrate superiority over works that were not designed for an unconstrained setting, and get closer to MoDi ([40]). MoDi is carefully molded for unconstrained settings, while our work can be applied to any (or no) constrain, and also provides editing capabilities.

6. Discussion

Section Summary: The MDM method offers a flexible way to generate realistic human motions for a range of tasks. Its straightforward design makes the model quick to train and effective at following guidance, while performing well regardless of small changes in its structure. The main drawback is slower generation due to many repeated steps, though this remains manageable in practice, and future work could add finer control over the outputs.

We have presented MDM, a method that lends itself to various human motion generation tasks. MDM is an untypical classifier-free diffusion model, featuring a transformer-encoder backbone, and predicting the signal, rather than the noise. This yields both a lightweight model, that is unburdening to train, and an accurate one, gaining much from the applicable geometric losses. Our experiments show superiority in conditioned generation, but also that this approach is not very sensitive to the choice of architecture.

A notable limitation of the diffusion approach is the long inference time, requiring about $1000$ forward passes for a single result. Since our motion model is small anyway, using dimensions order of magnitude smaller than images, our inference time shifts from less than a second to only about a minute, which is an acceptable compromise. As diffusion models continue to evolve, beside better compute, in the future we would be interested in seeing how to incorporate better control into the generation process, and widen the options for applications even further.

Acknowledgements

Section Summary: The authors thank Rinon Gal for his helpful suggestions and references. They note that the work received funding from the Israel Science Foundation via two grants, as well as support from Len Blavatnik, the Blavatnik family foundation, and Tel Aviv University’s innovation labs.

We thank Rinon Gal for his useful suggestions and references. This research was supported in part by the Israel Science Foundation (grants no. 2492/20 and 3441/21), Len Blavatnik and the Blavatnik family foundation, and The Tel Aviv University Innovation Laboratories (TILabs).

References

Section Summary: This section compiles a list of academic papers and preprints that serve as foundational sources for research on AI-driven human motion synthesis, text-conditioned generation, and related generative modeling techniques. The citations span core methods like diffusion models, variational autoencoders, and neural networks for pose estimation, drawn from major venues in computer vision and machine learning. They also include supporting works on datasets, motion prediction, and image or video generation that inform applications in realistic 3D character animation.

[1] Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022.

[2] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. arXiv preprint arXiv:2203.08063, 2022.

[3] Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecasting. In 2019 International Conference on 3D Vision (3DV), pp. 719–728. IEEE, 2019.

[4] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

[5] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.

[6] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.

[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.

[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.

[9] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022b.

[10] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.

[11] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837–2845, 2021.

[12] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015.

[13] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5253–5263, 2020.

[14] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (TOG), 39(4):60–1, 2020.

[15] Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. Skeleton-aware networks for deep motion retargeting. ACM Transactions on Graphics (TOG), 39(4):62–1, 2020.

[16] Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE. In International Conference on Computer Vision (ICCV), pp. 10985–10995, October 2021.

[17] Mingyi Shi, Kfir Aberman, Andreas Aristidou, Taku Komura, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM Transactions on Graphics (TOG), 40(1):1–15, 2020.

[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.

[19] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152–5161, 2022a.

[20] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236–252, 2016.

[21] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021–2029, 2020.

[22] Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-view human action recognition. In Proceedings of the 26th ACM international Conference on Multimedia, pp. 1510–1518, 2018.

[23] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.

[24] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10, 2022a.

[25] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pp. 4346–4354, 2015.

[26] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891–2900, 2017.

[27] Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer. Human motion prediction via spatio-temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7134–7143, 2019.

[28] Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Back to mlp: A simple baseline for human motion prediction. arXiv preprint arXiv:2207.01567, 2022b.

[29] Félix G Harvey and Christopher Pal. Recurrent transition networks for character locomotion. In SIGGRAPH Asia 2018 Technical Briefs, pp. 1–4. 2018.

[30] Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Convolutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), pp. 918–927. IEEE, 2020.

[31] Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan. Single-shot motion completion with transformer. arXiv preprint arXiv:2103.00776, 2021.

[32] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.

[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[34] Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(4):1–11, 2016.

[35] Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. arXiv preprint arXiv:2203.13694, 2022.

[36] Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In The IEEE International Conference on Computer Vision (ICCV), 2021.

[37] A Aristidou, A Yiannakidis, K Aberman, D Cohen-Or, A Shamir, and Y Chrysanthou. Rhythm is a dancer: Music-driven motion synthesis with global structure. IEEE Transactions on Visualization and Computer Graphics, 2022.

[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.

[39] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985, 2019.

[40] Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine-Hornung, and Daniel Cohen-Or. Modi: Unconditional motion synthesis from diverse data. arXiv preprint arXiv:2206.08010, 2022.

[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.

[42] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.

[43] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.

[44] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.

[45] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. arXiv preprint arXiv:2209.00349, 2022.

[46] Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV), pp. 565–574. IEEE, 2021.

[47] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pp. 5442–5451, October 2019.

[48] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 1–10. IEEE, 2021.

[49] Adobe Systems Inc. Mixamo, 2021. URL https://www.mixamo.com. Accessed: 2021-12-25.