Motion VAEs and Diffusion

Every earlier tool in this course takes motion you already have — a mocap clip, a blend of clips, a pose graph — and rearranges it. But a studio never has enough clips. The hero needs to stumble, recover, glance left, sit on a low wall, then spring up in a way no actor was ever recorded doing. Capturing every such moment is impossible; the space of human movement has an enormous long tail of one-off actions. The frontier question of this course is therefore: can a model generate brand-new, plausible, controllable motion — ideally directed in plain English, "a person walks forward, then sits down slowly"?

This page is about the two families of generative models that answer that question: motion VAEs, which learn a compact latent space of movement you can sample and interpolate, and motion diffusion models, which build motion by iteratively denoising from pure noise under a text or music condition. They chase the same goal — synthesise motion no one captured — by two very different mechanisms, and diffusion is currently the leading approach to text-to-motion.

The goal: generate, not replay

Playback and blending answer "what did we record?" A generative model answers "what is a plausible motion I have never seen?" To do that it has to learn the distribution of natural motion — which sequences of poses look human and which look broken — from a big dataset of mocap (AMASS, HumanML3D, and friends). Once a model holds that distribution, three superpowers follow:

Novelty & variety — draw many different plausible motions for the same request, instead of one fixed clip. Content at scale.
Control / conditioning — steer generation with a goal: a target location, an action label, a music track, or a natural-language text prompt.
Long-tail coverage — synthesise the rare, never-captured actions that make a character feel alive, because the model interpolates and extrapolates the movement it learned.

Motion VAEs: a navigable latent space

A variational autoencoder squeezes each motion through a narrow bottleneck. An encoder maps a pose (or a short window of poses) to a small vector z in a low-dimensional latent space; a decoder reconstructs the motion from z. The "variational" part forces that latent space to be smooth and roughly a unit Gaussian, so two things become true and useful: sampling a random z decodes to a plausible new pose, and interpolating between two codes z_A and z_B traces a continuous path of plausible in-between motions — a learned way to blend styles.

Ling et al. (2020) made this autoregressive: the decoder predicts the next pose from the current pose and a sampled latent, and you roll it forward one frame at a time to grow a whole clip:

x_{t+1} = \text{Decoder}(x_t,\, z_t), \qquad z_t \sim \mathcal{N}(0, I).

The masterstroke is what sits on top. Because every plausible motion corresponds to a point (or path) in the latent space, a separate controller / policy — trained with reinforcement learning — needs only to choose the latent z_t at each step to steer the character toward a goal (reach that target, keep balance, follow a heading). The hard problem "what are the joint angles?" is replaced by the easy problem "which point in this small, well-behaved space?" The VAE has become a compact, controllable motion prior.

The diffusion revolution

Diffusion models flipped the field around 2022. The idea is almost provocatively simple. Take a real motion and, over many steps, keep adding Gaussian noise until it is indistinguishable from random static — the forward process. Then train a network to run that backwards: given a noisy motion at step t, predict a cleaner version at step t-1. At generation time you start from pure noise and apply that learned denoiser T times; a coherent motion crystallises out of the fog.

Generation runs a reverse denoising chain x_T \to x_{T-1} \to \dots \to x_0, starting from noise x_T \sim \mathcal{N}(0, I).
Each step is conditioned on a signal c — a text prompt, an action label, or a music track — so the denoiser is x_{t-1} = \text{Denoise}(x_t,\, t,\, c).
MDM (Tevet et al., 2022) predicts the clean motion x_0 at every step, which makes it easy to inject geometric constraints (foot contacts, joint targets) during sampling.

Conditioned on text, this is today's leading text-to-motion method: "a person walks forward then sits" yields diverse, high-quality clips, and re-running the sampler gives a genuinely different valid performance each time. That combination of diversity and quality is exactly what earlier generators struggled to deliver together. Some variants close the physics gap by conditioning on, or refining with, a physics simulator / RL policy so the output actually respects contacts and balance (physics-conditioned diffusion).

Picturing the latent space

A VAE's latent space is abstract and high-dimensional, but the intuition survives in two dimensions. Each dot below is one plausible motion the model has learned; nearby dots are similar movements, so the cloud is a smooth atlas of motion. Pick two named codes — a walk z_A and a sit z_B — and the straight path between them is a continuous interpolation: decode points along it and you get a smooth blend from walking to sitting, none of which was a recorded clip. That path is the generative act.

Diffusion has no such fixed atlas to sample from; it manufactures each motion on demand by walking down from noise. Same destination — a novel plausible motion — reached by carving instead of by picking a point off a map.

Worked example: same goal, two mechanisms

Suppose we want a fresh "walk that turns into a sit". Watch how each family produces it.

Motion VAE. (1) Encode a recorded walk to a code z_A and a recorded sit to z_B. (2) To blend the two styles, walk the straight line z(\alpha) = (1-\alpha)\,z_A + \alpha\,z_B as \alpha goes 0 \to 1. (3) Decode each z(\alpha) (autoregressively rolling poses forward) to get a smooth walk-into-sit no clip contained. To make it reach a chair, a small RL controller picks the z_t at each step. One forward pass per frame — fast.

Diffusion. (1) Set the condition c = the text embedding of "a person walks forward then sits down." (2) Start from pure noise x_T \sim \mathcal{N}(0, I) — a garbage motion. (3) Apply the denoiser T times, x_{t-1} = \text{Denoise}(x_t, t, c), and the walk-then-sit sharpens out of the noise. Re-run with new noise for a different valid take. Dozens of network passes per clip — slower, but higher diversity and directable by language. Same goal; one picks a point, the other denoises from chaos.

The trade-offs

Neither family dominates; you choose by what the shot needs.

	Motion VAE	Diffusion
Sampling speed	Fast — one decode pass per frame; real-time friendly	Slow — many denoising steps per clip
Diversity / quality	Lower diversity; can look averaged / blurry	High diversity and quality; state of the art
Text-to-motion	Weak	Strong — the leading approach
Control	RL policy navigates the latent space	Conditioning + guidance during sampling

The practical picture: reach for a VAE-style latent policy when you need interactive, real-time characters (games), and for diffusion when you want the richest, language-directed variety and can afford offline generation. The frontier is squeezing diffusion's cost down — latent diffusion (denoise in a small VAE latent instead of raw pose space) and distillation (train a few-step or one-step sampler) are actively closing the speed gap. The two families are quietly converging.

Because the space of human movement is effectively infinite and deeply combinatorial. A character can walk at any speed, on any slope, carrying any load, in any mood, reacting to any nudge, transitioning into any next action — the number of distinct little performances explodes far past what any mocap stage could shoot. Capture gives you a dense sample near the common motions and almost nothing in the long tail of rare ones. A generative model's job is precisely to fill that tail: having learned the rules of plausible movement from the samples we do have, it can synthesise the stumble-recover-glance-sit that was never on the schedule — and do it a hundred different ways on request.

These models learn the statistics of motion, not the physics. They have no built-in notion of gravity, contact, or non-interpenetration — only "what usually comes next." So a plausible-looking clip can quietly break the world: foot-sliding (the planted foot skates because contact wasn't enforced), self-penetration (an arm passes through the torso), and floating (the character drifts a few centimetres off the ground). The eye catches these instantly and the illusion dies.

The fixes are to bolt physics back on: post-process with inverse kinematics to pin foot contacts, or run the output through a physics simulator; or better, use physics-conditioned / RL-refined training so plausibility is baked in. And remember the other cost: diffusion's iterative sampling is too slow for hard real-time — a per-frame budget of milliseconds can't afford dozens of denoising passes, which is why latent diffusion and distillation exist. Never ship raw generator output into a game loop and assume the feet will behave.

Where this sits

Generative motion is the endpoint of a ladder: from motion matching to neural controllers, which still stitch recorded motion, up to models that invent it. For where all of this is heading next, see the state of the art.