Learned Motion: from Motion Matching to Neural

Imagine a game character sprinting across broken terrain, cutting sideways the instant you flick the stick, planting a foot on a ledge and pushing off — all reading as one continuous, weighty performance. For a decade the state of the art behind that was motion matching: keep a huge library of motion-captured frames, and every single frame of gameplay, search that library for the recorded pose whose future best fits where the player wants to go, then jump to it. It looks fantastic. It also drags a database of millions of poses around in memory and pays a nearest-neighbour search 30–60 times a second.

This page is about the shift that answered the obvious question: what if a neural network could replace the database? Instead of storing every pose and looking one up, train a net to map (current character state + where the player wants to go) straight to the next pose — learning the whole motion library into a handful of megabytes of weights. We'll trace the landmark models (PFNN, Mode-Adaptive Networks, Learned Motion Matching), see why the naive version collapses into blurry mush, and set up the jump to motion VAEs and diffusion.

Recap: motion matching, and what it costs

Motion matching is beautifully simple. You hold a big flat pile of mocap frames. Each frame is tagged with a feature vector — the character's current pose and velocity, plus the near-future trajectory that actually followed it in the capture. At runtime you build a desired feature vector from the player's input (where they're steering) and the current pose, and you find the stored frame whose feature vector is nearest:

i^\star \;=\; \arg\min_{i}\; \big\lVert\, \mathbf{f}_i - \mathbf{f}_{\text{query}} \,\big\rVert^2 .

Play forward from i^\star, re-search a few frames later, blend across the jumps, and you get gorgeous, grounded motion with essentially no hand-authored state machine. The price is paid in two currencies:

Memory. Quality scales with coverage, so production databases balloon to millions of frames — often hundreds of megabytes to gigabytes of pose data that must sit in RAM.
Search. Every query is a nearest-neighbour lookup over that whole set, repeated many times per second, per character — costly to do, and awkward to scale to crowds.

There's a subtler cost too: the system can only ever reproduce poses it recorded. It never invents an in-between it wasn't given; it can only pick and blend among clips it already owns.

The neural alternative: learn the database away

The core idea is to treat the controller as a function and approximate it with a network. Feed in the character's current state and the desired trajectory/goal; read out the next pose. Written abstractly, with network weights \theta:

\mathbf{y}_{t+1} \;=\; \Phi_\theta\big(\, \mathbf{x}_t,\; \mathbf{g}_t \,\big),

where \mathbf{x}_t is the current state (pose, joint velocities, foot contacts, phase — more on that below), \mathbf{g}_t is the goal (the desired future trajectory, gait, and terrain profile), and \mathbf{y}_{t+1} is the predicted next pose. You train \theta on the very same mocap that would have filled the database, minimising the difference between the predicted pose and the recorded one. At runtime the database is gone: all that ships is \theta.

The payoffs are exactly the mirror image of motion matching's costs — tiny memory (a few MB of weights instead of a giant pose bank), a fixed, predictable cost per frame (one forward pass, no search over a growing set), and generalisation: a network can interpolate smoothly between clips it never literally saw, filling in motions the capture never recorded. But it doesn't come for free — you pay up front in training data and training time, and a badly-posed network fails in a spectacular way we'll meet in the "Watch out!" box.

Landmark model: the Phase-Functioned Neural Network (PFNN)

The breakthrough that made neural locomotion crisp was Holden, Komura and Saito's Phase-Functioned Neural Network (2017). Its insight is that locomotion is fundamentally cyclic: the left foot strikes, rolls, lifts; the right foot strikes, rolls, lifts. Encode where you are in that cycle as a single phase variable \phi \in [0, 2\pi), and let the network's weights themselves be a function of phase:

\mathbf{y}_{t+1} \;=\; \Phi\big(\mathbf{x}_t,\; \theta(\phi)\big), \qquad \theta(\phi) = \text{Catmull–Rom blend of a few control weight sets around the cycle}.

Rather than one fixed set of weights trying to cover both "foot planting" and "foot swinging" at once, PFNN stores a small number of control weight sets spaced around the phase circle and smoothly interpolates between them as \phi advances. The effect is a compact network that behaves like a different specialist at each moment of the stride, yet stays continuous. The result: high-quality, responsive locomotion over uneven terrain, from a model small enough to ship, with a fixed per-frame cost.

Why does the phase input matter so much? Because it tells the network which part of the stride it is in, it resolves the ambiguity that would otherwise make the net average incompatible poses together. That single scalar is the difference between crisp footfalls and the blurry damped mush of naive regression — the failure the "Watch out!" below dissects.

Beyond one clock: Mode-Adaptive Networks and gating experts

A single phase variable is perfect for a biped walking and running, but it strains on richer motion — a quadruped has multiple limbs on different, coupled rhythms (walk, trot, canter, gallop), and there's no one clean foot cycle to key everything off. Zhang, Starke, Komura and Yang's Mode-Adaptive Neural Networks (2018) generalised the idea: instead of a hand-defined phase driving the blend, a small gating network looks at the state and produces blending coefficients that mix a bank of expert weight sets:

\theta \;=\; \sum_{k=1}^{K} \alpha_k(\mathbf{x})\,\theta_k, \qquad \boldsymbol{\alpha}(\mathbf{x}) = \text{gating network output}, \quad \textstyle\sum_k \alpha_k = 1 .

The gating net learns when to lean on which expert — effectively discovering its own phases and gait modes rather than being told them. This mixture-of-experts idea is the backbone of much of the modern neural-character work (later systems layer local phases per limb on top of it), and it's the general answer to "one cyclic clock isn't enough."

Learned Motion Matching: keep the paradigm, drop the database

There's a third path that's especially loved in games because it preserves the exact feel and controllability of motion matching while shedding its memory cost. Holden and colleagues' Learned Motion Matching (2020) replaces the giant pose database with a set of small networks trained to imitate what motion matching would have done:

A compressor / decompressor (an autoencoder) squeezes each pose into a compact latent code and reconstructs it — so poses need not be stored explicitly.
A stepper network predicts how the latent feature evolves frame to frame while a clip plays out — replacing "read the next stored frame".
A projector network maps a query feature to the nearest good feature — replacing the nearest-neighbour search over the database.

Together these reproduce motion-matching-quality output at a fraction of the memory (databases shrink from hundreds of MB to a few MB of weights) and with a bounded per-frame cost — no search over a growing set. Crucially it keeps the motion-matching authoring workflow: you still think in terms of a feature database and trajectories, you just no longer ship the database.

Worked example: a database vs weights that interpolate

Put numbers on the trade-off. Take a locomotion set captured at 60\,\text{Hz}, giving 1{,}000{,}000 frames of skeleton animation. Suppose each frame stores a modest 60-DoF pose (positions + rotations + velocities) as 32-bit floats:

10^{6}\ \text{frames} \times 60\ \text{floats} \times 4\ \text{bytes} \;=\; 240{,}000{,}000\ \text{bytes} \;\approx\; 229\ \text{MB}

…and that's before the feature index and blend metadata a real system also carries. A PFNN trained on the same capture might have on the order of a few hundred thousand parameters per control weight set; even generously counted, the shipped weights land in the low single-digit megabytes — call it a 50\text{–}100\times reduction. Two things are happening at once:

Compression. The net stops storing individual frames and stores only the rule that generated them.
Generalisation. Ask motion matching for a gait at a speed and turn radius it never recorded, and it can only stitch the nearest recorded clips. The trained net evaluates \Phi_\theta and interpolates a pose it never saw — a genuinely new frame consistent with everything it learned.

And the phase input is what keeps that interpolation honest. Without it, when several plausible next poses exist (foot could be landing or lifting), the network — trained to minimise average error — hedges by predicting the average of them, which is a limp, foot-sliding non-pose. The phase tells it which of those futures applies, so it commits to a crisp one.

A tempting reflex: if regression to the mean blurs the motion, throw a bigger network at it. It doesn't help — and understanding why is the whole point. The blur isn't a capacity problem; it's a problem specification problem. If the input state genuinely maps to several different valid next poses, then the mathematically optimal single-output predictor is their mean, no matter how many parameters it has. A bigger net just fits that mean more precisely. The fix is never "more capacity" — it's to remove the ambiguity (add a phase or gait input so the mapping becomes one-to-one), or to model the distribution instead of a point (mixture density, or a generative model that samples a future rather than averaging). That's exactly the thread that leads into the next lesson on generative motion models.

The single most common way a first neural-motion attempt fails: build a plain network that takes the current state and directly regresses the next pose, train it with a mean-squared-error loss, and watch the output turn into blurry, damped, foot-sliding mush — motion that looks like it's wading through treacle, with no crisp footfalls.

The cause is multimodality. From a given state, many futures are plausible (the character could plant the left foot, or the right, or keep gliding). MSE training pushes the network toward the value that minimises average squared error over all those futures — which is their average. The average of a foot-down pose and a foot-up pose is a foot hovering mid-air: physically wrong, visually dead. The cures are exactly the models on this page and the next:

Phase conditioning (PFNN) — tell the net where in the cycle it is, so each state has one answer.
Mixture / expert models (Mode-Adaptive nets) — let different experts own different modes instead of blending them into one.
Autoregression with noise / generative sampling — sample a specific future rather than averaging all of them.

If your learned character walks like it's underwater, suspect mean-collapse before you suspect a bug.