Lip Sync

Turn the sound off on a well-animated talking character and something quietly amazing keeps working: you can almost read the lips. Turn it back on and the mouth and the voice lock together so completely that you never think about it. Get it slightly wrong — a badly dubbed film, a video-game cut-scene where the jaw flaps a beat behind the words — and it is instantly, viscerally off. Humans are expert lip-readers; we spend our lives watching mouths, so the bar for lip sync — making a character's mouth match speech — is brutally high.

This page follows the whole pipeline: from a stream of speech sounds to a moving mouth. Two ideas do most of the work. First, the sounds we hear and the shapes we see are not the same alphabet — many sounds share one mouth shape. Second, real mouths never snap between shapes; each sound smears into its neighbours. Miss either idea and you get the classic flappy-puppet mouth.

Phonemes you hear, visemes you see

Spoken language is built from phonemes — the smallest units of sound that change meaning. English has roughly 40 of them: the /p/ in "pat", the /b/ in "bat", the /m/ in "mat", the vowels, and so on. But your eyes can't hear the difference between /p/, /b/ and /m/: all three are made by pressing the lips shut. On screen they look identical.

A phoneme is a unit of sound — the auditory atom of speech (English has ~40).
A viseme is a unit of visible mouth shape — what the face looks like while making a sound.
The map phoneme → viseme is many-to-one: sounds that differ only in voicing or in the tongue/throat (invisible from outside) collapse together, so ~40 phonemes reduce to only ~10–15 visemes.

This collapse is a gift to the animator. You do not need forty mouth shapes; you need a dozen or so blendshape targets — a lips-closed shape, a wide "ee", a rounded "oo", an open "ah", the teeth-on-lip "f/v", and a handful more — and a lookup table that says which sound uses which. The table below shows the idea; note how whole groups of phonemes point at the same shape.

A phoneme → viseme table

Viseme (mouth shape)	Phonemes that share it	How it's made
Closed lips (bilabial)	/p/ /b/ /m/	lips pressed together
Lip-to-teeth	/f/ /v/	top teeth on bottom lip
Wide spread ("ee")	/iː/ /ɪ/ /j/	lips stretched, corners back
Rounded ("oo")	/uː/ /w/	lips pushed forward, small round hole
Open ("ah")	/ɑː/ /a/	jaw dropped, mouth wide open
Tongue-tip ("th")	/θ/ /ð/	tongue between the teeth
Alveolar	/t/ /d/ /n/ /l/ /s/ /z/	tongue at the ridge (lips barely change)

The last row is telling: half a dozen very different sounds — /t/, /d/, /n/, /l/, /s/, /z/ — are all made with the tongue behind the teeth while the lips do almost nothing, so they share one near-neutral viseme. Trying to give each its own distinct mouth shape would be wasted work, and worse, would make the mouth twitch on sounds that should look calm.

The pipeline

A classical lip-sync system runs in three stages:

Get a timed phoneme sequence. Either forced alignment (you already have the audio and a transcript, and an algorithm lines the known words up against the waveform to stamp each phoneme with a start and end time), or ASR (automatic speech recognition guesses the phonemes straight from audio when you have no transcript).
Map phonemes → visemes. Run each phoneme through the lookup table to get a sequence of target mouth shapes with timings.
Drive the mouth. Turn those viseme targets into weights on the blendshape targets over time — and here is where the craft lives, because you must blend, not cut.

Stage three is deceptively easy to do badly. The naive version sets one keyframe per phoneme: at the moment of the /m/, snap the mouth to fully closed; at the /a/, snap it fully open. It is correct sound-by-sound and it looks terrible — because that is not how real mouths move.

Coarticulation: sounds bleed into each other

Say the word "stew", then say "stee". The /s/ is the same phoneme both times — but watch your lips: for "stew" they are already rounding during the /s/, getting ready for the "oo" that is still two sounds away; for "stee" they are already spreading. Your mouth anticipates the future and lingers on the past. This overlap of gestures is called coarticulation, and it means a viseme is never an isolated pose — its real shape depends on its neighbours.

The visible shape for a sound is not fixed; it is pulled toward the shapes of the sounds before and after it.
The mouth anticipates upcoming sounds (starts forming "oo" early) and lingers on past ones — gestures overlap in time rather than lining up in a queue.
Animating one crisp keyframe per phoneme ignores this overlap and produces mechanical, "poppy" motion.

Two families of technique model the overlap. Dominance models (Cohen–Massaro) give each viseme a dominance function — a bump in time saying how strongly it pulls the mouth — and the final shape at any instant is the weighted blend of every nearby viseme by its dominance, so a strong, sharp sound like a closed-lip /m/ dominates its moment while a weak, slurred sound yields to its neighbours. In symbols, the mouth parameter is

m(t) \;=\; \frac{\sum_i D_i(t)\, T_i}{\sum_i D_i(t)},

where T_i is viseme i's target value and D_i(t) its dominance (a bump centred on that phoneme's time). The simpler, very common alternative is just to spline-blend the viseme targets — lay the target shapes down as keys and let smooth interpolation carry the mouth between them, which automatically overshoots and eases the way a spring would. Either way, and importantly, the jaw, lips and tongue move on their own overlapping schedules: the jaw may still be closing from the last word while the lips already round for the next.

Worked example: the word "mama"

Take a single, tiny word: "mama". Its phonemes are /m/ \to /a/ \to /m/ \to /a/ — closed, open, closed, open. Track just one number, the mouth-open parameter (0 = lips sealed, 1 = wide open), across the word.

The naive hard-cut version holds fully closed, then jumps to fully open, then slams shut, then flies open again — a square wave. On screen that reads as a hinge snapping: the "flappy puppet". The blended version treats those four values as viseme targets and lets the mouth flow between them, and crucially it anticipates: the mouth starts easing open a little before the vowel and has not fully sealed on the second /m/ before the last vowel is already pulling it back open. The curve becomes a smooth, breathing wave — and it reads as speech.

The two curves hit the same targets at the same phoneme times. The only difference is what happens between the phonemes — and that between is the entire difference between "robotic" and "alive". Spacing was the whole story for timing and easing, and it is the whole story here too.

The signature beginner mistake is one viseme per phoneme, hard-cut: snap the mouth to a crisp pose at every sound and hold it. It flunks both of this page's big ideas at once.

First, it ignores coarticulation — real mouths blend and anticipate, so hard cuts pop mechanically ("flappy puppet"). Fix: blend the viseme targets over time and let neighbours pull on each other (spline the targets, or use dominance weighting). Second, it forgets that phoneme → viseme is many-to-one — do not animate every phoneme with its own distinct shape. /p/ and /b/ and /m/ are the same closed viseme; /t/, /d/, /n/, /s/ barely move the lips at all. Giving each sound a unique pose makes the mouth chatter and buzz on sounds the eye should read as still. Fewer, blended shapes beat many, crisp ones — every time.

The classical pipeline needs a phoneme sequence first. Modern audio-driven systems skip that middle-man: a neural network learns the mapping from the raw audio waveform straight to mouth shapes, trained on hours of video of real people talking. NVIDIA's Audio2Face, the JALI model (which cleverly splits control into a jaw axis and a lip axis so it can dial in mumbling versus over-enunciating), and a wave of neural approaches all work this way. Because they learn from real performance, they pick up things a lookup table never could — the extra emphasis on a stressed word, the way an emotion reshapes the whole mouth, the little asymmetries of a real face. The phoneme table did not vanish; it got absorbed into the weights of a network that learned the same many-to-one map, and the coarticulation, directly from data.

Dubbing swaps the audio but keeps the original footage, so the actor's visible visemes were made for the original language's phonemes. When the new track says a rounded "oo" but the lips on screen are spread wide for the original word, your expert lip-reading brain flags the mismatch even if you can't say why. It is exactly a phoneme→viseme conflict — the sound and the shape disagree — which is why "visual dubbing" research now uses the audio-driven neural methods above to re-animate the actor's mouth to fit the new language.

Where this sits

Lip sync drives the mouth, but a talking face is more than a mouth: eyebrows, blinks, cheeks and the whole coding system of expression come next in facial animation and FACS, and the very blendshape and muscle targets we've been weighting are built in blendshape and muscle faces.