Temporal Coherence and Denoising

Pause a game frame and step forward one frame. Almost nothing has changed: the camera nudged a pixel, a character's arm swung a few degrees, a shadow crept. Ninety-something percent of the picture is identical to the frame you just drew. Redrawing all of it from scratch, sixty times a second, is heroic — and mostly wasted. This is the central fact renderers now build their whole strategy around: consecutive frames are nearly the same. We call it temporal coherence, and exploiting it is what makes modern real-time path tracing possible at all.

The plan of this page: first, why coherence lets us amortise expensive work across many frames instead of paying for it every frame; then temporal anti-aliasing (TAA), which accumulates samples over time by reprojecting old frames; then denoising, which turns a splotchy one-sample path trace into a clean image using geometry buffers as a guide; and finally the artefacts all of this fights — ghosting and flicker — and how history gets rejected when it goes stale.

The core move: amortise work over frames

Coherence. Frame N and frame N+1 differ only slightly, because the camera and objects move continuously — most surface points visible now were visible a frame ago.
Amortisation. Because of that overlap, an expensive quantity (lighting, an anti-aliased colour, a noise-free estimate) need not be recomputed in full each frame. We reuse last frame's result and fold in a little fresh work, spreading the cost across many frames.
Reprojection. To reuse a value, we must find where a point that is at screen pixel p now was on screen last frame — that displacement is its motion vector.

Every technique below is a variation on the same recipe: reproject (find the old position), reuse (blend the history in), and reject (throw the history away when it can't be trusted). Get those three right and one cheap sample per frame, blended over time, behaves like dozens of expensive ones.

Motion vectors: the address of last frame's pixel

A motion vector stored at pixel p says how far, in screen space, the surface seen there moved since the previous frame. If a point projects to p now and projected to p_{\text{prev}} last frame, the renderer stores

\mathbf{m}(p) = p - p_{\text{prev}} \qquad\Longrightarrow\qquad p_{\text{prev}} = p - \mathbf{m}(p).

The engine already knows this: it has the current and previous transforms of every object and camera, so it can transform each fragment by both and subtract. The motion vectors are written into a screen-sized buffer (a "velocity buffer") right alongside colour. To reuse last frame's value at p, you follow the vector backwards to p_{\text{prev}} and sample the previous frame there. That single lookup is the workhorse of the whole page.

Temporal anti-aliasing (TAA)

A single sample at a pixel's centre gives jagged edges: it either hits the triangle or misses, with nothing in between. Classic supersampling fixes this by taking many sub-pixel samples per frame — expensive. TAA spreads those samples across time instead. Each frame the camera is jittered by a sub-pixel offset (a low-discrepancy sequence like Halton picks the offsets so that over, say, eight frames they tile the pixel evenly). Frame by frame the renderer samples a slightly different spot inside every pixel.

Those samples are combined by reprojecting the accumulated history colour from last frame — follow the motion vector to p_{\text{prev}}, read the old blended colour — and mixing it with this frame's fresh sample by an exponential moving average:

c_{\text{out}}(p) = (1-\alpha)\,\underbrace{c_{\text{hist}}(p - \mathbf{m}(p))}_{\text{reprojected history}} \;+\; \alpha\,\underbrace{c_{\text{cur}}(p)}_{\text{this frame}} , \qquad \alpha \approx 0.1 .

With \alpha \approx 0.1 each output pixel is roughly the average of the last ten frames' jittered samples — an effective 10\times supersample for the price of one sample per frame. That is why TAA has become the default anti-aliaser in real-time rendering: it is nearly free and it also smooths shading noise, not just geometric edges.

The catch is the same one that haunts everything here: the reprojected history is only valid if that pixel really was showing the same surface last frame. When it wasn't, TAA smears — which brings us to the artefacts, but first, denoising, which pushes the very same idea much harder.

A picture of reprojection

Here is the one geometric idea the whole page rests on. The lower strip is the previous frame; the upper strip is the current frame. A surface point sits at pixel p in the current frame. Its motion vector \mathbf{m}(p) tells us it was at p - \mathbf{m}(p) last frame — so to reuse last frame's clean, accumulated colour we sample there and blend it into p.

Everything now depends on the arrow being right. If the motion vector points somewhere the surface was not — because it was hidden, or off screen, or the motion was mis-estimated — the history we fetch belongs to a different surface, and we get a smear.

Denoising a path-traced frame

Path tracing shoots random rays to estimate the light arriving at each pixel. With a full budget of thousands of samples the estimate converges to a clean image; with the one sample per pixel a real-time renderer can afford, the estimate is extremely noisy — a blizzard of bright and dark speckles, because the Monte Carlo variance is enormous at one sample. A denoiser is the post-process that reconstructs a clean image from that noisy estimate.

The trick that makes it work: the renderer is cheap to produce auxiliary geometry buffers that are not noisy at all —

Albedo — the surface's base colour, sampled directly from textures (noise-free).
Normal — the surface orientation at each pixel (noise-free).
Depth — distance from the camera (noise-free).

A good denoiser blurs the noisy lighting heavily, but only across pixels the guide buffers say belong to the same surface — similar normal, similar depth. It refuses to blur across a depth cliff or a normal discontinuity, so edges stay crisp while flat regions are smoothed. This is the idea behind the edge-avoiding À-Trous filter and its temporal cousin SVGF (Spatiotemporal Variance-Guided Filtering), which additionally accumulates samples over time by exactly the TAA reprojection above — a spatial blur guided by the geometry, plus a temporal accumulation guided by motion vectors.

The modern alternative is a machine-learned denoiser — NVIDIA's OptiX denoiser or Intel's Open Image Denoise (OIDN) — a neural network trained on pairs of noisy and clean renders that learns to map (noisy colour + albedo + normal) to the converged image. It often beats hand-written filters on detail, at the cost of a trained model and inference time.

The enabling trick: 1 spp + reproject + denoise

Trace one sample per pixel this frame; reproject and accumulate the previous frames' samples along the motion vectors (giving many effective samples for free); then run a spatiotemporal denoiser guided by albedo, normal and depth to reconstruct a clean image.

None of the three parts is sufficient alone: one sample is far too noisy, temporal accumulation alone still leaves residual noise and lags on motion, and a spatial denoiser alone over-blurs. Together they turn a noisy trickle of rays into a stable, clean, animated image — the pipeline behind real-time path-traced games and the "ray reconstruction" style upscalers.

It seems perverse to deliberately shake the camera by a fraction of a pixel every frame. But a fixed sample at each pixel centre can only ever tell you what is at the centre — it is blind to a triangle edge slicing through the pixel. By moving the sample point around inside the pixel over successive frames, the accumulated average sees the whole pixel's worth of coverage, and the edge resolves smoothly. The offsets aren't random: a low-discrepancy sequence (Halton, or a Poisson-disc set) is chosen so that after k frames the samples are spread as evenly as possible across the pixel, so the average converges fast. The jitter is undone when reading history — you sample where the surface actually was, not where the shaken camera pretends it is — so the image doesn't wobble; only the sampling pattern does.

Worked example: reproject, then decide

A pixel at screen position p=(640, 360) stores a motion vector \mathbf{m}=(8, -2) (the surface moved 8 px right and 2 px up since last frame). We want to reuse last frame's accumulated colour.

Step 1 — reproject. The previous screen position is p_{\text{prev}} = p - \mathbf{m} = (640-8,\; 360-(-2)) = (632,\, 362). Sample the history buffer at (632, 362).

Step 2 — validate. Compare the history's stored depth and normal at (632,362) against this frame's depth/normal at (640,360). If depth agrees to within tolerance and the normals point the same way, the history is the same surface — trust it.

Step 3 — blend (history valid). With \alpha=0.1,

c_{\text{out}} = 0.9\,c_{\text{hist}}(632,362) + 0.1\,c_{\text{cur}}(640,360).

Step 3′ — reject (history invalid). Suppose the depth at (632,362) was far nearer — last frame that pixel showed a foreground object that has since moved aside, disoccluding our surface. The history belongs to the wrong thing. We discard it and fall back to the current sample alone, c_{\text{out}} = c_{\text{cur}}, restarting accumulation there. That pixel is momentarily noisier — a price we happily pay to avoid dragging a wrong colour behind the object.

Reprojected history is wrong wherever a surface was hidden last frame or the motion vector is off. Two classic failures:

Disocclusion. A surface that was behind another object last frame has no valid history — there is literally no previous sample of it to reuse. Reproject anyway and you fetch the colour of whatever was in front, which trails behind the moving object as a smear.
Bad motion vectors. Fast motion, sub-pixel detail, shadows, reflections and transparency (which surface's velocity does a glass pane store?) all produce vectors that point to the wrong place. The stale colour they fetch lingers — a comet-tail ghost following bright moving highlights.

The fix is history rejection: at every pixel, sanity-check the reprojected history against the current geometry — depth mismatch, normal mismatch, or a colour far outside the local neighbourhood's range (neighbourhood colour clamping) — and when it fails, throw the history out and fall back to the current sample. You trade a flash of extra noise on those pixels for correctness. Too little rejection gives ghosting; too much gives flicker and lost detail — tuning that balance is most of the art of a good temporal filter.

A tiny \alpha (long history) gives the smoothest, least-noisy image — but it also makes the filter sluggish: any real change (a light flicking on, a highlight sweeping across metal) takes many frames to appear, so moving features lag and ghost. A large \alpha (short history) is responsive but noisy and prone to flicker — the shimmering, boiling look of an estimate that never settles. Good temporal filters make \alpha adaptive: near-static pixels get a long history (many effective samples), just-disoccluded or fast-moving pixels get a short one, and SVGF even uses the local variance to decide how hard to blur spatially. History length is a dial between ghosting and flicker, not a free lunch.

How fast does noise fall as history grows?

Monte Carlo error shrinks like 1/\sqrt{n} in the number of samples n. Accumulating history is exactly gathering more samples per pixel over time, so the noise in a temporally-accumulated frame falls the same way as the effective sample count — the number of frames folded in — rises. Drag the history length slider to set how many frames are accumulated and watch the noise curve: the first few frames buy the biggest drop, and the returns diminish. A disocclusion resets a pixel to the far left of this curve — one sample, maximum noise — which is exactly why freshly-revealed regions look grainy for a moment.

The 1/\sqrt{n} law is why denoisers matter: to halve the noise you must quadruple the samples, so at one sample per frame the naive wait is far too long. The denoiser fills the gap by borrowing spatial neighbours (guided by the geometry buffers) on top of the temporal accumulation.