The 4f Correlator and Optical Convolution

Convolution is arguably the most-executed mathematical operation on Earth. Every blur, sharpen and edge-detect, every audio filter, every layer of a convolutional neural network — all of it is the same sliding multiply–accumulate, and datacenters burn megawatts doing it. The previous lesson put a strange weapon on the table: a lens Fourier-transforms an image for free. The convolution theorem tells you what to do with such a weapon. Convolution in space is multiplication in frequency — and multiplication is the one operation optics does merely by passing light through something. So: transform with one lens, multiply by slipping a mask — a patterned transparency — into the Fourier plane, transform back with a second lens. Four focal lengths of bench: the 4f correlator, the closest thing analog optical computing has ever had to a killer app.

Anatomy of the machine

Lay two identical lenses of focal length f on an optical axis, 2f apart. Put the input transparency one focal length before the first lens; the output appears one focal length after the second — four focal lengths from end to end, hence the name. The middle plane — back focal plane of L_1, front focal plane of L_2 — is the Fourier plane, where the input exists briefly as its own spectrum. Whatever transmission function t(u,v) you park there multiplies that spectrum pointwise before L_2 undoes the transform:

with input g(x,y) and a Fourier-plane mask whose transmission is H\!\big(\tfrac{u}{\lambda f}, \tfrac{v}{\lambda f}\big), the output field is g_{\text{out}}(x, y) \;=\; (g * h)(-x, -y), the convolution of the input with the kernel h = \mathcal{F}^{-1}[H], in inverted coordinates;
with no mask at all (H \equiv 1), the output is g(-x,-y): two successive Fourier transforms give back the input, upside down;
the kernel size costs nothing: a 3×3 blur and a 501×501 texture template are both "one mask", applied in one pass of light;
choosing H = G^{*} (the conjugate spectrum of a target g) makes the output the correlation of the scene with the target — a matched filter, with a bright peak wherever the target appears.

The last bullet is why the word correlator stuck. Shine in a photograph of a harbour, put the matched filter for a particular ship in the Fourier plane, and the output plane lights up with a sharp peak at the ship's location — template matching over the whole scene, all positions tested simultaneously, in one transit of light.

Masks you can think with

Three filters cover most of the intuition. An iris (a hole around the centre) passes only low frequencies: the output is a blurred, smoothed image — convolution with a broad blur kernel. An opaque dot on the centre blocks the DC and low frequencies: what survives is edges and fine texture, glowing on a dark field — this is dark-field imaging, beloved of microscopists a century before anyone called it a high-pass filter. And a grating-like mask with transmission varying as a shifted sinusoid displaces features — the shift theorem made of plastic. Zernike's phase-contrast microscope, which won the 1953 Nobel Prize, is exactly a 4f system with a tiny phase-shifting dot in the Fourier plane: arguably the first analog optical computer to ship in volume.

Now the flagship demonstration. The program below is the 4f bench in software: a hand-rolled discrete Fourier transform plays each lens, a pointwise multiply plays the mask, and we check the result against direct convolution — the sliding multiply–accumulate a CPU would do:

// The 4f correlator in software: lens = DFT, mask = pointwise multiply, lens = DFT back. type C = { re: number; im: number }; const c = (re: number, im = 0): C => ({ re, im }); const add = (a: C, b: C): C => c(a.re + b.re, a.im + b.im); const mul = (a: C, b: C): C => c(a.re * b.re - a.im * b.im, a.re * b.im + a.im * b.re); // A lens, as arithmetic: the discrete Fourier transform (sign −1) and its inverse (+1). function dft(x: C[], sign: number): C[] { const N = x.length; const out: C[] = []; for (let k = 0; k < N; k++) { let acc = c(0); for (let n = 0; n < N; n++) { const ang = (sign * 2 * Math.PI * k * n) / N; acc = add(acc, mul(x[n], c(Math.cos(ang), Math.sin(ang)))); } out.push(sign > 0 ? c(acc.re / N, acc.im / N) : acc); // inverse carries the 1/N } return out; } const signal = [0, 0, 1, 3, 5, 3, 1, 0, 0, 0, 2, 4, 2, 0, 0, 0]; // the "scene", N = 16 const kernel = [1, 2, 3, 2, 1]; // a small blur kernel const N = signal.length; // Zero-pad the kernel to N — the mask must span the whole Fourier plane. const padded: C[] = signal.map((_, i) => c(i < kernel.length ? kernel[i] : 0)); // Optical route: transform, apply the mask, transform back. const S = dft(signal.map((v) => c(v)), -1); const H = dft(padded, -1); const optical = dft(S.map((s, k) => mul(s, H[k])), 1).map((z) => Math.round(z.re * 100) / 100); // Electronic route: direct circular convolution — N·K multiply–accumulates. const direct: number[] = []; for (let i = 0; i < N; i++) { let acc = 0; for (let j = 0; j < kernel.length; j++) acc += kernel[j] * signal[(i - j + N) % N]; direct.push(acc); } console.log("optical (2 DFTs + mask): " + optical.join(" ")); console.log("direct (slide + MAC): " + direct.join(" ")); let worst = 0; for (let i = 0; i < N; i++) worst = Math.max(worst, Math.abs(optical[i] - direct[i])); console.log("largest disagreement: " + worst);

Identical answers, two philosophies. The direct route costs N \cdot K multiply–accumulates and grows with the kernel; the "optical" route costs two transforms and a pointwise multiply — and on the bench, both transforms and the multiply are free. Only the loading of the scene and the reading of the result cost anything. Note one honest wrinkle the code surfaces: the DFT route computes circular convolution (the ends wrap around); an optical system's finite apertures impose their own version of the same edge caveat.

The CNN connection — and where it strains

A convolutional layer applies a bank of small kernels to an image, at every position. A 4f system applies one kernel to an image at every position, in one pass, with a kernel-size cost of zero. The mapping is irresistible, and it has been built many times: replace the fixed transparency with a spatial light modulator and the Fourier-plane mask becomes a programmable weight bank — an optical conv layer whose inference energy is dominated purely by I/O (optical neural networks covered the fully-connected cousin). The strain appears between layers. A CNN needs a nonlinearity after each convolution, and passive linear optics cannot supply one — so each layer's output must be detected, squashed electronically, and re-modulated onto light, paying the full I/O bill at every layer boundary (the nonlinearity problem). The economics therefore favour optics for the first, largest layer — huge images, big receptive fields, weights fixed at manufacture — with electronics taking over once the data has shrunk. Exactly this division of labour keeps reappearing in photonic-accelerator proposals, and you now know the reason it sits where it does.

There is a fraud hiding in the matched filter. The filter H = G^* is complex-valued — it must adjust the phase of every frequency component — but a photographic transparency can only attenuate: its transmission is a real number between 0 and 1. For years that seemed to kill the idea. In 1964, Anthony Vander Lugt, working at the University of Michigan's radar lab, produced the fix: record the mask holographically. Interfere the desired filter field with a tilted reference beam and photograph the fringes; the resulting real-valued transparency, placed in the Fourier plane, diffracts light into several orders, and one of those orders carries exactly the complex product G^* \cdot (input spectrum) — the impossible complex multiplication, smuggled inside an ordinary black-and-white photograph. The correlation peak appears off-axis, cleanly separated from the unwanted orders. Cold War money loved it: automatic target recognition — find the tank in the aerial photograph — funded optical correlators for decades, and fingerprint matchers and docking sensors built on Vander Lugt's trick flew well into the digital era. It is also your preview of the next lesson: holography is precisely the art of storing a complex field in a real medium.

The 4f system's output lives in inverted coordinates — two forward transforms in a row give g(-x,-y), not g(x,y) — and whether your machine computes convolution g * h or correlation g \star h depends on a conjugate: multiplying by H gives convolution with h, while multiplying by H^* gives correlation (the kernel slides without being flipped). The two differ exactly when the kernel is asymmetric — which is precisely the interesting case in matched filtering. Deep-learning practice adds a layer of comedy: the "convolution" in virtually every CNN framework is actually correlation (no kernel flip), harmless there because the network learns the kernels either way — but fatal if you naively transplant trained CNN weights onto an optical bench with the wrong sign convention. When an optical correlator's peak mysteriously lands at the mirror-image position, or a transplanted kernel sharpens instead of blurs, check the flip before checking the alignment.

Where this goes next

Everything above hinged on one component we waved into existence: a mask that stores a complex field. Vander Lugt made one photographically; the general technique — recording and replaying complete optical fields, amplitude and phase — is holography, and it turns out to be not just a way to make filters but a candidate memory technology, with terabytes per crystal and content-addressable recall. That is the next lesson.