Training Photonic Neural Networks

Here is the failure mode that greets everyone who builds their first photonic accelerator. Train a network in simulation — a pristine digital twin of the chip — to 98% accuracy. Compile the weights to phase settings, download them to the hardware, run the test set. Accuracy: 70-something. Nothing is broken; every component is "in spec". The digital twin simply isn't the device. Coupler ratios are a percent off, phase shifters have miscalibrated response curves, neighbouring heaters leak into each other, the activation (lesson 4) has a slightly different shape than the ReLU you trained with, and the whole ensemble drifts with the lab's air conditioning. Each error is small; a deep analog pipeline multiplies them together. Offline-trained weights degrade on real analog hardware — reliably, quantitatively, and worse as the system grows. This lesson is about the four families of fixes, which range from "measure the device harder" to the genuinely elegant: making the light itself run backpropagation.

Why small errors become big networks' problems

A mesh implements its matrix through many devices in series: light bound for any output traverses O(N) MZIs, and the matrix entry it realises is a product of every coupler and phase along the way. Independent per-device errors therefore accumulate — in the small-error regime the matrix error grows like \sigma\sqrt{N_{\text{dev}}} for per-device deviation \sigma — until it saturates at "a completely different matrix". The chart below shows this first-order scaling model: slide the mesh size up and watch how the same per-device quality delivers a rapidly worsening matrix.

Read the design consequence off the chart: doubling the mesh size at fixed accuracy demands halving the per-device error — devices must improve quadratically as systems scale. Since device physics will not oblige forever, the alternative is to stop demanding that the hardware match the model, and instead make the training absorb the mismatch. There are four ways, in escalating order of intimacy with the hardware.

The four families

Strategy	Forward pass	Gradient	Absorbs static error?	Absorbs noise/drift?
1. Calibrate-then-compile	—	Offline, on the twin	Only what calibration catches	No
2. Hardware-in-the-loop	On the chip	Digital backprop from measured outputs/activations	Yes — the real device is inside the loop	Partially (it trains on the noise it sees)
3. In-situ physical backprop	On the chip	Optical — an adjoint field runs backward through the same mesh	Yes	Partially
4. Physics-aware / noise-aware training	Simulated, with device model + injected noise	Offline, through the device model	Statistically (trains for robustness, not for one device)	Yes — by finding flat minima

Hardware-in-the-loop is the workhorse: run the forward pass on the physical chip, measure the outputs (and, where instrumented, the hidden activations), and compute weight updates digitally as if those measurements were the network's true state. Whatever static transformation the hardware actually applies — miscalibration included — is what the gradient sees, so training steers around the device's quirks instead of colliding with them. The catch is throughput: every gradient step spends real chip passes, and the naive version — finite differences, perturbing one parameter at a time — needs P + 1 forward passes per step. For a modest mesh with P = 10^4 phases and 10^4 training steps, that is 10^8 chip evaluations before you have trained anything. The classic escape is simultaneous perturbation (SPSA): perturb all parameters at once with a random \pm\delta pattern and estimate the directional gradient from just two passes per step, trading pass count for gradient variance. Between those extremes sit the modern physics-aware autodiff methods, which backpropagate through a differentiable model of the device while clamping it to hardware measurements.

Physics-aware training (family 4) attacks the problem before the hardware is even involved: during digital training, inject into every MAC the noise statistics of lesson 5 and the measured transfer curves of the real components. The optimiser then avoids sharp minima that depend on exact weights and settles into flat basins where perturbations cost little — the same flat-minima logic as quantisation-aware training. The resulting weights are not tuned to any single chip, but they are robust to the whole distribution of chips — which is exactly what you want for manufacturing at volume, where per-unit hardware-in-the-loop fine-tuning is a cost, not a feature.

The most beautiful idea in this corner of the field is due to Hughes, Fan and colleagues (2018): the mesh can compute its own gradients. Backpropagation's backward pass multiplies by W^{\mathsf T} — and a reciprocal optical mesh, run backwards, applies exactly that transpose. Send the forward field through; then inject the error vector from the output side, so an adjoint field propagates backward through the very same interferometers. At each phase shifter, the gradient of the loss with respect to that phase is proportional to a simple interference term between the local forward and adjoint fields — a quantity readable in place with a small power monitor. Three optical passes and every one of thousands of gradients has been measured, in parallel, by physics — no digital twin, no finite differences, and the measurement automatically includes the device's real imperfections. It remains a laboratory technique (it needs coherent access to both ends and per-device monitors), but as a proof of principle it settles an old question: analog hardware need not outsource its learning to a digital chaperone.

The recurring institutional mistake — it has burned real product teams, not just students — is to treat training as a one-time offline event followed by deployment of frozen weights, the way GPU inference works. On analog hardware that pipeline silently assumes three false things: that the digital model matches your specific die (fabrication variation says no), that the match is stable in time (thermal drift says no — lesson 3's rings re-tune continuously for a reason), and that accuracy at deployment equals accuracy measured at bring-up (ageing lasers and shifting bias points say no). A production photonic accelerator is a closed-loop system: it ships with calibration firmware, monitors, and a schedule of re-trims — and its training story must say which family above runs at the factory, which runs in the field, and how often. When a paper reports hardware accuracy, the first-order question is: measured how long after calibration? A number taken five minutes after a full re-trim and a number taken a week later can differ by more than the entire claimed advantage over electronic baselines — and honest papers now report both.

Where this goes next

Suppose every problem so far is solved: the matvec flies, the nonlinearity is paid for, the noise is budgeted, the training absorbs the hardware's sins. One question remains, and it is the one that decides whether any of this ships: does it actually save energy? The final lesson of this module draws up the complete energy ledger — laser, modulators, detectors and, above all, the DACs and ADCs — and finds the honest crossover point against a tensor core.