Hardware–Algorithm Co-Design

A photonic mesh multiplies matrices at the speed of light — and a matrix multiplier is not a computer. Every real workload is a braid of operations, only some of which optics does well, and the architect's job is to cut the braid in the right places: photonics where light is overwhelming, electronics where it is hopeless, and as few border crossings as the interface lesson will let you afford. This is the same discipline that produced the TPU and every other successful accelerator — the argument of domain-specific architecture — applied to a domain with unusually sharp edges. And it cuts both ways: the algorithm is not a fixed customer to be served but a design variable to be bent. Networks can be retrained to tolerate the precision optics offers, restructured to reuse the matrices optics holds, and rearranged to cross the analog border less often. Co-design means moving both sides toward the seam.

Drawing the partition

Start with the honest inventory. What follows is the consensus partition that essentially every credible photonic-AI architecture converges on:

Operation	Domain	Why
Large dense matrix–vector products	Photonic	O(N²) MACs per pass, energy amortised by N, latency = time of flight
Convolutions	Photonic	Lowered to matmul (im2col) or spread across wavelengths in a WDM engine
Nonlinear activations	Electronic	Optical nonlinearity is weak and power-hungry — the verdict of the nonlinearity lesson
Softmax, normalisation, residuals	Electronic	Exponentials, divisions and reductions across channels — cheap digitally, absurd optically
Weight and activation storage	Electronic	No dense, fast, cheap optical memory exists (see the vignette below)
Control, scheduling, calibration	Electronic	The analog computer's digital chaperone, per the previous lesson

The pattern is stark: photonics gets exactly the operations whose cost grows quadratically with width, electronics keeps everything that is linear, pointwise or stateful. That is not a defeat — in a large transformer, the quadratic parts are 90-plus percent of the arithmetic. It does mean the system's speedup is governed by an old and unforgiving law.

Amdahl, again

If a fraction p of the work runs on the mesh with speedup s, and the rest stays electronic at its old pace, the overall gain is

S \;=\; \frac{1}{(1-p) + p/s}, \qquad S \;\xrightarrow{\,s\to\infty\,}\; \frac{1}{1-p}.

Accelerate 90% of the work infinitely and you have gained a factor of ten, full stop — the surviving 10% is now your whole runtime. Worse, the electronic remainder in a photonic system is not idle bookkeeping: it includes every activation function, every softmax, and every DAC/ADC crossing, so p is decided by where you draw the partition and how often you cross it. Play with the chart: the lesson it teaches is that pushing the mesh's raw speedup s ever higher buys almost nothing once s \gg 1/(1-p) — the leverage is all in growing p, which is an algorithm problem, not a photonics problem.

Mapping a transformer onto light

Now the workload everyone actually cares about. A transformer layer contains three families of matrix multiplies, and they are not equally photonic. The projection matrices W_Q, W_K, W_V, W_O and the two MLP matrices are static weights: program them into a mesh once, stream every token of every request through them, amortise perfectly. But the heart of attention — QK^{\mathsf T} and the attention-weighted sum AV — multiplies activations by activations. Both operands are fresh for every input. A thermo-optic mesh that takes microseconds to reprogram cannot chase data that changes every nanosecond; serving these matmuls optically requires loading one operand through fast modulators at the symbol rate, which drags the full conversion toll back in — DACs on both sides now — or leaving QK^{\mathsf T} to the electronics. Between attention matmuls sit softmax (electronic, with two conversions) and the KV-cache (electronic memory, growing with context length). The pragmatic mapping that emerges: meshes hold the static weights — projections, MLPs, anything reused across a batch — while dynamic attention stays electronic, and the architecture lives or dies on how much of the model that static fraction covers. For convolutional networks the news is better: kernels are static weights par excellence, and a WDM engine can fan one kernel across many wavelengths at once — convolution is photonics' most natural workload, as the 4f correlator already hinted in the analog module.

Bending the algorithm toward the optics

The other half of co-design edits the workload itself. Four moves matter in practice. Train for low precision and noise: quantisation-aware and noise-aware training (the direct descendants of mixed-precision training) produce networks that keep their accuracy at the 4–6 bits an analog mesh can honestly deliver — turning the interface's exponential price list from a wall into a ramp. Prefer structured matrices: circulant and Toeplitz weight structures turn a matmul into a Fourier-domain elementwise product, which fixed optics performs essentially for free; pruning to structures the mesh natively holds beats pruning to arbitrary sparsity, which optics cannot exploit at all. Maximise reuse: schedule inference so each programmed layer sees the largest possible batch before the mesh is retuned — the batching logic of every weight-stationary accelerator, sharpened by the mesh's slow reprogramming. Minimise crossings: fuse consecutive linear operations into one optical pass (two chained matmuls are one matmul — precompute the product), and place the unavoidable nonlinearities where a single detect–redrive already had to happen. Each move is invisible in a FLOP count and decisive in an energy budget.

The recurring newcomer's proposal — and, historically, the pitch of more than one funding round — is to eliminate the electronic side entirely: optical nonlinearities between mesh layers, optical memory holding the weights, photons in, answers out. Each ingredient fails independently. Optical nonlinearity at useful strength demands power that erases the energy advantage (Module 7's post-mortem applies with full force). Optical memory dense and durable enough to hold a model does not exist. And an all-optical pipeline still needs calibration, control and I/O — the digital chaperone never leaves. The honest system is heterogeneous, and the design question is never "how do we remove the electronics?" but "how rarely can we visit it?". Treat any architecture whose block diagram contains no ADCs with the suspicion you would give a perpetual-motion machine: somewhere off the slide, a converter is paying the bill.

It is a fair question — the course did meet an optical memory. Module 6's holographic storage lesson showed a medium that stores a matrix as an interference pattern and performs the matmul in the act of readout — the dream fused into one device. Volume holograms achieved striking densities in the lab, and phase-change photonic cells today hold mesh weights non-volatilely. What none of them achieve together is the triad DRAM offers: density, unlimited fast rewriting, and cost. Holograms rewrite slowly and fatigue; phase-change cells endure limited cycles and hold few bits; and neither approaches electronic storage's dollars per gigabyte. So weights live in SRAM and DRAM, are spent into the mesh through DACs, and the KV-cache — the fastest-growing memory in modern inference — never leaves the electronic side at all. Memory, more than the matmul, is why the electronic sea will keep its islands for the foreseeable future.

Where this goes next

Co-design gives you a machine that is defensible on paper: mesh-resident weights, electronic nonlinearities, minimal crossings, an algorithm retrained to meet the hardware halfway. The next question is how to measure such a hybrid honestly — what its TOPS/W figure means, what got counted, and how to compare it against a GPU without fooling yourself or being fooled. That forensic craft is the next lesson.