Photonic Reservoir Computing
Module 5 left a bruise that never quite healed: photonic neural networks are glorious at
inference and miserable at training. Every weight lives in a physical component
that drifts, every gradient must be estimated through hardware imperfections, and
training
the mesh ends up fighting the very physics that made inference cheap.
Reservoir computing is the judo response: stop training the network.
Take any fixed, random, nonlinear dynamical system — the "reservoir" — drive it with your input
stream, and let it do whatever it naturally does. Its internal state becomes a rich, churning,
nonlinear echo of the input's recent history. Then train exactly one thing: a linear
readout that maps the observed state to the desired output. All the hard, physical,
drift-prone machinery is untouched and untrained; the entire learning problem collapses to linear
regression — one matrix solve, no gradients, no backpropagation through glass.
For photonics this is close to a perfect fit. The
previous
lesson stored static patterns in a medium; here the "memory" is dynamic — light
circulating in a loop — and the machine computes with its own transient dynamics. What
optics lacks (precise programmability) the recipe never asks for; what optics has in
embarrassing surplus (fast, cheap, high-dimensional nonlinear dynamics) is exactly the
ingredient.
The recipe, and the hardware trick that made it photonic
A classical (electronic) reservoir is a sparse random recurrent network with a few hundred nodes.
Building hundreds of coupled optical neurons sounds expensive — so the photonic community found a
beautiful shortcut: one nonlinear node plus a delay line. Feed
the node's output into a kilometre of fibre (round-trip time \tau) and
back into its input. Now sample the circulating signal every \theta
seconds: each of the N = \tau/\theta time slots behaves as a distinct
virtual node, coupled to its neighbours by the node's finite response time. One modulator
and a spool of fibre impersonate an N-neuron recurrent network,
time-multiplexed at gigahertz rates. To make the virtual nodes respond differently from
one another, the input is first multiplied by a fixed, random, fast-repeating
mask — without it, every slot would see the same drive and the network would
collapse to one effective neuron.
The readout: ridge regression or bust
Collect the reservoir's state vector \mathbf{x}(t) (the
N virtual-node values) at each time step into a matrix
X, and the training targets into Y. The
readout weights solve a linear least-squares problem — but never solve it naively. Reservoir
states are strongly correlated (neighbouring virtual nodes echo one another), so
XX^{\mathsf T} is ill-conditioned, and unregularised least squares
manufactures huge, delicately cancelling weights that amplify every microvolt of detector noise.
The standard cure is exactly the
Tikhonov
regularization you have already studied, here under its statistics alias
ridge regression:
W \;=\; Y X^{\mathsf T}\,\big(X X^{\mathsf T} + \lambda I\big)^{-1}.
The penalty \lambda trades training error against weight magnitude —
and small weights are not merely a statistical nicety here but a physical robustness
guarantee: the readout must keep working tomorrow, when the reservoir has drifted a little. In
practice \lambda is chosen by validation, and it is the single most
important hyperparameter the experimenter controls.
Watch the whole pipeline run. The program below builds a 24-virtual-node delay-loop reservoir with
an MZI-flavoured \sin^2 nonlinearity, and trains a ridge readout on a
task chosen to need both memory and nonlinearity — predicting
y(t) = u(t\!-\!1)\cdot u(t\!-\!2) from the stream
u(t). A linear readout of the raw input alone cannot do this at all;
the fixed random reservoir makes it a linear problem:
// A delay-loop reservoir + ridge readout, end to end.
const N = 24, T = 300, WASH = 5, SPLIT = 220;
let seed = 42; // deterministic pseudo-randomness
const rand = (): number => {
seed = (seed * 1664525 + 1013904223) % 4294967296;
return seed / 4294967296 - 0.5;
};
const mask: number[] = []; // the fixed random input mask
for (let i = 0; i < N; i++) mask.push(rand() > 0 ? 1 : -1);
const u: number[] = []; // the input stream
for (let t = 0; t < T; t++) u.push(rand());
// Reservoir dynamics: each virtual node is a sin² nonlinearity driven by the masked
// input plus the echo of its neighbour one loop ago. NOTHING here is ever trained.
const X: number[][] = [];
let prevRow: number[] = new Array(N).fill(0);
for (let t = 0; t < T; t++) {
const row: number[] = [];
for (let i = 0; i < N; i++) {
const drive = 2.2 * mask[i] * u[t] + 0.9 * prevRow[(i + N - 1) % N] + 0.25;
row.push(Math.sin(drive) ** 2);
}
X.push(row);
prevRow = row;
}
// Target needs memory AND nonlinearity: y(t) = u(t-1)·u(t-2).
const y: number[] = [];
for (let t = 0; t < T; t++) y.push(t < 2 ? 0 : u[t - 1] * u[t - 2]);
// Ridge regression on [states, bias]: solve (A + λI) w = b by Gaussian elimination.
function ridge(rows: number[][], target: number[], lambda: number): number[] {
const d = rows[0].length;
const A: number[][] = [];
const b: number[] = new Array(d).fill(0);
for (let i = 0; i < d; i++) A.push(new Array(d).fill(0));
for (let t = 0; t < rows.length; t++) {
for (let i = 0; i < d; i++) {
b[i] += rows[t][i] * target[t];
for (let j = 0; j < d; j++) A[i][j] += rows[t][i] * rows[t][j];
}
}
for (let i = 0; i < d; i++) A[i][i] += lambda;
for (let col = 0; col < d; col++) { // elimination
let piv = col;
for (let r = col + 1; r < d; r++) if (Math.abs(A[r][col]) > Math.abs(A[piv][col])) piv = r;
[A[col], A[piv]] = [A[piv], A[col]];
[b[col], b[piv]] = [b[piv], b[col]];
for (let r = col + 1; r < d; r++) {
const fac = A[r][col] / A[col][col];
for (let j = col; j < d; j++) A[r][j] -= fac * A[col][j];
b[r] -= fac * b[col];
}
}
const w = new Array(d).fill(0);
for (let i = d - 1; i >= 0; i--) {
let s = b[i];
for (let j = i + 1; j < d; j++) s -= A[i][j] * w[j];
w[i] = s / A[i][i];
}
return w;
}
const feats = (t: number): number[] => [...X[t], 1]; // states + bias
const trainRows: number[][] = [], trainY: number[] = [];
for (let t = WASH; t < SPLIT; t++) { trainRows.push(feats(t)); trainY.push(y[t]); }
const w = ridge(trainRows, trainY, 1e-4);
function nmse(from: number, to: number, model: (t: number) => number): number {
let err = 0, varY = 0, mean = 0, n = to - from;
for (let t = from; t < to; t++) mean += y[t] / n;
for (let t = from; t < to; t++) {
err += (model(t) - y[t]) ** 2;
varY += (y[t] - mean) ** 2;
}
return err / varY;
}
const reservoirOut = (t: number): number =>
feats(t).reduce((acc, xi, i) => acc + xi * w[i], 0);
console.log("reservoir readout, test NMSE: " + nmse(SPLIT, T, reservoirOut).toFixed(3));
// Baseline: the best LINEAR readout of the raw input u(t) alone (no reservoir).
const base = ridge(
u.slice(WASH, SPLIT).map((v) => [v, 1]),
y.slice(WASH, SPLIT), 1e-4);
console.log("linear-on-input baseline NMSE: " +
nmse(SPLIT, T, (t) => base[0] * u[t] + base[1]).toFixed(3));
The baseline scores an NMSE near 1 — no better than guessing the mean — while the reservoir
readout tracks the target well, despite none of its dynamics being designed for the
task. The fixed random system projected the input's history into a high-dimensional nonlinear
feature space where a hyperplane suffices; the only "learning" was finding the hyperplane.
What it's actually good at
Reservoirs are not general-purpose learners — they are temporal feature machines, and
their sweet spot is streaming signals that need modest precision at ferocious speed. The
headline photonic demonstrations are telling: channel equalization — undoing
the nonlinear distortion a fibre link inflicts on a signal — done optically at gigabaud rates on
the raw waveform; spoken-digit and header recognition at millions of words per
second; chaotic time-series prediction, the field's favourite benchmark since
the delay dynamics of the reservoir naturally match the delay structure of chaotic systems. The
common shape: the input is already a fast analog waveform (so no I/O conversion is needed on the
way in), the required output is low-dimensional (so the readout is cheap), and the value lies in
latency — the reservoir answers while a digital DSP would still be filling its input buffer.
Conversely, tasks needing long memory, exact symbolic manipulation, or high-precision arithmetic
are structurally wrong for it — the reservoir's fading memory and analog noise floor set hard
limits no readout can undo.
The idea seems too lazy to work — surely the network's weights must be about the task?
The resolution is one of the prettiest ideas in machine learning. A readout that is linear in the
state can represent any function that is linear in the features the state happens to
compute — so the reservoir's only job is to compute, incidentally, an enormous spray of diverse
nonlinear functions of the input's recent history. Random systems are astonishingly good at this:
what a designed system computes deliberately, a random high-dimensional system computes by
accident, many times over. This is the same mathematics as kernel methods and random-feature
regression — Cover's theorem guarantees that nonlinearly projecting data into enough dimensions
makes linearly-inseparable problems separable with high probability. Two properties, not design,
are required: separation (different input histories drive the state to different
places) and fading memory (the state forgets initial conditions, so it depends
on the recent input, not on how the machine was switched on — the "echo state property").
Almost any sufficiently rich dynamical system pitched between order and chaos qualifies — fibre
loops, silicon-photonic meshes, semiconductor lasers with feedback, and, in one famous 2003
experiment by Fernando and Sojakka, a literal bucket of water computing XOR with ripples.
Two classic ways to fool yourself with a reservoir. First: the moment you catch yourself tuning
the reservoir to the task — adjusting feedback strength, bias point or mask against the
test error, task by task — you have quietly started training a recurrent network by expensive
black-box search and forfeited the whole bargain; global hyperparameters are set once, by
validation, and then the reservoir is frozen. Second, subtler: with hundreds of virtual nodes the
linear readout has hundreds of free parameters, and unregularised least squares will happily
memorise your training set and its noise — reservoir papers live and die by the honesty of their
train/validation/test splits and their \lambda sweeps. The physical
version of the same sin: a readout trained at 9 a.m. slowly decays as temperature drifts the
reservoir away from the state distribution it was fit to. The saving grace is that retraining
costs one linear solve on fresh data — seconds, not GPU-days — which is precisely why the
architecture tolerates hardware that would be hopeless for a conventionally trained photonic
network.
Where this goes next
Reservoir computing uses analog optical dynamics as a feature map and lets electronics
draw the conclusions. The next lesson pushes the same philosophy to its extreme: build an optical
dynamical system whose equilibrium itself is the answer — a network of coupled
parametric oscillators that relaxes toward the ground state of a combinatorial optimization
problem. That machine is the
coherent
Ising machine.