A Layer of Neurons

A single neuron computes one weighted sum. Real networks rarely stop at one — they put many neurons side by side, all reading the very same input at once, and call the whole group a layer. Each neuron in the layer has its own private set of weights, so even though every neuron sees identical inputs, each can learn to notice something different — one might lean towards detecting an edge, another a colour, another a texture. Nothing is shared between them except the input they all look at.

This is exactly the situation matrix–vector multiplication was built for: many dot products against the same vector, computed together. A layer of neurons is that multiplication happening for real.

Picture a layer that looks at a fruit's weight and its colour brightness — two numbers in, several neurons reading both. One neuron's weights might end up large on weight and near-zero on brightness, so it effectively becomes a "heaviness detector." Another might do the opposite and become a "brightness detector." A third might weight both roughly equally and end up detecting something like "overall bigness." Same two inputs, three completely different questions asked of them, all computed in one pass — that's the whole value of putting neurons in a layer rather than using them one at a time.

Without matrix notation you'd have to write out one equation per neuron by hand — z_1 = w_{11}x_1 + w_{12}x_2 + \dots + b_1, then z_2 = w_{21}x_1 + w_{22}x_2 + \dots + b_2, and so on, one line for every neuron in the layer. \vec{z} = W\vec{x} + \vec{b} says the exact same thing in one line, no matter whether the layer has three neurons or three thousand. That's not just tidier notation — it's the reason a layer can be handed straight to a computer as a single instruction rather than a loop written out by hand.

Each output is a row · input

Three inputs feed a layer of two neurons. Pick an output neuron and its connections light up: its value is that row of W dotted with the input vector, plus a bias. Two neurons, two dot products — and together they are precisely W\vec{x} + \vec{b}.

Work it by hand with the numbers shown above: \vec{x} = (3, 1, 2), row 1 of W is (1, 0, 2) with bias 0.5, and row 2 is (-1, 2, 1) with bias -0.5. Neuron 1's raw output is (1)(3) + (0)(1) + (2)(2) + 0.5 = 7.5, and neuron 2's is (-1)(3) + (2)(1) + (1)(2) - 0.5 = 0.5. Two separate rows, two separate sums — computed together as one matrix–vector product.

Then squash each one

A raw weighted sum on its own is just a number — it still needs an activation function before it becomes the layer's real output. Apply the sigmoid, \sigma(z) = 1/(1+e^{-z}), to each of the two raw sums above:

a_1 = \sigma(7.5) \approx 0.9994, \qquad a_2 = \sigma(0.5) \approx 0.622.

Every entry of \vec{z} gets squashed the same way, independently — the first neuron ends up almost fully "on", the second only moderately so. That elementwise squash is what stops a stack of layers from secretly behaving like one giant straight line; the full story of why and which squashing function to use lives in activation functions.

Notice that squashing never mixes neurons together — neuron 1's activation depends only on neuron 1's own raw sum, never on neuron 2's. Whatever "detector" each neuron became through its weights stays that neuron's own business right through to the activation. Only the next layer, by reading every activation at once, gets to combine what the different neurons noticed.

Worked example: how many weights, really?

Take a smaller layer: 2 inputs, 3 neurons. How many individual weight numbers does the layer need to learn? It's tempting to guess 3 (one per neuron) or 2 (one per input) — but every one of the 3 neurons needs its own weight for each of the 2 inputs, so the true count is 3 \times 2 = 6 weights, arranged as a 3\times 2 matrix, plus 3 biases (one per neuron):

W = \begin{pmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \\ w_{31} & w_{32} \end{pmatrix}, \qquad \vec{b} = \begin{pmatrix} b_1 \\ b_2 \\ b_3 \end{pmatrix}.

Each row still belongs to one neuron, each column still belongs to one input — grow either number and the matrix grows to match. Swap in the earlier example's numbers (3 inputs, 2 neurons) and you get a 2 \times 3 matrix instead — six weights again, just arranged the other way round, because now it's 2 neurons each reading 3 inputs.

Why phrase it as a matrix

Because computers are devastatingly fast at matrix multiplication — it's exactly what GPUs are built for. Writing a layer as W\vec{x} means a whole layer of thousands of neurons runs as a single, blisteringly fast operation, and a whole batch of inputs becomes one matrix–matrix multiply. Linear algebra isn't just the notation for neural networks; it's the reason they're fast enough to be useful.

Put a number on it: a modest image-recognition layer might take 784 pixel values as input and feed 512 neurons. That's a weight matrix with 512 rows and 784 columns — 401,408 individual weights, computed for one image in a single matrix–vector multiply. A real network stacks several such layers and processes thousands of images at once; without the matrix framing, that's an unmanageable pile of individual sums.

There's no formula that hands you the perfect number — a layer's neuron count is a genuine design dial that engineers tune by hand and by experiment, the way a recipe's ingredient quantities get adjusted to taste. Too few neurons and the layer may be too simple to notice the patterns that matter (it can't even represent enough different "row detectors"); too many and the network gets slower to run, harder to train, and — as the earlier warning about network size mentioned — more prone to memorising its training examples instead of learning from them. Picking layer widths well is part science, part trial and error, and part hard-won experience.

A quick way to catch both mistakes at once: if someone tells you a layer has "10 neurons" but can't tell you how many inputs feed it, you don't yet know how big its weight matrix is — and if they hand you a weight matrix without saying which axis is rows and which is columns, you don't yet know which end the input vector plugs into. Always ask for both numbers.

GPUs — Graphics Processing Units — were built for one job: pushing millions of pixels onto a screen, sixty times a second, by doing enormous numbers of small matrix and vector calculations in parallel. Nobody designed them with neural networks in mind. But a layer of neurons is also just an enormous number of small matrix and vector calculations, done over and over — and it turns out a chip built for rendering dragons and racetracks is spectacularly good at it too. That happy coincidence, exploited from the early 2010s onward, is one of the unglamorous reasons deep learning suddenly became practical: the hardware to run it at scale was already sitting in gaming PCs.

A modern GPU can carry out many trillions of these little "multiply, then add" steps every second, spread across thousands of tiny cores working at once. A layer's W\vec{x} is precisely that operation, repeated once per weight — so hardware built to shade a few million pixels turned out, purely by luck of the maths, to be exactly what deep learning needed to grow up.

See it explained