Every piece is finally on the table: a network that can compute a prediction, a cost function that
scores how wrong that prediction was, backpropagation that turns the score into a gradient for every
weight, and gradient descent that uses the gradient to improve. Training is simply
wiring those four pieces into one repeating cycle — the very same
That five-step cycle, run over and over — often for hours, sometimes for weeks — is all that "training a neural network" means, from the tiniest toy classifier you can fit on one screen to the largest language models running on stadium-sized data centres. Nothing new gets invented at scale; the exact same loop just runs for far, far longer.
Remember the free-throw shooter from the training loop: shoot, see how far off, adjust, shoot
again. Let's replay that cycle once more, but this time every step gets its real, technical name.
Take the simplest possible "network" — a single weight
| Round | Forward: | Loss: | Backward: gradient | Update: new |
|---|---|---|---|---|
| 1 ( | ||||
| 2 ( | ||||
| 3 ( |
Watch the loss column:
These two classes spiral together — no straight line can separate them. Step through training and watch the network bend its decision boundary into a curve that wraps around the data, while the loss falls. A linear model could never do this; the hidden layers' non-linearity is what makes the curved boundary possible. Each step you drag through is one more round of exactly the forward → loss → backward → update cycle from the table above — just running on a network with far more than one weight.
Real training sets rarely fit in memory as one training example — they hold thousands, millions, sometimes billions of examples. Two words describe how that flood of data gets sliced up for the loop:
Put them together: a dataset of
"How many examples per batch?" turns out to matter a lot, and training comes in three common flavors depending on the answer:
| Flavor | Batch size | Character |
|---|---|---|
| Full-batch | the entire training set, every update | very stable, accurate gradient — but painfully slow, since nothing updates until the whole dataset has been scanned |
| Stochastic (single-example) | just one example per update | updates constantly and cheaply — but each gradient is a noisy, jittery estimate based on a single example |
| Mini-batch | a modest chunk (say, 32–256 examples) | the practical middle ground almost everyone actually uses: frequent updates, a reasonably smooth gradient, and batches sized to fit neatly on the training hardware |
When people casually say a network was trained with "batch size 64," they mean this mini-batch flavor — running steps 1–4 on 64 examples at a time, updating, then moving to the next 64, all the way through the epoch.
In practice, training is part science, part art: choosing the learning rate, the batch size, the
network shape, the
Settings chosen before training begins — the learning rate, the batch size, how many epochs to run, the shape of the network itself — are called hyperparameters, to distinguish them from the weights the loop itself learns. Picking good hyperparameters is often a trial-and-error search of its own: train several candidate settings, watch each one's validation loss, and keep whichever configuration generalizes best. A common safety net is early stopping — watching the validation loss epoch by epoch and halting training the moment it stops improving, even if the training loss is still falling, which is often the clearest early sign that overfitting is about to begin.
A falling loss curve is the goal, but training doesn't always cooperate. Three classic failure patterns, each needing a different fix:
There's also a cost to training that has nothing to do with the maths: it's genuinely expensive. Real, large networks need specialized hardware — GPUs or TPUs built to do millions of multiplications in parallel — and even then can take enormous amounts of computer time and electricity to finish.
The tiny examples on this page — one weight, a handful of points, a spiral of two colours — train completely in a fraction of a second on an ordinary laptop. Scale the very same forward–loss–backward–update loop up to a modern large language model, and training can take months of continuous computation across thousands of specialized chips, at a cost that runs into the tens or hundreds of millions of dollars in electricity and hardware time. Same four steps, same loop — just run an almost unimaginable number of times, on an almost unimaginable amount of data.
One habit unites toy projects and industrial-scale training runs alike: engineers stare, for hours, at a graph of the loss (or "loss curve") ticking downward over time — exactly the shrinking numbers you watched in the worked example's table above. Whether the run costs nothing or costs a fortune, that falling curve is the one signal everyone is watching for.
It's a strange kind of patience, watching a number get smaller for weeks on end — but it's exactly how nearly every model you've ever used got built: a search engine's ranking model, a phone's voice-to-text, a translation app, a chatbot. Somewhere, a loss curve like the one in this page's table crept downward, round after round, until someone decided it was good enough to ship.