The Training Loop

Think about learning to shoot basketball free throws. You don't read a manual and suddenly sink every shot. You take a shot, watch where it lands — short, long, veering left — and adjust your aim a little. Then you shoot again. Short by less this time. Adjust again. Shoot again. After a few hundred repeats of try, see how far off you were, adjust, your arm has quietly learned the right motion, without you ever writing down a formula for it.

Training a machine learning model is exactly this cycle, run by a computer instead of an arm, and repeated not hundreds but often millions of times. That cycle is called the training loop, and it has four steps:

Predict. Run the current model on a training example and see what it guesses.
Measure. Compare the guess to the true answer — how wrong was it? That "how wrong" number is called the loss.
Adjust. Nudge the model's internal numbers a little, in whichever direction makes that loss smaller next time.
Repeat. Go back to step one, with slightly better numbers than before.

That's it. There's no secret fifth step. A model that plays chess, translates languages, or recognises your voice was built by running this same humble loop — predict, measure, adjust, repeat — an enormous number of times.

Watching the error shrink, round by round

Let's shrink the loop down to something you can run entirely by hand. Suppose our "model" is the simplest one imaginable: it guesses y = m \cdot x for some number m that we're allowed to adjust. There's just one knob to turn.

Say the true pattern in the world is y = 3x, and we have one training example: when x = 2, the true answer is y = 6. We don't know the true pattern is "times 3" — we have to discover it by looping. Start with a bad guess, m = 1, and nudge by 0.5 whenever the prediction is too low:

Round	Current m	Predict	True	Error	Adjust
1	1.0	1.0 \times 2 = 2	6	4 (too low)	bump m up to 1.5
2	1.5	1.5 \times 2 = 3	6	3 (too low)	bump m up to 2.0
3	2.0	2.0 \times 2 = 4	6	2 (too low)	bump m up to 2.5
4	2.5	2.5 \times 2 = 5	6	1 (too low)	bump m up to 3.0

Look at the error column: 4, 3, 2, 1, \ldots — shrinking every single round, with no human ever telling the model "the answer is 3". It found m = 3 purely by predicting, measuring how wrong it was, and adjusting in the direction that helped. A real model has thousands or millions of knobs instead of one, and adjusts all of them at once, but the shrinking-error story is identical.

Watch it learn

Drag the step slider to advance the training loop on a slightly bigger example — a line being fit to nine data points instead of one. The line starts flat and useless; with each step it's nudged toward the data and the loss printed above the graph drops. You're watching predict–measure–adjust play out visually — the same loop, whether the model is a single line or a giant neural network.

What's an "epoch"?

Real training sets don't have one example — they have thousands, millions, sometimes billions. You could adjust the knobs after every single example, but it's common to gather up predictions and errors across the whole dataset before deciding how to adjust. One full pass through every training example, once each, is called an epoch.

"Training for 10 epochs" means the model looped over the entire dataset ten separate times, adjusting its numbers a little after each pass (or, often, many small adjustments within each pass). "Training" is simply running the loop, epoch after epoch, until the loss stops getting smaller.

Picture a dataset of 60{,}000 labelled photographs, being trained for 20 epochs. That's 60{,}000 predict–measure–adjust opportunities in each epoch, repeated 20 times over — well over a million individual trips around the loop, all to fine-tune the same handful of ideas: guess, check, nudge, go again. In practice, models rarely wait for a full epoch's worth of examples before adjusting — they typically look at a small batch at a time (a "mini-batch") and nudge after each one — but the underlying loop, and the reason it's called an epoch when the whole dataset has been covered once, stays exactly the same.

Yes to all three — and each mistake has its own name and its own fix:

Too few rounds: stop the loop early and the model simply hasn't finished learning yet. It stays inaccurate on everything, training data included — an under-learned model that never got the chance to shrink its error down.
Too many rounds: counter-intuitively, running the loop for far too long can make things worse in a different way. Instead of learning the general pattern, the model starts memorising accidental quirks and noise specific to its training examples — a problem with its own name, overfitting, which you'll meet properly soon.
Too big a nudge: each adjustment in the loop is usually kept deliberately small and cautious. A huge nudge can overshoot the good answer entirely and bounce the model's numbers wildly from one round to the next, sometimes making the loss worse. How big to make each nudge is controlled by a setting called the learning rate — another idea waiting just up ahead.

You can see the overshoot problem in the very same one-knob example from before. This time, instead of a cautious nudge of 0.5, try a reckless nudge of 4 whenever the prediction is too low or too high:

Round	Current m	Predict	True	Error
1	1.0	2	6	4 (too low) → wildly overshoot to m = 5.0
2	5.0	10	6	4 (too high!) → wildly overshoot back to m = 1.0
3	1.0	2	6	4 (too low) → back to m = 5.0

Notice the error never shrinks — it just bounces between 4 and 4 forever, flipping the model between an m that's far too small and one that's far too large. Compare that to the small, cautious nudges from the worked example earlier, where the error calmly shrank 4, 3, 2, 1, \ldots every round. Same loop, same data — the only difference is the size of the nudge, and it's the difference between learning and never settling down at all.

In 2016, DeepMind's AlphaGo beat one of the world's strongest Go players — a game with more possible positions than atoms in the observable universe, far too many to search by brute force. How? AlphaGo (and its successor, AlphaZero) trained by playing millions of games against itself, starting from knowing almost nothing.

Every single game was just one more spin of the exact loop you learned today: predict a good move, measure whether the game was eventually won or lost, adjust the internal numbers to make winning moves more likely next time, repeat. Nothing about the loop was different from our one-knob line — there were just vastly more knobs, and vastly more repeats. That's also the honest answer to "why does training a huge model take days or weeks of computer time?" — it's this same simple cycle, just run an almost unimaginable number of times.

The same loop, all the way up

Astonishingly, this four-step loop is essentially all of model training — from this toy line to systems with billions of knobs. What changes is only how the "adjust" step is computed, and that's the engine called gradient descent, which we build next. Master the loop and you understand the heartbeat of machine learning.