Somewhere around 2017, GPU architects looked at the workloads pouring into their machines and noticed something almost embarrassing: the overwhelming majority of the arithmetic was one operation, repeated beyond counting — multiply two small matrices, add the result to a third. Deep learning had turned the world's compute demand into a monoculture of matrix multiplication. And the general-purpose SIMT machine, magnificent as it is, pays a tax on every single multiply: fetch an instruction, decode it, schedule it, read operands, write results — a whole administrative apparatus wrapped around one floating-point operation per lane.

So the architects did what architects always do when one operation dominates: they gave it its own silicon. A tensor core (NVIDIA's name; AMD says matrix core, Intel says XMX engine — the idea is the same) is a small, fierce matrix-multiply unit embedded inside each SM, sitting right next to the ordinary vector pipes. This lesson is about what that unit computes, why it multiplies in narrow precision but accumulates in wide precision, and why the headline "10×" speedup is something you must earn with data layout — not something you get for free.

One instruction, one tile: D = A·B + C

The unit of work is not a scalar and not a vector — it is a tile. A single matrix-multiply-accumulate (MMA) instruction says: take an m \times k tile A, a k \times n tile B, multiply them, and add the result into an m \times n accumulator tile:

with typical shapes like 16 \times 16 \times 16. That one instruction performs 16 \cdot 16 \cdot 16 = 4096 multiply-accumulates — roughly 8192 floating-point operations — and it is issued once, to a whole warp, which executes it cooperatively: all 32 lanes together own the tiles and drive the matrix unit as a team. In machine code it looks almost insultingly small:

Compare the ledger. On the ordinary FP32 vector pipes, one instruction buys one fused multiply-add per lane — 32 MACs per warp instruction, each dragging its share of fetch, decode, scheduling and operand routing behind it. The MMA instruction buys thousands of MACs for the same administrative cost. This is the amortisation-of-control argument, and it is the entire justification for the hardware: the energy and area of a modern processor are mostly spent on deciding what to do, not on doing it. Decide once; compute four thousand times.

The tile, drawn

The figure shows a small tile multiply. Step through it: first one dot product, then all of them at once, then the part that surprises everyone — where the tiles actually live.

That last step deserves a pause. The tiles are not in some private matrix memory — they are spread across the warp's ordinary register file, as fragments: lane 0 holds a few cells of A, a few of B, a few accumulators of D; lane 1 holds different ones; no single lane ever owns a whole row. The layout is chosen so the matrix unit can drink operands from all 32 lanes' registers at once, through the same operand-collector plumbing that feeds the vector pipes — the register file, as ever, is the real interface between "the programme" and "the arithmetic". You never address a fragment cell individually; you hand the warp's fragments to the MMA instruction and trust the layout.

Mixed precision: multiply narrow, accumulate wide

Now look again at the instruction above: inputs in FP16, accumulator in FP32. That asymmetry is the second big idea. Halving the width of the multiplier inputs roughly quarters the multiplier silicon (multiplier area grows with the square of operand width), so FP16 or BF16 inputs let you pack four times the multipliers into the same area and power. Neural networks tolerate low-precision products remarkably well.

What they do not tolerate is a low-precision accumulator. A dot product in deep learning is a long sum — hundreds or thousands of terms. Watch what happens if the running total is kept in FP16, which carries only 11 significant bits: once the total reaches 2048, the gap between representable numbers becomes 2 — so adding 1.0 does nothing. The sum 1 + 1 + 1 + \dots climbs to 2048 and then flatlines forever, silently. Every term after that is rounded out of existence. Keeping the accumulator in FP32 (24 significant bits) pushes that cliff out by a factor of 2^{13} — far beyond any realistic sum length. Hence the rule stamped into the silicon: multiply narrow, accumulate wide.

For inference the inputs shrink further — FP8 and INT8 formats double the multiplier count yet again (a trained network being used tolerates coarser products than one being trained) — but the accumulator stays wide (FP32, or INT32 for integer paths). The accumulator is always the last thing to give up its bits.

Watch the drift happen

Don't take the cliff on faith — run it. This demo quantises values to an FP16-like format (10 explicit mantissa bits) and accumulates a long dot product two ways: once re-quantising the running total to FP16 after every add, once keeping it wide.

Half the sum simply vanished — no error, no warning, just a wrong answer. Real networks sum messier values than 1.0, so the drift is subtler, but the mechanism is identical: a narrow accumulator rounds away small addends once the total is large. That is why tensor cores wire the accumulation path wide even when the multipliers are narrow.

The throughput arithmetic

How much does all this buy? Take a representative modern SM. Its FP32 vector pipes sustain about 128 FMAs per cycle — 256 flops. Its four tensor cores, fed FP16 fragments, sustain roughly 1024 FMAs per cycle between them — 2048 flops, an 8\times jump; drop to FP8 and the ratio doubles again to 16\times. Multiply across a hundred-plus SMs and the tensor path is where essentially all of a modern GPU's headline petaflops live. The vector pipes have become the supporting cast: address arithmetic, activations, normalisation, everything that isn't the matmul.

Cousin of the systolic array

If this smells familiar, it should: it is the systolic-array story, replayed inside a general-purpose machine. The TPU's answer to the matmul monoculture was one enormous array — a 256 \times 256 grid that is the whole chip, gloriously efficient and able to do nothing else. The GPU's answer is many small engines: a few modest matrix units per SM, embedded in a sea of programmable SIMT hardware that handles everything around the matmul — the data staging, the activations, the odd-shaped layers, the code nobody predicted. Tensor cores are the deliberate midpoint on the flexibility–efficiency axis: less efficient than the TPU on pure dense matmul, vastly more efficient than vector pipes, and still surrounded by a machine that can run anything.

What actually runs on them: deep-learning training and inference above all — dense layers directly, and convolutions lowered into matrix multiplies so the same engines serve both (see for the software side of the precision game). Increasingly, HPC codes join in: iterative solvers that do the bulk of the work in FP16 on tensor cores and polish the answer back to FP64 accuracy — the cheap-arithmetic-plus-careful-refinement trick escaping the AI ghetto.