A CPU core is a sprinter obsessed with latency: it will spend a billion transistors on out-of-order execution, branch prediction and giant caches so that one thread finishes as fast as physically possible. A GPU makes the opposite bet. It doesn't care how long any single thread takes; it cares about throughput — total work per second across thousands of threads. It spends its transistors on arithmetic units, not on making one thread clever, and it hides every stall not by predicting around it but by having so many threads ready that there is always something else to run.
This is why a GPU can have tens of thousands of threads in flight and a CPU has a few dozen. It's a different animal for a different job: the CPU wants your email to feel instant; the GPU wants to multiply two enormous matrices, shade a million pixels, or train a network — jobs made of the same operation repeated on mountains of independent data.
NVIDIA calls the GPU's execution model SIMT — Single Instruction, Multiple Threads. To the programmer you write one ordinary-looking function (a "kernel") describing what one thread does, and launch it across a grid of thousands of threads. But underneath, the hardware bundles threads into groups of 32 (NVIDIA calls a group a warp; AMD uses 32- or 64-wide wavefronts) and runs every thread in a warp in lock-step, executing the same instruction on its own data at the same time.
So SIMT is really SIMD wearing a friendlier mask. The lanes of a packed-SIMD unit are exposed to you as independent "threads," which is far nicer to program — each thread has its own registers and its own index — but the machine underneath still issues one instruction per warp per cycle and applies it across all 32 lanes. You get SIMD efficiency with a thread's-eye programming model.
A GPU is a grid of Streaming Multiprocessors (SMs; AMD calls them Compute Units). Each SM holds many warps resident at once — their registers all live in a huge register file — and a warp scheduler that, every cycle, picks a warp that is ready and issues its next instruction. The memory hierarchy is tuned for bandwidth: a small fast shared memory (scratchpad) and L1 per SM, an L2 shared across the chip, and high-bandwidth DRAM (GDDR or HBM) feeding the whole thing.
A DRAM access costs hundreds of cycles. A CPU fights that with big caches and out-of-order execution to find independent work. A GPU does something audacious: when a warp issues a memory load and stalls, the scheduler simply switches to another ready warp — instantly, because all warps' state is already sitting in the register file. With enough resident warps, there is always one ready to run, and the memory latency is completely hidden behind other warps' arithmetic. This is why GPUs need so many threads: not for the parallel result, but to have enough spare work to cover the stalls.
The fraction of the possible warp slots you actually keep resident is called occupancy. Too few warps and a memory stall shows through as idle arithmetic units; enough warps and the latency vanishes. The model below shows the tipping point — how many warps you need before the units stay busy:
Because a warp shares one instruction stream, branches are its Achilles' heel. If some threads in a warp
take the
| CPU (latency-oriented) | GPU (throughput-oriented) | |
|---|---|---|
| Goal | finish one thread ASAP | maximise total work/second |
| Cores / threads | a few big cores, dozens of threads | thousands of tiny threads (warps) |
| Transistor budget spent on | OoO, branch prediction, big caches | arithmetic units, huge register file |
| Hides memory latency by | caches + out-of-order execution | switching to another ready warp |
| Hates | cache misses | warp divergence (branchy code) |
GPUs were built to shade pixels: apply the same lighting maths to millions of independent fragments — embarrassingly parallel, arithmetic-heavy, latency-tolerant. It turned out that a neural network's core operation, matrix multiplication, has exactly that shape: the same multiply-accumulate over vast regular arrays with no branching. When NVIDIA exposed the hardware for general computing with CUDA (2007), researchers discovered a graphics card was an astonishingly cheap matrix engine. The modern AI boom rides on a machine designed for video games — and today's data-center GPUs add tensor cores, units that do a whole small matrix-multiply per instruction, doubling down on the coincidence.
Two traps. First, a GPU "thread" is really a SIMD lane dressed up — thousands of them share instruction streams in warps; it is nothing like a heavyweight OS thread with its own program counter and stack switched by the kernel. Second, launching more threads does not make any single thread finish sooner — a GPU is slow per-thread on purpose. More threads raise throughput and occupancy (so stalls stay hidden), not latency. If you need one answer fast and there's no parallelism to exploit, a CPU will beat a GPU every time.