An ahead-of-time compiler translates the whole program once, before it runs, knowing nothing about the inputs it will see. A just-in-time (JIT) compiler does the opposite: it starts by interpreting, and compiles code to native machine instructions while the program is already running — armed with something an AOT compiler can only dream of: live profile data about what this particular execution actually does. Which branch is taken 99% of the time? What concrete type flows through this polymorphic call? Which loop is hot? The JIT sees it happen and compiles for the observed reality, not the worst case.

This is the engine behind Java's HotSpot, JavaScript's V8, .NET's CLR, LuaJIT and PyPy. It is why a dynamically-typed language, seemingly doomed to slow interpretation, can approach the speed of C on hot code. The whole design is a bet: spend compilation effort only where it pays off, speculate on what you have observed, and keep an escape hatch for when the speculation turns out wrong.

Hot-path detection: don't compile what you won't rerun

Compiling costs time now to save time later — so it only pays for code that runs many times. A JIT therefore begins every method in a cheap bytecode interpreter and attaches counters: increment on each method entry and each loop back-edge. When a counter crosses a threshold, that method or loop is declared hot and handed to the compiler. The 90/10 rule — most time in a small fraction of the code — is what makes this profitable: you interpret the cold 90% of methods essentially for free and lavish optimisation on the hot 10%.

Tiered compilation: a ladder of effort

A single "interpret vs compile" switch is too blunt. Modern engines use tiered compilation: several execution modes of increasing quality and increasing compile cost, and code climbs the ladder as it proves itself hot. HotSpot and V8 both do this.

• Tier 0 — interpreter. Starts instantly, runs slowly, gathers profile data.
• Tier 1 — baseline JIT. Compiles fast, optimises little; gets hot code off the interpreter quickly and keeps collecting profiles.
• Tier 2 — optimising JIT. Slow to compile, produces excellent code, using the accumulated profile to speculate aggressively (inlining, type specialisation, unboxing).

The ladder balances two opposing costs. Jumping straight to the optimising tier would stall startup on code that turns out cold; staying in the interpreter forever would leave peak performance on the table. Tiering gives fast startup and high peak throughput by moving each piece of code only as far up the ladder as its hotness justifies.

Speculation, deoptimisation and OSR

The optimising tier's power comes from profile-guided speculation: it compiles code that is correct only if an assumption observed so far continues to hold. "This variable has always been an integer, so compile integer arithmetic." "This call site has only ever targeted Cat.speak, so inline it." These assumptions let it generate code as tight as a statically typed language's — but they might be wrong on a future iteration.

So every speculation is protected by a cheap guard: a quick check (is it still an int? is the receiver still a Cat?). If a guard fails, the engine performs deoptimisation — it discards the optimised code and reconstructs the exact interpreter state (all the locals and the operand stack) at that program point, then resumes in the interpreter as if the fast code had never run. The dual manoeuvre is on-stack replacement (OSR): swapping a running activation from one tier to another mid-execution — crucial for a long-running loop that becomes hot while it is still spinning, since you cannot wait for it to return to upgrade it.

Inline caches: making dynamic dispatch cheap

In a dynamic language, x.foo() must look up which foo to call based on x's runtime type — a hash-map lookup on every call, murderous in a loop. The classic cure is the inline cache (IC): cache the result of the lookup at the call site. The first time, do the full lookup and remember "type T → method m" right there in the code. Next time, if the receiver is still T (a one-word guard), jump straight to m.

• A monomorphic IC caches one type — perfect when a site always sees the same type.
• A polymorphic IC (PIC) caches a handful, checking each in turn.
• A site that sees too many types goes megamorphic and falls back to the full lookup.

Inline caches are also a profile source: a call site that stayed monomorphic is a golden signal to the optimising tier that it can safely inline that one target under a type guard.

Method JITs vs trace JITs

Two philosophies for choosing the unit of compilation:

• Method JITs (HotSpot, V8) compile whole methods, following the control-flow graph and speculating with profiles. This is the mainstream design.
• Trace JITs (early TraceMonkey, PyPy, LuaJIT) compile a single hot trace — the actual straight-line path taken through a loop iteration, across method boundaries — turning it into branch-free code with side-exit guards. Traces inline naturally and expose enormous optimisation on the hot path, but suffer when control flow is irregular (trace explosion) and pay for many side exits.

Hotness and tiering, in code

A toy tier controller: run methods through counters and promote them across tiers as they heat up. It models the essential policy — cheap first, expensive only when earned.

hot climbs to the optimising tier once its counter passes 100; cold never leaves the interpreter — precisely the effort-where-it-pays discipline every real JIT enforces.