Hardware-Assisted Virtualization
For a decade, virtualizing x86 meant heroic software: VMware's
binary
translator rewriting kernel code on the fly, or Xen porting the guest. Both existed only
because the CPU couldn't do textbook
trap-and-emulate.
Then, in 2005–06, Intel shipped VT-x and AMD shipped AMD-V (SVM) —
virtualization extensions that added, in silicon, exactly what Popek and Goldberg said x86 was missing.
Overnight, an unmodified guest could be virtualized cleanly, and the trap-and-emulate ideal came true on
the world's most stubborn ISA. This lesson is how that hardware works.
A new axis: root vs non-root mode
The elegant idea behind VT-x/AMD-V is to stop fighting over the four privilege rings and instead add an
orthogonal dimension. The CPU now has two worlds:
- VMX root mode — where the VMM/hypervisor runs. This is the "real"
machine; the hypervisor here has full control.
- VMX non-root mode — where the guest runs. Crucially, non-root mode
has its own full set of rings 0–3. So the guest kernel runs in
non-root ring 0 — it is genuinely in ring 0, exactly where it expects to be,
no de-privileging or ring compression needed. The guest is no longer lied to about its ring.
This solves the leak problem at a stroke. The old trouble was that de-privileging the guest made
sensitive instructions misbehave (a \texttt{popf} silently dropping the
interrupt flag). Now the guest kernel really is in ring 0 of its own world, so those instructions behave
correctly for the guest — and the hardware can be configured to intercept precisely the
operations the hypervisor cares about, forcing a controlled exit to root mode. Sensitivity and
privilege are reconciled by hardware.
Crossing between the worlds
Movement between root and non-root is a hardware transition, not a trap in the old sense:
- A VM-entry takes the CPU from root into non-root — the hypervisor executes
\texttt{VMLAUNCH} (first time) or
\texttt{VMRESUME} (subsequent) and the guest starts/continues running.
- A VM-exit takes the CPU from non-root back into root — triggered when the guest does
something the hypervisor asked to be notified about. The hardware atomically saves the entire
guest processor state and loads the host's, then jumps to the hypervisor's exit handler.
All of this hangs off one in-memory data structure: the VMCS (VM Control Structure;
AMD calls it the VMCB). It holds a guest-state area (the registers, control regs, RIP, RSP to
run the guest with), a host-state area (what to restore on exit), and a set of
execution controls — bitmaps and fields that tell the CPU which events should
cause a VM-exit. The hypervisor programs the VMCS, executes
\texttt{VMLAUNCH}, and the CPU does the rest.
What causes a VM-exit?
The hypervisor tunes the VMCS controls to exit on exactly the events it must mediate, and no more —
because every exit is expensive (a full state swap, thousands of cycles). Typical
exit causes:
| Guest event | Why the VMM must see it | Frequency |
|---|
| \texttt{CPUID} | the VMM presents a virtual CPU feature set to the guest | rare |
| access to a control register / \texttt{HLT} | the VMM virtualizes CPU config and idle | occasional |
| I/O port or MMIO to a virtual device | the VMM emulates the device | workload-dependent |
| EPT violation (nested page fault) | the VMM must fix up guest-physical→host-physical mapping | can be hot |
| external interrupt / timer | the VMM owns the real interrupt controller and reschedules | frequent |
| explicit \texttt{VMCALL} | a paravirtual hypercall from a cooperating guest | as designed |
Ordinary guest instructions — arithmetic, memory access to already-mapped pages, even most privileged
instructions the guest runs in its own ring 0 — cause no exit at all; they run at full
native speed in non-root mode. The whole performance game of a hardware VMM is minimising VM-exits.
Why this "made trap-and-emulate clean again"
Compare the three eras. Old trap-and-emulate needed de-privileging and broke on x86's leaks. Binary
translation fixed the leaks but at the cost of a complex JIT and translated-code overhead. Hardware
assist gives you the best of both: the guest runs unmodified and in its own
ring 0 (no translation, no leaks), while a hardware-defined exit mechanism intercepts exactly the
sensitive events the VMM chose — genuine trap-and-emulate, but with the "trap" upgraded to a precise,
programmable VM-exit. The VMM code shrinks from a binary translator to a comparatively simple exit
handler.
There was a catch in the first generation (2006): a VM-exit was slower than a VMware
binary-translation callout, so on syscall/kernel-heavy workloads early VT-x could lose to
software. Intel and AMD spent the next several generations driving exit latency down (from ~3000+ cycles
toward a few hundred) and — critically — adding
nested
paging (EPT/NPT) so that the single most frequent exit — the page fault — could be handled
in hardware without any exit at all. That is the next lesson.
Counting the cost of exits
A hardware VMM's speed is dominated by how often the guest exits and how much each exit costs. The
simulation models a one-second slice of guest execution, tallies the exits by cause, and reports what
fraction of time is stolen by the VMM. Notice how a modest per-exit cost, multiplied by a
high exit rate, dominates everything.
// How much of a second does a hardware VMM lose to VM-exits?
const CYCLES_PER_SEC = 3_000_000_000; // a 3 GHz core
const EXIT_COST = 1200; // cycles for one VM-exit + handler + VM-entry
// Exit events during a 1-second guest slice (illustrative rates for a busy guest).
const exits = [
{ cause: "external interrupt/timer", perSec: 250_000 },
{ cause: "I/O to virtual devices", perSec: 120_000 },
{ cause: "EPT violation (page fix)", perSec: 60_000 },
{ cause: "CPUID / control regs", perSec: 2_000 },
];
let total = 0;
for (const e of exits) {
const cost = e.perSec * EXIT_COST;
total += e.perSec;
console.log(`${e.cause.padEnd(28)} ${e.perSec.toLocaleString().padStart(9)} exits -> ${(cost / 1e6).toFixed(1)}M cycles`);
}
const lostCycles = total * EXIT_COST;
const overhead = (lostCycles / CYCLES_PER_SEC) * 100;
console.log(`\ntotal VM-exits/sec : ${total.toLocaleString()}`);
console.log(`cycles lost to exits: ${(lostCycles / 1e6).toFixed(0)}M of ${CYCLES_PER_SEC / 1e6}M`);
console.log(`virtualization overhead: ${overhead.toFixed(1)}% of the CPU`);
console.log("\n=> the biggest wins come from ELIMINATING whole classes of exit (e.g. EPT removes");
console.log(" page-fault exits; posted interrupts remove interrupt exits), not shaving per-exit cost.");
The design law
- A new root (VMM) vs non-root (guest) mode axis, orthogonal to
rings 0–3, lets the guest kernel run genuinely in its own ring 0 — no de-privileging, no
leaks.
- VM-entry (\texttt{VMLAUNCH}/\texttt{VMRESUME})
runs the guest; a configured event causes a VM-exit that atomically saves guest and
restores host state, jumping to the VMM.
- The VMCS/VMCB holds guest state, host state, and controls selecting which events
exit — so the VMM intercepts exactly what it needs.
- Performance is governed by the VM-exit rate × per-exit cost; the big wins come
from removing whole classes of exit (nested paging, posted interrupts), not just making
each exit cheaper.
When Intel launched VT-x in 2006, many benchmarks showed VMware's mature binary translator
beating the shiny new hardware. The reason is instructive: a translated
\texttt{popf} was an inline call inside the translation cache —
maybe a few hundred cycles — whereas an early VM-exit flushed pipelines and swapped a large processor
state through memory, costing several thousand. So on a workload that hammered privileged operations,
fewer but heavier hardware exits lost to many but lighter software callouts. Hardware
won decisively only once (a) exit latency fell generation on generation, and (b) nested paging removed
the most common exit entirely. It's a lovely reminder that "hardware support" is not automatically
faster — the whole system's exit rate is what matters.
People often call the hypervisor "ring −1", as if VMX root mode were a fifth, more-privileged ring below
ring 0. It isn't. Root vs non-root is a separate, orthogonal mode bit: both
worlds contain a full set of rings 0–3. The guest kernel runs in non-root ring 0 and the guest apps in
non-root ring 3; the hypervisor runs in root ring 0. "Ring −1" is a catchy shorthand for "more
privileged than the guest's ring 0", but taking it literally leads you to imagine a nested ring
hierarchy that the hardware does not have. The extra privilege comes from the world, not from a
deeper ring.
Where next
Hardware fixed the CPU side of virtualization. But there are two more hard illusions to build:
the guest's memory
(two levels of address translation, and the nested paging that keeps page-fault exits out of the VMM)
and its
I/O
devices. Both lean heavily on the root/non-root machinery you just learned.