Hypervisor Architectures

We now have all the mechanisms — CPU virtualization, memory virtualization, I/O virtualization. The hypervisor (or VMM) is the piece of software that puts them together and actually runs the virtual machines. But where does that software sit? The answer shapes performance, security, and the entire cloud. There are two classic answers and one modern twist that quietly won.

Type-1 and Type-2 — the classic split

A Type-1 or bare-metal hypervisor runs directly on the hardware, with no host OS beneath it; the VMs run on top. It owns the machine, so it is lean, fast, and has a small trusted computing base — the choice for servers and the cloud (VMware ESXi, Xen, Microsoft Hyper-V).

A Type-2 or hosted hypervisor runs as an ordinary application on top of a normal operating system (VirtualBox, VMware Workstation, Parallels). It is wonderfully convenient — you install it like any program on your laptop and keep using your desktop — but it goes through the host OS for scheduling and I/O, adding a layer of overhead. It is the choice for developers and desktops.

KVM — the twist that ate the distinction

Then Linux blurred the line. KVM (Kernel-based Virtual Machine) is not a separate hypervisor program at all — it is a kernel module that lets the Linux kernel use the hardware VT-x/AMD-V modes directly. Load it, and Linux itself becomes a Type-1 hypervisor: a VM is just a Linux process whose threads are virtual CPUs, scheduled by the ordinary CFS scheduler, using the ordinary memory management. It is arguably both types at once — bare-metal performance because it uses hardware modes directly, yet backed by a full general-purpose OS. KVM (with QEMU for device models and virtio) powers most of the world's public clouds.

Type-1 (bare-metal) — runs directly on hardware, no host OS; lean, fast, small TCB (ESXi, Xen, Hyper-V); the server/cloud choice;
Type-2 (hosted) — an application on a normal OS; convenient but adds a host-OS layer (VirtualBox, VMware Workstation); the desktop choice;
KVM — a kernel module that turns Linux into a Type-1 hypervisor; a VM is a Linux process, scheduled and memory-managed like any other;
the deeper the hypervisor sits (closer to hardware), the lower the overhead and the smaller the attack surface.

Live migration — moving a running machine

One of virtualization's most magical tricks is live migration: moving a running VM from one physical host to another with barely a hiccup — used constantly in the cloud to drain a host for maintenance or rebalance load. The dominant technique is pre-copy:

copy all the VM's memory pages to the destination while it keeps running;
the running VM keeps dirtying pages, so re-copy just the pages that changed since the last round — each round is smaller than the last;
when the remaining dirty set is tiny, briefly stop the VM, copy that last little bit and the CPU state, and resume on the destination — a pause of only milliseconds.

It converges only if the VM dirties memory slower than the network can copy it; a write-heavy VM can thrash the pre-copy loop, which is why there is also post-copy (start on the destination immediately, fault missing pages across). Watch pre-copy shrink the dirty set round by round.

// Pre-copy live migration: each round copies the pages dirtied during the previous round. // It converges when the dirty set is small enough for a sub-second stop-and-copy. let dirty = 1_000_000; // pages to send initially (whole memory) const NET = 300_000; // pages/round the network can copy const DIRTY_RATE = 0.25; // fraction of copied pages the running VM re-dirties let round = 0, sent = 0; while (dirty > 20_000 && round < 12) { round++; const copy = Math.min(dirty, NET); sent += copy; dirty = Math.round(copy * DIRTY_RATE); // pages re-dirtied while we copied console.log(`round ${round}: copied ${copy}, re-dirtied ${dirty} remaining`); } console.log(`STOP-AND-COPY the last ${dirty} pages -> resume on destination (pause ~ms)`); console.log(`total pages sent: ${sent}`);

Yes — it is called nested virtualization, and it is more than a party trick. It is how you run a Windows/WSL2 VM inside a cloud VM, how CI systems test hypervisors, and how confidential-computing VMs layer. The hard part is that only the bottom hypervisor really has the hardware VT-x mode; the nested (L1) hypervisor thinks it does, so the L0 hypervisor has to emulate the virtualization hardware for it — shadowing the guest's VMCS structures and multiplexing the single real hardware mode. It works, with a performance tax that grows with each level. That the abstraction composes at all — a VM believing it is bare metal all the way down — is a small marvel of the fidelity property.

The textbook ranking — bare-metal beats hosted — is a fine rule of thumb, but KVM breaks the neat picture. KVM is hosted inside a full Linux kernel, which by the old taxonomy sounds like Type-2, yet it runs guests through the hardware virtualization modes directly and delivers essentially bare-metal performance — so it is usually classified Type-1. The lesson: the "Type-1 vs Type-2" box matters less than where the guest's instructions and I/O actually execute. If the fast path goes straight through hardware modes and virtio/passthrough, the label on the box barely matters; if it detours through an emulator or a host-OS syscall on every operation, it will crawl regardless of its "type."