BGP and the Internet Backbone

So far, routing has been a maths problem: model the network as a weighted graph, run Dijkstra or Bellman–Ford, find least cost. That works beautifully inside one organisation, where a single administrator sets the link costs and everyone shares the goal of shortest paths. But the Internet is not one organisation. It is roughly a hundred thousand separate networks — ISPs, universities, Google, Netflix, your national telco — each independently run, each called an Autonomous System (AS) and tagged with a number (an ASN). These ASes distrust each other, compete commercially, and absolutely do not share a cost metric.

When Comcast decides how to reach a Vodafone customer, "shortest path" is almost beside the point. The real questions are: who am I willing to send traffic through? who pays whom? which neighbour's route makes me money rather than costs me money? Routing between ASes is governed by policy and economics, not distance — and the single protocol that glues all hundred thousand networks into one Internet is BGP, the Border Gateway Protocol. It has been called "the most important protocol you've never heard of," and also "held together with duct tape and good intentions." Both are true.

Intra-domain vs inter-domain: two different jobs

The Internet's routing is deliberately split into two levels, and confusing them is the first hurdle.

Intra-domain (interior) routing — inside a single AS, run by protocols like OSPF or RIP from the previous lesson. One administrator, one metric, goal = shortest path. This is the weighted-graph world.
Inter-domain (exterior) routing — between ASes, run by BGP. Many administrators, no shared metric, goal = enforce each AS's policy while maintaining global reachability. This is the world of contracts and trust.

A packet from your laptop to a distant server typically rides interior routing to the edge of your ISP, then hops AS-to-AS across the backbone under BGP, then interior routing again inside the destination's network. BGP's job is stitching the ASes together; OSPF's job is finding your way around within one.

BGP is a path-vector protocol

BGP is a cousin of distance-vector — but with a crucial upgrade. Instead of advertising a distance ("z is 4 away"), a BGP router advertises the entire sequence of ASes a route passes through — the AS-PATH. This makes it a path-vector protocol. An advertisement says, in effect: "to reach prefix 203.0.113.0/24, come to me, and the path from here is AS-PATH: 300 200 100."

Carrying the whole path buys two things distance-vector could not:

Instant loop detection. Before accepting a route, an AS checks whether its own ASN already appears in the AS-PATH. If it does, accepting would create a loop, so it rejects the route outright. No count-to-infinity, no phantom echoes — the loop is visible in the path itself.
Policy hooks. Seeing the full path lets an AS make decisions based on who is on it — "I won't carry traffic that transits my competitor," or "prefer routes through my customer over routes through my provider."

eBGP and iBGP: two flavours of the same protocol

BGP runs in two modes, distinguished by whether the two routers speaking it are in the same AS.

eBGP (external BGP) — spoken between routers in different ASes, across the boundary. This is where routes actually enter and leave an AS, and where policy is applied.
iBGP (internal BGP) — spoken between routers within the same AS, to distribute the externally-learned routes to every border router, so the whole AS agrees on how to reach the outside world. (iBGP carries the external routes; the AS's own interior routing — OSPF — still handles reaching those border routers.)

A neat way to hold it: eBGP learns routes from the world; iBGP spreads that knowledge around the house.

Policy and money: why the shortest path often loses

Here is the heart of inter-domain routing. Because there is no shared cost metric, a BGP router selecting among candidate routes runs down a list of attributes, and the very first tiebreaker — before path length is even considered — is local preference, a number an AS assigns purely by its own business policy. AS-PATH length is only a later tiebreaker. So a longer path can, and routinely does, win.

Why would you deliberately pick a longer path? Money. AS relationships come in three commercial flavours:

Customer → provider. The customer pays the provider for transit to the rest of the Internet.
Peer ↔ peer. Two ASes exchange traffic between their own customers for free (settlement-free peering), but neither carries the other's transit traffic onward.
Provider → customer. The provider is paid to carry the customer's traffic everywhere.

From these fall the golden rules of routing economics. An AS prefers customer routes over peer routes over provider routes — because sending via a customer earns money, via a peer is free, and via a provider costs money. And an AS will not act as a free transit between two of its providers or peers. This produces the famous valley-free property: a valid Internet path goes "up" the provider hierarchy from the source, optionally across one peer link at the top, then strictly "down" to the destination — never up-down-up (which would mean an AS paying to carry someone else's traffic for nothing). Other attributes fine-tune this: MED (multi-exit discriminator) lets a neighbour hint which of several links into it to prefer, and AS-PATH length breaks remaining ties.

The instinct drilled in by Dijkstra and Bellman–Ford — "routing finds the shortest path" — is exactly wrong for BGP, and it is the misconception to unlearn here. BGP's route-selection process consults local preference first, and local preference is set by business policy, not distance. Only if local preferences tie does BGP fall to shorter AS-PATH, and even then several more policy attributes can intervene. So the path your packets take across the Internet is frequently not the geographically or hop-count shortest one — it is the one that is cheapest or most advantageous for the ASes involved. Traffic between two European cities has been known to detour through another continent because that path was, to some AS along the way, commercially preferable. If you ever try to explain a real Internet path purely by distance, you will be baffled; explain it by money and policy and it snaps into focus. BGP optimises for economics and control; shortest-path is at best a late tiebreaker.

Fragile by design: slow convergence, leaks and hijacks

BGP holds the Internet together, but it is startlingly delicate, for two connected reasons: it converges slowly, and it trusts what it is told.

Slow convergence. Because acceptance depends on policy and every AS re-runs selection as advertisements ripple in, a single failure can trigger a storm of successive path explorations ("path hunting") before the network settles — sometimes minutes of instability across the backbone.
Route leaks. An AS that is supposed to keep two relationships separate accidentally re-advertises routes it shouldn't — e.g. announces its provider's routes to a peer — and suddenly floods of traffic take a wrong, overloaded turn. In 2017 a leak from a single ISP disrupted large parts of the Internet for hours.
Prefix hijacks. BGP has almost no built-in authentication: historically, an AS could simply announce a prefix it does not own, and — if the false announcement is more specific (longest-prefix match!) or more policy-preferred — much of the Internet will believe it and route that traffic to the impostor. Famous incidents include a 2008 case where Pakistan Telecom's attempt to block YouTube domestically leaked worldwide and took YouTube offline globally for a couple of hours, and repeated hijacks used to steal cryptocurrency traffic. Mitigations like RPKI (which cryptographically certifies which AS may originate which prefix) are being deployed, but slowly — BGP's original design simply assumed everyone was honest.

That is the paradox of the backbone: the protocol that unites a hundred thousand mutually-distrustful networks was itself built on trust. It works astonishingly well most of the time, and spectacularly badly on the rare days it doesn't — and understanding why it fails is understanding that its routing decisions were never really about distance at all.

A "default-free" backbone router — one that carries a route to every destination on the Internet with no default fallback — holds the entire global routing table: well over 900,000 IPv4 prefixes and climbing (plus a large IPv6 table). Each must be stored, and each update re-run through policy. The table's relentless growth is a real operational worry — it strains router memory (there have been "512K day" outages when tables outgrew hardware limits) and is a big part of why route aggregation, the CIDR supernetting you met earlier, matters so much: every prefix two ASes can combine into one is a prefix nearly a million routers don't have to store.