Two messages arrive on your phone. The first says: "the sun rose this morning." The second says: "school is cancelled today." Both are short sentences — but one of them is worthless and the other changes your whole day. Why? The first told you something you already knew was practically certain; the second told you something unlikely. In 1948 Claude Shannon turned this everyday intuition into a number: information is surprise, and surprise can be measured, in bits. The theory he built in one legendary paper underpins every ZIP file, every video stream, every phone call — and, as we will keep discovering, a surprising amount of physics.

Measuring surprise: \log_2(1/p)

Suppose an event has probability p. Shannon defined the information content (or surprisal) of seeing it actually happen as

Why a logarithm? Because it is the only choice that makes information behave the way it obviously should:

• Certain events carry nothing. If p = 1 then \log_2(1/1) = 0 bits — "the sun rose" is free.
• Rarer means more. As p shrinks, \log_2(1/p) grows — the cancelled school day beats the sunrise.
• Independent surprises add. Two independent events with probabilities p and q happen together with probability pq, and \log_2\frac{1}{pq} = \log_2\frac{1}{p} + \log_2\frac{1}{q}. Learning both = the sum of learning each. Only a logarithm turns multiplication of probabilities into addition of information.

The unit comes out beautifully. A fair coin flip has p = \tfrac12 per outcome, so each flip carries \log_2 2 = 1 — exactly one bit. The bit is not just a circuit voltage; it is the natural unit of surprise. And there is a lovely game hiding in the formula: \log_2 n is the number of yes/no questions you need to pin down one possibility out of n equally likely ones. Guessing a secret number from 1–64? Six good questions ("is it bigger than 32?" …) always suffice, because \log_2 64 = 6. Information content is the length of the perfect game of twenty questions.

Entropy: the average surprise of a source

A single event has a surprisal. A source — anything that keeps emitting symbols, like a coin, a die, or someone typing English — has an average surprisal, taken over its probabilities. That average is the Shannon entropy:

Each outcome contributes its surprise \log_2(1/p_i), weighted by how often it occurs, p_i. Let's work some examples:

• A fair coin. H = \tfrac12\log_2 2 + \tfrac12\log_2 2 = 1 bit per flip. Maximum uncertainty for a two-outcome source.
• A biased coin with heads 90% of the time. H = 0.9\log_2\frac{1}{0.9} + 0.1\log_2\frac{1}{0.1} \approx 0.9(0.152) + 0.1(3.32) \approx 0.47 bits. Less than half a bit! The frequent heads are barely surprising, and the rare surprising tails hardly ever contribute.
• A fair six-sided die. Six equal outcomes: H = \log_2 6 \approx 2.58 bits per roll.
• English letters. If all 26 letters were equally likely, \log_2 26 \approx 4.70 bits each. But they aren't — E is common, Z is rare — and feeding the real frequencies into the formula gives about 4.1 bits per letter. (Real English is even more predictable than that, because letters depend on their neighbours: exploit whole words and context and you get nearer 1 bit per letter — which is why predictive text works.)

Notice the pattern: uniform is maximal. Any lopsidedness in the probabilities makes the source more predictable and drags the entropy down, all the way to 0 bits for a source that always says the same thing.

The binary entropy curve

For a coin with heads-probability p, the entropy is the binary entropy function

Slide p below and watch the dashed line track the entropy of your coin. The curve peaks at exactly 1 bit when p = \tfrac12 (a fair coin is the least predictable coin) and falls symmetrically to 0 at both ends — a coin that always, or never, lands heads tells you nothing at all.

Entropy is the speed limit of compression

Here is why entropy is an engineering number and not just a philosophical one. Suppose your source emits symbols with entropy H bits per symbol, and you want to store its output in as few bits as possible — give short codes to common symbols, long codes to rare ones, exactly as Huffman coding does. How well can you possibly do? Shannon answered exactly:

This is why entropy deserves the name "information": it is the size of what the source is really telling you, once every scrap of redundancy has been squeezed out. Our 90%-heads coin has entropy \approx 0.47 bits per flip, so a clever code can record a million of its flips in about 470,000 bits — under half a bit per flip — and no code in the universe can beat that. A fair coin's flips, at a full 1 bit each, cannot be compressed at all: perfectly random data is incompressible, precisely because every bit of it is a genuine surprise.