Rotary Position Embeddings

A transformer's attention is permutation-blind — on its own it has no idea which token came first. The original fix was to add a positional encoding vector to each token. Rotary Position Embeddings (RoPE) take a cleverer route: instead of adding position information, they rotate each query and key vector by an angle that depends on its position. It's the position scheme behind most modern large language models — LLaMA, GPT-NeoX, PaLM and many more.

Encode position as a rotation

Split each query and key vector into pairs of coordinates, and treat each pair as a little 2-D vector. RoPE rotates the pair at position m by an angle m\theta — proportional to how far along the sequence the token sits (with different pairs rotating at different frequencies \theta, like the hands of many clocks turning at different speeds).

The magic appears when attention takes a dot product between a query at position m and a key at position n. Because rotating one vector by m\theta and the other by n\theta leaves a dot product that depends only on the difference of the angles, the attention score ends up depending on the relative position m - n — how far apart the tokens are — rather than their absolute places. Relative position, which is what language actually cares about, falls out for free.

Encodes position by rotating query/key vectors by an angle proportional to position — no vector is added.
The resulting attention score depends on relative position m - n, not absolute position.
It's applied to queries and keys (not values), and extrapolates gracefully to longer sequences than seen in training.

An added positional vector fixes an absolute slot ("you are token 5"), and a model trained only up to length 2048 has never seen "token 4000", so it struggles to extrapolate. A rotation instead bakes in relative distance ("you are 3 apart"), which is the same whether the pair sits at positions 5–8 or 4000–4003. That relativity is why RoPE-based models generalise to longer contexts so much more gracefully, and why techniques for stretching context further (like interpolating the rotation frequencies) plug into it so naturally.

RoPE rotates the queries and keys — the things whose dot product forms the attention scores — not the value vectors.
It carries no learned parameters: the rotation angles are fixed by position and frequency. Its information lives in the relative angle between tokens, not an absolute code you can read off a single vector.

Worked example: same gap, same angle

Follow a single frequency slice of RoPE where each position step rotates a coordinate pair by \theta = 0.1 radians (a real model uses many different \theta values, one per pair, but tracking one slice keeps the arithmetic simple).

Take a query at position m = 5 and a key at position n = 8. The query pair is rotated by 5\theta = 0.5 radians and the key pair by 8\theta = 0.8 radians. Their dot product ends up depending only on the angle between them: (n - m)\theta = 3\theta = 0.3 radians.

Now slide both tokens 100 places further into the sequence: a query at m = 105 and a key at n = 108. The absolute rotations are now 10.5 and 10.8 radians — nothing like the first pair — but the difference is still 3\theta = 0.3 radians, exactly matching before. Since the attention score depends only on that angle difference, the model produces the identical "3 tokens apart" signal whether the gap sits near the start of a short prompt or a thousand tokens into a long one — precisely why RoPE-based models extrapolate to lengths they never trained on.