The Cross Product and Normals

Games live on perpendicular vectors: which way a wall faces, which way is "out", which way the light should bounce. The cross product is the machine that builds them. Feed it two vectors and it returns a third that is perpendicular to both, with a length equal to the area of the parallelogram they span:

\lVert \vec{a} \times \vec{b}\rVert = \lVert\vec{a}\rVert\,\lVert\vec{b}\rVert\,\sin\theta, \qquad (\vec{a}\times\vec{b}) \perp \vec{a}, \quad (\vec{a}\times\vec{b}) \perp \vec{b}.

Perpendicularity is the whole point. Its killer application in graphics is the surface normal — the arrow sticking straight out of a triangle — which we now build line by line.

The triangle normal, line by line

A triangle is three points A, B, C. We want the direction it faces: an arrow perpendicular to the whole flat triangle.

Step 1 — two edges from one corner. Subtract points to get two edge vectors lying in the triangle's plane (point − point = vector):

\vec{e}_1 = B - A, \qquad \vec{e}_2 = C - A.

Step 2 — cross them. Both edges lie in the triangle, so anything perpendicular to both is perpendicular to the triangle. That is exactly what the cross product hands back:

\vec{n} = \vec{e}_1 \times \vec{e}_2 = (B - A)\times(C - A).

Step 3 — normalise to a pure direction. Its raw length is the parallelogram area (twice the triangle's), which we don't want polluting a direction. Divide by the length to get a unit normal — see why the length is the area:

\hat{n} = \frac{\vec{n}}{\lVert \vec{n}\rVert}.

Step 4 — the winding order picks the side. Because \vec{a}\times\vec{b} = -(\vec{b}\times\vec{a}), swapping B and C flips \hat{n} to the other side. So the order you list the vertices — the winding order, clockwise or counter-clockwise — decides which face is "out". Every engine fixes one convention (usually counter-clockwise = front) and sticks to it religiously.

(B-A)\times(C-A) = -\,\big[(C-A)\times(B-A)\big] \quad\Rightarrow\quad \text{vertex order } \Leftrightarrow \text{ which way is out.} For \vec{a}, \vec{b}\in\mathbb{R}^3 at angle \theta:

Half of every closed model faces away from the camera at any instant — the far side of a sphere, the back of a wall. Drawing it is wasted work, so the GPU culls it. The test is a dot product against the normal. Let \hat{n} be the triangle's outward normal and \vec{v} the direction from the triangle toward the camera. The face points toward the camera only when

\vec{v}\cdot\hat{n} > 0 \quad (\text{visible}), \qquad \vec{v}\cdot\hat{n} \le 0 \quad (\text{facing away — skip it}).

One cross product to build the normal, one dot product to test it, and the engine throws away roughly half the triangles before shading a single pixel. The same \hat{n} then drives lighting (\max(0,\hat{n}\cdot\hat{L})) and collision response ("push the player back out along the normal"). It is the most reused vector in the pipeline — and it all starts with (B-A)\times(C-A).

The formula \vec a\times\vec b is defined the same way everywhere, but which way it points depends on the handedness of the coordinate system you're computing in. Feed the identical two vectors — same numbers, same order — into a right-handed system and a left-handed one, and the two computed normals point in exactly opposite directions.

This is a classic source of inverted, inside-out-looking lighting (and backwards backface culling) when porting geometry between engines that disagree on handedness — say, exporting a mesh from a right-handed modelling tool into a left-handed game engine, or vice versa. The triangle data and the (B-A)\times(C-A) formula look completely unchanged, yet every normal in the scene now faces into the surface instead of out of it. The usual fix is to negate one axis or reverse the triangle's winding order when the geometry crosses the handedness boundary, so the normals point outward again.

See it explained