Inside your program, a chat message is a tidy object: { user: "alice", text: "hi", ts: 42 } — three fields, sitting at some address in RAM, wired together by pointers your language manages for you. But a socket cannot carry an object. A socket carries bytes: a flat, one-dimensional river of numbers 0–255, with no fields, no pointers, and — this is the sting in the tail — no marks showing where one message stops and the next begins. Getting your data onto that river, and safely off it again at the far end, is two distinct jobs that beginners forever confuse.

• Serialization (a.k.a. marshalling) — turning an in-memory object into a flat byte sequence, and parsing it back. This is what the bytes mean.
• Framing — marking where each message begins and ends on a boundary-less TCP stream, so the receiver can chop the river back into the messages you sent. This is where one message ends.

You need both, and they are independent choices. This page builds each from the ground up, and its centrepiece is a working deframer — the exact code every networked program secretly contains — that reassembles messages no matter how cruelly the network chops the byte stream.

Serialization: text vs binary

To serialize is to answer "how do I write this object as bytes so the other side can rebuild it?" The great divide is between text formats and binary formats.

Text formats — JSON, XML, YAML. The bytes are human-readable characters. { "user": "alice", "text": "hi", "ts": 42 } is literally those characters, UTF-8 encoded. The huge win is that the format is self-describing: the field names travel with the data, so anyone — a person, a new service, a debugger — can read it without a prior agreement. The cost is size and speed: you ship the string "user" on every single message, and parsing text into numbers is comparatively slow.

Binary formats — Protocol Buffers, Avro, Thrift, MessagePack. The bytes are a compact machine encoding. Schema-based binary formats like Protocol Buffers take this furthest: both sides share a .proto schema that says "field 1 is a string called user, field 2 is a string called text, field 3 is an int called ts." On the wire you then send only field number + value — never the names. That is dramatically smaller and faster to parse, but the bytes are not self-describing: without the schema they are meaningless, and both ends must agree on it in advance.

Schema evolution is the quiet reason big systems lean binary. Protobuf gives every field a number, and old code simply skips field numbers it doesn't recognise. So you can add a field 4 next year and last year's servers keep working — forward and backward compatible by design. Hand-rolled binary formats that hard-code byte offsets have no such grace: insert a field in the middle and every old reader shatters.

Framing: where does each message end?

Suppose you have serialized three messages and written them to a TCP socket. The receiver gets a stream of bytes with the boundaries erased — it must re-discover them. There are exactly two classic strategies, and essentially every real protocol picks one.

• Length prefixing. Before each message, write its length. The frame is [length][payload]. The reader reads the length header, then reads exactly that many payload bytes. Pro: the payload can contain any bytes — length prefixing does not care what's inside. Con: you must know the length up front, and you must decide how many bytes the length field itself takes (2 bytes caps a message at 65 535; 4 bytes at ~4 GB).
• Delimiters. Put a special marker between messages — a newline (\n) for line-based text protocols (HTTP headers, SMTP, Redis's RESP), or \r\n\r\n to end a header block. The reader scans for the delimiter. Pro: simple, streamable, human-readable, no need to know the length in advance. Con: the delimiter must not appear in the payload — or you must escape it, or forbid it. This is why line-based protocols are awkward for arbitrary binary data.

The centrepiece: a deframer that survives arbitrary chunking

Here is the crux. TCP delivers bytes in whatever chunk sizes it likes — a data event might hand you half a length header, or three messages glued together, or one byte at a time. A correct deframer therefore cannot assume anything about chunk boundaries. It must buffer incoming bytes and repeatedly ask: "do I now hold a complete frame? If so, emit it and keep the leftover; if not, wait for more." The code below implements a length-prefixed framer and a stateful deframer, then feeds it a stream sliced at the worst possible places — including mid-header.

Trace the state: chunk 0 is 2 bytes — not even a full 4-byte header — so feed buffers and returns, emitting nothing. Only once enough bytes have accumulated does the while loop fire, and it keeps firing as long as complete frames remain, leaving any partial tail in buf for next time. That buffer-and-loop is the heart of every real network reader.

Putting it together

Serialization and framing compose: you serialize the object to a payload (JSON, protobuf, whatever), then frame the payload for the stream (length-prefix it, or delimit it). The receiver reverses both: deframe to recover the payload bytes, then deserialize to rebuild the object. Many real protocols even do both at once — HTTP frames its headers with \r\n delimiters and its body with a Content-Length length prefix, all in one message, which is exactly the machinery you'll assemble on the HTTP-server page.