Low-Level Systems Programming

Serialization Concepts

Serialization converts in-memory Rust data structures into a linear sequence of bytes (or text tokens) for storage or transmission, while deserialization reconstructs typed values from those bytes after validating structure and invariants.

In-Memory Rust Struct (`Order { id: 42, symbol: "BTC", qty: 100 }`)
       |
       +---[ Text Format (JSON / TOML) ]--------> `{"id":42,"symbol":"BTC","qty":100}`
       |     (Self-describing, human-readable)
       |
       +---[ Compact Binary (Postcard / Bincode)]-> `[0x2A, 0x03, b'B', b'T', b'C', 0x64]`
       |     (Schema-driven, minimal size)
       |
       +---[ Zero-Copy Archive (`rkyv` / `zerocopy`)]-> Aligned byte buffer cast directly to `&ArchivedOrder`!

Comparing Serialization Architectures in Rust

Architecture Ecosystem Examples Wire Size Deserialization Cost Schema Evolution Support
Self-Describing Text serde_json, toml Large (field names repeated as strings) Medium (UTF-8 + number parsing) High (easy to add/ignore optional fields)
Tagged Binary (TLV) prost (Protobuf), rmp-serde (MessagePack), ciborium (CBOR) Small (numeric field tags or compact maps) Low High (forward & backward compatible via field tags)
Schema-Less Positional Binary bincode, postcard (popular in no_std) Minimal (zero field tags; fields written in order) Very Low Low (reordering or inserting fields breaks old readers)
Zero-Copy / In-Place rkyv, zerocopy, bytemuck Medium (includes alignment padding & relative pointers) $O(1)$ / Near-Zero (validates buffer and casts pointer without allocating!) Low (tied to exact layout representation)

Designing a Trait-Based Codec (Encode and Decode)

How do serialization libraries like serde, bincode, and postcard serialize arbitrary nested structs and collections without runtime reflection? They combine two core traits—a type-driven Encode/Serialize trait and a Decode/Deserialize trait—composed recursively across primitives, slices, and structs:

use std::io::{self, ErrorKind, Read, Write};

pub trait WireEncode {
    fn encode<W: Write>(&self, writer: &mut W) -> io::Result<()>;
}

pub trait WireDecode: Sized {
    fn decode<R: Read>(reader: &mut R) -> io::Result<Self>;
}

impl WireEncode for u32 {
    fn encode<W: Write>(&self, writer: &mut W) -> io::Result<()> {
        writer.write_all(&self.to_le_bytes())
    }
}

impl WireDecode for u32 {
    fn decode<R: Read>(reader: &mut R) -> io::Result<Self> {
        let mut buf = [0u8; 4];
        reader.read_exact(&mut buf)?;
        Ok(u32::from_le_bytes(buf))
    }
}

impl WireEncode for String {
    fn encode<W: Write>(&self, writer: &mut W) -> io::Result<()> {
        let len = u32::try_from(self.len())
            .map_err(|_| io::Error::new(ErrorKind::InvalidInput, "string too long"))?;
        len.encode(writer)?;
        writer.write_all(self.as_bytes())
    }
}

impl WireDecode for String {
    fn decode<R: Read>(reader: &mut R) -> io::Result<Self> {
        let len = u32::decode(reader)? as usize;
        if len > 1024 * 1024 {
            return Err(io::Error::new(ErrorKind::InvalidData, "string exceeds 1 MiB cap"));
        }
        let mut bytes = vec![0u8; len];
        reader.read_exact(&mut bytes)?;
        String::from_utf8(bytes).map_err(|e| io::Error::new(ErrorKind::InvalidData, e))
    }
}

#[derive(Debug, PartialEq, Eq)]
pub struct TelemetrySample {
    pub sensor_id: u32,
    pub metric_name: String,
    pub value: u32,
}

impl WireEncode for TelemetrySample {
    fn encode<W: Write>(&self, writer: &mut W) -> io::Result<()> {
        self.sensor_id.encode(writer)?;
        self.metric_name.encode(writer)?;
        self.value.encode(writer)
    }
}

impl WireDecode for TelemetrySample {
    fn decode<R: Read>(reader: &mut R) -> io::Result<Self> {
        Ok(Self {
            sensor_id: u32::decode(reader)?,
            metric_name: String::decode(reader)?,
            value: u32::decode(reader)?,
        })
    }
}

fn main() -> io::Result<()> {
    let original = TelemetrySample {
        sensor_id: 7,
        metric_name: String::from("cpu_temp_celsius"),
        value: 62,
    };

    let mut buffer = Vec::new();
    original.encode(&mut buffer)?;

    let mut slice_reader = buffer.as_slice();
    let decoded = TelemetrySample::decode(&mut slice_reader)?;

    assert_eq!(original, decoded);
    println!(
        "Serialized into {} bytes and round-tripped: {decoded:?}",
        buffer.len()
    );
    Ok(())
}

Serialized into 28 bytes and round-tripped: TelemetrySample { sensor_id: 7, metric_name: "cpu_temp_celsius", value: 62 }