Binary Data Processing

Binary data processing parses and constructs compact, byte-oriented data formats—such as network frames, storage page headers, ELF/WASM object files, and sensor bitfields—directly from &[u8] slices without intermediate string conversions or heap allocations.

Binary Cursor Parsing over `&[u8]`:

  [ 0x89, 0x50, 0x4E, 0x47 | 0x00, 0x01 | 0x00, 0x00, 0x04, 0x00 | ... payload ... ]
  |<--- 4B Magic Header -->|<-- u16 --->|<----- u32 Length ----->|
  Cursor advances slice window step-by-step using `split_at_checked`!

Safe Zero-Copy Binary Cursor with split_at_checked

Stabilized in Rust 1.80, slice::split_at_checked(mid) (along with split_first_chunk::<N>()) splits a slice into Some((&slice[..mid], &slice[mid..])) or returns None if mid > slice.len(), eliminating any possibility of out-of-bounds slice panics when advancing through an untrusted binary buffer!

#[derive(Debug, PartialEq, Eq)]
pub enum WireError {
    UnexpectedEof,
    InvalidMagic([u8; 4]),
}

/// Zero-copy binary reader that advances a `&'a [u8]` slice window safely.
pub struct ByteCursor<'a> {
    remaining: &'a [u8],
}

impl<'a> ByteCursor<'a> {
    pub const fn new(bytes: &'a [u8]) -> Self {
        Self { remaining: bytes }
    }

    pub const fn remaining_len(&self) -> usize {
        self.remaining.len()
    }

    /// Consumes an exact fixed-size array `[u8; N]` from the front of the cursor.
    pub fn read_array<const N: usize>(&mut self) -> Result<[u8; N], WireError> {
        let (chunk, rest) = self
            .remaining
            .split_first_chunk::<N>()
            .ok_or(WireError::UnexpectedEof)?;
        self.remaining = rest;
        Ok(*chunk)
    }

    pub fn read_u8(&mut self) -> Result<u8, WireError> {
        Ok(u8::from_be_bytes(self.read_array::<1>()?))
    }

    pub fn read_u16_be(&mut self) -> Result<u16, WireError> {
        Ok(u16::from_be_bytes(self.read_array::<2>()?))
    }

    pub fn read_u32_be(&mut self) -> Result<u32, WireError> {
        Ok(u32::from_be_bytes(self.read_array::<4>()?))
    }

    /// Borrows `len` bytes directly from the underlying slice without copying!
    pub fn read_slice(&mut self, len: usize) -> Result<&'a [u8], WireError> {
        let (head, tail) = self
            .remaining
            .split_at_checked(len)
            .ok_or(WireError::UnexpectedEof)?;
        self.remaining = tail;
        Ok(head)
    }
}

#[derive(Debug, PartialEq, Eq)]
pub struct WalRecord<'a> {
    pub flags: u8,
    pub epoch: u16,
    pub payload: &'a [u8],
}

pub fn decode_wal_record(raw: &[u8]) -> Result<WalRecord<'_>, WireError> {
    let mut cur = ByteCursor::new(raw);
    let magic = cur.read_array::<4>()?;
    if &magic != b"WAL1" {
        return Err(WireError::InvalidMagic(magic));
    }

    let flags = cur.read_u8()?;
    let epoch = cur.read_u16_be()?;
    let len = cur.read_u32_be()? as usize;
    let payload = cur.read_slice(len)?;

    Ok(WalRecord {
        flags,
        epoch,
        payload,
    })
}

fn main() {
    let wire_bytes = b"WAL1\x03\x00\x2A\x00\x00\x00\x05hello";
    let record = decode_wal_record(wire_bytes).unwrap();

    println!(
        "Decoded WAL record: flags={:#x}, epoch={}, payload={:?}",
        record.flags,
        record.epoch,
        std::str::from_utf8(record.payload).unwrap()
    );
}

Decoded WAL record: flags=0x3, epoch=42, payload="hello"

Packing and Unpacking Bitfields

Protocols like IPv4, TCP, and DNS pack multiple small integer fields into a single u8 or u16 header byte using bitwise shifts (<<, >>) and masks (&, |):

/// Packs a 4-bit version (`0..=15`), 3-bit priority (`0..=7`), and 1-bit urgent flag into 1 byte.
pub const fn pack_control_byte(version: u8, priority: u8, urgent: bool) -> u8 {
    ((version & 0x0F) << 4) | ((priority & 0x07) << 1) | (urgent as u8)
}

pub const fn unpack_control_byte(byte: u8) -> (u8, u8, bool) {
    let version = (byte >> 4) & 0x0F;
    let priority = (byte >> 1) & 0x07;
    let urgent = (byte & 0x01) != 0;
    (version, priority, urgent)
}

fn main() {
    let packed = pack_control_byte(2, 5, true);
    let (ver, prio, urg) = unpack_control_byte(packed);
    println!("Packed byte={packed:#010b} -> version={ver}, priority={prio}, urgent={urg}");
}

Packed byte=0b00101011 -> version=2, priority=5, urgent=true