Strings and Text Processing

UTF-8 Representation

All Rust strings (String and str) are guaranteed by their type invariant to contain valid UTF-8, a variable-width encoding that represents every Unicode scalar value using 1 to 4 bytes.

UTF-8 Byte Encoding Layout

UTF-8 encodes a Unicode scalar value (char) into 1, 2, 3, or 4 bytes based on its code point range:

Code Point Range Byte 1 Byte 2 Byte 3 Byte 4 Example (char)
U+0000 .. U+007F (ASCII) 0xxxxxxx — — — 'A' (0x41, 1 byte)
U+0080 .. U+07FF 110xxxxx 10xxxxxx — — 'é' (0xC3 0xA9, 2 bytes)
U+0800 .. U+FFFF 1110xxxx 10xxxxxx 10xxxxxx — 'ক' (0xE0 0xA6 0x95, 3 bytes)
U+10000 .. U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx '🦀' (0xF0 0x9F 0xA6 0x80, 4 bytes)

Notice the self-synchronizing bit pattern of UTF-8:

  • ASCII bytes always start with 0 (0x00..=0x7F) and never appear as part of a multi-byte character.
  • Leading bytes of a multi-byte character start with two or more 1 bits (110..., 1110..., 11110...), indicating the exact byte length of the character.
  • Continuation bytes always start with the bit prefix 10 (0x80..=0xBF).

Inspecting the Raw UTF-8 Bytes of a String

fn inspect_utf8(text: &str) {
    println!("Text: \"{text}\" ({} bytes total)", text.len());
    for ch in text.chars() {
        let mut buf = [0_u8; 4];
        let encoded = ch.encode_utf8(&mut buf);
        println!(
            "  '{}' (U+{:04X}) -> {} byte(s): {:02X?}",
            ch,
            ch as u32,
            encoded.len(),
            encoded.as_bytes()
        );
    }
}

fn main() {
    inspect_utf8("Aéক🦀");
}

Text: "Aéক🦀" (10 bytes total)
  'A' (U+0041) -> 1 byte(s): [41]
  'é' (U+00E9) -> 2 byte(s): [C3, A9]
  'ক' (U+0995) -> 3 byte(s): [E0, A6, 95]
  '🦀' (U+1F980) -> 4 byte(s): [F0, 9F, A6, 80]

Why Direct Integer Indexing (s[0]) Is Forbidden (E0277)

Because UTF-8 is variable-width, the $n$-th character does not necessarily live at byte offset $n$ or $4n$, and returning a single byte at s[1] in "Aé" (0xC3) would yield an incomplete surrogate/leading byte rather than a character. To prevent accidental $O(n)$ scans disguised as $O(1)$ indexing or misinterpreting byte offsets as character indices, Rust does not implement Index<usize> on str or String:

// ❌ does not compile
fn main() {
    let s = String::from("hello");
    let first = s[0];
}

error[E0277]: the type `str` cannot be indexed by `{integer}`
 --> src/main.rs:4:19
  |
4 |     let first = s[0];
  |                   ^ string indices are ranges of `usize`
  |
  = help: the trait `SliceIndex<str>` is not implemented for `{integer}`
  = note: you can use `.chars().nth()` or `.bytes().nth()`