Strings and Text Processing

Unicode Normalization Considerations

In Unicode, many visually identical characters can be represented by different sequences of code points (such as a single precomposed character vs. a base letter followed by a combining mark). Because Rust's standard ==, Ord, and Hash implementations on str and String compare raw UTF-8 bytes, visually identical strings in different normalization forms compare as not equal.

Precomposed (NFC) vs. Decomposed (NFD) Representations

Consider the character é (Latin small letter e with acute):

  • NFC (Canonical Decomposition, followed by Canonical Composition): A single code point U+00E9 ('é', 2 UTF-8 bytes [0xC3, 0xA9]).
  • NFD (Canonical Decomposition): Two code points—ASCII 'e' (U+0065, 1 byte [0x65]) followed by the combining acute accent U+0301 (2 bytes [0xCC, 0x81]), totaling 3 UTF-8 bytes.
NFC ("\u{00E9}"):         [ 0xC3, 0xA9 ]        (1 char, 2 bytes)
NFD ("e\u{0301}"):        [ 0x65, 0xCC, 0x81 ]  (2 chars, 3 bytes)

fn main() {
    let nfc = "caf\u{00E9}";      // Precomposed 'é' (typical on Linux/Windows/Web)
    let nfd = "cafe\u{0301}";     // Decomposed 'e' + combining acute (typical on macOS HFS+/APFS filenames)

    println!("NFC: \"{nfc}\" (bytes = {:?}, chars = {})", nfc.as_bytes(), nfc.chars().count());
    println!("NFD: \"{nfd}\" (bytes = {:?}, chars = {})", nfd.as_bytes(), nfd.chars().count());
    println!("nfc == nfd (byte equality): {}", nfc == nfd);
}

NFC: "café" (bytes = [99, 97, 102, 195, 169], chars = 4)
NFD: "café" (bytes = [99, 97, 102, 101, 204, 129], chars = 5)
nfc == nfd (byte equality): false

The Four Standard Unicode Normalization Forms

Form Name Behavior Typical Use Case
NFC Canonical Composition Decomposes then recomposes into shortest canonical code points W3C / Web standard text storage, usernames, display text
NFD Canonical Decomposition Splits precomposed characters into base + combining marks Internal text processing, stripping accents, macOS APFS filenames
NFKC Compatibility Composition Replaces compatibility variants (e.g., 'fi' ligature -> "fi", '²' -> '2', fullwidth 'Ａ' -> 'A') then composes Identifier matching (UAX #31), search indexing, security deduplication
NFKD Compatibility Decomposition Compatibility decomposition without recomposition Fuzzy full-text search and transliteration

Why the Standard Library Does Not Bundle Normalization Tables

Full Unicode normalization tables (unicode-normalization crate) are large and evolve across Unicode versions, whereas core and std keep str equality and hashing fast ($O(n)$ byte memcmp). When comparing user-supplied identifiers, passwords, or cross-platform filenames:

  1. Normalize inputs at the boundary (typically to NFC for general text or NFKC for identifiers/search queries) using the unicode-normalization crate (s.nfc().collect::<String>()).
  2. Store and hash only the normalized String so standard == and HashMap lookups remain $O(n)$ byte comparisons.