Strings and Text Processing

Character Count vs. Byte Count

Unicode text has three distinct levels of representation—UTF-8 bytes, Unicode scalar values (char), and extended grapheme clusters—each yielding a different count for non-ASCII or composite text.

The Three Levels of Text Granularity

Level Rust API Time Complexity What It Counts
1. UTF-8 Bytes s.len() or s.bytes().count() $O(1)$ Raw encoded bytes (1..=4 bytes per char)
2. Unicode Scalar Values (char) s.chars().count() $O(n)$ Code points (U+0000..=U+D7FF and U+E000..=U+10FFFF), including standalone combining marks and zero-width joiners
3. Extended Grapheme Clusters External unicode-segmentation crate (s.graphemes(true).count()) $O(n)$ User-perceived visual characters (a base character plus any combining accents or emoji sequences)

How Combining Marks and Emoji Sequences Diverge

A single visual character on screen (a grapheme cluster) can be composed of multiple Unicode scalar values (chars):

  1. Decomposed accents: "e\u{0301}" (visual é) consists of two chars—'e' (U+0065, 1 byte) followed by the combining acute accent U+0301 (2 bytes)—totaling 3 bytes and 2 chars for 1 visual character.
  2. Indic consonant + vowel signs: "কি" consists of 'ক' (U+0995, 3 bytes) + vowel sign 'ি' (U+09BF, 3 bytes)—totaling 6 bytes and 2 chars.
  3. Zero-Width Joiner (ZWJ) emoji sequences: "👩‍💻" (woman technologist) is '👩' (U+1F469, 4 bytes) + ZWJ (U+200D, 3 bytes) + '💻' (U+1F4BB, 4 bytes)—totaling 11 bytes and 3 chars for 1 visual emoji.
fn analyze(label: &str, s: &str) {
    let chars: Vec<char> = s.chars().collect();
    println!(
        "{label:<22} -> bytes: {:>2}, chars: {} {:?}",
        s.len(),
        chars.len(),
        chars
    );
}

fn main() {
    analyze("Precomposed 'é'", "\u{00E9}");
    analyze("Decomposed 'e' + '◌́'", "e\u{0301}");
    analyze("Bengali 'কি'", "কি");
    analyze("Emoji ZWJ '👩‍💻'", "👩\u{200D}💻");
}

Precomposed 'é'        -> bytes:  2, chars: 1 ['é']
Decomposed 'e' + '◌́'   -> bytes:  3, chars: 2 ['e', '\u{301}']
Bengali 'কি'           -> bytes:  6, chars: 2 ['ক', 'ি']
Emoji ZWJ '👩‍💻'         -> bytes: 11, chars: 3 ['👩', '\u{200d}', '💻']

Fast Path When a String Is Pure ASCII (s.is_ascii())

When s.is_ascii() returns true (every byte is in 0x00..=0x7F), each char is guaranteed to be exactly 1 byte, so s.len() equals s.chars().count():

fn char_count_fast(s: &str) -> usize {
    if s.is_ascii() {
        s.len() // O(1) when ASCII
    } else {
        s.chars().count() // O(n) UTF-8 scan when non-ASCII bytes are present
    }
}

fn main() {
    println!("ASCII count:   {}", char_count_fast("hello_world"));
    println!("Unicode count: {}", char_count_fast("হ্যালো"));
}

ASCII count:   11
Unicode count: 6