Unicode and UTF-8

Every string in Rust (&str and String) is guaranteed by the type system to be valid UTF-8—a variable-width encoding that represents every character in the Unicode standard using 1 to 4 bytes.

Unicode Code Points vs UTF-8 Encoding

Unicode is a universal character set that assigns a unique integer—called a code point (written U+0000 through U+10FFFF)—to every symbol across human writing systems, technical notation, and emoji.
UTF-8 (Unicode Transformation Format — 8-bit) is the byte encoding that translates those code points into sequences of 1 to 4 bytes (u8) in memory.

Code Point Range	UTF-8 Byte Length	Byte 1 Binary Pattern	Byte 2	Byte 3	Byte 4	Examples
`U+0000` .. `U+007F`	1 byte	`0xxxxxxx`	—	—	—	`'A'` (`U+0041`), `'0'`, `'\n'` (100% ASCII compatible)
`U+0080` .. `U+07FF`	2 bytes	`110xxxxx`	`10xxxxxx`	—	—	`'é'` (`U+00E9`), `'π'` (`U+03C0`), `'ش'`
`U+0800` .. `U+FFFF`	3 bytes	`1110xxxx`	`10xxxxxx`	`10xxxxxx`	—	`'ক'` (`U+0995`), `'中'` (`U+4E2D`), `'€'` (`U+20AC`)
`U+10000` .. `U+10FFFF`	4 bytes	`11110xxx`	`10xxxxxx`	`10xxxxxx`	`10xxxxxx`	`'🦀'` (`U+1F980`), `'𝄞'` (`U+1D11E`)

Variable-Width UTF-8 Byte Layout in Memory for "Aπক🦀" (10 bytes total, 4 chars):

  'A' (U+0041)     'π' (U+03C0)         'ক' (U+0995)              '🦀' (U+1F980)
  (1 byte)         (2 bytes)            (3 bytes)                 (4 bytes)
+------------+   +------------+----+   +------------+----+----+   +------------+----+----+----+
| 0x41       |   | 0xCF       |0x80|   | 0xE0       |0xA6|0x95|   | 0xF0       |0x9F|0xA6|0x80|
+------------+   +------------+----+   +------------+----+----+   +------------+----+----+----+
  Byte [0]         Bytes [1..3]          Bytes [3..6]               Bytes [6..10]

fn main() {
    let sample = "Aπক🦀";

    println!("String:            {sample}");
    println!("Byte length (.len): {} bytes", sample.len());
    println!("Char count:         {} chars", sample.chars().count());
    println!("Raw UTF-8 bytes:    {:02X?}", sample.as_bytes());
}

String:            Aπক🦀
Byte length (.len): 10 bytes
Char count:         4 chars
Raw UTF-8 bytes:    [41, CF, 80, E0, A6, 95, F0, 9F, A6, 80]

Why UTF-8 Is Self-Synchronizing and ASCII-Compatible

Look closely at the leading bits of every UTF-8 byte in the table above:

ASCII bytes (0x00..=0x7F) always start with bit 0. An ASCII byte value can never appear as part of a multi-byte character. Searching for b'/' or b'\n' in a UTF-8 buffer can never accidentally match the second or third byte of a non-ASCII character.
Leading bytes of multi-byte sequences start with 110, 1110, or 11110—the number of leading 1 bits tells you exactly how many bytes (2, 3, or 4) belong to that character.
Continuation bytes always start with 10 (0x80..=0xBF). Looking at any arbitrary byte in a string immediately tells you whether it is a valid character boundary or the middle of a multi-byte character (str::is_char_boundary).

Three Views of a Rust String: Bytes, Scalar Values (`char`), and Grapheme Clusters

When working with Unicode text, there is no single universal definition of a "character." Rust distinguishes three levels of representation:

Representation Level	How to Access in Rust	Element Type	Example on `"नमस्ते"` (Hindi "Namaste")
1. UTF-8 Bytes	`s.bytes()` or `s.as_bytes()`	`u8`	18 bytes (`[224, 164, 168, ...]`, 3 bytes per code point)
2. Unicode Scalar Values	`s.chars()`	`char` (4 bytes each)	6 `char`s: `['न', 'म', 'स', '्', 'त', 'े']` (4 consonants + 2 combining vowel/virama marks)
3. Extended Grapheme Clusters	`unicode-segmentation` crate (`.graphemes(true)`)	`&str`	4 user-perceived characters: `["न", "म", "स्ते", ...]`

fn main() {
    let word = "नमस्ते";

    println!("Byte length: {}", word.len());
    println!("Chars ({}):  {:?}", word.chars().count(), word.chars().collect::<Vec<char>>());

    // Iterating with `.char_indices()` gives the exact starting byte offset of each `char`:
    for (byte_offset, ch) in word.char_indices() {
        println!("  byte {byte_offset:2}: '{ch}' (U+{:04X}, {} bytes)", ch as u32, ch.len_utf8());
    }
}

Byte length: 18
Chars (6):  ['न', 'म', 'स', '्', 'त', 'े']
  byte  0: 'न' (U+0928, 3 bytes)
  byte  3: 'म' (U+092E, 3 bytes)
  byte  6: 'स' (U+0938, 3 bytes)
  byte  9: '्' (U+094D, 3 bytes)
  byte 12: 'त' (U+0924, 3 bytes)
  byte 15: 'े' (U+0947, 3 bytes)

Why Direct Integer Indexing (`s[i]`) Is Forbidden on Strings

In Rust, you cannot index a string with a single integer (let c = s[0];):

// ❌ does not compile
fn main() {
    let greeting = "Здравствуйте";
    let first = greeting[0];
}

error[E0277]: the type `str` cannot be indexed by `{integer}`
 --> src/main.rs:3:26
  |
3 |     let first = greeting[0];
  |                          ^ string indices are ranges of `usize`
  |
  = help: the trait `SliceIndex<str>` is not implemented for `{integer}`
  = note: you can use `.chars().nth()` or `.bytes().nth()`

Rust forbids s[0] for two reasons:

Ambiguous meaning: In "Здравствуйте", Cyrillic 'З' is 2 bytes ([0xD0, 0x97]). Returning raw byte 208 (0xD0) at index 0 would surprise anyone expecting the character 'З'.
$O(1)$ indexing expectation: Because UTF-8 is variable-width (1 to 4 bytes per char), finding the $n$-th char requires scanning from the beginning of the string in $O(n)$ time. Rust refuses to hide an $O(n)$ scan behind s[i] syntax that looks like an $O(1)$ array lookup.

Slicing Strings on Valid Char Boundaries

You can slice a string using a byte range (&s[start..end]), provided both start and end land on valid UTF-8 character boundaries. Slicing in the middle of a multi-byte character panics at runtime, whereas s.get(start..end) safely returns None:

fn main() {
    let crab = "🦀 Rust"; // '🦀' occupies bytes 0..4; ' ' is byte 4; "Rust" is bytes 5..9

    let emoji_slice: &str = &crab[0..4];
    let word_slice: &str = &crab[5..9];
    println!("Slices: '{emoji_slice}' and '{word_slice}'");

    // Safe non-panicking range lookup with `.get()`:
    println!("crab.get(0..2) (mid-emoji!): {:?}", crab.get(0..2));
    println!("crab.get(0..4) (valid):      {:?}", crab.get(0..4));
}

Slices: '🦀' and 'Rust'
crab.get(0..2) (mid-emoji!): None
crab.get(0..4) (valid):      Some("🦀")

Encoding and Decoding UTF-8 Explicitly

fn main() {
    // 1. Encode a single `char` into a stack buffer of up to 4 UTF-8 bytes
    let mut buf = [0_u8; 4];
    let encoded_str: &str = '🦀'.encode_utf8(&mut buf);
    println!("Encoded '🦀': {:?} ({encoded_str})", encoded_str.as_bytes());

    // 2. Validate raw bytes as UTF-8 (`std::str::from_utf8`)
    let valid_bytes = &[0xF0, 0x9F, 0xA6, 0x80];
    let invalid_bytes = &[0xFF, 0xFF];

    println!("from_utf8(valid):   {:?}", std::str::from_utf8(valid_bytes));
    println!("from_utf8(invalid): {:?}", std::str::from_utf8(invalid_bytes).is_err());

    // 3. Lossy conversion replaces invalid UTF-8 sequences with '' (U+FFFD)
    println!("from_utf8_lossy:    {}", String::from_utf8_lossy(b"Hello \xFF World"));
}

Encoded '🦀': [240, 159, 166, 128] (🦀)
from_utf8(valid):   Ok("🦀")
from_utf8(invalid): true
from_utf8_lossy:    Hello  World