Token Trees

A token tree (tt) is the fundamental structural unit passed into and out of Rust macros. Every token stream produced by the Rust lexer is organized into a sequence of token trees where all delimiters—parentheses (...), square brackets [...], and curly braces {...}—are guaranteed to be balanced.

The Two Kinds of Token Trees

A single token tree is either:

1. A single non-delimiter token (leaf token): an identifier (foo), literal (42, "text"), keyword (fn, let), or punctuation token (+, ::, ->, ;).
2. A delimited group: a matching pair of (...), [...], or {...} containing zero or more nested token trees inside it.

Expression:  `compute(a + 1, [x, y])`

Parsed as 2 Top-Level Token Trees (`$tt:tt`):
  Token Tree #1 (Single Identifier): `compute`
  Token Tree #2 (Delimited Group):   `( a + 1 , [ x , y ] )`
                                        | | | |   ^^^^^^^
                                        | | | |   Nested Delimited Token Tree `[ x , y ]`
                                        | | | +-- Punct `,`
                                        | | +---- Literal `1`
                                        | +------ Punct `+`
                                        +-------- Ident `a`

Notice that (a + 1, [x, y]) is a single token tree ($args:tt) at the outer level because the parentheses group everything inside them into one balanced node!

macro_rules! count_token_trees {
    () => { 0_usize };
    ($first:tt $($rest:tt)*) => {
        1_usize + count_token_trees!($($rest)*)
    };
}

fn main() {
    // 4 single tokens: `a`, `+`, `b`, `;`
    let flat = count_token_trees!(a + b ;);

    // Only 2 token trees at the top level: `compute` and `(a + 1, [x, y])`!
    let grouped = count_token_trees!(compute(a + 1, [x, y]));

    println!("flat token tree count:    {}", flat);
    println!("grouped token tree count: {}", grouped);
}

flat token tree count:    4
grouped token tree count: 2

Why Rust Macros Require Balanced Delimiters

Because the lexer groups tokens into token trees before any macro runs, it is syntactically impossible to invoke a Rust macro with unmatched delimiters (such as bad_macro!( { ) or open_brace!() paired with close_brace!()). This guarantee ensures that:

• Syntax highlighters, formatters (rustfmt), and IDEs (rust-analyzer) can parse an entire source file without expanding external procedural macros first.
• A macro expansion can never corrupt the brace structure of the surrounding function or module.

Invisible AST Group Delimiters ($:expr vs. $:tt)

When macro_rules! captures a fragment using a typed fragment specifier other than :tt, :ident, or :lifetime (for example, $e:expr, $t:ty, or $p:pat), the compiler wraps the captured AST node in an opaque AST group (invisible parentheses).

Once $e:expr is captured, it is treated as a single opaque :tt token tree if forwarded to a helper macro, and its internal tokens can no longer be destructured token-by-token (though it matches $next:expr or $single:tt). If a macro needs to inspect or split the raw tokens inside an argument before parsing it as an expression, capture it as $($tokens:tt)+ first:

macro_rules! forward_as_tt {
    ($single_tt:tt) => {
        println!("matched as a single tt: {}", stringify!($single_tt));
    };
}

macro_rules! capture_expr_then_forward {
    ($e:expr) => {
        // Even though `1 + 2 * 3` was 5 tokens originally, capturing it as `$e:expr`
        // bundles it into a single opaque token tree when forwarded to `forward_as_tt!`:
        forward_as_tt!($e);
    };
}

fn main() {
    capture_expr_then_forward!(1 + 2 * 3);
}

matched as a single tt: 1 + 2 * 3