Macros

Token Streams

A TokenStream (proc_macro::TokenStream or proc_macro2::TokenStream) is the fundamental input and output currency of procedural macros: an ordered sequence of TokenTree nodes produced by the compiler's lexer.

The Four Variants of TokenTree

Instead of passing raw source characters (&str) or a flat list of parentheses to macros, the Rust lexer groups balanced delimiters ((), [], {}) into nested Group trees before invoking any macro:

pub enum TokenTree {
    Group(Group),     // Balanced `( ... )`, `[ ... ]`, or `{ ... }` containing an inner TokenStream
    Ident(Ident),     // Identifier or keyword: `struct`, `Point`, `x`, `u32`
    Punct(Punct),     // Single punctuation character: `+`, `-`, `:`, `;`, `>` (with Spacing::Joint or Alone)
    Literal(Literal), // String, char, byte, integer, or float literal: `"hi"`, `42`, `3.14`
}

Source Code:   `fn add(a: i32) -> i32 { a + 1 }`

TokenStream (7 top-level TokenTrees):
  [0] Ident("fn")
  [1] Ident("add")
  [2] Group(Parenthesis, TokenStream: [ Ident("a"), Punct(':'), Ident("i32") ])
  [3] Punct('-', Spacing::Joint)
  [4] Punct('>', Spacing::Alone)
  [5] Ident("i32")
  [6] Group(Brace,       TokenStream: [ Ident("a"), Punct('+'), Literal(1) ])

Notice that multi-character operators like ->, ::, +=, and && are represented as consecutive Punct tokens whose Spacing is Spacing::Joint on all characters except the final one (Spacing::Alone).

Every TokenTree Carries a Span

Every Group, Ident, Punct, and Literal inside a TokenStream carries a Span recording its exact source file, byte range, and hygiene context. When a procedural macro constructs new tokens (or emits a compile_error!) and attaches the input token's Span, compiler diagnostics underline the exact offending token in the user's source file rather than pointing blindly to the macro call as a whole.

Constructing Token Streams with quote!

Writing TokenTree::Ident(...) and TokenTree::Punct(...) calls manually is verbose. In practice, procedural macros use the quote::quote! macro to construct a proc_macro2::TokenStream using #var interpolation and #(#slice)* repetition:

// Inside a procedural macro using `quote`:
/*
let field_names = [&ident_a, &ident_b];
let tokens = quote! {
    impl Summary for #struct_name {
        fn fields(&self) -> &[&'static str] {
            &[ #( stringify!(#field_names) ),* ]
        }
    }
};
*/

// Resulting Rust code produced by the TokenStream above:
struct Record {
    id: u64,
    active: bool,
}

trait Summary {
    fn fields(&self) -> &[&'static str];
}

impl Summary for Record {
    fn fields(&self) -> &[&'static str] {
        &[stringify!(id), stringify!(active)]
    }
}

fn main() {
    let rec = Record {
        id: 1,
        active: true,
    };
    println!(
        "Record (id={}, active={}) fields: {:?}",
        rec.id,
        rec.active,
        rec.fields()
    );
}

Record (id=1, active=true) fields: ["id", "active"]