Declarative and Procedural Macros in Depth

Parsing Token Streams

While the compiler passes a raw proc_macro::TokenStream (a flat sequence of Group, Ident, Punct, and Literal token trees) into a procedural macro, real-world procedural macros use the standard proc-macro2, syn, and quote triad to parse token streams into typed syntax trees and turn Rust AST nodes back into output token streams.

The proc-macro2, syn, and quote Triad

Input `proc_macro::TokenStream` (from `rustc`)
       |
       |  `syn::parse_macro_input!(input as DeriveInput)`
       v
Typed AST (`syn::DeriveInput`, `syn::ItemFn`, or custom `syn::parse::Parse`)
       |
       |  Inspect fields, generics, and attributes; interpolate into `quote! { ... }`
       v
`proc_macro2::TokenStream`
       |
       |  `.into()`
       v
Output `proc_macro::TokenStream` (returned to `rustc`)
Crate Primary Role Why It Exists
proc-macro2 Wrapper around proc_macro::TokenStream and Span proc_macro types panic if used outside the compiler's proc-macro execution context; proc-macro2 allows unit-testing macro logic in normal cargo test binaries!
syn Recursive-descent parser for Rust syntax and custom DSLs Parses TokenStream into rich AST structs (DeriveInput, ItemFn, Expr, Type, Generics)
quote Quasi-quoting macro quote! { ... } with #var and #(#items)* interpolation Converts Rust syntax with interpolated AST variables back into a proc_macro2::TokenStream

Implementing a Custom Token-Stream Parser in Standard Rust

To understand how syn::parse::Parse works under the hood, here is a complete, runnable recursive-descent token parser over token trees that parses key: Type = default_expr declarations and generates a struct definition:

#[derive(Debug, PartialEq)]
enum SimpleToken<'a> {
    Ident(&'a str),
    Colon,
    Equals,
    Literal(&'a str),
}

#[derive(Debug)]
struct FieldSpec<'a> {
    name: &'a str,
    ty: &'a str,
    default_lit: &'a str,
}

fn parse_field_spec<'a>(tokens: &[SimpleToken<'a>]) -> Result<FieldSpec<'a>, &'static str> {
    match tokens {
        [
            SimpleToken::Ident(name),
            SimpleToken::Colon,
            SimpleToken::Ident(ty),
            SimpleToken::Equals,
            SimpleToken::Literal(default_lit),
        ] => Ok(FieldSpec {
            name,
            ty,
            default_lit,
        }),
        _ => Err("expected `<name>: <Type> = <literal>`"),
    }
}

fn main() {
    let stream = [
        SimpleToken::Ident("port"),
        SimpleToken::Colon,
        SimpleToken::Ident("u16"),
        SimpleToken::Equals,
        SimpleToken::Literal("8080"),
    ];

    let spec = parse_field_spec(&stream).unwrap();
    println!(
        "parsed FieldSpec -> pub {}: {} (default = {})",
        spec.name, spec.ty, spec.default_lit
    );
}

parsed FieldSpec -> pub port: u16 (default = 8080)

Parsing Custom Syntax with syn::parse::Parse

In a procedural macro crate using syn, any custom DSL struct can implement syn::parse::Parse:

// Inside a `proc-macro = true` crate with `syn`:
/*
use syn::parse::{Parse, ParseStream};
use syn::{Expr, Ident, Result, Token};

struct KeyValueInput {
    key: Ident,
    _arrow: Token![=>],
    value: Expr,
}

impl Parse for KeyValueInput {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(KeyValueInput {
            key: input.parse()?,
            _arrow: input.parse()?,
            value: input.parse()?,
        })
    }
}
*/
fn main() {
    println!("syn::parse::Parse advances a cursor over ParseStream token by token.");
}

syn::parse::Parse advances a cursor over ParseStream token by token.