Crate syn [] [src]

Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.

Currently this library is geared toward the custom derive use case but contains some APIs that may be useful for Rust procedural macros more generally.

Version requirement: Syn supports any compiler version back to Rust's very first support for procedural macros in Rust 1.15.0. Some features especially around error reporting are only available in newer compilers or on the nightly channel.

Example of a custom derive

The canonical custom derive using Syn looks like this. We write an ordinary Rust function tagged with a proc_macro_derive attribute and the name of the trait we are deriving. Any time that derive appears in the user's code, the Rust compiler passes their data structure as tokens into our macro. We get to execute arbitrary Rust code to figure out what to do with those tokens, then hand some tokens back to the compiler to compile into the user's crate.

[dependencies]
syn = "0.14"
quote = "0.6"

[lib]
proc-macro = true
extern crate proc_macro;
extern crate syn;

#[macro_use]
extern crate quote;

use proc_macro::TokenStream;
use syn::DeriveInput;

#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
    // Parse the input tokens into a syntax tree
    let input: DeriveInput = syn::parse(input).unwrap();

    // Build the output, possibly using quasi-quotation
    let expanded = quote! {
        // ...
    };

    // Hand the output tokens back to the compiler
    expanded.into()
}

The heapsize example directory shows a complete working Macros 1.1 implementation of a custom derive. It works on any Rust compiler >=1.15.0. The example derives a HeapSize trait which computes an estimate of the amount of heap memory owned by a value.

pub trait HeapSize {
    /// Total number of bytes of heap memory owned by `self`.
    fn heap_size_of_children(&self) -> usize;
}

The custom derive allows users to write #[derive(HeapSize)] on data structures in their program.

#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
    a: Box<T>,
    b: u8,
    c: &'a str,
    d: String,
}

Spans and error reporting

The heapsize2 example directory is an extension of the heapsize example that demonstrates some of the hygiene and error reporting properties of Macros 2.0. This example currently requires a nightly Rust compiler >=1.24.0-nightly but we are working to stabilize all of the APIs involved.

The token-based procedural macro API provides great control over where the compiler's error messages are displayed in user code. Consider the error the user sees if one of their field types does not implement HeapSize.

#[derive(HeapSize)]
struct Broken {
    ok: String,
    bad: std::thread::Thread,
}

In the Macros 1.1 string-based procedural macro world, the resulting error would point unhelpfully to the invocation of the derive macro and not to the actual problematic field.

error[E0599]: no method named `heap_size_of_children` found for type `std::thread::Thread` in the current scope
 --> src/main.rs:4:10
  |
4 | #[derive(HeapSize)]
  |          ^^^^^^^^

By tracking span information all the way through the expansion of a procedural macro as shown in the heapsize2 example, token-based macros in Syn are able to trigger errors that directly pinpoint the source of the problem.

error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
 --> src/main.rs:7:5
  |
7 |     bad: std::thread::Thread,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`

Parsing a custom syntax using combinators

The lazy-static example directory shows the implementation of a functionlike!(...) procedural macro in which the input tokens are parsed using nom-style parser combinators.

The example reimplements the popular lazy_static crate from crates.io as a procedural macro.

lazy_static! {
    static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}

The implementation shows how to trigger custom warnings and error messages on the macro input.

warning: come on, pick a more creative name
  --> src/main.rs:10:16
   |
10 |     static ref FOO: String = "lazy_static".to_owned();
   |                ^^^

Debugging

When developing a procedural macro it can be helpful to look at what the generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded or the cargo expand subcommand.

To show the expanded code for some crate that uses your procedural macro, run cargo expand from that crate. To show the expanded code for one of your own test cases, run cargo expand --test the_test_case where the last argument is the name of the test file without the .rs extension.

This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust's new Custom Derive system.

Optional features

Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.

Modules

buffer

A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor.

punctuated

A punctuated sequence of syntax tree nodes separated by punctuation.

spanned

A trait that can provide the Span of the complete contents of a syntax tree node.

synom

Parsing interface for parsing a token stream into a syntax tree node.

token

Tokens representing Rust punctuation, keywords, and delimiters.

visit

Syntax tree traversal to walk a shared borrow of a syntax tree.

Macros

Token

A type-macro that expands to the name of the Rust type representation of a given token.

alt

Run a series of parsers, returning the result of the first one which succeeds.

braces

Parse inside of { } curly braces.

brackets

Parse inside of [ ] square brackets.

call

Invoke the given parser function with zero or more arguments.

cond

Execute a parser only if a condition is met, otherwise return None.

cond_reduce

Execute a parser only if a condition is met, otherwise fail to parse.

custom_keyword

Parse the given word as a keyword.

do_parse

Run a series of parsers, optionally naming each intermediate result, followed by a step to combine the intermediate results.

epsilon

Parses nothing and always succeeds.

input_end

Parse nothing and succeed only if the end of the enclosing block has been reached.

keyword

Parse a single Rust keyword token.

many0

Parse zero or more values using the given parser.

map

Transform the result of a parser by applying a function or closure.

named

Define a parser function with the signature expected by syn parser combinators.

not

Invert the result of a parser by parsing successfully if the given parser fails to parse and vice versa.

option

Turn a failed parse into None and a successful parse into Some.

parens

Parse inside of ( ) parentheses.

parse_quote

Quasi-quotation macro that accepts input like the quote! macro but uses type inference to figure out a return type for those tokens.

punct

Parse a single Rust punctuation token.

reject

Unconditionally fail to parse anything.

switch

Pattern-match the result of a parser to select which other parser to run.

syn

Parse any type that implements the Synom trait.

tuple

Run a series of parsers and produce all of the results in a tuple.

value

Produce the given value without parsing anything.

Structs

Abi

The binary interface of a function: extern "C".

AngleBracketedGenericArguments

Angle bracketed arguments of a path segment: the <K, V> in HashMap<K, V>.

Attribute

An attribute like #[repr(transparent)].

BareFnArg

An argument in a function type: the usize in fn(usize) -> bool.

Binding

A binding (equality constraint) on an associated type: Item = u8.

BoundLifetimes

A set of bound lifetimes: for<'a, 'b, 'c>.

ConstParam

A const generic parameter: const LENGTH: usize.

DataEnum

An enum input to a proc_macro_derive macro.

DataStruct

A struct input to a proc_macro_derive macro.

DataUnion

A tagged union input to a proc_macro_derive macro.

DeriveInput

Data structure sent to a proc_macro_derive macro.

ExprArray

A slice literal expression: [a, b, c, d].

ExprAssign

An assignment expression: a = compute().

ExprAssignOp

A compound assignment expression: counter += 1.

ExprBinary

A binary operation: a + b, a * b.

ExprBlock

A blocked scope: { ... }.

ExprBox

A box expression: box f.

ExprBreak

A break, with an optional label to break and an optional expression.

ExprCall

A function call expression: invoke(a, b).

ExprCast

A cast expression: foo as f64.

ExprCatch

A catch expression: do catch { ... }.

ExprClosure

A closure expression: |a, b| a + b.

ExprContinue

A continue, with an optional label.

ExprField

Access of a named struct field (obj.k) or unnamed tuple struct field (obj.0).

ExprForLoop

A for loop: for pat in expr { ... }.

ExprGroup

An expression contained within invisible delimiters.

ExprIf

An if expression with an optional else block: if expr { ... } else { ... }.

ExprIfLet

An if let expression with an optional else block: if let pat = expr { ... } else { ... }.

ExprInPlace

A placement expression: place <- value.

ExprIndex

A square bracketed indexing expression: vector[2].

ExprLit

A literal in place of an expression: 1, "foo".

ExprLoop

Conditionless loop: loop { ... }.

ExprMacro

A macro invocation expression: format!("{}", q).

ExprMatch

A match expression: match n { Some(n) => {}, None => {} }.

ExprMethodCall

A method call expression: x.foo::<T>(a, b).

ExprParen

A parenthesized expression: (a + b).

ExprPath

A path like std::mem::replace possibly containing generic parameters and a qualified self-type.

ExprRange

A range expression: 1..2, 1.., ..2, 1..=2, ..=2.

ExprReference

A referencing operation: &a or &mut a.

ExprRepeat

An array literal constructed from one repeated element: [0u8; N].

ExprReturn

A return, with an optional value to be returned.

ExprStruct

A struct literal expression: Point { x: 1, y: 1 }.

ExprTry

A try-expression: expr?.

ExprTuple

A tuple expression: (a, b, c, d).

ExprType

A type ascription expression: foo: f64.

ExprUnary

A unary operation: !x, *x.

ExprUnsafe

An unsafe block: unsafe { ... }.

ExprVerbatim

Tokens in expression position not interpreted by Syn.

ExprWhile

A while loop: while expr { ... }.

ExprWhileLet

A while-let loop: while let pat = expr { ... }.

ExprYield

A yield expression: yield expr.

Field

A field of a struct or enum variant.

FieldsNamed

Named fields of a struct or struct variant such as Point { x: f64, y: f64 }.

FieldsUnnamed

Unnamed fields of a tuple struct or tuple variant such as Some(T).

Generics

Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc.

Ident

A word of Rust code, which may be a keyword or legal variable name.

ImplGenerics

Returned by Generics::split_for_impl.

Index

The index of an unnamed tuple struct field.

Lifetime

A Rust lifetime: 'a.

LifetimeDef

A lifetime definition: 'a: 'b + 'c + 'd.

LitBool

A boolean literal: true or false.

LitByte

A byte literal: b'f'.

LitByteStr

A byte string literal: b"foo".

LitChar

A character literal: 'a'.

LitFloat

A floating point literal: 1f64 or 1.0e10f64.

LitInt

An integer literal: 1 or 1u16.

LitStr

A UTF-8 string literal: "foo".

LitVerbatim

A raw token literal not interpreted by Syn, possibly because it represents an integer larger than 64 bits.

Macro

A macro invocation: println!("{}", mac).

MetaList

A structured list within an attribute, like derive(Copy, Clone).

MetaNameValue

A name-value pair within an attribute, like feature = "nightly".

ParenthesizedGenericArguments

Arguments of a function path segment: the (A, B) -> C in Fn(A,B) -> C.

Path

A path at which a named item is exported: std::collections::HashMap.

PathSegment

A segment of a path together with any path arguments on that segment.

PathTokens

A helper for printing a self-type qualified path as tokens.

PredicateEq

An equality predicate in a where clause (unsupported).

PredicateLifetime

A lifetime predicate in a where clause: 'a: 'b + 'c.

PredicateType

A type predicate in a where clause: for<'c> Foo<'c>: Trait<'c>.

QSelf

The explicit Self type in a qualified path: the T in <T as Display>::fmt.

TraitBound

A trait used as a bound on a type parameter.

Turbofish

Returned by TypeGenerics::as_turbofish.

TypeArray

A fixed size array type: [T; n].

TypeBareFn

A bare function type: fn(usize) -> bool.

TypeGenerics

Returned by Generics::split_for_impl.

TypeGroup

A type contained within invisible delimiters.

TypeImplTrait

An impl Bound1 + Bound2 + Bound3 type where Bound is a trait or a lifetime.

TypeInfer

Indication that a type should be inferred by the compiler: _.

TypeMacro

A macro in the type position.

TypeNever

The never type: !.

TypeParam

A generic type parameter: T: Into<String>.

TypeParen

A parenthesized type equivalent to the inner type.

TypePath

A path like std::slice::Iter, optionally qualified with a self-type as in <Vec<T> as SomeTrait>::Associated.

TypePtr

A raw pointer type: *const T or *mut T.

TypeReference

A reference type: &'a T or &'a mut T.

TypeSlice

A dynamically sized slice type: [T].

TypeTraitObject

A trait object type Bound1 + Bound2 + Bound3 where Bound is a trait or a lifetime.

TypeTuple

A tuple type: (A, B, C, String).

TypeVerbatim

Tokens in type position not interpreted by Syn.

Variant

An enum variant.

VisCrate

A crate-level visibility: crate.

VisPublic

A public visibility level: pub.

VisRestricted

A visibility level restricted to some path: pub(self) or pub(super) or pub(crate) or pub(in some::module).

WhereClause

A where clause in a definition: where T: Deserialize<'de>, D: 'static.

Enums

AttrStyle

Distinguishes between attributes that decorate an item and attributes that are contained within an item.

BareFnArgName

Name of an argument in a function type: the n in fn(n: usize).

BinOp

A binary operator: +, +=, &.

Data

The storage of a struct, enum or union data structure.

Expr

A Rust expression.

Fields

Data stored within an enum variant or struct.

FloatSuffix

The suffix on a floating point literal if any, like the f32 in 1.0f32.

GenericArgument

An individual generic argument, like 'a, T, or Item = T.

GenericParam

A generic type parameter, lifetime, or const generic: T: Into<String>, 'a: 'b, const LEN: usize.

IntSuffix

The suffix on an integer literal if any, like the u8 in 127u8.

Lit

A Rust literal such as a string or integer or boolean.

MacroDelimiter

A grouping token that surrounds a macro body: m!(...) or m!{...} or m![...].

Member

A struct or tuple struct field accessed in a struct literal or field expression.

Meta

Content of a compile-time structured attribute.

NestedMeta

Element of a compile-time attribute list.

PathArguments

Angle bracketed or parenthesized arguments of a path segment.

ReturnType

Return type of a function signature.

StrStyle

The style of a string literal, either plain quoted or a raw string like r##"data"##.

TraitBoundModifier

A modifier on a trait bound, currently only used for the ? in ?Sized.

Type

The possible types that a Rust value could have.

TypeParamBound

A trait or lifetime used as a bound on a type parameter.

UnOp

A unary operator: *, !, -.

Visibility

The visibility level of an item: inherited or pub or pub(restricted).

WherePredicate

A single predicate in a where clause: T: Deserialize<'de>.

Functions

parse

Parse tokens of source code into the chosen syntax tree node.

parse2

Parse a proc-macro2 token stream into the chosen syntax tree node.

parse_str

Parse a string of Rust code into the chosen syntax tree node.