Struct ParseBuffer

pub struct ParseBuffer<'a> { /* private fields */ }

Cursor position within a buffered token stream.

This type is more commonly used through the type alias ParseStream which is an alias for &ParseBuffer.

ParseStream is the input type for all parser functions in Syn. They have the signature fn(ParseStream) -> Result<T>.

Calling a parser function

There is no public way to construct a ParseBuffer. Instead, if you are looking to invoke a parser function that requires ParseStream as input, you will need to go through one of the public parsing entry points.

• The parse_macro_input! macro if parsing input of a procedural macro;
• One of the syn::parse* functions; or
• A method of the Parser trait.

Implementations

impl<'a> ParseBuffer<'a>

pub fn parse<T>(&self) -> Result<T, Error>

where T: Parse,

Parses a syntax tree node of type T, advancing the position of our parse stream past it.

pub fn call<T>( &'a self, function: fn(_: &'a ParseBuffer<'a>) -> Result<T, Error>, ) -> Result<T, Error>

Calls the given parser function to parse a syntax tree node of type T from this stream.

Example

The parser below invokes Attribute::parse_outer to parse a vector of zero or more outer attributes.

use syn::{Attribute, Ident, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses a unit struct with attributes.
//
//     #[path = "s.tmpl"]
//     struct S;
struct UnitStruct {
    attrs: Vec<Attribute>,
    struct_token: Token![struct],
    name: Ident,
    semi_token: Token![;],
}

impl Parse for UnitStruct {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(UnitStruct {
            attrs: input.call(Attribute::parse_outer)?,
            struct_token: input.parse()?,
            name: input.parse()?,
            semi_token: input.parse()?,
        })
    }
}

pub fn peek<T>(&self, token: T) -> bool

where T: Peek,

Looks at the next token in the parse stream to determine whether it matches the requested type of token.

Does not advance the position of the parse stream.

Syntax

Note that this method does not use turbofish syntax. Pass the peek type inside of parentheses.

• input.peek(Token![struct])
• input.peek(Token![==])
• input.peek(syn::Ident) (does not accept keywords)
• input.peek(syn::Ident::peek_any)
• input.peek(Lifetime)
• input.peek(token::Brace)

Example

In this example we finish parsing the list of supertraits when the next token in the input is either where or an opening curly brace.

use syn::{braced, token, Generics, Ident, Result, Token, TypeParamBound};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

// Parses a trait definition containing no associated items.
//
//     trait Marker<'de, T>: A + B<'de> where Box<T>: Clone {}
struct MarkerTrait {
    trait_token: Token![trait],
    ident: Ident,
    generics: Generics,
    colon_token: Option<Token![:]>,
    supertraits: Punctuated<TypeParamBound, Token![+]>,
    brace_token: token::Brace,
}

impl Parse for MarkerTrait {
    fn parse(input: ParseStream) -> Result<Self> {
        let trait_token: Token![trait] = input.parse()?;
        let ident: Ident = input.parse()?;
        let mut generics: Generics = input.parse()?;
        let colon_token: Option<Token![:]> = input.parse()?;

        let mut supertraits = Punctuated::new();
        if colon_token.is_some() {
            loop {
                supertraits.push_value(input.parse()?);
                if input.peek(Token![where]) || input.peek(token::Brace) {
                    break;
                }
                supertraits.push_punct(input.parse()?);
            }
        }

        generics.where_clause = input.parse()?;
        let content;
        let empty_brace_token = braced!(content in input);

        Ok(MarkerTrait {
            trait_token,
            ident,
            generics,
            colon_token,
            supertraits,
            brace_token: empty_brace_token,
        })
    }
}

pub fn peek2<T>(&self, token: T) -> bool

where T: Peek,

Looks at the second-next token in the parse stream.

This is commonly useful as a way to implement contextual keywords.

Example

This example needs to use peek2 because the symbol union is not a keyword in Rust. We can’t use just peek and decide to parse a union if the very next token is union, because someone is free to write a mod union and a macro invocation that looks like union::some_macro! { ... }. In other words union is a contextual keyword.

use syn::{Ident, ItemUnion, Macro, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses either a union or a macro invocation.
enum UnionOrMacro {
    // union MaybeUninit<T> { uninit: (), value: T }
    Union(ItemUnion),
    // lazy_static! { ... }
    Macro(Macro),
}

impl Parse for UnionOrMacro {
    fn parse(input: ParseStream) -> Result<Self> {
        if input.peek(Token![union]) && input.peek2(Ident) {
            input.parse().map(UnionOrMacro::Union)
        } else {
            input.parse().map(UnionOrMacro::Macro)
        }
    }
}

pub fn peek3<T>(&self, token: T) -> bool

where T: Peek,

Looks at the third-next token in the parse stream.

pub fn parse_terminated<T, P>( &'a self, parser: fn(_: &'a ParseBuffer<'a>) -> Result<T, Error>, separator: P, ) -> Result<Punctuated<T, <P as Peek>::Token>, Error>

where P: Peek, <P as Peek>::Token: Parse,

Parses zero or more occurrences of T separated by punctuation of type P, with optional trailing punctuation.

Parsing continues until the end of this parse stream. The entire content of this parse stream must consist of T and P.

Example

use syn::{parenthesized, token, Ident, Result, Token, Type};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

// Parse a simplified tuple struct syntax like:
//
//     struct S(A, B);
struct TupleStruct {
    struct_token: Token![struct],
    ident: Ident,
    paren_token: token::Paren,
    fields: Punctuated<Type, Token![,]>,
    semi_token: Token![;],
}

impl Parse for TupleStruct {
    fn parse(input: ParseStream) -> Result<Self> {
        let content;
        Ok(TupleStruct {
            struct_token: input.parse()?,
            ident: input.parse()?,
            paren_token: parenthesized!(content in input),
            fields: content.parse_terminated(Type::parse, Token![,])?,
            semi_token: input.parse()?,
        })
    }
}

See also

If your separator is anything more complicated than an invocation of the Token! macro, this method won’t be applicable and you can instead directly use Punctuated’s parser functions: parse_terminated, parse_separated_nonempty etc.

use syn::{custom_keyword, Expr, Result, Token};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

mod kw {
    syn::custom_keyword!(fin);
}

struct Fin(kw::fin, Token![;]);

impl Parse for Fin {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(Self(input.parse()?, input.parse()?))
    }
}

struct Thing {
    steps: Punctuated<Expr, Fin>,
}

impl Parse for Thing {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(Thing {
            steps: Punctuated::parse_terminated(input)?,
        })
        // or equivalently, this means the same thing:
            steps: input.call(Punctuated::parse_terminated)?,
    }
}

pub fn is_empty(&self) -> bool

Returns whether there are no more tokens remaining to be parsed from this stream.

This method returns true upon reaching the end of the content within a set of delimiters, as well as at the end of the tokens provided to the outermost parsing entry point.

This is equivalent to .peek(syn::parse::End). Use .peek2(End) or .peek3(End) to look for the end of a parse stream further ahead than the current position.

Example

use syn::{braced, token, Ident, Item, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses a Rust `mod m { ... }` containing zero or more items.
struct Mod {
    mod_token: Token![mod],
    name: Ident,
    brace_token: token::Brace,
    items: Vec<Item>,
}

impl Parse for Mod {
    fn parse(input: ParseStream) -> Result<Self> {
        let content;
        Ok(Mod {
            mod_token: input.parse()?,
            name: input.parse()?,
            brace_token: braced!(content in input),
            items: {
                let mut items = Vec::new();
                while !content.is_empty() {
                    items.push(content.parse()?);
                }
                items
            },
        })
    }
}

pub fn lookahead1(&self) -> Lookahead1<'a>

Constructs a helper for peeking at the next token in this stream and building an error message if it is not one of a set of expected tokens.

Example

use syn::{ConstParam, Ident, Lifetime, LifetimeParam, Result, Token, TypeParam};
use syn::parse::{Parse, ParseStream};

// A generic parameter, a single one of the comma-separated elements inside
// angle brackets in:
//
//     fn f<T: Clone, 'a, 'b: 'a, const N: usize>() { ... }
//
// On invalid input, lookahead gives us a reasonable error message.
//
//     error: expected one of: identifier, lifetime, `const`
//       |
//     5 |     fn f<!Sized>() {}
//       |          ^
enum GenericParam {
    Type(TypeParam),
    Lifetime(LifetimeParam),
    Const(ConstParam),
}

impl Parse for GenericParam {
    fn parse(input: ParseStream) -> Result<Self> {
        let lookahead = input.lookahead1();
        if lookahead.peek(Ident) {
            input.parse().map(GenericParam::Type)
        } else if lookahead.peek(Lifetime) {
            input.parse().map(GenericParam::Lifetime)
        } else if lookahead.peek(Token![const]) {
            input.parse().map(GenericParam::Const)
        } else {
            Err(lookahead.error())
        }
    }
}

pub fn fork(&self) -> ParseBuffer<'a>

Forks a parse stream so that parsing tokens out of either the original or the fork does not advance the position of the other.

Performance

Forking a parse stream is a cheap fixed amount of work and does not involve copying token buffers. Where you might hit performance problems is if your macro ends up parsing a large amount of content more than once.

// Do not do this.
if input.fork().parse::<Expr>().is_ok() {
    return input.parse::<Expr>();
}

As a rule, avoid parsing an unbounded amount of tokens out of a forked parse stream. Only use a fork when the amount of work performed against the fork is small and bounded.

When complex speculative parsing against the forked stream is unavoidable, use parse::discouraged::Speculative to advance the original stream once the fork’s parse is determined to have been successful.

For a lower level way to perform speculative parsing at the token level, consider using ParseStream::step instead.

Example

The parse implementation shown here parses possibly restricted pub visibilities.

• pub
• pub(crate)
• pub(self)
• pub(super)
• pub(in some::path)

To handle the case of visibilities inside of tuple structs, the parser needs to distinguish parentheses that specify visibility restrictions from parentheses that form part of a tuple type.

struct S(pub(crate) A, pub (B, C));

In this example input the first tuple struct element of S has pub(crate) visibility while the second tuple struct element has pub visibility; the parentheses around (B, C) are part of the type rather than part of a visibility restriction.

The parser uses a forked parse stream to check the first token inside of parentheses after the pub keyword. This is a small bounded amount of work performed against the forked parse stream.

use syn::{parenthesized, token, Ident, Path, Result, Token};
use syn::ext::IdentExt;
use syn::parse::{Parse, ParseStream};

struct PubVisibility {
    pub_token: Token![pub],
    restricted: Option<Restricted>,
}

struct Restricted {
    paren_token: token::Paren,
    in_token: Option<Token![in]>,
    path: Path,
}

impl Parse for PubVisibility {
    fn parse(input: ParseStream) -> Result<Self> {
        let pub_token: Token![pub] = input.parse()?;

        if input.peek(token::Paren) {
            let ahead = input.fork();
            let mut content;
            parenthesized!(content in ahead);

            if content.peek(Token![crate])
                || content.peek(Token![self])
                || content.peek(Token![super])
            {
                return Ok(PubVisibility {
                    pub_token,
                    restricted: Some(Restricted {
                        paren_token: parenthesized!(content in input),
                        in_token: None,
                        path: Path::from(content.call(Ident::parse_any)?),
                    }),
                });
            } else if content.peek(Token![in]) {
                return Ok(PubVisibility {
                    pub_token,
                    restricted: Some(Restricted {
                        paren_token: parenthesized!(content in input),
                        in_token: Some(content.parse()?),
                        path: content.call(Path::parse_mod_style)?,
                    }),
                });
            }
        }

        Ok(PubVisibility {
            pub_token,
            restricted: None,
        })
    }
}

pub fn error<T>(&self, message: T) -> Error

where T: Display,

Triggers an error at the current position of the parse stream.

Example

use syn::{Expr, Result, Token};
use syn::parse::{Parse, ParseStream};

// Some kind of loop: `while` or `for` or `loop`.
struct Loop {
    expr: Expr,
}

impl Parse for Loop {
    fn parse(input: ParseStream) -> Result<Self> {
        if input.peek(Token![while])
            || input.peek(Token![for])
            || input.peek(Token![loop])
        {
            Ok(Loop {
                expr: input.parse()?,
            })
        } else {
            Err(input.error("expected some kind of loop"))
        }
    }
}

pub fn step<F, R>(&self, function: F) -> Result<R, Error>

where F: for<'c> FnOnce(StepCursor<'c, 'a>) -> Result<(R, Cursor<'c>), Error>,

Speculatively parses tokens from this parse stream, advancing the position of this stream only if parsing succeeds.

This is a powerful low-level API used for defining the Parse impls of the basic built-in token types. It is not something that will be used widely outside of the Syn codebase.

Example

use proc_macro2::TokenTree;
use syn::Result;
use syn::parse::ParseStream;

// This function advances the stream past the next occurrence of `@`. If
// no `@` is present in the stream, the stream position is unchanged and
// an error is returned.
fn skip_past_next_at(input: ParseStream) -> Result<()> {
    input.step(|cursor| {
        let mut rest = *cursor;
        while let Some((tt, next)) = rest.token_tree() {
            match &tt {
                TokenTree::Punct(punct) if punct.as_char() == '@' => {
                    return Ok(((), next));
                }
                _ => rest = next,
            }
        }
        Err(cursor.error("no `@` was found after this point"))
    })
}

pub fn span(&self) -> Span

Returns the Span of the next token in the parse stream, or Span::call_site() if this parse stream has completely exhausted its input TokenStream.

pub fn cursor(&self) -> Cursor<'a>

Provides low-level access to the token representation underlying this parse stream.

Cursors are immutable so no operations you perform against the cursor will affect the state of this parse stream.

Example

use proc_macro2::TokenStream;
use syn::buffer::Cursor;
use syn::parse::{ParseStream, Result};

// Run a parser that returns T, but get its output as TokenStream instead of T.
// This works without T needing to implement ToTokens.
fn recognize_token_stream<T>(
    recognizer: fn(ParseStream) -> Result<T>,
) -> impl Fn(ParseStream) -> Result<TokenStream> {
    move |input| {
        let begin = input.cursor();
        recognizer(input)?;
        let end = input.cursor();
        Ok(tokens_between(begin, end))
    }
}

// Collect tokens between two cursors as a TokenStream.
fn tokens_between(begin: Cursor, end: Cursor) -> TokenStream {
    assert!(begin <= end);

    let mut cursor = begin;
    let mut tokens = TokenStream::new();
    while cursor < end {
        let (token, next) = cursor.token_tree().unwrap();
        tokens.extend(std::iter::once(token));
        cursor = next;
    }
    tokens
}

fn main() {
    use quote::quote;
    use syn::parse::{Parse, Parser};
    use syn::Token;

    // Parse syn::Type as a TokenStream, surrounded by angle brackets.
    fn example(input: ParseStream) -> Result<TokenStream> {
        let _langle: Token![<] = input.parse()?;
        let ty = recognize_token_stream(syn::Type::parse)(input)?;
        let _rangle: Token![>] = input.parse()?;
        Ok(ty)
    }

    let tokens = quote! { <fn() -> u8> };
    println!("{}", example.parse2(tokens).unwrap());
}

Trait Implementations

impl<'a> AnyDelimiter for ParseBuffer<'a>

fn parse_any_delimiter( &self, ) -> Result<(Delimiter, DelimSpan, ParseBuffer<'_>), Error>

Returns the delimiter, the span of the delimiter token, and the nested contents for further parsing.

impl<'a> Debug for ParseBuffer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'a> Display for ParseBuffer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'a> Drop for ParseBuffer<'a>

fn drop(&mut self)

Executes the destructor for this type.

impl<'a> Speculative for ParseBuffer<'a>

fn advance_to(&self, fork: &ParseBuffer<'a>)

Advance this parse stream to the position of a forked parse stream.

impl<'a> RefUnwindSafe for ParseBuffer<'a>

impl<'a> UnwindSafe for ParseBuffer<'a>