Enum dmgrs::lexer::Lexeme

pub enum Lexeme {
    Error,
    HorizontalWhitespace,
    EndOfLineComment,
    StartMultiComment,
    EndMultiComment,
    EndOfLine,
    Punct(char),
    KwConst,
    KwFn,
    KwLoop,
    KwIf,
    KwElse,
    KwContinue,
    KwBreak,
    KwStatic,
    Ident(&'static str),
    Str(&'static str),
    BinaryLiteral(u16),
    DecimalLiteral(u16),
    HexLiteral(u16),
}

The possible tokens within a Dmgrs program.

At this early stage of the processing all keywords, instructions, and register names just count as an “ident”. A simple string interning mechanism is used, so it’s not a super allocation fest to do this.

Variants

Error

An error during lexing.

HorizontalWhitespace

Spaces and tabs, but not line endings.

EndOfLineComment

Starts with // and eats characters until the end of line.

StartMultiComment

Multi-line comments start with /*

EndMultiComment

Multi-line comments end with */

EndOfLine

A line ends with \r\n (windows), \n (unix), or \r (pre-unix mac).

Punct(char)

A punctuation character (using the [:punct:] regex class) that does not match any other case.

KwConst

Starts a constant declaration

KwFn

Starts a function declaration

KwLoop

Used at the opening of a loop

KwIf

Used for branching around code blocks

KwElse

Used for branching around code blocks

KwContinue

Continue a loop (jump to the start)

KwBreak

Break a loop (jump to the end)

KwStatic

Starts a static declaration

Ident(&'static str)

A standard identifier in C-style langs: [_a-zA-Z][_a-zA-Z0-9]*

The lone character _ ends up matching as an Ident rather than a Punctuation.

Str(&'static str)

Holds all the stuff between ".

This allows \" and \\ within the string literal that’s collected, but doesn’t actually handle escape sequence processing.

Thanks to Quirl, who made this regex: “works by specifying all of the escape sequences you allow (here just \", and \\ for a \ itself) up front and then requiring the rest of the literal to not be a quote or escape (that would start with \)”

BinaryLiteral(u16)

The digits of a binary literal (no prefix).

• Binary literals are a digit sequence (0-1) prefixed with % or 0b
• Use of _ is allowed in the literal, but they don’t change the value.

DecimalLiteral(u16)

The digits of a decimal literal.

• Decimal literals are a digit sequence (0-9) with no prefix.
• Use of _ is allowed in the literal, but they don’t change the value.

HexLiteral(u16)

The digits of a hex literal (no prefix).

• Hex literals are a digit sequence (0-9a-fA-F) prefixed with $ or 0x
• Use of _ is allowed in the literal, but they don’t change the value.

Trait Implementations

impl Clone for Lexeme

fn clone(&self) -> Lexeme

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Lexeme

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Lexeme

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for Lexeme

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given Hasher.

impl<'s> Logos<'s> for Lexeme

type Extras = ()

Associated type Extras for the particular lexer. This can be set using #[logos(extras = MyExtras)] and accessed inside callbacks.

type Source = str

Source type this token can be lexed from. This will default to str, unless one of the defined patterns explicitly uses non-unicode byte values or byte slices, in which case that implementation will use [u8].

const ERROR: Self = Lexeme::Error

Helper const of the variant marked as #[error].

fn lex(lex: &mut Lexer<'s, Self>)

The heart of Logos. Called by the Lexer. The implementation for this function is generated by the logos-derive crate.

fn lexer(source: &'source Self::Source) -> Lexer<'source, Self> where
    Self::Extras: Default, 

Create a new instance of a Lexer that will produce tokens implementing this Logos.

fn lexer_with_extras(
    source: &'source Self::Source,
    extras: Self::Extras
) -> Lexer<'source, Self>

Create a new instance of a Lexer with the provided Extras that will produce tokens implementing this Logos.

impl Ord for Lexeme

fn cmp(&self, other: &Lexeme) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self where
    Self: PartialOrd<Self>, 

Restrict a value to a certain interval.

impl PartialEq<Lexeme> for Lexeme

fn eq(&self, other: &Lexeme) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd<Lexeme> for Lexeme

fn partial_cmp(&self, other: &Lexeme) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl Copy for Lexeme

impl Eq for Lexeme

impl StructuralEq for Lexeme

impl StructuralPartialEq for Lexeme