Module com.maroontress.clione
Package com.maroontress.clione

Interface Token

All Known Implementing Classes:
DefaultToken
public interface Token
The preprocessing token.

Token objects can have children, which means they can be in a tree structure. For tokens that the LexicalParser.next() method returns, tokens of type TokenType.DIRECTIVE only have children.

The Token object has its type, span, and characters. The type is one of the constants defined in TokenType, the span represents the range of the source file where the token occurs, and the characters are SourceChar objects that compose it.

Note that the Token object is an immutable object.

  • Method Summary

    Modifier and Type
    Method
    Description
    List<SourceChar>
    getChars()
    Returns the characters that compose this token.
    List<Token>
    getChildren()
    Returns the child tokens.
    SourceSpan
    getSpan()
    Returns a new span representing the range of this token in the source file.
    TokenType
    getType()
    Returns the type of this token.
    String
    getValue()
    Returns a new string representing this token.
    String
    toString()
    Returns a new string representation of this token that is easy for a person to read.
    Token
    withChildren(Collection<Token> newChildren)
    Returns a new token that has the same content of this token but has the specified child tokens.
    Token
    withType(TokenType newType)
    Returns a new token that has the same content of this token but has the specified token type.

  • Method Details

    • getValue

      String getValue()
      Returns a new string representing this token.

      The string that this method returns does not have the clue of the token type and does not include the content of the child tokens.

      Returns:
      The new string representing this token.

    • getSpan

      SourceSpan getSpan()
      Returns a new span representing the range of this token in the source file.
      Returns:
      A new span representing the range of this token.

    • getChars

      List<SourceChar> getChars()
      Returns the characters that compose this token.
      Returns:
      The unmodifiable list containing the characters that compose this token.

    • getType

      TokenType getType()
      Returns the type of this token.
      Returns:
      The type of this token.

    • getChildren

      List<Token> getChildren()
      Returns the child tokens.

      This method returns an empty list if this token has no child tokens.

      Returns:
      The unmodifiable list containing the child tokens.

    • withType

      Token withType(TokenType newType)
      Returns a new token that has the same content of this token but has the specified token type.
      Parameters:
      newType - The token type of the new token.
      Returns:
      The new token.

    • withChildren

      Token withChildren(Collection<Token> newChildren)
      Returns a new token that has the same content of this token but has the specified child tokens.
      Parameters:
      newChildren - The child tokens of the new token.
      Returns:
      The new token.

    • toString

      String toString()
      Returns a new string representation of this token that is easy for a person to read.

      This method returns a string equal to the value of:

              "[value=" + getValue() + ", span=" + getSpan() + ", "
            + "chars=" + getChars() + ", type=" + getType() + ", "
            + "children=" + getChildren() + "]";
      Overrides:
      toString in class Object
      Returns:
      The new string representation of this token.