Module com.maroontress.clione
Package com.maroontress.clione

Class Tokens

java.lang.Object
com.maroontress.clione.Tokens
public final class Tokens extends Object
The utility class for operations on token sequence.

  • Method Details

    • reparseLineDigitsAndFilename

      public static List<Token> reparseLineDigitsAndFilename(Collection<Token> tokens, String filename, Set<String> reservedWords)
      Reparses the given tokens as a line directive.
      Parameters:
      tokens - The tokens.
      filename - The filename.
      reservedWords - The set of the reserved words.
      Returns:
      The list of the new tokens.

    • reparseIncludeFilename

      public static List<Token> reparseIncludeFilename(Collection<Token> tokens, String filename, Set<String> reservedWords)
      Reparses the given tokens as an include directive.
      Parameters:
      tokens - The tokens.
      filename - The filename.
      reservedWords - The set of the reserved words.
      Returns:
      The list of the new tokens.

    • stringize

      public static Token stringize(List<Token> tokens, SourceLocation where)
      Converts the given tokens into a string literal token.

      This method implements the behavior of the C preprocessor stringizing operation with the following precise rules:

      • Tokens whose type is TokenType.DELIMITER or TokenType.COMMENT are treated as whitespace for the purpose of stringizing.
      • All leading and trailing whitespace tokens are removed.
      • Any sequence of one or more whitespace tokens between non-whitespace tokens is collapsed to a single ASCII space (U+0020).
      • Whitespace that belongs to embedded string-literal tokens (i.e., tokens that are not whitespace) is preserved and is not collapsed.

      After the above trimming/collapsing, the remaining token texts are concatenated in token order. The concatenated text is then escaped: every backslash ('\') and double quote ('"') in the text is prefixed with a backslash, and the whole result is enclosed in double quotes to produce a C-style string literal.

      Newly created characters of the produced string literal token is assigned the provided SourceLocation (the same location is used for the characters).

      Parameters:
      tokens - The immutable list of tokens to be stringized.
      where - The source location to assign to newly created characters of the resulting string literal token.
      Returns:
      A new Token of type TokenType.STRING that represents the stringized form of the input tokens.

    • concatenate

      public static Token concatenate(Token left, Token right, Set<String> reservedWords)
      Concatenates the two tokens and returns a newly created token.

      The characters of left and right are appended in that order to form a single token text. The resulting token's type is determined as follows:

      1. If the concatenated token text is equal to one of the strings in reservedWords, the resulting token type is TokenType.RESERVED.
      2. Otherwise, the method attempts to parse the concatenated text as a single token. If parsing succeeds and yields exactly one token, the resulting token type is set to that parsed type.
      3. If parsing produces more than one token, or parsing fails, the resulting token type is TokenType.UNKNOWN.
      Parameters:
      left - The left token.
      right - The right token.
      reservedWords - A set of the reserved words.
      Returns:
      The new token.

    • isDelimiterOrComment

      public static boolean isDelimiterOrComment(Token token)
      Checks whether the given token is a delimiter or a comment.
      Parameters:
      token - The token to be checked.
      Returns:
      true if the token is a delimiter or a comment, otherwise false.

    • isKeywordOrIdentifier

      public static boolean isKeywordOrIdentifier(Token token)
      Checks whether the given token is a keyword or an identifier.
      Parameters:
      token - The token to be checked.
      Returns:
      true if the token is a keyword or an identifier, otherwise false.

    • normalizeToken

      public static Token normalizeToken(Token token, Set<String> reservedWords)
      Normalizes the given token.

      If the token is an identifier and its value is in the given set of the reserved words, this method returns a new token that is the same as the given token except that its type is TokenType.RESERVED.

      Otherwise, this method returns the given token with no modifications.

      Parameters:
      token - The token.
      reservedWords - The set of the reserved words.
      Returns:
      The new token.