Module com.maroontress.clione
Package com.maroontress.clione

Class Tokens

java.lang.Object
com.maroontress.clione.Tokens
public final class Tokens extends Object
The utility class for operations on token sequence.

  • Method Details

    • stringize

      public static Token stringize(List<Token> tokens, SourceLocation where)
      Converts the given tokens into a string literal token.

      This method implements the behavior of the C preprocessor stringizing operation with the following precise rules:

      • Tokens whose type is TokenType.DELIMITER or TokenType.COMMENT are treated as whitespace for the purpose of stringizing.
      • All leading and trailing whitespace tokens are removed.
      • Any sequence of one or more whitespace tokens between non-whitespace tokens is collapsed to a single ASCII space (U+0020).
      • Whitespace that belongs to embedded string-literal tokens (i.e., tokens that are not whitespace) is preserved and is not collapsed.

      After the above trimming/collapsing, the remaining token texts are concatenated in token order. The concatenated text is then escaped: every backslash ('\') and double quote ('"') in the text is prefixed with a backslash, and the whole result is enclosed in double quotes to produce a C-style string literal.

      Newly created characters of the produced string literal token is assigned the provided SourceLocation (the same location is used for the characters).

      Parameters:
      tokens - The immutable list of tokens to be stringized.
      where - The source location to assign to newly created characters of the resulting string literal token.
      Returns:
      A new Token of type TokenType.STRING that represents the stringized form of the input tokens.

    • concatenate

      public static Token concatenate(Token left, Token right, Set<String> reservedWords)
      Concatenates the two tokens and returns a newly created token.

      The characters of left and right are appended in that order to form a single token text. The resulting token's type is determined as follows:

      1. If the concatenated token text is equal to one of the strings in reservedWords, the resulting token type is TokenType.RESERVED.
      2. Otherwise, the method attempts to parse the concatenated text as a single token. If parsing succeeds and yields exactly one token, the resulting token type is set to that parsed type.
      3. If parsing produces more than one token, or parsing fails, the resulting token type is TokenType.UNKNOWN.
      Parameters:
      left - The left token.
      right - The right token.
      reservedWords - A set of the reserved words.
      Returns:
      The new token.