Module com.maroontress.clione
Package com.maroontress.clione.impl

Interface DefaultTokenizer

Functional Interface:
This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
@FunctionalInterface public interface DefaultTokenizer
The function that changes the state of the specified Transcriber object with the specified SourceChar object, lets the Transcriber object read characters from its source and store a new token in its builder, and returns the token type of the stored token.
See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    TokenType
    apply(Transcriber x, SourceChar c)
    Returns the token type of the token composed of the specified SourceChar and if needed the characters supplied from the specified Transcriber object.

  • Method Details

    • apply

      TokenType apply(Transcriber x, SourceChar c) throws IOException
      Returns the token type of the token composed of the specified SourceChar and if needed the characters supplied from the specified Transcriber object.

      The transcriber may read characters from its source to build a new token. It stores the building token in its TokenBuilder object. So use Transcriber.toToken(TokenType) method to get the new token object as follows:

              Token newToken(Transcriber x, SourceChar c,
                       DefaultTokenizer otherwise) throws IOException {
            var type = otherwise.apply(x, c);
            return x.toToken(type);
        }

      Note that this function does not return null unlike Tokenizer.apply(Transcriber).

      Parameters:
      x - The transcriber.
      c - The first character that the Transcriber has read from its source but has not yet been stored to its builder.
      Returns:
      The token type.
      Throws:
      IOException - If an I/O error occurs.