Module com.maroontress.clione
Package com.maroontress.clione.impl

Interface Tokenizer

Functional Interface:
This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
@FunctionalInterface public interface Tokenizer
The function that changes the state of the specified Transcriber object, lets it read characters from its source and store a new token in its builder, and returns the token type of the stored token.
See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    TokenType
    apply(Transcriber x)
    Returns the token type of the token that the specified Transcriber object reads.

  • Method Details

    • apply

      TokenType apply(Transcriber x) throws IOException
      Returns the token type of the token that the specified Transcriber object reads.

      The transcriber reads characters from its source to build a new token. It stores the building token in its TokenBuilder object. So use Transcriber.toToken(TokenType) method to get the new token object as follows:

              Token newToken(Transcriber x, Tokenizer tokenizer) throws IOException {
            var type = tokenizer.apply(x);
            if (type == null) {
                return null;
            }
            return x.toToken(type);
        }

      Note that this function may return null unlike DefaultTokenizer.apply(Transcriber, SourceChar).

      Parameters:
      x - The transcriber.
      Returns:
      null if the transcriber's source has reached EOF. Otherwise, the token type.
      Throws:
      IOException - If an I/O error occurs.