Finding Parts of Text - Natural Language Processing with Java - page 55

Using the StreamTokenizer class

The StreamTokenizer class, found in the java.io package, is designed to tokenize
an input stream. It is an older class and is not as flexible as the StringTokenizer class
discussed in the next section. An instance of the class is normally created based on a file
and will tokenize the text found in the file. It can be constructed using a string.

The class uses a nextToken method to return the next token in the stream. The token re-
turned is an integer. The value of the integer reflects the type of token returned. Based on
the token type, the token can be handled in different ways.

The StreamTokenizer class fields are shown in the following table:

Field Data Type Meaning

nval double

Contains a number if the current token is a number

sval String

Contains the token if the current token is a word token

TT_EOF static int A constant for the end of the stream

TT_EOL static int A constant for the end of the line

TT_NUMBER static int The number of tokens read

TT_WORD static int A constant indicating a word token

ttype int

The type of token read

In this example, a tokenizer is created followed by the declaration of the isEOF variable,
which is used to terminate the loop. The nextToken method returns the token type.
Based on the token type, numeric and string tokens are displayed:

try {
StreamTokenizer tokenizer = new StreamTokenizer(
newStringReader("Let's pause, and then reflect."));
boolean isEOF = false;
while (!isEOF) {