The I/O Package - The Java Programming Language - page 784

• TT_WORD : A word was scanned. The String field sval contains the
word that was found.
• TT_NUMBER : A number was scanned. The double field nval contains
the value of the number. Only decimal floating-point numbers
(with or without a decimal point) are recognized. The tokenizer
does not understand 3.4e79 as a floating-point number, nor 0xffff
as a hexadecimal number.
• TT_EOL : An end-of-line was found.
• TT_EOF : The end-of-file was reached.
The input text is assumed to consist of bytes in the range \u0000 to
\u00FF Unicode characters outside this range are not handled correctly.
Input is composed of both special and ordinary characters. Special char-
acters are those that the tokenizer treats speciallynamely whitespace,
characters that make up numbers, characters that make up words, and
so on. Any other character is considered ordinary. When an ordinary
character is the next in the input, its token type is itself. For example,
if the character '¿' is encountered in the input and is not special, the
token return type (and the ttype field) is the int value of the character
'¿' .
As one example, let's look at a method that sums the numeric values in
a character stream it is given:
static double sumStream(Reader source) throws IOException {
StreamTokenizer in = new StreamTokenizer(source);
double result = 0.0;
while (in.nextToken() != StreamTokenizer.TT_EOF) {
if (in.ttype == StreamTokenizer.TT_NUMBER)
result += in.nval;
}
return result;
}