If eolFlag is true, the end-of-line characters are returned as tokens. If eolFlag is false, the end-
of-line characters are ignored.
The wordChars( ) method is used to specify the range of characters that can be used in
words. Its general form is shown here:
void wordChars(int start, int end)
Here, start and end specify the range of valid characters. In this program, characters in the
range 33 to 255 are valid word characters.
The whitespace characters are specified using whitespaceChars( ). It has this general form:
void whitespaceChars(int start, int end)
Here, start and end specify the range of valid whitespace characters.
The next token is obtained from the input stream by calling nextToken( ). It returns the
type of token.
StreamTokenizer defines four int constants: TT_EOF, TT_EOL, TT_NUMBER, and
TT_WORD. There are three instance variables. nval is a public double used to hold the
values of numbers as they are recognized. sval is a public String used to hold the value of
any words as they are recognized. ttype is a public int indicating the type of token that has
just been read by the nextToken( ) method. If the token is a word, ttype equals TT_WORD.
If the token is a number, ttype equals TT_NUMBER. If the token is a single character, ttype
contains its value. If an end-of-line condition has been encountered, ttype equals TT_EOL.
(This assumes that eolIsSignificant( ) was invoked with a true argument.) If the end of the
stream has been encountered, ttype equals TT_EOF.
The word count program revised to use a StreamTokenizer is shown here:
// Enhanced word count program that uses a StreamTokenizer
import java.io.*;
class WordCount
{
public static
int words=0;
public static
int lines=0;
public static
int chars=0;
public static void wc(Reader r) throws IOException {
StreamTokenizer tok = new StreamTokenizer(r);
tok.resetSyntax();
tok.wordChars(33, 255);
tok.whitespaceChars(0, ' ');
tok.eolIsSignificant(true);
while (tok.nextToken() != tok.TT_EOF) {
switch (tok.ttype) {
case StreamTokenizer.TT_EOL:
lines++;
chars++;
break;
case StreamTokenizer.TT_WORD:
words++;
Search WWH :
