The I/O Package - The Java Programming Language - page 788

Resets the syntax table so that all characters are ordinary. If
you do this and then start reading the stream, nextToken al-
ways returns the next character in the stream, just as when
you invoke InputStream.read .
There are no methods to ask the character class of a given character
or to add new classes of characters. Here are the default settings for a
newly created StreamTokenizer object:
wordChars('a', 'z'); // lower case ASCII letters
wordChars('A', 'Z'); // upper case ASCII letters
wordChars(128 + 32, 255); // "high" non-ASCII values
whitespaceChars(0, ' '); // ASCII control codes
commentChar('/');
quoteChar('"');
quoteChar('\'');
parseNumbers();
This leaves the ordinary characters consisting of most of the punctuation
and arithmetic characters ( ; , : , [ , { , + , = , and so forth).
The changes made to the character classes are cumulative, so, for ex-
ample, invoking wordChars with two different ranges of characters defines
both ranges as word characters. To replace a range you must first mark
the old range as ordinary and then add the new range. Resetting the
syntax table clears all settings, so if you want to return to the default
settings, for example, you must manually make the invocations listed
above.
Other methods control the basic behavior of the tokenizer:
public void eolIsSignificant(boolean flag)
If flag is true , ends of lines are significant and TT_EOL may be
returned by nextToken . If false , ends of lines are treated as
whitespace and TT_EOL is never returned. The default is false .