Understanding Streams - Beginning Java - page 391

As I've said, the int value that is returned by the nextToken() method indicates what kind of data item
was read. It can be any of the constant static variables shown in Table 8-7 defined in the StreamTokenizer
class:

TABLE 8-7 : Constant Static Variables
TOKEN
VALUE

DESCRIPTION

TT_NUMBER The token is a number that has been stored in the public field nval of type double in the tokenizer object.
TT_WORD The token is a word that has been stored in the public field sval of type String in the tokenizer object.
TT_EOF

The end of the stream has been reached.

An end-of-line character has been read. This is set only if the eolIsSignificant() method has been
called with the argument, true . Otherwise, end-of-line characters are treated as whitespace and ignored.

TT_EOL

If a quoted string is read from the stream, the value that is returned by nextToken() is the quote character
used for the string as type int — either a single quote or a double quote. In this case, you retrieve the ref-
erence to the string that was read from the sval member of the tokenizer object. The value indicating what
kind of token was read last is also available from a public data member ttype , of the StreamTokenizer
object, which is of type int .

Customizing a Stream Tokenizer
You can modify default tokenizing mode by calling one or other of the methods found in Table 8-8 .

TABLE 8-8 : Methods that Modify Tokenizing Modes
METHOD DESCRIPTION

Resets the state of the tokenizer object so no characters have any special significance. This has
the effect that all characters are regarded as ordinary and are read from the stream as single
characters so no tokens are identified. The value of each character is stored in the ttype field.

resetSyntax()

Sets the character ch as an ordinary character. An ordinary character is a character that has no
special significance. It is read as a single character whose value is stored in the ttype field.
Calling this method does not alter the state of characters other than the argument value.

ordinaryChar(
int ch)

Causes all characters from low to hi inclusive to be treated as ordinary characters. Calling this
method does not alter the state of characters other than those specified by the argument values.

ordinaryChars(
int low,
int hi)

Causes all characters from low to hi inclusive to be treated as whitespace characters. Unless
they appear in a string, whitespace characters are treated as delimiters between tokens. Calling
this method does not alter the state of characters other than those specified by the argument
values.

whitespaceChars(
int low,
int hi)

Specifies that the characters from low to hi inclusive are word characters. A word is at least
one of these characters. Calling this method does not alter the state of characters other than
those specified by the argument values.

wordChars(
int low,
int hi)

Specifies that ch is a character that indicates the start of a comment. All characters to the end
of the line following the character ch are ignored. Calling this method does not alter the state
of characters other than the argument value.

commentChar(
int ch)

Specifies that matching pairs of the character ch enclose a string. Calling this method does not
alter the state of characters other than the argument value.

quoteChar(
int ch)

If the argument is false , this switches off recognizing comments between /* and */ . A true
argument switches it on again.

slashStarComments(
boolean flag)