Java Reference
In-Depth Information
int token = tokenizer.nextToken();
switch (token) {
case StreamTokenizer.TT_EOF:
isEOF = true;
break;
case StreamTokenizer.TT_EOL:
break;
case StreamTokenizer.TT_WORD:
System.out.println(tokenizer.sval);
break;
case StreamTokenizer.TT_NUMBER:
System.out.println(tokenizer.nval);
break;
default:
System.out.println((char) token);
}
}
} catch (IOException ex) {
// Handle the exception
}
When executed, we get the following output:
Let
'
This is not what we would normally expect. The problem is that the tokenizer uses apo-
strophes (single quote character) and double quotes to denote quoted text. Since there is
no corresponding match, it consumes the rest of the string.
We can use the ordinaryChar method to specify which characters should be treated as
common characters. The single quote and comma characters are designated as ordinary
characters here:
tokenizer.ordinaryChar('\'');
tokenizer.ordinaryChar(',');
When these statements are added to the previous code and executed, we get the following
output:
Search WWH ::




Custom Search