Java Reference
In-Depth Information
Simple Java tokenizers
There are several Java classes that support simple tokenization; some of them are as fol-
lows:
•
Scanner
•
String
•
BreakIterator
•
StreamTokenizer
•
StringTokenizer
Although these classes provide limited support, it is useful to understand how they can be
used. For some tasks, these classes will suffice. Why use a more difficult to understand and
less efficient approach when a core Java class can do the job? We will cover each of these
classes as they support the tokenization process.
The
StreamTokenizer
and
StringTokenizer
classes should not be used for new
development. Instead, the
String
class'
split
method is usually a better choice. They
have been included here in case you run across them and wonder whether they should be
used or not.
