Arrays and Strings - Beginning Java 2 SDK - page 151

The second phase extracts the substrings in sequence from the beginning of the string, and stores them
in an array of String variables that has count elements. Following each substring from the first to the
penultimate is a separator, so we use the version of the substring() method that accepts two index
arguments for these. The last substring is signaled by a failure to find the separator character when
index will be -1. In this case we use the substring() method with a single argument to extract the
substring through to the end of the string text .
The third phase simply outputs the contents of the array by displaying each element in turn, using
a for loop.
What we have been doing here is breaking a string up into tokens - substrings in other words - that are
separated by delimiters - characters that separate one token from the next. This is a sufficiently frequent
requirement that Java provides you with an easier way to do this - using the StringTokenizer class.
Using a String Tokenizer
We can use an object of the StringTokenizer class to do what we did in the previous example. You
can construct a StringTokenizer that can process a given string like this:
String text = "To be or not to be"; // String to be segmented
StringTokenizer st = new StringTokenizer(text); // Create a tokenizer for it
The tokenizer object st that we have created here will assume that a delimiter can be a space, a tab, a
newline character, a carriage return, or a form-feed character. It is also possible to specify your own set
of delimiters when you create the tokenizer object. For example, if we only wanted a comma or a space
to be considered as a delimiter we could create the tokenizer with the statement:
StringTokenizer st = new StringTokenizer(text, " ,"); // Tokenize using , or space
The second argument is a string containing all the characters that are to be considered as delimiters in
the string text .
First of all, you can call the countTokens() method for the StringTokenizer object to determine
how many tokens the string contains. This is handy when you want to store the tokens away in an array
as it gives you the means to create the array ahead of time, like this:
String[] subStr = new String[st.countTokens()];
The countTokens() method returns an int value that is the number of tokens in the string -
assuming you haven't extracted any in the way we will see next. If you have extracted tokens, the value
returned will be the number remaining in the string. Now we have an array that is just large enough to
accommodate all the tokens in the string text . All we have to do is extract them.
You can use the StringTokenizer object to pass once through the string to extract each of the tokens
in turn. Calling the nextToken() method for the StringTokenizer object will return a reference to
a String object that is the next token in the string being processed. We could therefore extract all the
tokens like this: