Lexical Analysis - Introduction to Compiler Construction in a Java World - page 35

FIGURE 2.4 A state transition diagram for recognizing the separator ; and the operators
== , = , ! , and * .
nextCh();
returnnewTokenInfo(STAR,line);
...
}
White Space
Before attempting to recognize the next incoming token, one wants to skip over all white
space. In j--, as in Java, white space is defined as the ASCII SP characters (spaces), HT
(horizontal tabs), FF (form feeds), and line terminators; in j-- (as in Java), we can denote
these characters as `' , `t' , `f' , `b' , `r' , and `n' , respectively. Skipping over white space
is done from the start state, as illustrated in Figure 2.5.
The code for this is simple enough, and comes at the start of a method for reading the
next incoming token:
while(isWhitespace(ch)){
nextCh();
}