Lexical Analysis - Introduction to Compiler Construction in a Java World - page 29

Chapter2
LexicalAnalysis
2.1 Introduction
The first step in compiling a program is to break it into tokens. For example, given the j--
program
packagepass;
importjava.lang.System;
publicclassFactorial{
//Twomethodsandafield
publicstaticintfactorial(intn){
if(n<=0)
return1;
else
returnn*factorial(n-1);
}
publicstaticvoidmain(String[]args){
intx=n;
System.out.println(x+"!="+factorial(x));
}
staticintn=5;
}
we want to produce the sequence of tokens package , pass , ; , import , java , . , lang , . ,
System , ; , public , class , Factorial , { , public , static , int , factorial , ( , int , n,) , { ,
if , ( , n , <= , 0 , ) , } , return , 1 , ; , else , return , n , * , factorial , ( , n , - , 1 , ) , } , ; , } ,
public , static , void , main , ( , String , [ , ] , args , ) , } , { , int , x , = , n , ; , System , . , out ,
. , println , ( , x , + , "!=" , + , factorial , ( , x , ) , ) , } , ; , } , static , int , n , = , 5 , ; , and } .
Notice how we have broken down the program's text into its component elements. We
call these the lexical tokens (or lexical elements) which are described by the language's
lexical syntax. Notice also how the comment has been ignored in the sequence of tokens;
this is because comments have no meaning to the compiler or to the results of executing
the program. We call this process of recognizing tokens lexical analysis, or more simply
scanning.
The lexical syntax of a programming language is usually described separately. For ex-
ample, in j-- we describe identiers as \a letter, underscore, or dollar-sign, followed by zero
or more letters, digits, underscores or dollar-signs." We can also describe these tokens more
formally, using what are called regular expressions. We visit regular expressions later in
Section 2.3. For now, let us be a little less formal.
29