Database Reference
In-Depth Information
loss (it can be very annoying to accidentally overwrite the output of a long job with that of
another).
Next, we specify the map and reduce types to use via the
setMapperClass()
and
setReducerClass()
methods.
The
setOutputKeyClass()
and
setOutputValueClass()
methods control the
output types for the reduce function, and must match what the
Reduce
class produces.
The map output types default to the same types, so they do not need to be set if the map-
per produces the same types as the reducer (as it does in our case). However, if they are
different, the map output types must be set using the
setMapOutputKeyClass()
and
setMapOutputValueClass()
methods.
The input types are controlled via the input format, which we have not explicitly set be-
cause we are using the default
TextInputFormat
.
After setting the classes that define the map and reduce functions, we are ready to run the
job. The
waitForCompletion()
method on
Job
submits the job and waits for it to
finish. The single argument to the method is a flag indicating whether verbose output is
generated. When
true
, the job writes information about its progress to the console.
The return value of the
waitForCompletion()
method is a Boolean indicating suc-
cess (
true
) or failure (
false
), which we translate into the program's exit code of
0
or
1
.
NOTE
The Java MapReduce API used in this section, and throughout the topic, is called the “new API”; it re-
places the older, functionally equivalent API. The differences between the two APIs are explained in
Ap-
pendix D
, along with tips on how to convert between the two APIs. You can also find the old API equi-
valent of the maximum temperature application there.
A test run
After writing a MapReduce job, it's normal to try it out on a small dataset to flush out any
immediate problems with the code. First, install Hadoop in standalone mode (there are in-
structions for how to do this in
Appendix A
)
. This is the mode in which Hadoop runs us-
ing the local filesystem with a local job runner. Then, install and compile the examples us-
ing the instructions on the topic's website.
Let's test it on the five-line sample discussed earlier (the output has been slightly re-
formatted to fit the page, and some lines have been removed):
