Database Reference
In-Depth Information
<value>false</value>
</property>
<property>
<name>mapred.reduce.slowstart.completed.maps</name>
<value>0.8</value>
</property>
</configuration>
Let's discuss each property in detail:
•
mapred.job.tracker
: This property defines the host and port on which the
jobtracker runs. All communication with the jobtracker is done over the host and
port.
•
mapred.job.tracker.http.address
: This property defines the web ad-
dress of the jobtracker. This web location helps in the monitoring of jobs submit-
ted to the cluster.
•
mapreduce.job.counters.max
: Internally, Hadoop maintains several
counters, for example,
JobCounter
and
TaskCounter
to count the job and
task-related information during their process. However, it is also possible for de-
velopers to define their own counters. This liberty could cause issues if the num-
ber of counters is not controlled, as the jobtracker maintains these counters glob-
ally. This property helps in limiting the number of counters that can be generated.
•
mapred.output.compress
: This is a Boolean property, and if set to
true
,
it will compress the job's output file.
•
mapred.output.compression.type
: This property defines the type of
compression that can be set. The options are
NONE
,
RECORD
, or
BLOCK
.
•
mapred.output.compression.codec
: This property defines the codec to
be used for compression of the job's output file.
•
mapred.map.output.compression.codec
: This property defines the
codec that should be used to compress the map output files.
•
mapred.compress.map.output
: This property, if set to
true
, can com-
press the map output files before it is sent across the network.
•
io.sort.mb
: This property defines the memory set to perform the in-memory
sorting and is useful when tuning to reduce the number of spilled records.
•
io.sort.factor
: When output data generated from a map task is small
enough to fit into a tasktracker's memory, it is retained there and all operations are
done in-memory. However, if the data is larger than the tasktracker's memory, it is
spilled (written) to the disk. This property defines the number of open file handles
that will be used when sorting files.
