Database Reference
In-Depth Information
Benchmarking a Hadoop Cluster
Is the cluster set up correctly? The best way to answer this question is empirically: run
some jobs and confirm that you get the expected results. Benchmarks make good tests be-
cause you also get numbers that you can compare with other clusters as a sanity check on
whether your new cluster is performing roughly as expected. And you can tune a cluster us-
ing benchmark results to squeeze the best performance out of it. This is often done with
monitoring systems in place (see
Monitoring
)
, so you can see how resources are being used
across the cluster.
To get the best results, you should run benchmarks on a cluster that is not being used by
others. In practice, this will be just before it is put into service and users start relying on it.
Once users have scheduled periodic jobs on a cluster, it is generally impossible to find a
time when the cluster is not being used (unless you arrange downtime with users), so you
should run benchmarks to your satisfaction before this happens.
Experience has shown that most hardware failures for new systems are hard drive failures.
By running I/O-intensive benchmarks — such as the ones described next — you can “burn
in” the cluster before it goes live.
Hadoop Benchmarks
Hadoop comes with several benchmarks that you can run very easily with minimal setup
cost. Benchmarks are packaged in the tests JAR file, and you can get a list of them, with
descriptions, by invoking the JAR file with no arguments:
%
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/
hadoop-mapreduce-*-tests.jar
Most of the benchmarks show usage instructions when invoked with no arguments. For ex-
ample:
%
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/
hadoop-mapreduce-*-tests.jar \
TestDFSIO
TestDFSIO.1.7
Missing arguments.
Usage: TestDFSIO [genericOptions] -read [-random | -backward |
-skip [-skipSize Size]] | -write | -append | -clean [-compression
codecClassName]
[-nrFiles N] [-size Size[B|KB|MB|GB|TB]] [-resFile resultFileName]
[-bufferSize Bytes] [-rootDir]
