MapReduce - Hadoop: The Definitive Guide - page 82

The DistributedFileSystem returns an FSDataInputStream (an input stream
that supports file seeks) to the client for it to read data from. FSDataInputStream in
turn wraps a DFSInputStream , which manages the datanode and namenode I/O.
The client then calls read() on the stream (step 3). DFSInputStream , which has
stored the datanode addresses for the first few blocks in the file, then connects to the first
(closest) datanode for the first block in the file. Data is streamed from the datanode back
to the client, which calls read() repeatedly on the stream (step 4). When the end of the
block is reached, DFSInputStream will close the connection to the datanode, then find
the best datanode for the next block (step 5). This happens transparently to the client,
which from its point of view is just reading a continuous stream.
Blocks are read in order, with the DFSInputStream opening new connections to datan-
odes as the client reads through the stream. It will also call the namenode to retrieve the
datanode locations for the next batch of blocks as needed. When the client has finished
reading, it calls close() on the FSDataInputStream (step 6).
During reading, if the DFSInputStream encounters an error while communicating
with a datanode, it will try the next closest one for that block. It will also remember datan-
odes that have failed so that it doesn't needlessly retry them for later blocks. The DFSIn-
putStream also verifies checksums for the data transferred to it from the datanode. If a
corrupted block is found, the DFSInputStream attempts to read a replica of the block
from another datanode; it also reports the corrupted block to the namenode.
One important aspect of this design is that the client contacts datanodes directly to retrieve
data and is guided by the namenode to the best datanode for each block. This design al-
lows HDFS to scale to a large number of concurrent clients because the data traffic is
spread across all the datanodes in the cluster. Meanwhile, the namenode merely has to ser-
vice block location requests (which it stores in memory, making them very efficient) and
does not, for example, serve data, which would quickly become a bottleneck as the num-
ber of clients grew.