Storage Provisioning and Networking - Deploying and Managing a Cloud Infrastructure - page 272

NameNode Conventions
Some of the most important Hadoop NameNode conventions and concepts are introduced
in the following list:
Communication Protocols All HDFS communication uses the TCP/IP protocol suite.
A client using the Client Protocol connects to a configurable TCP port on the NameNode.
The DataNodes use DataNode Protocol to talk to the NameNode. Both Client Protocol
and DataNode Protocol are wrapped by the Java Remote Procedure Call (Java RPC). The
NameNode does not initiate any RPC calls. The design is such that it can only respond to
RPC requests issued by the clients or DataNodes. This intrinsically makes the DataNodes
a lot more secure if a NameNode is compromised.
Safemode Upon startup, the NameNode enters a special Safemode state. Replication can-
not occur when the NameNode is in the Safemode state. The NameNode receives a heart-
beat and a block report from the DataNodes. A block is deemed safely replicated when the
minimum number of replicas of that data block has checked in with the NameNode. The
NameNode exits Safemode once a configurable percentage of safely replicated data blocks
have checked in with the NameNode. The NameNode then compiles a list of all those data
blocks that have lower than a specified number of replicas. The NameNode then replicates
those data blocks to other DataNodes.
Metadata Persistence NameNode uses a transaction log called EditLog to record every
change that occurs to the HDFS namespace and file system metadata. EditLog stores an
entry to record the creation of a new file, changing the replication factor of a file and other
such operations. The entire file system namespace, mappings of blocks to files, and file
properties are stored in a file called FsImage . Both EditLog and FsImage are stored on the
local file system of the NameNode.
The image of the entire file system namespace and a file Blockmap is kept in memory. The
metadata system is designed to be compact, and 4 GB of main memory is plenty to support
a very large number of directories and files.
How this works is that as soon as the NameNode starts up, it reads the FsImage and EditLog
from the disk. The NameNode applies all the transactions from EditLog to the FsImage copy
kept in memory. It also flushes out this new version, as soon as it is stable, to the disk so that
in case of failure of the node, the latest changes have already been written. The EditLog is
then truncated, which completes the establishment of a checkpoint.
It is important to note here the role of the DataNode, which is oblivious to the knowledge
of HDFS files. Each block of HDFS data is kept in a separate file in its location file system.
By design, the DataNode does not create all the files in the same directory but rather uses a
heuristic to determine an optimal number of files per directory. The DataNode also creates
an appropriate subdirectory structure, which is helpful if there is a large quantity of files.
Recall that the performance of I/O operations directly depends on the number of items
that need to be read for a given command execution. As the DataNode starts up, it sends