Database Reference
In-Depth Information
of moving the data around, while still getting the benefit of spreading
computation over a larger number of computers.
Although HDFS is not directly responsible for performing the computation
work, it does provide an interface that allows applications to place their
computing tasks closer to the data. MapReduce is a prime example of this.
It works hand in hand with HDFS to distribute computational tasks to
DataNodes so that the work can be performed as close to the data as
possible. Only the results of the data actually need to be transmitted across
the network.
NOTE
You can think of moving the computation closer to the data as being
similar to the way that many relational database engines optimize
queries by moving filter criteria as close to the original data as possible.
For example, if you are joining two tables, and then filtering results by a
column from one table, the engine will often move that filter to the
initial retrieval of data from the table, thus reducing the amount of data
it has to join with the second table.
NameNodes and DataNodes
HDFS uses two primary types of nodes. A NameNode acts as a master node,
managingthefilesystemandaccesstofilesfromclients.DataNodesmanage
the physical storage of data.
Each HDFS cluster has a single NameNode. The NameNode acts as the
coordinator for the cluster and is responsible for managing and
communicating to all the DataNodes in a cluster. It manages all the
metadata associated with the cluster. One of its primary responsibilities is
to manage the file system namespace, which is the layer that presents the
distributed data stored in HDFS as though it is in a traditional file system
organized as folders and files. The NameNode manages any file system
namespace operations, such as creating or deleting files and folders.
Thefilesystemnamespacepresentstheappearance thatthedataisstoredin
a folder/file structure, but the data is actually split into multiple blocks that
are stored onDataNodes. TheNameNode controls whichblocksofdata map
