Database Reference
In-Depth Information
rules. Essentially, when it comes to big data, this governance team owns the
roadmap for your IT organization to get it from its current state to some
preferable future state. The need for the governance team is that there is a
lot of change occurring around data in general and big data in particular.
Having a team focused on where your organization needs to go allows the
operations team to focus on the current state and implementing the rules
established by the governance team.
The role of the operations team is to implement the above. The operations
team is responsible for auditing and logging the information determined by
thegovernanceteamasbeingimportant.Theoperationsteamisresponsible
for security and privacy of the data as outlined by the governance team.
The operations team is responsible for implementing the quotas determined
for users and providing access to approved users as determined by the
governance team. Ideally, you want different people on the operations team
and on the governance team.
Creating Operational Analytics
Hadoop is a complex distributed system, and monitoring it is not a trivial
task.AvarietyofinformationsourceswithinHadoopprovideformonitoring
and debugging the various services. To create an operational analytics
solution, you must collect these monitors and store them for correlation and
trending analysis.
A solution included with HDP is the HDP monitoring dashboard. This
solution uses two monitoring systems, called Ganglia and Nagios, to
combine certain metrics provided by Hadoop into graphs and alerts that
are more easily understood by administrators and managers alike. Using
the HDP monitoring dashboard, you can communicate the state of various
cluster services and also diagnose common problems.
When it comes to monitoring Hadoop, in many respects it differs little
from monitoring a database management system. I like to refer to several
system resources as the canaries in the coal mine. These resources—CPU,
disk, network, and memory—are vital to the performance of any system
retrieving, analyzing, and moving data. The utilization of these resources
both as a point-in-time resource, and viewing them as a trend over time
provides valuable insight into the state and health of a system.
