Database Reference
In-Depth Information
Monitoring Applications
The BigInsights Web Console provides context-sensitive views of the cluster,
so that people only see what they need to based on their associated role. If
someone with the security role of “user” runs an application, they only see
their own statistics in the Application History view. This particular view is
optimized to show a high level of application status information, hiding the
lower-level workflow and job information.
People with administrator roles can see the state of all applications in the
Application History view and the Application Status monitoring dashboard,
and are able to drill into individual workflows and jobs for debugging or
performance testing reasons. The Application Status pane has views for
Workflows and Jobs. Each of those views lists every active and completed
workflow and job. You can drill into each workflow or job to get further
details, and from there also see related elements.
Compression
When dealing with the large volumes of data that are expected in a Hadoop
setting, the idea of compression is appealing. On the one hand, you can save
a great deal of space (especially when you consider that every storage block
is replicated three times by default in Hadoop); on the other hand, data trans-
fer speeds are improved because of lower data volumes on the wire. You
should consider two important items before choosing a compression scheme:
splittable compression
and the
compression and decompression speeds
of the com-
pression algorithm that you're using.
Splittable Compression
In Hadoop, files are split (divided) if they are larger than the cluster's block
size setting (normally one split for each block). For uncompressed files, this
means that individual file splits can be processed in parallel by different
mappers. Figure 5-8 shows an uncompressed file with the vertical lines
Big data
represents
utilization,
and IBM
is uniquely
positioned
to help clients
design,
develop and
execute
a big data
strategy.
a new era in data
exploration and
Figure 5-8
An uncompressed splittable ile in Hadoop
