Databases Reference
In-Depth Information
If we do not want to use in-memory data grid and still achieve real-time analysis,
there is an alternative approach as well. In a typical Hadoop implementation, you will
notice map-reduce jobs are executed in a scheduled manner to run against the data
stored in HDFS. HFlame enhances Hadoop core with real-time streaming analysis
capability. In traditional Hadoop, a map-reduce job processes only the current snapshot
of available data and ends right after it finished processing the snapshot. Processing
of any new contents requires scheduling of another map-reduce job. With HFlame
enhanced Hadoop, map-reduce jobs can optionally be configured to run in continuous
mode. Which essentially means that map-reduce job doesn't end even if there are no
more new contents available. As soon as new data is pushed in HDFS, continuously
running map-reduce jobs are notified, which immediately passes the new contents
through map-reduce process and extract insights.
HFlame supports following behavior:
1.
HFlame runs on top of customer's Hadoop installation.
HFlame is an incremental add on to existing Hadoop clusters.
2.
No new API. Completely driven by configuration.
3.
HFlame's real time map-reduce jobs are completely fault
tolerant. In the event of any failure, failed components are
automatically scheduled on other available Hadoop nodes.
4.
HFlame guarantees no data loss. If any component of
map-reduce job or Hadoop infrastructure fails in the middle,
automatic job/component's recovery procedure will take of
care starting the data processing from exactly the same place
where it failed.
5.
Allows building a complex mesh of real time map-reduce
jobs to support data analysis requirements that cannot be
described in single map-reduce process.
6.
Supports data analysis frameworks like PIG, HIVE.
7.
Real time map-reduce jobs can optionally be run in batch
mode, i.e., reduce tasks, accumulate data for a certain amount
of time, and then produce the aggregated results.
Figure
8-6
explains the flow of real-time map-reduce job.
