Database Reference
In-Depth Information
of points in each blob. The total number of bytes may also be decreased if you compress the
data being inserted. If you can arrange to have 1,000 data points or more per blob, ingest
rates can be very high. As mentioned in
Chapter 3
, in one test with one data point per second
and one-hour time windows, ingestion into a 4-node storage tier in a 10-node MapR cluster
exceeded 100 million data points per second. This rate is more than 1,000 times faster than
the system was able to ingest data without direct blob insertion.
To accomplish this high-performance style of data insertion with live data arriving at high
velocity as opposed to historical data, it is necessary to augment the native Open TSDB with
capabilities such as those provided by the open source extensions developed by MapR and
described in more detail in the following section.
Figure 4-2
gives us a look inside the modi-
fied time series daemon (TSD) as modified for direct blob insertion. These open source
modifications will work on databases built with Apache HBase or with MapR-DB.
Figure 4-2. Changes inside the TSD when using extensions to Open TSDB that enable high-speed
ingestion of rapid streaming data. Data is ingested initially to the storage tier in the blob-oriented
format that stores many data points per row.
