Database Reference
In-Depth Information
Real-time insights and actions on time-series data
Taste-Graph-based real-time recommendation system
For
the
other
use
cases,
check
out
Use Case: Real-Time Insights and Immediate Actions
We needed the capability to turn the enormous volumes of data that the site gen-
erates into useful insights. These insights had to be in real time, as multiple other
systems needed to act in real time based on this information. The system must be
able to handle terabytes of new data every day and hundreds of billions of writes.
Other basic quality requirements, such as availability, scalability, and multi-data-
center support, also needed to be met. This data is also of time-series nature, and
we should be able to support efficient temporal queries on it. Cassandra was an
obvious fit for this use case. However, we don't yet use this system for deep ana-
lytics. For deep analytics, machine learning, and offline reporting, we move data
into our data warehouse environment based on Teradata, Hadoop, MicroStrategy,
and many other business intelligence tools.
System Overview
As shown in
Figure 12.1
, raw data from the business event stream, including
data on checkout, payment, shipping, and refunds, flows into multiple Cassandra
clusters, where it is stored for several months or even years. From there, it feeds
the fraud prevention platform, affiliate pricing engine, order tracking, real-time
reporting, and other systems. We deploy multiple techniques, such as distributed
counters, complex event processing, in-memory aggregations, and even combina-
tions of these techniques, to do real-time computations. The common pattern used
in all techniques is data stored in pre-aggregated form based on the target use case.
