Database Reference
In-Depth Information
about the number of visitors as it is the various products they browse and
the correlations between them.
To analyze this data, a number of specialized log-processing tools were
introduced and marketed. With the rise of Big Data and tools like Hadoop,
much of the web analytics infrastructure shifted to these large batch-based
systems. They were used to implement recommendation systems and other
analysis. It also became clear that it was possible to conduct experiments on
the structure of websites to see how they affected various metrics of interest.
This is called A/B testing because—in the same way an optometrist tries to
determine the best prescription—two choices are pitted against each other
to determine which is best. These tests were mostly conducted sequentially,
but this has a number of problems, not the least of which is the amount of
time needed to conduct the study.
As more and more organizations mined their website data, the need to
reduce the time in the feedback loop and to collect data on a more continual
basis became more important. Using the tools of the system-monitoring
community, it became possible to also collect this data in real time and
perform things like A/B tests in parallel rather than in sequence. As the
number of dimensions being measured and the need for appropriate
auditing (due to the metrics being used for billing) increased, the analytics
community developed much of the streaming infrastructure found in this
topic to safely move data from their web servers spread around the world to
processing and billing systems.
This sort of data still accounts for a vast source of information from a variety
of sources, although it is usually contained within an organization rather
than made publicly available. Applications range from simple aggregation
for billing to the real-time optimization of product recommendations based
on current browsing history (or viewing history, in the case of a company
like Netflix).
Online Advertising
A major user and generator of real-time data is online advertising. The
original forms of online advertising were similar to their print counterparts
with “buys” set up months in advance. With the rise of the advertising
exchange and real-time bidding infrastructure, the advertising market has
become much more fluid for a large and growing portion of traffic.
