Databases Reference
In-Depth Information
From “Overlapping Experiment Infrastructure: More,
Better, Faster Experimentation”
The design goals for our experiment infrastructure are therefore:
more, better, faster.
More
We need scalability to run more experiments simultaneously.
However, we also need flexibility: different experiments need
different configurations and different sizes to be able to measure
statistically significant effects. Some experiments only need to
change a subset of traffic, say Japanese traffic only, and need to
be sized appropriately. Other experiments may change all traffic
and produce a large change in metrics, and so can be run on less
traffic.
Better
Invalid experiments should not be allowed run on live traffic.
Valid but bad experiments (e.g., buggy or unintentionally pro‐
ducing really poor results) should be caught quickly and disabled.
Standardized metrics should be easily available for all experi‐
ments so that experiment comparisons are fair: two experiment‐
ers should use the same filters to remove robot traffic when
calculating a metric such as CTR.
Faster
It should be easy and quick to set up an experiment; easy enough
that a non-engineer can do so without writing any code. Metrics
should be available quickly so that experiments can be evaluated
quickly. Simple iterations should be quick to do. Ideally, the sys‐
tem should not just support experiments, but also controlled
ramp-ups, i.e., gradually ramping up a change to all traffic in a
systematic and well-understood way.
That experimental infrastructure has a large team working on it and
analyzing the results of the experiments on a full-time basis, so this is
nontrivial. To make matters more complicated, now that we're in an
age of social networks, we can no longer assume that users are inde‐
pendent (which is part of the randomization assumption underlying
experiments). So, for example, Rachel might be in the treatment group
of an experiment Facebook is running (which is impossible because
Rachel isn't actually on Facebook, but just pretend), which lets Rachel
