Caitlin Smallwood - Data Scientists at Work - page 29

It's fortunate that I've worked in a lot of different industries and have a lot
of years of experience under my belt because I kind of know how much
resource it takes to do what I envision we will need to do in all the different
areas. And it's always a moving target. It never goes backwards. I'm never
worried about overhiring.
Gutierrez: What's a specific project that you have worked on recently?
Smallwood: I'll talk about a project that pertains to both experimentation
and predictive modeling. Experimentation is a core part of how Netflix evolves
our product. We really believe in this approach toward our product innovation,
and so it's important that we are firm on how we measure those experiments,
because there are many of them and they're precious to everyone.
One of the key metrics that we leverage for measuring our experiments is
the number of viewing hours. How many hours did a particular customer
spend watching Netflix over, let's say, a 28-day period, because that's really the
key measure of engagement. If you're spending time on Netflix, it's extremely
highly correlated with retention. Though retention is our core metric, I'm
going to talk about hours in relation to this project.
For a long time we just measured total hours and percentage of customers
that stream more than n hours at a couple of different thresholds. It's a really
good way to think about hours because you get a sense of the distribution.
However, it's much more impactful if we get people who are streaming only
3 hours a month to stream 4 hours a month than it is if we get people who
are already streaming 30 hours a month to stream 31 hours a month. We
really thought about this and we weren't happy with treating all customers
equally when we were assessing hours.
We wanted a way to compare statistically how algorithm A versus algorithm
B is impacting the lighter streamers, but we didn't want to do it at just one
threshold. We wanted to do it at all the thresholds. To solve this, we essen-
tially built a predictive model that has to do with the relationship between
hours and retention at all parts of the distribution.
We came up with a metric that we call the “streaming score.” The streaming
score is related to both the number of hours that a customer streamed and
how long they've been with Netflix, as well as the relationship with retention.
The streaming score also takes into account viewing region, because people
in the US are very different than people in Latin America in terms of how
much TV they watch. So we built this model and now we have this metric,
the streaming score, with test statistics around it that let us get a much more
granular, impactful, and essential view of the streaming.
Gutierrez: How was it built?
Smallwood: First, we did a lot of research on different ways we could model
it. It's a little bit complicated because customers get billed once a month,
whereas the streaming is happening daily. So you have this marker on the day