Database Reference
In-Depth Information
which is determining if people stay month after month. When we increase
customer retention in an A/B test, then you get to remove all judgment and
people's egos and everything else. The data speaks for itself. That's the easiest
measure of success. We're lucky that a lot of the algorithms that we work on
are amenable to tests. So we'll build version A and set it up versus version B
of, say, our ranking algorithm, and then we'll just test it to find out if it is bet-
ter or not better in the eyes of our customer. It's awesome that we have that
opportunity.
The harder situation is when our models are used as input to human deci-
sions. A great example of this is when we try to predict customer demand
for titles we don't have yet. The information the predictive models generate
is used for decision making in our Beverly Hills office, where they're negotiat-
ing with studios about what content to acquire. For instance, if they have the
choice of five different titles, which one should they pick for Netflix given
limited amounts of money?
That's a very hard problem, as it's a particularly hard space to model because
it's so dependent on what other titles we have in our catalog and many fac-
tors about our member base and their interests in different kinds of content.
In this situation, it's harder to measure the success of a model, because you
never know at the point when you're making the decision whether the model
is right or wrong. The content acquiring model has been a very successful
model, but that doesn't mean it's accurate for every single content decision.
We monitor how successful this type of model is in aggregate over time.
The way we measure success for this type of model is more about how the
company internally views the usefulness of the model. We monitor things like:
Are they using it and do they mostly trust it? Do they ask for feature enhance-
ments? Do they want more out of it? Do they come to our team looking for
more modeling to solve other problems? So it's a softer measure of success
for that kind of modeling.
Gutierrez:
Is there a number of models you need to create per year?
Smallwood:
This goes back to the previous point of low process. There
isn't a requirement on my team to push out a certain number of models per
year. We have very few, if any, requirements. What we do have are responsi-
bilities. For me personally, it's my responsibility to figure out how to evolve
the team and our work to keep pace with all the innovation going on across
the company.
For example, I see that with the movement toward more originals at Netflix, a
different set of content decisions are going to need to be made. From this I know
that my team needs to beef up our talent and level of resources in that area, so
that we can help them make some really difficult decisions. I need to understand
what the company strategy is in all the different parts of the business.
