Database Reference
In-Depth Information
you decide to cancel your Netflix account. You may stop streaming but you
also have service until the end of your billing cycle, so there's a little bit of a
nebulous relationship there. This relationship made us think about doing a
survival model.
We ended up implementing a custom model for this particular metric. But we
did a bunch of research, and then we built a couple different versions of the
model. We studied both versions of the model for both accuracy [AUC] and
performance on A/B test results. And, again to the point of learning from each
other, as a group we debated the results and decided what we thought was a
good version to model.
Gutierrez: How did you set out the research path?
Smallwood: A lot of the research ideas came from different people on the
team. Across the team there are enough people with different methodolo-
gies that they're familiar with that we have a lot of ideas of how to do things.
Many folks on the team are great at keeping up with research in the industry,
so they'll bring new techniques back to us as a team and share them with
everybody. This means we've got good coverage of new techniques that are
popping up everywhere, both within our team and the research that is hap-
pening elsewhere.
So first we brainstormed about the classes of techniques that would be good
here. The different techniques depend on the characteristics of the data, so
we made sure to really understand all the key dimensions that we cared about
and how the data was distributed. We also worked to really understand which
dimensions were actually important to this problem. As this process evolved,
one person on the team—she's this brilliant statistician—was really leading
the effort, so she started hammering away at the problem.
Gutierrez: How did she take the lead? Was she assigned, or did it just evolve
naturally?
Smallwood: A lot of things just naturally fell into place, as she was very pas-
sionate about this problem and had the time to work on it. She and I really
saw eye-to-eye that this was an important thing to solve for Netflix, especially
since we both agreed that we really hadn't quite been looking at it the right
way before. We also both understood that it would be a little bit of a hard sell,
even at Netflix, because it's been the way of looking at tests for so long. So not
only did she have the perfect kind of brain for this particular kind of problem,
she also understood the implications of tackling and leading the project.
Gutierrez: How do you balance time, projects, and priorities?
Smallwood: Within my group, we have several teams that are focused on
different parts of the business as their primary focus. It's not the only thing
 
Search WWH ::




Custom Search