Database Reference
In-Depth Information
Gutierrez:
How do you choose which problems to solve?
Tunkelang:
I start by thinking about the potential impact using optimistic
assumptions. How many people will care if we deliver an optimal solution, and
how much better is an optimal solution than what we have today? I try to find
the right measure of impact and establish an upper bound. If that's not high
enough, then the problem is probably not worth solving.
Then I try to make more conservative assumptions about the expected impact.
If we only make it halfway from where we are today to optimal, is it still worth
it? What about 10 percent of the way? If more conservative assumptions give
me pause, then I try to make our opportunity analysis more rigorous.
Finally, I try to estimate how much work it will take to validate our assump-
tions. Not how much work it will take to solve the problem, but how much
work it will take to remove most of the uncertainty about the project's suc-
cess. I strongly favor projects that allow for rapid mitigation of uncertainty.
Sometimes this means fast failure, and other times it means early promises of
success.
Gutierrez:
How do you think about whether you're modeling the right
thing?
Tunkelang:
As George Box said, “All models are wrong, but some are useful.”
It's tempting to make models as realistic as possible, but it's a temptation
I try to resist. I'd rather have a simple model that is easy to explain. Of course,
using simple models means having to consider the risks of confusing them
for reality.
For example, when I'm working on search quality, it's tempting to model a click
as a search success and an abandoned search results page as a search failure.
But not all clicks are successful. For example, the searcher may need to click
to see more information to determine that the result is irrelevant. On the
other side, not all abandoned searches are failures. For example, the searcher
may be satisfied with the information presented in the search summaries.
So we work to keep in mind that models aren't supposed to be photorealistic.
Rather, they give you something that's close enough to reality to be an ade-
quate proxy, yet simple enough to be measured and optimized. And, as long
as we work with models in good faith and keep our eyes open for glaring
divergences from reality, our models tend to be kind to us.
Gutierrez:
How do you think about whether you have the right data?
Tunkelang:
There is no substitute for common sense. You have to look at all of
the data you have and validate it against the data you don't have. For example, if
your data tells you that the most in-demand skills are all in a particular industry,
you should check to make sure your data doesn't overrepresent that industry.
