Recommendation Engines: Building a User-Facing Data Product at Scale - Doing Data Science - page 200

various algorithms when building a large-scale engineering system
and infrastructure that powers a user-facing product.
Matt graduated from MIT in CS, worked at SiteAdvisor, and co-
founded Hunch as its CTO. Hunch is a website that gives you recom‐
mendations of any kind. When they started out, it worked like this:
they'd ask people a bunch of questions (people seem to love answering
questions), and then someone could ask the engine questions like,
“What cell phone should I buy?” or, “Where should I go on a trip?”
and it would give them advice. They use machine learning to give
better and better advice. Matt's role there was doing the R&D for the
underlying recommendation engine.
At first, they focused on trying to make the questions as fun as possible.
Then, of course, they saw things needing to be asked that would be
extremely informative as well, so they added those. Then they found
that they could ask merely 20 questions and then predict the rest of
them with 80% accuracy. They were questions that you might imagine
and some that are surprising, like whether people were competitive
versus uncompetitive, introverted versus extroverted, thinking versus
perceiving, etc.—not unlike MBTI .
Eventually Hunch expanded into more of an API model where they
crawl the Web for data rather than asking people direct questions. The
service can also be used by third parties to personalize content for a
given site—a nice business proposition that led to eBay acquiring
Hunch.com .
Matt has been building code since he was a kid, so he considers soft‐
ware engineering to be his strong suit. Hunch requires cross-domain
experience so he doesn't consider himself a domain expert in any fo‐
cused way, except for recommendation systems themselves.
The best quote Matt gave us was this: “Forming a data team is kind of
like planning a heist.” He means that you need people with all sorts of
skills, and that one person probably can't do everything by herself.
(Think Ocean's Eleven , but sexier.)
A Real-World Recommendation Engine
Recommendation engines are used all the time—what movie would
you like, knowing other movies you liked? What topic would you like,
keeping in mind past purchases? What kind of vacation are you likely
to embark on, considering past trips?