Database Reference
In-Depth Information
Gutierrez:
Have you faced any challenges with giving your customers the
model results?
Radinsky:
After we gather the data and construct the various classifiers, what
we give to our customers are scores about people to whom they are selling.
I come from a search engine background, as I previously worked on building
different algorithms for Bing. And there it wasn't that big of a problem. You
have a query and you just give back to the user a ranked result. We do this
as well—our customers also get ranked list of people to call. The challenge
we faced here is that sometimes the salesperson doesn't want to call whom
I'm telling him to call. They'll push back and ask, “Why is this person an A and
this person a B?” You can't just create this statistical model and just give it to
the salesperson and expect them to say, “Oh, yeah, that's right”—because they
want to understand why certain people are ranked a certain way.
Also, the salesperson will have some kind of prior knowledge that they want
to see included in the model. It's a big difference between what is statistically
right and what is perceived right. I actually call it emotional artificial intel-
ligence because we work with people. Our algorithm and the results it pro-
duces have to work with people. So we ended up doing a clustering that is not
only statistically right, but also takes into account all of the emotional feelings
that the person who is working with the algorithm is taking into account.
One of our algorithms to address this issue is as follows. We score of a lot
of prospective people for the salesperson to call on from A to D. Then we
monitor whether somebody we've scored as an A is skipped for somebody
who's a B. If they were skipped, then I would say that the explanation for the
A classification was probably wrong. This new insight is then going to be fed
back to our classifier. The model is then going to retrain itself and build a bet-
ter snippet or a better explanation.
More than that, most of our customers just want to give feedback to the
system, so we've been using a lot of reinforcement learning algorithms.
Salespeople sometimes tell me, “This person is not an A. I know they're not
an A. This person's definitely a B.” And even if we know it's wrong based on
historical data, the salesperson knows something, right? So we have to work
with people when we build our classifiers. Our challenge is to achieve this
interaction in an automated way—and this is an interesting challenge.
Gutierrez:
What do you find exciting about this?
Radinsky:
The part of it that makes me excited is that we can help compa-
nies perform much better because we see their sales operation from a macro
point of view. This is a completely different view than they have because we
see it from the data point of view, which helps us to help them make bet-
ter decisions. In the last six months, we've mostly been building and scaling
because we have more and more customers coming in, and the system has to
be built and working for all of them all the time.
