Database Reference
In-Depth Information
involved, which is why we revisit the results at least once a week. Often we
have a deep-dive on the models, results, and performance. We might discuss
something like: “We feel that this campaign isn't performing as well as it could.
Can you tell us whether we should change the frequencies or whether we
have to change the bid strategy in a certain way?” And so I spend some of
my time working with people upstairs, some account management time, just
translating what I know about how these things work into strategies on how
to best run a certain campaign.
Gutierrez:
What about your prototype building tasks?
Perlich:
I have lots of favorite little toy problems. Sometimes these are devel-
oped into new prototypes. These toy problems encompass things like feed-
back saying we were actually performing too well on certain campaigns, or
feedback that we are not necessarily realizing the margin that we could on
other campaigns. Much of the prototype work is looking and asking if there
are there some smart ways of going into our system, looking at the data, and
seeing if there are strategies we should employ to help our customers and
our margins.
Once I decide to build a prototype, I have some idea of what to do. I imple-
ment it and then I test it out on some small data set. We call these small data
sets our “sandbox,” as they do not use the production system of data. So I
run the prototype and I see how that would work in the sandbox. When that
comes back as good, then we will try it out with bigger data sets.
That was the case recently, where we felt something was wrong with the rela-
tionship between our bid price and the price we end up paying. The auctions
work a bit like eBay where the winner is supposedly only paying the second
highest bid price. For these campaigns we wanted a black list of publishers
that we should not bid on the bid list because, for some reason, the bids that
we were winning were really expensive and then they were performing really
badly. In this context we discovered that it seemed like we could reduce the
bid price for these types of requests and we would save margin, and for some
of the campaigns, the performance would even go up. I do not know why
this was happening or what was causing it, so this was a prototype that we
ended up building and it was used to select campaigns to be put on different
bid strategies. But it started out as my little pet project with a toy model that
I worked on for three days until we had a prototype. It does not add value
consistently on all campaigns so we have not automated it but keep it rather
as a tool to use as needed.
Gutierrez:
What kind of tools do you use?
Perlich:
We are very much a self-made shop, so we build the things that we
need. I am not really hardcore on top of what the production side looks like,
so I will talk about our side. On our side we have two main technology areas.
