Database Reference
In-Depth Information
he would ask for things and I would give them to him, and he'd be like “No,
no, no. This isn't right.” And I'd be like, “I don't understand, that's exactly what
you asked for.” Or I'd bounce ideas off of him like, “I'm thinking of using this
normalization technique, what do you think we should use?” And he wouldn't
be able to speak my language. Eventually, we learned to work together, how to
communicate with each other, and meet deadlines with non-pretty versions
of the product.
Once we had the project moving along, it was easier to understand the grand
vision of what we were doing. Then we got a data artist involved and all of
a sudden we could really take the data I was making and make it pretty and
understandable. Through this process, we were able to solidify what we were
trying to accomplish, because before that I was making graphs somewhat hap-
hazardly to show different outcomes. Once we had a more polished project
and could see how it was changing and what made it special, we were really
able to focus. Then I was able to work on making the process faster and we
brought in some other engineers to help me make the process run in a few
hours—instead of a day—for all of our customers.
Gutierrez:
How did you think about what kind of data and modeling you
needed for the project?
Smith:
As simple as it sounds, it was a great deal of trial and error. The idea
was to first try to understand at a very basic level what people were doing. So
as a first step, we had to try to get all of the data that we could into one place.
Once we had this data, we could then do very simple summary statistics to
get a glimpse of what the data showed.
We started with questions like: How much data do we need? And how much
do we have? And then we'd go back to math to understand the statistical con-
fidence we would be able to generate. Once we understood those numbers,
we could then figure out how many links we needed to see in what amount
of time, so that it would be useful for our customers. For some companies, we
could do it by month, and for other companies, we could do it by week. Once
we were certain about the data and had confidence in it, we had to choose the
models that would generate the best and most intuitive answers to a person
looking at the results. There was never one right answer. It was more of a mat-
ter of figuring out what we could communicate and with what bias.
Gutierrez:
What do you mean by what you can communicate?
Smith:
What I mean is that in this project we are doing a lot of communi-
cation to people who might be seeing the data for the first time and/or the
summary statistics of that data. So we had to focus on what type of things we
are communicating that makes sense to the average person. You can explain
to them all of the math, but unless they can translate that into what's going on
in the graph, they're not going to get it.
