Databases Reference
In-Depth Information
Terminology: Crowdsourcing and Mechanical Turks
These are a couple of terms that have started creeping into the ver‐
nacular over the past few years.
Although crowdsourcing—the concept of using many people to solve
a problem independently—is not new, the term was only fairly re‐
cently coined in 2006. The basic idea is a that a challenge is issued and
contestants compete to find the best solution.
The Wisdom of Crowds
was a topic written by James Suriowiecki (Anchor, 2004) with the
central thesis that, on average, crowds of people will make better de‐
cisions than experts, a related phenomenon. It is only under certain
conditions (independence of the individuals rather than group-think
where a group of people talking to each other can influence each other
into wildly incorrect solutions), where groups of people can arrive at
the correct solution. And only certain problems are well-suited to this
approach.
Amazon Mechanical Turk is an online crowdsourcing service where
humans are given tasks. For example, there might be a set of images
that need to be labeled as “happy” or “sad.” These labels could then
be used as the basis of a training set for a supervised learning problem.
An algorithm could then be trained on these human-labeled images
to automatically label new images. So the central idea of Mechanical
Turk is to have humans do fairly routine tasks to help machines, with
the goal of the machines then automating tasks to help the humans!
Any researcher with a task they need automated can use Amazon
Mechanical Turk as long as they provide compensation for the hu‐
mans. And any human can sign up and be part of the crowdsourcing
service, although there are some quality control issues—if the re‐
searcher realizes the human is just labeling every other image as
“happy” and not actually looking at the images, then the human won't
be used anymore for labeling.
Mechanical Turk is an example of artificial artificial intelligence (yes,
double up on the “artificial”), in that the humans are helping the ma‐
chines helping the humans.
