Information Technology Reference
In-Depth Information
agent beliefs and are thus easier to apply in practice, for example for encouraging truth-
fulness in opinion polls.
2
Truthful Reporting through Scoring Rules
In many cases, agents are asked to provide information about an outcome that will even-
tually become known with certainty. For example, experts may predict the weather, the
future of the economy, or the completion date of a project. When this is the case, in-
centives for reporting this information truthfully can be provided through
proper scor-
ing rules
([3]). Agents provide information in the form of a probability distribution on
different possible outcomes. Once the true outcome becomes known, they get paid a
reward that depends on how well their prediction matched the observed outcome. This
reward is computed by a scoring rule that takes the report and the true outcome as argu-
ments. A scoring rule is called
proper
if it provides the highest expected reward exactly
when the agent reports its probability distribution truthfully.
Assume that the task is to predict which of
k
outcomes
o
1
,..,
o
k
will actually occur,
and that an expert agent has a probability distribution
p
=(
p
(
o
1
)
,..,
p
(
o
k
))
for the true
outcome. The agent reports this distribution as
q
=(
q
1
,..,
q
k
)
. We would like to provide
incentives so that it is optimal to report
q
=
p
.
This can be provided for example using the
quadratic scoring rule
:
pay
(
o
t
,
q
)=
a
+
b
2
q
t
−
j
=
1
q
j
k
where
o
t
is the outcome that actually occured and
a
is a non-negative and
b
a positive
constant. It is straightforward to show that this scoring rule is proper in that the expected
payment:
k
i
=
1
p
(
o
i
)
pay
(
o
i
,
q
)
=
a
+
b
2
E
[
pay
](
q
)=
k
j
=
1
p
(
o
j
)
k
j
=
1
q
j
k
j
=
1
p
(
o
j
)
q
j
−
=
a
+
b
(
2
p
·
q
−|
q
|
)
is maximized by maximizing
p
q
, which is the case exactly when the vectors
p
and
q
are aligned. Thus, reporting truthfully is a dominant strategy for agents.
As an example, consider predicting whether the next day's weather will be good (g)
or bad (b) as a vector of two probabilities (p(g),p(b)). Let the scoring rule be
pay
(
o
t
,
q
)=
1
+
2
q
t
−
·
j
=
1
q
j
k
An expert's true belief could be that the weather will be good with probability 0.8, and
bad with probability 0.2. Now consider the expected payoff for reporting this distribu-
tion truthfully. If the weather turns out to be good, the expert receives a payment of
