Information Technology Reference
In-Depth Information
pay
(
g
,
(
0
.
8
,
0
.
2
)) =
1
+
2
·
0
.
8
−
0
.
68
=
1
.
92; if it turns out to be bad, the payment is
0
.
68
=
0
.
72. Thus, the expected payoff for truthfully
reporting the probability distribution is:
pay
(
b
,
(
0
.
8
,
0
.
2
)) =
1
+
2
·
0
.
2
−
0
.
8
pay
(
g
,
(
0
.
8
,
0
.
2
))+
0
.
2
pay
(
b
,
(
0
.
8
,
0
.
2
))=
1
.
68
Now consider a false report, for example
(
0
.
5
,
0
.
5
)
. Now the reward in case of good
and bad weather is identical and equal to
pay
(
g
/
b
,
(
0
.
5
,
0
.
5
)) =
1
+
2
0
.
5
=
1
.
5,
and thus the expected payment is also equal to 1
.
5. This is significantly less than what
is expected for truthful reporting.
There are other proper scoring rules, such as the logarithmic scoring rule:
·
0
.
5
−
pay
(
o
t
,
q
)=
a
+
b
ln
q
t
where
o
t
is the outcome that actually occured and
a
is a non-negative and
b
a positive
constant. These may lead to lower expected payments or wider margins for truth-telling,
but can have other drawbacks. For example, with the logarithmic scoring rule payments
can become negative.
Proper scoring rules can also be constructed for eliciting averages and other prop-
erties of distributions. Recently, [4] have characterized the questions to which truthful
answers can be elicited using scoring rules.
3
The Peer Prediction Method
Proper scoring rules can be applied whenever the ground truth that is being observed
can eventually be verified. However, there are many cases where this condition is not
satisfied. Consider for example ratings reported for products and services on the inter-
net: it is not possible to independently verify whether these ratings were given truthfully.
Similarly, measurements taken by sensors would often not be verifiable by other means.
A similar situation exists when reporting opinions about hypothetical scenarios, such as
what would happen if interest rates were raised by different degrees: since only one of
these scenarios will actually be implemented, predictions about the others cannot be
verified.
However, in such cases it is still possible to make truthful reporting an equilibrium
strategy for agents by applying a proper scoring rule based on the prediction of another
agent, called a reference report. Provided the other agent made a truthful prediction
and both have the same knowledge and observing the same signals, truthful reporting is
the best response. Thus, for a population of agents with the same knowledge, reporting
truthfully is a Nash equilibrium. This is called the
peer prediction method
in ([5]).
As an example, consider reporting the quality of service received by a plumber. Two
agents A and B both report on the quality of service they received. The key idea is
that the quality of service A received will influence its expectation of the quality that
B received: if A observed good service, then its belief for the probability
p
(
g
|
g
)
that
B also received good service is higher than the value
p
(
g
|
b
)
if A received bad service.
Assume for this example that
p
(
g
b
)=
0
.
4.
Now we apply the same scoring rule mechanism we mentioned earlier, but consider
B's report the ground truth. If A observed good service, its probability distribution
|
g
)=
0
.
8and
p
(
g
|
