Trust in Online Collaborative IS - Community-Built Databases: Research and Development - page 348

R i ( t )to R i ( T ) is obtained using the initial values R i (0) ¼ 0.2 for all anonymous

users, and R i (0) 0.45 for all registered users (data not shown). These initial

values are used in the rest of the chapter.
Finally, it is worth noting that if Model 3 were to perform well on the classifica-
tion task (vandals vs. admins), this would provide further indirect evidence that
Model 3 is self-consistent and may perform well on other users too, since the update
equation a5t time t + 1 for Model 3 uses the predicted reputation for users other
than vandals or admins at time t .

¼

14.3.1 User Reputation Results

In this section, we evaluate the reputation models on our dataset extracted from
English Wikipedia history, as described in Sect. 14.5 .

14.3.1.1 Evaluation on Ground-Truth Data

We first analyze the performance of the reputation models on two major popula-
tions: vandals and admins. Vandals are users who have been blocked by the
Wikipedia Committee because they performed edits in violation of Wikipedia
rules by engaging in vandalism. The “admin” title is conferred to users selected
by the Wikipedia Committee due to their helpful, long-term contributions.
Although Model 1, Model 2, and Model 3 have at most one free parameter ( )

and can be applied directly to estimate the reputation R i ( t ) of any user at any time,
here we first use the output R i ( T ) to derive a classifier to separate vandals and
admins. Table 14.1 shows the AUC (Area Under the Curve) values corresponding
to the ROC curves of the three corresponding classifiers. The table shows that all
the three models perform well and their classification performances are compara-
ble. To further analyze classification performance on a broader set of users, we
extend the test populations beyond the extreme of vandals and admins to all blocked
users on one side and to good users of Wikipedia on the other side.
All blocked users are a superset of the vandals. According to Wikipedia, in
addition to vandalism, user blocking can happen because of other reasons such as

a

Table 14.1 AUC values for the three reputation models
Admins-
vandals

Good users-
vandals

Admins-
blocked users

Good users-
blocked users

Model 1 0.9751 0.9839 0.9196 0.9220

Model 2 0.9753 0.9769 0.9094 0.9153

Model 3 0.9742 0.9762 0.9073 0.9125