Large-Scale Network Traffic Analysis for Estimating the Size of IP Addresses and Detecting Traffic Anomalies - Large Scale and Big Data: Processing and Management - page 455

computed. The fraud score is consistently high and stable over time, while the qual-
ity score of the filtered traffic remains an order of magnitude lower than the quality
score of the unfiltered traffic for the same group of publishers.

14.5.4.8 Overlap with Other Blacklists
Figure 14.12 illustrates the overlap between IPs filtered by the IP size histogram filter
and IPs listed in Gmail blacklist [25] and in Spamhaus Exploit blacklist (XBL) [24].
For each day, a blacklist of IPs that sent abusive clicks during that day was compiled.
The x -axis represents the time difference between the day the blacklist was compiled
and the day the Gmail and Spamhaus blacklists were compiled.
A zero value indicates that blacklists associated with the same day are compared.
Negative values indicate the compiled blacklist is some days older than the blacklist
compiled by Gmail or Spamhaus XBL. Positive values indicate the opposite sce-
nario. The y -axis represents the percentage of IPs detected with the compiled system
that are also found in other blacklists. Interestingly, a large percentage of abusive
clicks are generated by IPs that also generate other kinds of abusive traffic, such as
spam emails. In particular, up to 45% of abusive clicks are generated by source IPs
listed either in Gmail blacklist or in Spamhaus XBL.

14.5.5 F lagging e ntities
The IP size histogram filter described in Section 14.5.4 can distinguish between a set
of legitimate and a set of abusive clicks by automatically detecting anomalous spikes
in a distribution associated with low quality click traffic. To avoid detection, a fraud-
ster may spread its clicks across various buckets to avoid generating high probability
regions in few buckets. This warrants a method that examines the entire distribution.
In this section, the IP size distributions associated with entities are considered.
An entity can be a user-agent, an e-mail domain, a publisher, a city, a country, and so

45

40

35

30

XBL + Gmail
Gmail
XBL

25

20

15

10 -6

-4 -2 0 2 4 6

Time difference

FIGURE 14.12 Percentage of abusive clicks generated by IPs listed on the Gmail blacklist

or on XBL.