Explainable and Non-explainable Discrimination in Classification - Discrimination and Privacy in the Information Society - page 156

this correct rate is applied, i.e., if we accepted 20% of male applicants and 20%
of female applicants to medicine and if we accepted 40% of male applicants and
40% of female applicants to computer science. Figure 8.3 illustrates the situation.
Observe that the total number of accepted applicants has not changed, 60 applicants
were accepted before and 60 are accepted now.

Fig. 8.3 Corrected data, empty dots indicate the corrected decisions

We can see from the figure that the total acceptance rates of males and females
are still different, 36% and 24% correspondingly. However, as long as we treat males
and females equally within each program, there is no illegal discrimination. Thus all
this 12% difference is explainable and tolerable. Recall, that there is 22% over all
discrimination in the original data shown in Figure 8.2. Thus, as we see that 12%
is explainable and thus the remaining 22% 10% is non-explainable, and

we should aim to eliminate only this illegal part of discrimination when training a
classifier. See (Zliobaite et al., 2011) for further technical information on how to
calculate the explainable discrimination.

− 10% =

8.3.3 Illustration of the Redlining Effect

Suppose that it is no longer allowed to discriminate females directly, the gender
information is kept hidden from the admission committee to avoid the gender dis-
crimination. The committee will treat male and female applicants within medicine
and within computer science equally. However, knowing the fact that females prefer
to apply to medicine, it is still possible to discriminate indirectly (without knowing
the gender of an applicant). A decision maker who wants to discriminate, may re-
duce the overall acceptance rates to medicine and increase the acceptance rate to
computer science. This phenomenon is known as the redlining (Tootell, 1996).
Figure 8.4 illustrates the situation with our university example. Recall that in our
example 80 females chose to apply to medicine and 20 to computer science, while
20 males chose medicine and 80 chose computer science. Within each program both
genders are treated equally. Figure 8.4 plots the situation when an adversary varies
the acceptance rates within each program (keeping the total number of accepted
people fixed to 60 as in the original example). The black dots illustrate the accep-
tance rates where all the difference between males and females is explainable (20%
for medicine and 40% for computer science as calculated in the previous section). If