Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures - Discrimination and Privacy in the Information Society - page 42

Chapter 3
Why Unbiased Computational Processes Can
Lead to Discriminative Decision Procedures
Toon Calders and Indrė Žliobaitė 1
Abstract. Nowadays, more and more decision procedures are supported or even
guided by automated processes. An important technique in this automation is data
mining. In this chapter we study how such automatically generated decision sup-
port models may exhibit discriminatory behavior towards certain groups based
upon, e.g., gender or ethnicity. Surprisingly, such behavior may even be observed
when sensitive information is removed or suppressed and the whole procedure is
guided by neutral arguments such as predictive accuracy only. The reason for this
phenomenon is that most data mining methods are based upon assumptions that
are not always satisfied in reality, namely, that the data is correct and represents
the population well. In this chapter we discuss the implicit modeling assumptions
made by most data mining algorithms and show situations in which they are not
satisfied. Then we outline three realistic scenarios in which an unbiased process
can lead to discriminatory models. The effects of the implicit assumptions not be-
ing fulfilled are illustrated by examples. The chapter concludes with an outline of
the main challenges and problems to be solved.
3.1 Introduction
Data mining is becoming an increasingly important component in the construction
of decision procedures (See Chapter 2 of this topic). More and more historical data
is becoming available, from which automatically decision procedures can be de-
rived. For example, based on historical data, an insurance company could apply