The Data Fluent Culture - Data Fluency: Empowering Your Organization with Effective Data Communication - page 146

reliability
Selection Consider subject-related variables that may impact

how people respond to interventions.

History Consider anything during the time frame of the

intervention that may impact results.

Maturation Consider that people change and grow. What may

look like a result of an intervention is purely par-
ticipant maturation.

Selection-Maturation
Interaction

Consider that different populations will mature at
different rates.

Repeated testing Consider that prior exposure to the intervention

may affect results of subsequent measures. For
example, people can become test-wise and are
thus less affected by the intervention.

Instrumentation Consider the reliability of the instrument used to

gather the data or the precision of the researcher
who takes the measurements. Inter-rater reliability
helps with this.

Regression to the mean Consider that extreme cases are likely to move in

one direction over the course of the pre-post test
sequence . . . always toward the average.

Experimenter Bias Consider the reality that expectations and invest-

ment in a project will impact the outcome of a
study.

generalizability
Pretest Consider that a pretest is artificial when mov-

ing outside to the broader population. When we
take the “treatment” to the external population,
there is no pretest. Here, the pretest is part of the
treatment.

Subject selection Consider the impact of this sampling issue. Even

if a sample of people is randomly assigned within
an experiment, if they don't represent the larger
community in terms of demographics, the experi-
ment is not generalizable.

Reactive effects/the
Hawthorne effect

Consider that the attention the subjects receive
in the experiment is impacting the outcome—this
won't happen in the larger community—therefore
an implementation dip will occur.