Environmental Engineering Reference
In-Depth Information
ProblemsWithStatistics
Einstein is quoted as saying, “Not everything that can be counted counts,
and not everything that counts can be counted.” Qualitative data do not have
to be quantified to be meaningful or legitimate. Descriptive passages are as
persuasive as a
p
value in ethnography, if not more so. The use of statistics,
specifically, in ethnography raises many issues. Meeting the assumptions that
a specific test requires may be a particularly sticky problem. One of the most
common assumptions of inferential statistics is that the sample is random.
Typically, ethnography uses stratified judgmental sampling rather than a truly
randomized selection. The use of parametric statistics requires large samples.
Most ethnographers work with small groups, however. The issues of expertise
and appropriateness raise further difficulties.
In many instances, sophisticated statistical approaches are inappropriate in
ethnography in particular and in social science in general. The first criterion is
the appropriateness of the tool for the problem. The second criterion—a sub-
set of the first—is the methodological soundness of the application. A third
criterion involves ethics. Is use of a certain tool at a given time with a certain
population ethical? The ethical question is discussed in Chapter 7.
No design or technique is good or bad per se. The application, however, can
be useful or useless and appropriate or inappropriate. The use of an experi-
mental design and related statistical formulas to study the impact of an educa-
tional program or treatment on a population of former dropouts, near dropouts,
and “push outs” (those the schools are no longer obligated to serve because
they are too old or too disruptive) is conceptually sound. In the abstract, this
approach could shed light on possible gains in math and reading scores by
students in the program (compared with scores of students in the control
group). The application of this design to generate sophisticated statistical
inferences about most educational programs, however, is inappropriate on
strict methodological grounds. The assumptions of the design are rarely met.
A most valid experimental design with human subjects involves a double-blind
arrangement. The individual delivering the treatment, the individual receiving
it, and the individual in the control group do not know who is really receiving
treatment. In most educational treatments, teachers know whether or not they
are delivering an educational treatment, and the students know whether or not
they have been accepted into the educational program. Instead of a double-
blind experiment, the treatment group receives a positive treatment; rejected
students receive a negative treatment. Thus, students receiving treatment may
react with a Hawthorne effect, whereas rejected students may react with a John
Henry effect—overcompensating to demonstrate that they can do well despite
the rejection. These forms of reactivity and contamination severely undermine
