Geoscience Reference
In-Depth Information
where the reference label is assumed to be correct. The source of reference data may be aerial
photography, ground visit, or videography. Discussion will be limited to the case in which the
assessment unit for comparing the map and reference label is a pixel. Similar issues apply to
sampling both pixels and polygons, but a greater assortment of design options has been developed
for pixel-based assessments. Most of the chapter will focus on site-specific accuracy, which is
accuracy determined on a pixel-by-pixel basis. In contrast, nonsite-specific accuracy provides a
comparison aggregated over some spatial extent. For example, in a nonsite-specific assessment, the
area of forest mapped for a county would be compared to the true area of forest in that county.
Errors of omission for a particular class may be compensated for by errors of commission from
other classes such that nonsite-specific accuracy may be high even if site-specific accuracy is poor.
Site-specific accuracy may be viewed as spatially explicit, whereas nonsite-specific accuracy
addresses map quality in a spatially aggregated framework.
A sampling design is a set of rules for selecting which pixels will be visited to obtain the
reference data. Congalton (1991), Janssen and van der Wel (1994), Congalton and Green (1999),
and Stehman (1999) provide overviews of the basic sampling designs available for accuracy
assessment. Although these articles describe designs that may serve well for small-area, limited-
objective assessments, they do not convey the broad diversity of design options that must be drawn
upon to meet the demands of large-area mapping efforts with multiple accuracy objectives. An
objective here is to expand the discussion of sampling design to encompass alternatives available
for more demanding, complex accuracy assessment problems.
The diversity of accuracy assessment objectives makes it important to specify which objectives
a particular assessment is designed to address. Objectives may be categorized into three general
classes: (1) description of the accuracy of a completed map, (2) comparison of different classifiers,
and (3) assessment of sources of classification error. This chapter focuses on the descriptive
objective. Recent examples illustrating descriptive accuracy assessments of large-area LC maps
include Edwards et al. (1998), Muller et al. (1998), Scepan (1999), Zhu et al. (2000), Yang et al.
(2001), and Laba et al. (2002). The foundation of a descriptive accuracy assessment is the error
matrix and the variety of summary measures computed from the error matrix, such as overall, user's
and producer's accuracies, commission and omission error probabilities, measures of chance-
corrected agreement, and measures of map value or utility
.
Additional descriptive objectives are often pursued. Because classification schemes are often
hierarchical (Anderson et al., 1976), descriptive summaries may be required for each level of the
hierarchy. For large-area LC maps, there is frequently interest in accuracy of various subregions,
for example, a state or province within a national map, or a county or watershed within a state
or regional map. Each identified subregion could be characterized by an error matrix and accom-
panying summary measures. Describing spatial patterns of classification error is yet another
objective. Reporting accuracy for various subsets of the data, for example, homogeneous 3
3
pixel blocks, edge pixels, or interior pixels may address this objective. Another potential objective
would be to describe accuracy for various aggregations of the data. For example, if a map
constructed with a 30-m pixel resolution is converted to a 90-m pixel resolution, what is the
accuracy of the 90-m product? Lastly, nonsite-specific accuracy may be of interest. For example,
if a primary application of the map were to provide LC proportions for a 5-
¥
5-km spatial unit
(e.g., Jones et al., 2001), nonsite-specific accuracy would be of interest. Nonsite-specific accuracy
has typically been thought of as applying to the entire map (Congalton and Green, 1999). However,
when viewed in the wider context of how maps are used, nonsite-specific accuracy at various
spatial extents becomes relevant.
The basic elements of a statistically rigorous sampling strategy are encapsulated in the speci-
fication of a probability sampling design, accompanied by consistent estimation following principles
of Horvitz-Thompson estimation. These fundamental characteristics of statistical rigor are detailed
in Stehman (2001). Choosing a sampling design for accuracy assessment may be guided by the
following additional design criteria: (1) adequate precision for key estimates, (2) cost-effectiveness,
¥