Other Programmatic Elements Identified by the Committee - Agriculture Forestry and Fishing Research at NIOSH - page 142

conducted by the Alaska Field Station provides a good example of continuous pro-
gram evaluation consistent with the NIOSH logic model: research on commercial
fishing safety is well defined and was designed by using surveillance data, which
can also be used to track the outcomes of the project. Those data were also used
to make changes in project activities to improve safety. The end outcome of the
program is a reduction in deaths associated with fishing activities in Alaska. The
Agricultural Center Evaluation Project is also a good example of a NIOSH evalu-
ation initiative. The report published in January 2007 provided recommendations
to NIOSH. The one overarching recommendation was to continue to support this
collaborative evaluation effort.
There does not appear to be a similar strategy or conceptual framework for
evaluation of most AFF activities. The briefing document made it difficult to sort
the information into appropriate sections in any coherent way. For example, sur-
veillance discussions were scattered throughout the document, making it difficult
to determine what was considered surveillance, hazard assessment, and so on.
Most important, it seems that no infrastructure has been developed in NIOSH
for consistently capturing the activities of AFF projects that can be used for ef-
ficient, effective evaluation. The briefing book contains NIOSH's best effort to go
back and find needed information, but much of the information that must reside
somewhere at NIOSH is not included; for example, the original requests for ap-
plications (RFAs) for the programs launched in 1990 are not in the packet although
the later RFAs calling for Ag Center applications are.
The NIOSH Operational Logic Model described in the evidence package indi-
cates that evaluation would occur during most of the steps of the model (Figure 1-3
on page 28 of NIOSH, 2006a). Although the evidence package does report outputs,
intermediate outputs, and outcomes in many cases, there is little information on
evaluating the effectiveness of the reported items. The one exception, as mentioned
earlier, is the Alaska commercial-fishing project, which seems to provide evidence
of effective evaluation.
NIOSH staff readily admitted the difficulty of measuring outcomes. This is
demonstrated in the evidence package by NIOSH stating that “in many instances
it is difficult to effectively trace the contribution of NIOSH to the end outcomes.
Many groups contribute to reducing occupational injuries and illnesses and to
creating safer places to work. Still, NIOSH is strongly committed to developing ob-
jective measures of its real-world performance. If the best measures of performance
relate to motivating and enabling others to work safer, this in no way diminishes
the importance of the accomplishment (NIOSH, 2006a).” Under “End Outcomes,”
the evidence package notes that “evidence of reductions in occupational hazard
exposures, illnesses, and/or injuries as a result of the AFF Program research is
elusive at best” (page 101 in NIOSH, 2006a).