Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health - Agriculture Forestry and Fishing Research at NIOSH - page 231

the research activities of industry and other federal agencies and the political and
regulatory environment, which can affect all components of the research program
(Figure 2). For purposes of this review, the results of inputs and external factors are
the program research activities, outputs, and associated transfer activities that may
result in intermediate outcomes and possibly eventual end outcomes.
The FC has used the NIOSH logic model to develop the flow chart to define
the scope and steps of an EC evaluation. The FC's vision of how a program evalu-
ation should occur is incorporated in a summary manner in the flow chart and
discussed extensively in later sections. For example, the FC identified two types
of outcomes: (a) intermediate outcomes , which represent implementations (what
external stakeholders, such as employers, do in reaction to the products of NIOSH
work, including new regulations, widely accepted guidelines, introduction of con-
trol technologies in the workplace, changes in employer or worker behaviors, and
changes in diagnostic practices of health-care providers), and (b) end outcomes ,
which are improvements (reductions in work-related injuries, illnesses, and haz-
ardous exposures). For the purpose of evaluation, the FC does not differentiate
between NIOSH's “intermediate customer” and “final customer” activities (Figure
1); instead it combines them into a single category (Box E, Review and Assessment
of Intermediate Outcomes, Figure 2). Training and development programs were ap-
propriately defined as outputs by NIOSH in the logic model, but the FC finds more
value in focusing on response to such offerings as intermediate outcomes (Box E)
in the flow chart. The number of workers exposed to training activities represents
a type of implementation of NIOSH outputs in the workplace. In evaluating each
program or major subprogram, the EC must collect, analyze, and evaluate infor-
mation on items described in each of the boxes of Figure 2. Further details on the
evaluation are described in Section III of this document.

II.B. Steps in Program Evaluation

The FC has concluded that useful evaluation requires: (a) a disciplined focus on
a small number of questions or hypotheses typically related to program goals, per-
formance criteria, and performance standards; (b) a rigorous method for answering
the questions or testing the hypotheses; and (c) a credible procedure for developing
qualitative and quantitative assessments. The evaluation process developed by the
FC is summarized here and described in detail in Section III of this document.

1. Gather appropriate information from NIOSH and other sources.

2. Determine timeframe that the evaluation will cover (see III.B.1).

3. Identify program-area major challenges and objectives (see III.B.2).

All NIOSH research programs, whether based on health outcomes or
sectors, are designed to be responsive to the safety and health problems