Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health - Agriculture Forestry and Fishing Research at NIOSH - page 254

tion. Its purpose is to encourage the EC to summarize its work in one place and
to concentrate on the subprograms and the items that will contribute to the final
impact and relevance scores.
To set the context for this step in the evaluation of the impact of the research
program in preparation to respond to charge 1, the EC will first need to consider
the available evidence of changes in work-related risks and adverse effects and
external factors related to the changes. That information should be organized as a
prose response to items 1.1 and 1.2 in Table 4.
Next, the EC should review the responses to the questions in Sections III.B.6
through III.B.8 and systematically rate the impact of the research program and
its subprograms by responding to items 1.3-1.7 in Table 4. To complete the table,
the EC response should use one of the following five terms: “major or important,”
“moderate,” “likely,” “limited,” or “none” (since 1995). The EC should evaluate
separately the impact of the research and the impact of transfer activities. High
ratings on items 1.3-1.7 require the committee's judgment that the program has
contributed to outcomes. For example, outcomes have occurred earlier than they
would have or are better than they would have been in the absence of the research
program, or outcomes would have occurred in the absence of external factors be-
yond NIOSH's control or ability to plan around.
The EC should then assess the relevance of the research program and subpro-
grams in preparation for addressing charge 2. The EC should review the responses
to the questions in Sections III.B.2 through III.B.5 and rate the relevance of the
research program and its subprograms by responding to items 2.1 and 2.2 in Table
4. The same five terms should be used (“major or important,” “moderate,” “likely,”
“limited,” or “none”) to evaluate separately the relevance of the research and the
relevance of the transfer activities. Transfer activities occur in two contexts: (1)
NIOSH efforts to translate intellectual products into practice and (2) efforts by
stakeholders to take advantage of NIOSH products.
Final Program Ratings
To provide the final assessment of the research program for charge 1 (impact)
and charge 2 (relevance), the ECs will use their expert judgment, their responses
to the questions in Table 4, and any other appropriate information to arrive at one
overall rating for the impact of the research program and one for its relevance to
the improvement of occupational safety and health. In light of substantial differ-
ences among the types of research programs that will be reviewed and the chal-
lenge to arrive at a summative evaluation of both impact and relevance, however,
the FC chose not to attempt to construct a single algorithm to produce the two
final ratings.