Environmental Engineering Reference
In-Depth Information
technical evaluation needs to assess whether the user interface is able to define and
conduct properly the runs of the underlying components. The requirements of a
user interface depend strongly on the background of the users and therefore needs
to be tested by potential users.
In the case of the SEAMLESS-IF GUI role games were the main tool for testing.
These role games involved both groups and individuals of SEAMLESS researchers
that acted as integrative modellers who interact with policy experts for an impact
assessment with SEAMLESS-IF. Varying with the level of development of the
integrated framework these tests evolved during the project from defining a policy
question with the GUI (pre-modelling), to quantifying scenarios and running
models through the GUI (modelling) and to testing the GUI functionalities for
displaying and analyzing results (post-modelling). Outcomes from the testing of the
SEAMLESS-IF procedure on preferences of potential end-users on display of results
were used in this last step. Each round of testing resulted in a set of recommendations
to the GUI developers on screens and functionality, which was prioritized in the
light of the needs for implementing the test cases.
System Evaluation
The third type of evaluation consists of assessing whether the framework as a whole
gives acceptable results that are useful for end-users. A key question in this analysis
would also be whether the integrated framework as a whole provides different
insights from those gained from an analysis based on the individual components.
In the case of SEAMLESS-IF the integrated framework was not yet available
at the time of writing, preventing evaluations of the system as a whole. Below
the outlines are sketched of the way in which such a system evaluation could
be done for the three types of components. Comparing the results obtained for
the test case with the framework and with the use of its individual components is
the ultimate step of this evaluation.
System Evaluation of Procedures for Using the Integrated
Framework
When performing a system evaluation of the procedures for integrated assessment
the focus will be on whether the procedure succeeds in translating a policy question
of end-users into assessed scenarios that provide satisfactory and useful information
to support policy decisions. This assessment would encompass not only the quality
of the definition of the problem and analysis of results in relation to the policy question,
but also the time needed for such an assessment in relation to the organization of
integrated assessments by end-users.
