Information Technology Reference
In-Depth Information
system for the chosen level of abstraction? These quality-related questions
precede verification and validation.
3.
Verification.
Which passive and active tests were conducted to verify that
the model is behaving in the way it is intended to behave? Social scientists
also call this internal validity. Verification tests include but are not limited
to the following: Code walkthrough, debugging, unit testing, profiling, and
other common procedures used in software development (Sergeant, 2004).
What where the results of such verification tests? Quality assessment should
cover investigation of which verification procedures were in fact used, since
results can range widely depending on the extent of verification methods
employed. Unfortunately, most social simulations are reported without much
(or any) information regarding verification procedures, as if “results speak
for themselves.”
4.
Validation.
Similarly, validation of a social simulation, what social scientist
call external validation (or establishing external validity), consists of a suite
of tests, not a single procedure. Such tests are important for assessing quality
in a social simulation. Which tests (histograms, RMSE for assessing goodness
of fit, time series, spatial analysis, network structures, and other forms of real
vs. artificial pattern matching tests) were conducted to validate the model?
What were the results? Validation tests are often the focus of reporting
results, at the expense of all other phases in the lifecycle of a social simulation
model.
5.
Analysis.
The preceding aspects provide a basis for establishing overall con-
fidence in a given model. What is the level of confidence in the model's re-
sults, given the combined set of verification and validation tests? If networks
are present and significant in the focal system, does the model exploit theory
and research in social network analysis (Wasserman and Faust, 2005)? Does
the model facilitate analysis of complexity in the system of nonlinear inter-
actions and emergent properties? Which features of complexity (emergence,
phase transitions, power-laws or other heavy-tailed distributions, criticality,
long-range dynamics, near-decomposability, serial-parallel systems, or other
structural features) are relevant to the particular model? If spatial features
are significant, does the simulation employ appropriate spatial metrics and
statistical tools for spatial data? What is the overall analytical plan in terms
of simulation runs and how is it justified? How does computational analysis
advance fundamental or applied understanding of social systems? In terms
of overall effectiveness, does the model render what is necessary for answer-
ing the initial research question or class of research questions? This differs
from eciency. In terms of the simulation's computational facilities, does the
model possess the necessary functionality for conducting extensive compu-
tational analysis to answer the research questions or go even beyond? How
powerful is the model in terms of enabling critical or insightful experiments?
For example, in terms of parameter exploration (evolutionary computation)
and record-keeping. What is the quality of the physical infrastructure that
renders the most effective simulation experience?
