A procedural model for ontological analyses - Information Systems Foundations: Constructing and Criticising - page 156

with the selected modelling grammar remains a problem. Unless the ontology and the
grammar are specified in the same language, it will be up to the coder to 'mentally con-
vert' the two specifications into each other for comparison purposes, which adds a
subjective element to the analysis. Different languages can also lead to different levels
of detail and further complicate the analysis. In any case, they make a more automated
comparison practically impossible. This is the typical situation in nearly all previous
analyses.
The three further shortcomings identified below are related to the process of the ontolo-
gical analysis and refer to what should be analysed, how it should be analysed as well
as who should conduct the analysis.
Lack of completeness
The first decision that has to be made in the process of an ontological analysis is on the
scope and depth of the analysis. Even though most ontologies have been discussed for
many decades, they still undergo modifications and extensions. It is up to the researcher
to clearly specify the selected version of the ontology and the scope and level of detail
of the analysis. In our work in the area of Web Services, for example, it was often not
clear what constructs form the core of the standard and, in fact, two researchers who
conducted independent analyses of the same Web Services standard selected a different
number of constructs.
Moreover, many ontological analyses focus solely on the constructs of the ontology and
the constructs of the grammar but do not sufficiently consider the relationships between
these constructs. The difficulty of clearly specifying the boundaries of the analysis, as
well as the limited consideration of relationships between the ontological constructs,
can lead to a lack of completeness.
Lack of guidance
After the scope and the level of detail of the analysis have been specified, it is typically
up to the coder to decide on the procedure of the analysis, i.e. in what sequence will
the ontological constructs and relationships be analysed? Currently, there are hardly
any recommendations on where to start the analysis. This lack of procedural clarity
underlies most analyses and has two consequences. First, a novice analyst lacks guidance
in the process of conducting the ontological evaluation. Second, the procedure of the
analysis can potentially have an impact on the results of the analysis. Thus, it is possible
that two analyses of the same modelling grammar using the same ontological base, but
that follow different processes, may lead to different outcomes.
Lack of objectivity
An ontological analysis of a grammar requires not only detailed knowledge of the selected
ontology and grammar, but also a good understanding of the languages in which the
ontology and the grammar are specified. This requirement explains why most analyses
are carried out by single researchers as opposed to research teams. Consequently, these
analyses are based on the individual interpretations of the involved researcher, which
adds significant subjectivity to the results. This problem is further compounded by the
fact that, unlike other qualitative research projects, ontological analyses typically do
not include attempts to further validate the results.
The five shortcomings identified above have a common flavour in that they heavily
depend on the researcher conducting the ontological evaluation. Three further shortcom-
ings have been identified, namely lack of result representation, lack of result classification