Information Technology Reference
In-Depth Information
Fig. 7.
Correctness and Completeness metrics
The results of the evaluation are shown in Table 1. The consultants' estimates for
performing the service identification without tool support ranged from 2:30 hours to
4:20 hours. The experiment revealed that using the S³C Solution Explorer consultants
needed between 45 minutes and 1:20 hours to actually conduct the task. Consultants
were able to indentify correct services
apart from one exception; all identified
services were included in the standard solution. Services completeness ranged from
50% to 80%. On the average consultants identified 62% of the relevant services.
−
Table 1.
InsideAx Evaluation Results
Correctness
Completeness
Consultant 1
02:30
01:05
100%
100%
100%
60%
80%
50%
Consultant 2
04:00
00:45
100%
100%
100%
60%
60%
66%
Consultant 3
04:20 01:20 75% 100% 100% 60% 60% 66%
The results reveal that the S
3
C Solution Explorer enables consultants to discover
relevant services in significantly less time than their estimates for service identifica-
tion without tool support. In debriefing meetings consultants said that service identifi-
cation with the help of S
3
C Solution Explorer was faster because services were
presented using a ranked list with matching probabilities.
The two experienced consultants had a 100% correctness rate. In the debriefing
meeting they argued that having service descriptions and requirements presented next
to each other did support their matching process and did speed up the actual decision.
However, they also mentioned that their experience did support them in taking deci-
sions. The two experienced consultants argued that their knowledge about ERP sys-
tems did in particular support them in taking the right decision when the provided
service description was limited (e.g. the Inventory Transaction Service was described
as “Describes the inventory transactions document”). The less experienced consultant
(Consultant 3) argued that he needed more time to make decisions. He said that in
most of the cases the provided service description did allow him to make a correct
decision. However, he also mentioned that more information on the services would
have been helpful and would have made him more confident in his decisions.
