Information Technology Reference
In-Depth Information
3.5.9.
Evaluations
Here, it is a matter of a global evaluation combining the IS part and PS parts.
More specifically, different evaluations were carried out. Three of them are
described in the following; they are within the context of the PREDIM
Mouver.Perso project (Mobility and mUltimodality travelers studying in the Nord-
Pas-de-Calais region - personalized multimodal information system) [GRI 07],
bringing together LAMIH, INRETS and Archimed.
3.5.9.1.
Functional and technical evaluations
The functional evaluations were fairly quick as they had already been carried out
in the implementation phase of the service. We checked in particular that the
functionalities of the service at the end of the integration phase still conformed to
those defined in the stage of capturing functional needs. Technical evaluations of
the service were carried out. The conformity of the HTML code produced in relation
to several browsers was also checked.
3.5.9.2.
Performance evaluation
We carried out tests to evaluate the performance of the personalized itinerary
search service. These tests involved the gathering of service response times for an
itinerary search request. We measured the average response time in relation to the
number of users contained in the user base. Figure 3.23 presents the response times
according to the filtering method used with the profile management agents. A
cognitive method (filtering based on the profile of the user: last made choice) and
two social methods (the first is that of the majority vote; the second is a method of
collaborative filtering based on the preferences and behaviors of users on the basis
of a Bayesian network, put forward in [ANL 06a]) were tested. We expressly chose
these three methods as they are included in the activity model of the profile
management agent for the choice of itinerary.
The results obtained show that when it is a matter of a majority vote, the
response times increase according to the number of responses registered in the
system, but that these times remain acceptable (less than a second for 500 users).
For filtering based on a Bayesian network, the response time is exponential from
100 users. Above 100 users, the response time exceeds 3 seconds. We believe we
can improve performances by improving the algorithm implemented. For filtering
based on the user profile, the number of users registered in the system does not
influence the performance of the system. Other performance evaluations have been
envisaged, for example evaluation of the performance of the service in relation to
the number of users simultaneously connected to the service or study of the impact
of the physical distribution of PerSyst agents on the global performance of the IS.
