Information Technology Reference
In-Depth Information
user behavior though prepared log sequences that consist of activities and mes-
sages. The sequences serve as input to the Probabilistic Process Engine which is
initiated with the original scenario process.
6.1 Experiment Setup and Success Metrics
Two quality metrics measure the success of our learning and adaptation mech-
anism. The Message Classification Error (
MCE
in range [0
,
1] ) determines for
each incoming unknown message during the experiments how close our message
prediction algorithm gets.
MCE
= 0 when the actual message type and highest
ranked predicted message type are identical, otherwise we extract the matching
message type from the ranking list and take the inverse of its expected occur-
rence probability 1
p
occur
) (i.e., the lower its probability, the higher the
MCE
).
Suppose for an incoming message the algorithm produces following ranking re-
sult [(7) : 0
.
5]
,
[(8) : 0
.
4]
,
[(2) : 0
.
1]: MCE would yield 0
.
6 when the message is
actually of type (8). For measuring the effect towards the user, we apply the
Activity Recommendation Error (
ARE
in range [0
,
1]) which is analogue to the
MCE
: when the next element in the log sequence is an activity, we retrieve an
activity recommendation and locate the matching activity. We keep the activ-
ity recommendation mechanism intentionally simple to focus only on the effect
of the message state model and probabilistic flow model. Active activities are
ranked based on their time since activation, thus the longer an activity remains
unfinished, the higher it will score.
Two activity message log sequences describe an evolution of the initial process
model. The
Quote
(3) message is no longer used, the
Agreement
(5a) message
is delayed until completion of the
PrepareShipment
(F) activity which also trig-
gers the
Authorization Of Invoice
message (6). The
Invoice
(7) is produced when
executing
Priority Dispatch
(J). The
Delivery Note
(8) only applies to
Regular
Dispatch
(H). During all experiments, we set EWMA coecients
α
=0
.
3and
β
=0
.
4 which is a trade-off between rapid uptake of novel user behavior and ro-
bustness against one-time deviations. We set
β>α
as the arcs need to learn and
forget message changes quicker than the state model (which tracks the message
probabilities across the whole process and not just a single, local arc occurrence).
LogSequenceA:1,A,2,B,C,D,4,E,5b,F,6,5a,G,J,7
LogSequenceB:1,A,2,D,4,B,C,E,5b,F,6,5a,G,H,8
−
6.2 Results
In experiment 1 (Figure 6 a and c), we play each of the log sequences 30 times
against the scenario process (dotted lines, green +, and blue x) — as well as
alternating each sequence (full lines, red circles) — and determine overall
MCE
p
and
ARE
p
(sum of all recommendation errors within one process instance).
MCE
p
rapidly drops to zero within a few iterations for individual log se-
quences A and B. The interleaving sequences take longer to produce a stable
process model as sequences A and B display an opposing user behavior (to the
