Applications to Safe Human–Robot Interaction - 3D Computer Vision: Efficient Methods and Applications - page 304

Fig. 7.20 Two-level architecture of the action recognition stage

Hahn et al. ( 2009 , 2010b ) propose a recognition stage for working actions, the
description of which is adopted here from Hahn et al. ( 2010b ). The method is based
on the classification and matching of three-dimensional trajectories given by the
sequence of three-dimensional positions of the hand. When the mean-shift method
is used, the trajectories of all moving objects in the scene are analysed. The cyclic
sequence of working actions in an engine assembly scenario is known to the system
but may be interrupted by 'unknown' motion patterns. To allow an online action
recognition, a sliding window approach is applied. Due to the fact that the system
is designed for safe human-robot interaction, a recognition stage with two levels
is implemented (Fig. 7.20 ). At the first level, a decision is made as to whether the
human worker performs a known working action (regular mode) or an unknown
motion (safety mode) based on a set of trajectory classifiers. In the safety mode
(level 1), the system may prepare to slow down or halt the industrial robot. The
regular mode (level 2) defines the cyclic working process performed by the human
worker. It is implemented as an HMM in which the hidden state is continuously
estimated by a particle filter.
The state of level 1 according to Fig. 7.20 is determined by a set of classifiers
based on features extracted from the trajectory data in the sliding window of a size
of eight time steps for all tracked objects. Movements between two working actions
(transfer motion) are recognised by a transfer classifier. Since it is known at which
position the worker has to tighten a screw or to fit a plug, a second classifier (work-
ing action classifier) is used for recognising working actions by incorporating spatial
context for the actions 'screw 1', 'screw 2', 'clean', and 'plug'. The reference loca-
tions are obtained based on the known three-dimensional pose of the engine. A third
classifier (distance classifier) is applied to the result of the working action classifier
and decides whether the recognised working action is a known one, since such mo-
tion patterns can only occur close to the three-dimensional object associated with
that action. The combination of the three classifiers results in an output discriminant
vector for the six classes.
According to the description by Hahn et al. ( 2010b ), the decision as to whether
the system is in safety mode or in regular mode is made based on the result of the
distance classifier and the matching accuracy of the particle weights in level 2, where
the observed trajectories are analysed with respect to the occurrence of known work-
ing actions. Similar to the work of Li et al. ( 2006 ), a particle filter-based matching of
a non-stationary HMM is applied in order to recognise the sequence of working ac-
tions. The HMM of level 2 (cf. Fig. 7.20 ) is derived from the known cyclic working