Action-Driven Perception for a Humanoid - Agents and Artificial Intelligence - page 84

Fig. 3. Scenario. a) Toy bricks in front of the humanoid robot Nao. The toy bricks exist in four
different shapes, have an identical color and are either light-weight (15 g )orheavy(50 g ). This
results in a total of eight categories that have to be distinguished by the robot. b) Rotation move-
ment with the star-shaped object captured by the robot camera. In the upper row the raw camera
image is shown, whereas the bottom row depicts the preprocessed image that is used to compute
the visual features.

twice, followed by lifting the object up and down three times (thereby altering the pitch
of the shoulder joint by 11.5 ◦ ) and, finally, turning it again twice.
After an action has been completed, the raw image of the lower camera of the Nao
robot is captured, whereas the electric current of the shoulder pitch servo motor is
recorded constantly (sampling frequency 10 Hz ) over the entire movement interval. For
each object category 10 single trial time series are recorded in the described way and
processed in real-time. This yields 80 bi-modal time series in total.

3.2 Data Processing

For the proprioceptive measurements only the mean values are computed for the time
intervals in between movements. The visual processing, on the other hand, involves
several steps (Fig. 3 b), which are accomplished using OpenCV [12]. First, the raw color
image is converted to a binary image using a color threshold. Next, the convex hull is
computed and, based on that, the contour belonging to the toy brick is extracted [13].
For the identified contour the first Hu moment is calculated [14]. Finally, the visual
measurements are scaled to be within the interval [ − 0 . 5 , 0 . 5] .

3.3 Training and Test Data

For testing, the data of single trials are used, i.e. 10 2-D time series per object category
(one dimension for each modality). However, for training, a prototype is determined for
each object category and modality (Fig. 4). To obtain this subclass representative, the
mean value of pooled single trials, with regard to identical object properties, is com-
puted. This means that, for instance, all circular-shaped objects are combined ( n =20 )