Skin Color in Face Analysis (Face Image Modeling and Representation) (Face Recognition) Part 3

Color Cue for Face Detection

As mentioned above, color is a useful cue for face detection as it can greatly reduce the search area by selecting only the skin-like regions. However, it is obvious that the use of skin color only is not enough to distinguish between faces and other objects with a skin-like appearance (such as hands, wood, etc.).

Fig. 9.15 The face tracking based on Raja et al.’s method failed and adapted to a nonfacial target. The left image displays the “localized face”. The right image shows the pixels selected by the current skin color model. The red box shows the pixels used for refreshing the model

Fig. 9.16 The constraint suggested by Raja et al.’s selects a nonpresentative set of skin pixels

Therefore, other procedures are needed to verify whether the selected regions are (or contain) faces or not. Depending on the robustness of the skin model and changes in the illumination conditions, one can notice two cases:

•    Case #1: The initial skin color detection step produces consistently reliable results. The skin color model is valid for the illumination conditions, the camera and its settings. The skin color model can be designed either for stable, controlled illumination (typical case) or for variable illumination (skin locus). In such cases, it is generally enough to consider each connected resultant component from the skin detection as a face candidate. Then, one can verify the “faceness” of the candidate by simple and fast heuristics.

•    Case #2: The initial skin color detection step produces unsatisfactory results or even fails. In this case, the skin color model does not correspond to the prevailing illumination, used camera or settings of the camera. One can hope that the results would indicate the locations of the faces, but their size estimation is too unreliable. Therefore, a different method for face detection (either an appearance-based or feature-based one) should be used when searching for the faces in and around the detected skin regions.

In both cases, the use of color accelerates the detection process. In the following, we review some methods based on color information for detecting faces. Most of the color-based face detectors start by determining the skin pixels which are then grouped using connected component analysis. Then, for each connected component, the best fit ellipse is computed using geometric moments, for example. The skin components which verify some shape and size constraints are selected as face candidates. Finally, features (such as eyes and mouth) are searched for inside each face candidate based on the observation that holes inside the face candidate are due to these features being different from skin color. Therefore, most of the color-based face detection methods mainly differ in the selection of the color space and the design of the skin model. In this context, as seen in Sect. 9.5, many methods for skin modeling in different color spaces have been proposed. For comparison studies, refer to [35, 46] and [34].

Among the works using color for face detection is Hsu et al.’s system which consists of two major modules: (1) face localization for finding face candidates, and (2) facial feature detection for verifying detected face candidates [15]. For finding the face candidates, the skin tone pixels are labeled using an elliptical skin model in the YCbCr color space, after applying a lighting compensation technique. The detected skin tone pixels are iteratively segmented using local color variance into connected components which are then grouped into face candidates. Then, the facial feature detection module constructs eye, mouth and face boundary maps to verify the face candidates. Good detection results have been reported on several test images. However, no comparative study has been made thus far.

In [7], Garcia and Tziritas presented another approach for detecting faces in color images. First, color clustering and filtering using approximations of the YCbCr and HSV skin color subspaces are applied to the original image, providing quantized skin color regions. Then a merging stage is iteratively performed on the set of homogeneous skin color regions in the color quantized image, in order to provide a set of face candidates. Finally, constraints related to shape and size of faces are applied, and face intensity texture is analyzed by performing a wavelet packet decomposition on each face area candidate in order to detect human faces. The authors have reported a detection rate of 94.23% and a false dismissal rate of 5.76% on a data set of 100 images containing 104 faces. Though the method can handle nonconstrained scene conditions, such as the presence of a complex background and uncontrolled illumination, its main drawback lies on that fact that it is computationally expensive due to its complicated segmentation algorithm and time-consuming wavelet packet analysis.

Sobottka and Pitas presented a method for face localization and facial feature extraction using shape and color [42]. First, color segmentation in HSV space is performed to locate skin-like regions. After facial feature extraction, connected component analysis and best fit ellipse calculation, a set of face candidates are obtained. To verify the “faceness” of each candidate, a set of eleven lowest-order-geometric moments is computed and used as inputs to a neural network. The authors reported a detection rate of 85% on a test set of 100 images.

Fig. 9.17 Examples of face detection results using the color-based face detector in [10]

In [11], Haiyuan et al. presented a different approach for detecting faces in color images. Instead of searching for facial features to verify the face candidates, the authors modeled the face pattern as a composition of a skin part and a hair part. They made two fuzzy models to describe the skin color and hair color in CIE XYZ color space. The two models are used to extract the skin color regions and the hair color regions which are compared with the prebuilt head-shape models by using a fuzzy theory based pattern-matching method to detect the faces.

In [10], Hadid et al. presented an efficient color-based face detector, using the skin locus model to extract skin-like region candidates, and then performing the selection by simple yet efficient refining stages. After ellipse fitting and orientation normalization, a set of criteria (face symmetry, presence of some facial features, variance of pixel intensities and connected component arrangement) are evaluated to keep only facial regions. The refining stages are organized in a cascade to achieve high accuracy and to keep the system fast. The system was able to detect faces and deal with different conditions (size, orientation, illumination and complex background). Figure 9.17 shows some detection examples performed by the system under different conditions.

Several other approaches using color information for detecting and tracking faces and facial features in still images and video sequences have been proposed [13, 54].

Fig. 9.18 Examples of face detection results using the color-based face detector in [9]

It appears that most of the methods have not been tested under practical illumination changes (usually only mild changes are considered), which makes them belonging to the first category (Case #1) described above.

More recently, to detect faces in natural and unconstrained environments, Hadid and Pietikanen [9] proposed an approach which considers the fact that color is a very powerful and useful cue for face detection, but unfortunately, it may also produce unsatisfactory results or even fail. The proposed approach consists of first preprocessing the images to find the potential skin regions, avoiding thus scanning the whole image when searching for faces, and then performing an exhaustive search in and around the detected skin regions. The exhaustive search is performed using a two-stage SVM based approach, exploiting the discrimination power of the Local Binary Patterns (LBP) features. The obtained results are interesting in the sense that the proposed approach inherits the speed from the color-based methods and the efficiency from the gray scale-based ones. Some detection results are shown in Fig. 9.18.

One problem of color-based face detectors lies in the fact that they are generally camera specific. Most of the methods have reported their results on specific and limited data sets and this fact does not facilitate performing a comparative analysis between the methods.

Currently, most methods for face detection rely only on gray scale information even when color images are available. Generally these methods scan the images at all possible locations and scales and then classify the sub-windows either as face or nonface, yielding in more robust but also computationally more expensive processing methods, especially with large-sized images. Among robust approaches based only on gray scale information is Viola and Jones’s approach [49]. The approach uses Haar-like features and AdaBoost as a fast training algorithm. AdaBoost is used to select the most prominent features among a large number of extracted features and construct a strong classifier from boosting a set of weak classifiers. Such systems generally run in real-time for small-sized images (e.g., 240 x 320 pixels), but tend to be slow for larger images. Including other cues such color or motion information may thus be very useful for speeding-up the detection process.

Color Cue for Face Recognition

The role of color information in the recognition of nonface objects has been the subject of much debate. However, there has only been a small amount of work which examines its contribution to face recognition. Most of the work has only focused on the luminance structure of the face, thus ignoring color cues, due to several reasons.

The first reason lies in the lack of evidence from human perception studies about the role of color in face recognition. Indeed, a notable study in this regard was done in [23], in which the authors found that the observers were able to quite normally process even those faces that had been subjected to hue-reversals. Color seemed to contribute no significant recognition advantages beyond the luminance information. In another piece of work [56], it is explained that the possible reason for a lack of observed color contribution in these studies is the availability of strong shape cues which make the contribution of color not very evident. The authors then investigated the role of color by designing experiments in which the shape cues were progressively degraded. They concluded that the luminance structure of the face is undoubtedly of great significance for recognition, but that color cues are not entirely discarded by the face recognition process. They suggested that color does play a role under degraded conditions by facilitating low-level facial image analysis such as better estimations of the boundaries, shape and sizes of facial features [56].

A second possible reason for a lack of work on color-based face recognition relates to the difficulties of associating illumination with white balancing of cameras. Indeed, as discussed in Sect. 9.3, illumination is still a challenging problem in automatic face recognition, therefore, there is no need to further complicate the task.

A third possible reason for ignoring color cues in the development of automatic recognition systems is the lack of color image databases1 available for the testing of the proposed algorithms, in addition to the unwillingness to develop methods which cannot be used with the already existing monochrome databases and applications.

However, the few attempts to use color in automatic face recognition includes the work conducted by Torres et al. [48] who extended the eigenface approach to color by computing the principal components from each color component independently in three different color spaces (RGB, YUV and HSV). The final classification is achieved using a weighted sum of the Mahalanobis distances computed for each color component. In their experiments using one small database (59 images), the authors noticed performance improvements for the recognition rates when using YUV (88.14%) and HSV (88.14%) color spaces, while a RGB color space provided the same results (84.75%) when using R, G or B separately and exactly the same results as using the luminance Y only. Therefore, they concluded that color is important for face recognition. However, the experiments are very limited, as only one small face database is used and the simple eigenface approach is tested.

In another piece of work that deals with color for face recognition [20], it has been argued that a performance enhancement could be obtained if a suitable conversion from color images to a monochromatic form would be adopted. The authors derived a transformation from color to gray-scale images using three different methods (PCA, linear regression and genetic algorithms). They compared their results with those obtained after converting the color images to a monochromatic form by using a simple transformationand they noticed a performance enhancement of 4% to 14% using a database of 280 images. However, the database considered in the experiments is rather small, thus, one should test the generalization performance of the proposed transformation on a larger set of images from different sources.

In [40], Rajapakse et al. considered an approach based on Nonnegative Matrix Factorization (NMF) and compared the face recognition results using color and gray scale images. On a test set of 100 face images, the authors have claimed a performance enhancement when using also color information for recognition.

In [19], Jones has attempted to extend the Gabor-based approach for face recognition to color images by defining the concept of quaternions (four component hypercomplex numbers). On a relatively limited set of experiments, the author has reported a performance enhancement on the order of 3% to 17% when using the proposed quaternion Gabor-based approach instead of the conventional monochromatic Gabor-based method.

Very recently, color face recognition has been revisited by many researchers, with an aim to discover the efficient use of color for boosting the face recognition performance. For instance, inspired by the psychophysical studies indicating that color does play a role in recognizing faces under degraded conditions, Choi et al. [58] carried out extensive experiments and studied the effect of color information on the recognition of low-resolution face images (e.g., less than 20 x 20 pixels). By comparing the performance of grayscale and color features, the results showed that color information can significantly improve the recognition performance.

Yang et al. [55] compared the discriminative power of several color spaces for face recognition and found out that different color spaces display different discriminating power. Experiments on a large scale face recognition grand challenge (FRGC) problem also revealed that the RGB and XYZ color spaces are weaker than the I1I2I3, YUV, YIQ color spaces for face recognition. The authors proposed then color space normalization techniques for enhancing the discriminative power of different color spaces.

For color based face verification, Chan et al. [2] proposed a discriminative descriptor encoding the color information of the face images. The descriptor is formed by projecting the local face image acquired by multispectral LBP operators, into LDA space. The overall similarity score is obtained by fusing local similarity scores of the regional descriptors. The method has been tested on the XM2VTS and FRGC 2.0 databases with very promising results.

Liu and his colleagues extensively investigated the problem of color face recognition and reported very good results on FRGC database (Version 2 Experiment 4) [27-30, 52, 53]. For instance, in [27], the authors first derived new (uncorrelated, independent and discriminating) color spaces from the RGB color space by means of linear transformations. Then, vectors are formed in these color spaces by concatenating their component images to form augmented pattern vectors, whose dimensionality is reduced by PCA. Finally, an enhanced Fisher model (EFM) is used for recognition. The obtained results are better than those of methods using grayscale or RGB color images. In [29], the authors considered a hybrid color space by combining the R component image of the RGB color space and the chromatic components I and Q of the YIQ color space. Experiments on the Face Recognition Grand Challenge (FRGC) version 2 Experiment 4 showed the hybrid color space significantly improves face recognition performance due to the complementary characteristics of its component images. Since most of the experiments conducted by Liu and his team were mainly using the FRGC database, it is of interest to see how well the proposed methods generalize to other databases and settings.

Conclusions

Color is a useful cue in facial image analysis. Its use for skin segmentation and face detection is probably the most obvious, while its contribution to face recognition is not very clear. The first important issues when planning the use of color in facial image analysis are the selection of a color space and the design of a skin model. Several approaches have been proposed for these purposes, but unfortunately, there is no optimal choice. The choice made depends on the requirement of the application and also on the environment (illumination conditions, camera calibration, etc.).

Once a skin model has been defined, the contribution of color to face detection, not surprisingly, plays an important role in pre-processing the images and in the selection of the skin-like areas. Then, other refining stages can also be launched in order to find faces among skin-like regions. Color-based face detectors could be significantly much faster than other detectors which are based solely on gray-scale information, especially with large-sized images.

In relation to the contribution of color to face recognition, the issue is still under debate and among the open questions are: is color information useful for face recognition at all? If yes, how the three different spectral channels of face images should be combined to take advantages of the color information? What is the optimal color space which provides the highest discriminative power, etc.? The current results suggest that color cue has not yet shown its full potential and need further investigation. Therefore, it perhaps makes sense for current automatic face recognition systems not to rely on color for recognition because its contribution is not well established yet.