Expanding the Local Binary Pattern to Multispectral Images Using Total Orderings (Computer Vision,Imaging and Computer Graphics) Part 1

Abstract. Texture is an important feature for image analysis, segmentation or classification. Since more and more image segmentation problems involve multi-and even hyperspectral data, it becomes necessary to define multispectral texture features. We propose here a natural extension of the classical Local Binary Pattern (LBP) operator to the case of multispectral images. The Loc al Multispectral Binary Pattern (LMBP) operator is based on the definition of total orderings in the multispectral image space and on an extension of the standard univariate LBP.

It allows the computation of both a multispectral texture structure coefficient and a multispectral contrast parameter distribution. Results are demonstrated in the case of the segmentation of brain tissues from multispectral MR images, and compared to other multispectral texture features.

Keywords: Local binary pattern, Multispectral image, Segmentation, Texture,Total orderings.

Introduction

Texture analysis plays an important role in several applications, from remote sensing to medical image processing, industrial applications or document processing. Four major issues are commonly expected: feature extraction, texture discrimination, texture classification and shape from texture. To achieve these analysis, several methods are available, using statistical (autocorrelation, co-occurrence), geometrical (structural techniques), model-based (MRF or fractal) or signal processing based approaches (spatial, Fourier, Gabor or wavelet filtering) (see for example [30] for a review). Among all these methods, the Local Binary Pattern (LBP) operator offers an efficient way of analyzing textures [22]. It relies on a simple but efficient theoretical framework, and combines both structural and statistical properties.


In the prementioned potential applications, multispectral data is more and more available and the extension of texture analysis methods becomes natural, either from a theoretical or an applicative point of view. For example, it is expected that a multispectral texture analysis on remote sensing images will improve segmentation of e.g. land cover types [23].

Several works already attempt to do multispectral texture analysis. Pla et al. [25] used a Multi-spectral Local Differences Texem (MLDT) model and an inter-scale post fusion process allowing multispectral images with textures to be segmented. Münzenmayer et al. [21] proposed an extension of the gray level sum- and difference histogram features to the case of the fusion of texture and color information. This particular issue has also been addressed using Gaussian Markov Random Fields [6], statistical modeling approaches [9,16], generalized [14] or spectral [11] co-occurrence matrices or even color codebooks [28].

We propose here to extend the LBP operator to the case of multispectral images. Only few works addressed the multispectral texture analysis problem using Local Binary Pattern. Maenpaa et al. [20] processed color and texture information using an Opponent Color Local Binary Pattern operator (OCLBP) and concluded that color and texture are phenomena that can – or even should – be treated separately. Lucieer et al. [18] proposed a measure based on LBP, and considered spatial interactions not only in one image (intra-plane) but also between bands (inter-plane). The neighborhood set of a pixel was here composed of the local neighborhood in all images. Song et al. [29] considered the classical univariate LBP formula, but applied the operator on the Euclidean norms of feature vectors. Han et al. [10] as for them proposed an improved multi channel local binary pattern in RGB color space by concatenating histograms produced by monochannel LBP at three channels. Contrary to these work, we do not apply the univariate LBP on scalar values derived from the multispectral dataset, but directly propose a multispectral LBP operator, based on a total ordering computed either in the images space, or in a derived vectorial space.

The paper is organized as follows: section 2 first recall the original univariate LBP operator, and some basic definitions on orderings in a vectorial space. It then introduces the LMBP – Local Multispectral Binary Pattern, that uses both of these notions. Section 3 presents and analyzes some preliminary results of the operator on data stemming from multispectral Magnetic Resonance Images.

Local Multispectral Binary Pattern

Local Binary Pattern

Ojala et al. [22] described the texture T as the joint distribution of the gray levels of P + 1 image pixels:tmp5839136_thumbwhere  gc is the gray level value of the center pixel, surrounded by P equally spaces pixels of gray levels gp, located on a circle of radius R. Gray values gp were interpolated if neighbors didn’t fit on the pixel grid. They then defined the Local Binary Pattern (LBP), a grayscale invariant and rotation invariant operator:

tmp5839138_thumb

where

tmp5839139_thumb

and σ(.) is the sign function. The uniformity function U(LBPp,R) corresponds to the number of spatial transitions in the neighborhood: the larger it is, the more likely a spatial transition occurs in the local pattern.

If LBPtp1R1 captures the spatial structure of the texture, it does not handle the strength of the pattern. To do so, a contrast measure was defined as

tmp5839140_thumb

and textures may then be characterized by the joint distribution of LBPtp1R1 and CP,R. Several extensions have been proposed to these features (e.g. [32], multi-resolution LBP [19,22], center-symmetric local binary pattern [12]), and numerous applications have been adressed using these techniques (e.g. face recognition [1], segmentation of remote-sensing images [31], visual inspection [24] or classification of outdoor images [7] or textures [17]).

The LBP operator relies on the sign function σ(.), and then on an ordering relation on the gray level space. Since there is no natural ordering for vector spaces, such as those produced by multispectral imaging, the extension of LBP to multispectral data is not straightforward. Some authors already defined multispectral LBP by combining intra- and inter-plane LBP relations [18], or by considering the univariate LBP on vector norms [29], but to our knowledge no LBP operator has directly be defined on vectorial data. We thus propose in the following to define the LMBP – Local Multispectral Binary Pattern – operator, based on LBP and on total orderings on R".

We first recall basic definitions on orders and then introduce the LMBP operator.

Total Orderings in R”

We first recall some basic definitions.

Definition 1. Let <P be a binary relation on a set P. <P is a pre-order if it is:

tmp5839141_thumb

Definition 2. Let <P be a binary relation on a set P. <P is a partial order if it is a pre-order and for all x,y G P, if x <P y and y <P x then x = y (antisymmetry).

Definition 3. Let <P be a partial order on a set P. <P is a total order if and only if for all x,y G P, x <P y or y <P x.

If it is straightforward to define orders for scalar values, the definition of partial -or total-orders for vector valued data is not so easy: if data stems from RGB images (P = R3),each channel being coded on 8 bits, each pixel can have one of the 224 possible vectorial values, hence defining 224! possible total orderings.

Examples of Space Filling Curves in the case n=2

Fig. 1. Examples of Space Filling Curves in the case n=2

One has then to find another way to introduce order in Rn. Barnett [3] defined four ways to order vectors: the marginal approach, the partial approach, the conditional order and the dimension reduction. This last technique is an usual way to proceed [8], and consists either in defining an order using a distance in Rn of each vector to a reference, or in projecting vectors into a vectorial space R9 where an order can be defined. In this latter case, the projection is defined by an application h : Rn ^ R9, and [4] proved that:

- h defines an ordering relation <n in Rn if and only if h is injective (and h can be supposed to be bijective if R9 is restricted to h(Rn))

- <n defines a total order in Rn if and only if there exist h : Rn ^ R9 bijective defining <n on Rn and q=1

-    <n defines a total order in Rn if and only if <n defines a space filling curve of Rn

Total ordering may then be identically handled by the definition of h, or by the construction of a space filling curve of Rn (see figure 1 for examples in the case n = 2). In the following, we propose as a preliminary study to define <n using an appropriate h.

The LMBP Operator

Figure 2 presents an overview of the algorithm. Each step is detailed in the following subsections.

Subspace Analysis. Since numerous information may be available from the original dataset of Rp, and since the p original images may be dependent, it may be useful to conduct feature extraction before defining the vector ordering. Several techniques are available to extract features (Principal or Independent Component Analysis, Minimum Noise Fraction, or nonlinear techniques such as Isomap or Local Linear Embedding). In a preliminary study, we used a simple and linear technique, the Principal Component Analysis (PCA), that will directly impact the choice of the h function: if X G Mm,P(R) denotes the matrix of the original data (m being the number of pixels, p the number of images and X.ti the ith column of X), the principal components are computed from the projections of the original data onto the eigenvectors of ZtZ, where Z = (X – 1μ)Ό is the matrix of centered and reduced data,tmp5839143_thumbis the vector of image mean values andtmp5839144_thumbis    the standard deviation of variable i. PCA allows n < p new variables to be computed, explaining of the variance of the original data.

Overview of the algorithm most

Fig. 2. Overview of the algorithm mosttmp5839148_thumb

The resulting information is stored intmp5839149_thumb

Definition of h. Recall thattmp5839150_thumban easily way to compute an ordering relation in R" is to derive it from h using the canonical ordering relation oftmp5839151_thumb

tmp5839156_thumb

This ordering relation is a partial order, and in the context of image processing may lead to several drawbacks: some vectors may not be ordered, and notions of Sup and Inf are not defined (and are compulsory in areas such as mathematical morphology). In our context, we thus decide to define a total ordering in R", then using q = 1 and h injective. Several choices are then possible (e.g. the bit mixing approach [4]), and we benefit from the new basis computed from PCA to use the lexicographic order in R", defined as:

tmp5839157_thumb

Figure 3 presents the corresponding space filling (n=2) curve and the associated h function, if each image is coded on b bits.

Computation of LMBP. Once h has been defined, an ordering relation on R" can be proposed as:

lexicographic order: (a): h function and (b): the corresponding 2D space filling curve (n=2)

Fig.3. lexicographic order: (a): h function and (b): the corresponding 2D space filling curve (n=2)

tmp5839159_thumb

Unfortunately, enforcing a total ordering on Rn makes h discontinuous: the Netto theorem [26] indeed proved that any bijective application from a manifold of dimension n to a manifold of dimensiontmp5839160_thumbis    discontinuous.    Thus,    h is not a linear function,and does not commute with linear functions. It is thus not straightforward to assess h(x) — h(y), given x — y in Rn, and any linear combination of vectors transformed by h should be avoided. If this is not really a problem for the definition of the LBP operator tmp5839161_thumbshould easily be replaced withtmp5839162_thumbbecause the minus sign only means to compare h(gp) with h(gc) using the ordering defined by h), the extension of Cp,R to the case of multispectral images is more problematic, because of both g and the deviation to this mean value. In order to be compliant with the discontinuity of h, we thus avoid any linear combination, and replace g by the median value mp of the gp’s. The LBMP operator is then

tmp5839166_thumb

where

tmp5839167_thumb

and the contrast operator

tmp5839168_thumb

CP,R is still a combination oftmp5839169_thumb. but we expect the nonlinearity introduced by mp to reduce the discontinuity effects.

If h is computed from the lexicographic order, the sign functionstmp5839171_thumbin the previous expressions reduce to

tmp5839173_thumb

The first components of x and y play here an important role, as their difference is weighted bytmp5839174_thumbbut the other components, with weightstmp5839175_thumbmay

invert the sign of the argument of the σ function. Subspace analysis is thus a crucial step in the whole process for the selection of relevant components.

From LMBP to Segmentation. Once LMBP and contrast have been computed for each pixel location, we were interested in image segmentation using these features. Several approaches can be performed, e.g.:

-    incorporate these operators in a vector describing the pixel properties, with other relevant values (e.g. values of the first principal components). The set of these vectors then serves as an input of an unsupervised clustering algorithm

-    compute a local 2D joint distribution (LMBP,CP,R) for each pixel, and use an adapted metric to cluster pixels.

In this preliminary study, we chose to use the first alternative. More precisely, a classical K-means algorithm was used as a clustering method, using the Euclidean metric to cluster feature vectors that the next section will detail.

Next post:

Previous post: