Abstract. We provide a deformable model for particle analysis. We investigate particle images from a backlit microscope system where particles suffer from out-of-focus blur. The blur is a result of particles being in front or behind the focus plane, and the out-of-focus gives a bias towards overestimating the particle size. This can be handled by only including the particles in focus, but most of the depicted particles will be left out of the analysis, which weakens the statistical estimate of the monitored process. We propose a new method for particle analysis. The model incorporates particle shape, size and intensity, which enables an estimation of the out-of-focus blur of the particle. Using the particle model parameters in a regression model we are able to infer 3D information about individual particles. Based on the defocus information we are able to infer the true size and shape of the particles. We demonstrate the capabilities of our model on both real and simulated data, and our approach shows promising results for a reliable particle analysis. The potential is more process information obtained over shorter sampling time.
Keywords: Particle analysis, Deconvolution, Depth estimation, Microscopic imaging.
Introduction
Knowledge about individual particles can be essential in industrial process optimization. We address the problem of analyzing images of individual particles. Application examples include suspended particles in for example a fermentation process, oil droplets in water, coal particles in a power plant, and spray particles in air. Two image examples are given in Fig. 1. A vision-based system can provide knowledge about particle distribution, size and shape, and these parameters are often essential for system design or process control. Here the choice of the analysis method and the quality of the images are essential elements, so both analysis method and image acquisition should be chosen carefully.
Motivation
The motivation of our work is an industrial endoscopic inspection system equipped with a probe that can be placed inside the process1. Images are acquired from the tip of the probe, which also contains a light source placed in front of the camera.
Fig. 1. Examples of particle images. (a) spherical transparent particles all 25 μm in diameter, and (b) a typical image to be analyzed depicting spray particles.
The resulting camera setup depicts particles as shadows, see Fig. 1. The visual appearance of the particles depends on the optical properties of the camera setup, the distance of the particles to the focus plane, and the physical reflectance properties of the particles. The depth of field of the camera optics is narrow and the particles get blurred as they move away from the focus plane, which introduces uncertainty of the particle characterization, c.f. Fig. 2. A tendency for overestimating particle size is reported in [11]. Employing a strategy where only in-focus particles are analyzed can be a good solution [5,10], but in situations with few particles or short inspection time this approach will give an uncertain estimate due to low sample size. Therefore it can be necessary to perform the analysis of the blurred particles as well.
Related Work
Our particle analysis approach combines the two elements of deblurring and shape characterization. The three dimensional nature of our problem does not allow the image acquisition system to have all particles in focus. Therefore we need to handle the out-of-focus blur, but this also provide information about the depth of the particles, which we use for estimating the spatial information about the particles. We will now address related work on deblurring followed by a discussion on particle shape modeling.
Deblurring. In a linear system the image formation can be described as the linear convolution of the object distribution and the point spread function (PSF). Hence, to reduce the blur from out-of-focus light, ideally the mathematical process of deconvolution can be applied. However, noise can easily be enhanced if one just implements a direct inverse operation, so the inverse has to be regularized. Different regularizers can be employed, for example iteratively deconvolving the image [14], [16], or using a Wiener filter [20]. Alternatively, a maximum entropy solution can be chosen, which aims at being mostly consistent with data [15], [19]. These methods assume a known PSF. When this is not the case, blind deconvolution can in some cases be applied recovering both the PSF and the deconvolved image. Typically this is solved by an optimization criterion based on known physical properties of the depicted object [9].
These methods are based on the assumption of a known – possibly space-dependent -PSF for the image. For many optical systems it is difficult to calculate a theoretical PSF with sufficiently accuracy to be used for deconvolution.
Fig. 2. Illustration of the particles relative to the focus plane. (a) particles in the 3D volume. (b) illustration of appearance change as a function of the distance to the focus plane.
Also it can be quite difficult to measure it experimentally with sufficient resolution and accuracy. In our case the particles of concern are illuminated from the back and in this respect it resembles the case of bright light microscopy. Such an imaging system is not exactly a linear device but in practice it is almost so. However, in the bright field setting the ’’simple” PSF is compounded by absorptive, refractive and dispersal effects, making it rather difficult to measure and calculate it.
One method for local image deblurring, which is needed for our problem, include iteratively estimating the blur kernel and updating the image accordingly in a Bayesian framework [17]. Another approach is to segment the image and estimate an individual blur kernel for the segments [2,12]. Blur also contains information about the depicted objects. This has been used by [4,18], where they obtain motion information by modeling blur. With a successful deblurring, e.g. based on one of these methods, we will still have to identify the individual particles. Instead, we suggest here to build a particle model.
Particle Modeling. Most particles have a fairly simple structure, typically being convex and close to circular or elliptical. This observation can be used for designing a particle model. In [6] a particle model is build for nanoparticles based on images obtained from an electron microscope. An elliptical model is aligned with the particles by maximizing the contrast between the average intensity of the particle and a surrounding narrow band. Particles in these images are naturally in focus.
Ghaemi et al. [7] analyze spray particles using a simple elliptical model. However, only in-focus particles are analyzed, and out-of-focus particles are pointed out as a cause of error. In addition, they mention the discretization on the CCD chip to be problematic, and argue that particles should be at least 40-60 pixels across to enable a good shape characterization. Blaisot and Yon [1] analyze diesel spray particles based on an image model derived from the point spread function for individual particles. They detect individual particles and detect the average particle diameter by applying a threshold based on the maximum and minimum intensity in a window around the particle. Furthermore they compensate for the out-of-focus blur by two additional thresholds, which they use for estimating the points spread. Furthermore they introduce morphological shape characterizations, that describes how much the shape deviates from a sphere.
We employ a deformable particle model for characterizing particles and account for out-of-focus blur. The parameters of our model encodes the shape of the particle in a natural manner. Based on the assumption that images are smooth we a able to obtain reliable shape and size information from particles smaller than 40-60 pixels in diameter. The main focus of this paper is our particle model, which we use for characterizing particle shape, size and blur. In Section 3 we describe our particle model and how it can be used for particle characterization. We experimentally validate the particle model in Section 4. Lastly, in Section 5 we discuss the obtained results, and we conclude the work in Section 6.
Contribution
This paper is an extended version of our previous paper [3], where we demonstrated that particles size, shape and their distance to the focus plane reliably can be inferred using our deformable model. The contributions of this paper are
1. A modified and more robust particle model.
2. A higher level of detail in the model description.
3. Demonstration of the model performance on spray data.
These extensions makes the model more applicable for particle characterization and easier to implement and use in practical applications.
Method
The goal of the proposed method is to obtain information about the true size and shape of an out-of-focus particle. Our idea is to learn particle appearance from observations of particles with known position relative to the focus plane. By comparing the appearance of an unknown particle to the training set, we can predict how the particle would appear, if it was in focus. As a result we obtain information about the true particle size and shape.
To facilitate this, the particles must be characterized in a way that describes the appearance as a function of blur well. Furthermore, particles should be easy to compare. We will now give a short description of how particles are depicted, and then explain the details of our particle model and descriptor. Finally we describe the statistical model for depth estimation.
Experimental Setup. The particle analysis is based on backlight where the particles appear as shadows. Real image examples are shown in Fig. 1. In Fig. 2 the experimental setup is illustrated. Notice that all particles in Fig. 1(a) are the same size of 25^m, but the blur makes them appear very different. Out-of-focus blur occurs both in front and behind the focus plane, but it is hard to tell if an observed particle is in front or behind, because the blur looks the same. As a consequence we have chosen to model the particles as a function of absolute distance to the focus plan, which is shown in Fig. 2. In Section 4 we experimentally show that these are reasonable assumptions.
Fig. 3. Intensity sampling with the particle model. Radial sampling pattern of our model with 10 sampling steps from the center point, marked with red, to the end of the radial line marked with blue. There are 8 radial sampling lines in this example (a). Each radial sampling line can be deformed by stretching or compressing the line while keeping equal distance between the sampling points (b). This stretch of the individual sampling lines is what deforms our model. In (c) the model is placed over a particle and in (d) the model is deformed by changing the sampling lines to fit the particle.
Fig. 4. Particle alignment and deformation. The image shows the particle, the red dot is the center, and the blue dots are the radial endpoints. The red curves show the intensity pattern along the individual radial lines and the blue is the average. The blob is initialized in (a), translated in (b), deformed in (c), the size is found in marked with green points (d) resulting in the segment in (e). Note that despite a very poor initial alignment the model finds the object very precisely. Also note how uniform the intensity pattern becomes by deformation.
Particle Analysis Model. The objective is to design a model that encodes information about the particle’s size, shape and blur. Our model is based on the observation that particles show close to radial symmetry up to scaling. If we sample along line segments from the center of the particle, we expect to see the same intensity pattern or a scaled version of this pattern. This is the idea that we base our particle model on, which is illustrated in Fig. 3.
Our particle prediction is based on the following
where
are the observed spatial position, shape and image appearance respectively, f is the function mapping observations to the vector Y containing the model prediction (st,rt, dt) of size, shape and distance to the focus plane, respectively. We will now give the details of the particle model and then explain how the parameters of this model are used for predicting the particle characteristics.
We sample n radial lines form the center coordinate co placed with equal angle around the center point. A particle descriptor is obtained by sampling the image intensity along these radial lines at m equidistant positions relative to the lengths of the radial lines. This intensity descriptor is denoted Io. The length of the radial lines are stored in the ro vector, which characterizes the particle shape.
Alignment with Image Data. Adapting the model to the image observations is done in the following four steps:
1. Detection of particles based on scale space blob detection [13].
2. Rigid model translation – the radial lines are not changed.
3. Coarse radial alignment using radial threshold.
4. Radial alignment using minimum intensity difference.
The particle model has to be initialized by a rough estimate of the particle size and position, and we have chosen to use scale space blob detection, see [13]. For computational reasons other blob-detectors could also be chosen. The blob detection supplies a subset of interest points in the image, and the rest of the image processing will take place around these points. As a preprocessing step for removal of noise we convolve the image with a Gaussian kernel with standard deviation σ.
Blob detection gives a particle center position, based on the response of an isotropic Laplacian operator. Many natural particles are anisotropic, so the center position can be inaccurate. To account for this inaccuracy we change the particles position by employing an optimization criterion based on radial symmetry and intensity variance. The reasoning for the first criterion is that particles are typically radially symmetric. Based on that we initiate our particle model with radial lines of equal length. We expect the radial lines to have highest similarity when they are sampled from the particle center, also for anisotropic particles. The variation criterion is based on the fact that the intensity descriptor has high variation when sampled on a particle and low otherwise. This turns out to be very important for the robustness of the alignment. The minimization problem becomes
where I is the mean intensity descriptor, and the sum of normed descriptor differences is weighed by η. σγ is the standard deviation of the mean descriptor, which is weighed by ξ. This alignment is optimized using simple gradient decent, by moving in the steepest decent direction until an optimum is reached. The procedure is repeated with finer step size, until a desired precision is obtained.
After an optimal particle position has been found, the particle shape is optimized to the image data by changing the length of the radial sampling lines. We employ two steps, to obtain a precise alignment. First we observe that the radial intensity samples are scaled in the radial direction and we employ a simple procedure to identify the scale parameter, which is illustrated in Fig. 5.
Fig. 5. Estimating the radial scale using threshold. The two curves illustrates two radially sampled lines, where one is a scaled version of the other. The horizontal axis is radial distance and the vertical is image intensity. The areas of the rectangles spanned by (I, n) and (I, r2) are proportional to the scale between the two curves. We employ a threshold to find this proportion, and we weigh the thresholded values in the radial direction to account for image noise.
This is baed on thresholding the radially sampled intensity values. We want to find the relative scale between two intensity curves First we estimate
where j is the number of sampling steps, G(j) is the normalize Gaussian where G( j ) = 
is chosen such that
s is chosen relative the number of radial sampling points (s = 0.5m). Ij is a thresholded intensity sample where
and t is a threshold value chosen as the mean intensity I over all sample lines. Ij is the sampled radial intensity. The scale ai is found by
The threshold based shape alignment is repeated five times, to obtain a good alignment. After this initial alignment the radial sample lines are adjusted
hereby minimizing the difference between the average descriptor and the individual radial descriptors, but taking length of neighboring radial lines Ti and rj into account, where N is the two neighboring radial lines. A large value of the weight parameter κ will push the shape towards a sphere. This optimization is done similarly to the positioning, also using gradient decent and refining the step size when a minimum is reached. The length of the final radial lines are normalized to sum to the same as original radial lines lengths.
The particle model results in an observed characterization as follows
containing the center position denoted co which is a 2D vector, the length of the radial line segments denoted ro which is a n-dimensional vector, and the intensity pattern Io which is m-dimensional. It is estimated as the mean
where I- is the radial pattern of line segment i. It should be noted that the difference between the line patterns have been minimized, so we model the remaining difference as noise, and as a result the averaging will smooth this noise and make the estimate robust.
Modeling the particle will create an independent characterization of the size, shape and blur, which is illustrated in Fig. 3. Particle shape is encoded in the length of the radial line segments, and the particle size can be obtained from a combination of the radial intensity pattern and the length of the line segments. The intensity pattern Io has a shape that bends off to become indistinguishable from the background, see Fig. 4, and the particle boundary is estimated at this point. We found a function of the total variation to be good way of estimating this. We estimate the total variation as the sum of absolute differences of Io and we obtain the distance as
which is the normed total variation. The constant c influences the estimated size of the particle.
Statistical Analysis. The blur is encoded in the radial pattern descriptor (Io), which we use as input for estimating the distance to the focus plane. We use a linear ridge regression to obtain the depth. The model is
‘ is the coefficients of
the regression model. We obtain the model parameters from a training set with known distance to the focus plane by solving
is the distance of the training data. See for example [8] for a detailed description of ridge regression.
