Image Block Error Recovery Using Adaptive Patch_Based Inpainting (Computer Vision,Imaging and Computer Graphics) Part 2

Combination Strategy

In single-directional inpainting, the error of restored pixels will increase along the recovery direction because of unconfident recovered pixel results [11]. Different recovery orders may introduce different error propagation patterns. The error from one recovery order can be compensated by results from the other order. Therefore, we can reconstruct the image through merging the result from different recovery orders as:

where wn(p) is the weighting coefficient controlling the contribution of the nth recovery order for the pixel p, and f(pn) is the value of the recovered pixel in the nth order.

In this paper, the weight is associated with the confidence of the recovery performance in a specific recovery order. The confidence consists two items: similarity confidence Sn(p) and reliablility confidence Rn(p).

The similarity confidence Sn(p) can be expressed by the Gaussian function of the Euclidean distance between the source patch and the target patch, and we simplified it as:

whereis    the difference between the target patch and the best match source patch, which is defined in (2), and the parameter a regulates the relative influence of the difference on the weights. It is set to 0.125, experimentally.

The reliability confidence Rn(p) measures the amount of reliable information surrounding the pixel p. Our aim is to give higher weight to the pixel whose patch has more pixels which are known or have already been recovered. Rn(p) is defined as:

Table 1. The impact of patch size for block loss (dB)

To	Lena	Baboon	Pepper	Barbara
1	26.63	20.94	28.34	24.13
2	27.35	21.25	29.10	26.01
3	27.33	21.24	29.31	26.40

where Ψφ) is the target patch centered on the pixel p, A( !Pip)) is the area of the patch Ψφ), i.e. the number of pixels in the patch. Initially, we define Rn(p) = 0 if p is a missing pixel, Rn(p) = 1 if not.

For each pixel p to be recovered, we define its weight for a specific recovery order associated with the product of above two terms:

The weighting provides an efficient and flexible way to select the appropriate pixels contributing to the estimation of the lost pixel for the final result.

We compute the confidence for all the recovery order for each lost pixel, and normalize the weight coefficients as:

After obtaining these weight coefficients, we recover the lost pixels through combining the intermediate results from all the recovery orders.

Experiments and Results

In order to illustrate the performance of our error recovery method, we take many experiments on test images: Lena, Baboon, Pepper and Barbara. We consider the situation of the 16×16 block since the image or video are often encoded in 16×16 block size. Different block-loss situations are investigated in the paper: isolated block loss and consecutive block loss. For objective evaluation, we use a modified peak signal-to-noise ratio (PSNR) as the objective measure in our experiments, which is defined just on the corrupted areas instead of the entire image:

where fo(p) and fr(p) are the pixel values in the original and the recovered image, and M is the number of the lost pixels. We first give the implementation details in our experiments and then compare it with several existing error recovery algorithms.

Implementation Details

For the implementation of the proposed algorithm, there remain some choices, which include the following:

1)    The patch size. The size of the patch affects how well the filled pixels capture the local characteristics of the known region. The patch size is controlled by To, which is defined in (1).

2)    The filling manner. The filling manner means whether the pixel centered on the target patch (pixel-filling) or the unknown part of the target patch (patch-filling) will be filled in the step 3 of the patch based inpainting algorithm.

In the first experiment, we illustrate how the choice of patch size affects the recovery performance. We fix the search range as 32×32 with pixel-filling. And we investigate the isolated block loss situation with about 10% loss rate. Table I shows the evolution of PSNR values with different patch size using pixel-filling. Smaller patch size allows more matching possibilities, thus implies weaker statistical constraints. Up to a certain limit, bigger patch size can capture the texture characteristics better, however with much higher computation complexity. From the results in table 1, To=2 is a good balance.

In the second experiment, we demonstrate the impact of the filling manner on the recovery performance. Table 2 shows the performance of the two different filling manners with To=2. It can be seen that pixel-filling shows better performance than the patch-filling and the gap ranges from 0.07 to 0.98dB.

Table 2. The impact of filling manner for block loss (dB)

Filling manner	Lena	Baboon	Pepper	Barbara
Pixel	27.35	21.25	29.10	26.01
Patch	27.06	20.88	28.12	25.94

Comparison Results

To demonstrate the effectiveness of our algorithm, we compare it with several previous existing error recovery algorithms: bilinear interpolation (BI), the orientation adaptive sequential interpolation (OASI) [11], order-based inpainting (OI) [14]. For our algorithm, in the experiment, we set To=2 and use pixel-filling manner in the recovery process.

Table 3 and Table 4 give the PSNR comparisons between the compared methods and our algorithm under the following two loss situations: the isolated block loss (about 10%) and consecutive block loss (about 25%). It can be seen that we have achieved 1.09-3.17 dB improvement in the case of isolated block loss and 1.16-3.05 dB improvement in the case of consecutive block loss over OASI.

To subjectively evaluate the results, Fig.6 shows the comparison of the reconstructed images for Barbara by the compared methods and our algorithm in the situation of isolated block loss. It can be observed that our new approach has achieved significant improvements in the area of complex texture structures. For better subjective evaluation, we show some enlarged examples for sharp edge areas, texture areas and very complex areas in Fig. 7. The visual quality of the recovered blocks are very good even when the areas contain a lot of detail information.

Fig. 6. Reconstructed images for Barbara for isolated block loss

Table 3. Performance comparison for isolated block loss (dB)

Image	BI	OI	OASI	Ours
Lena	24.03	23.74	25.98	27.35
Baboon	20.25	18.46	20.16	21.25
Pepper	24.85	24.34	26.67	29.10
Barbara	20.69	21.62	22.84	26.01

Table 4. Performance comparison for consecutive block loss (dB)

Image	BI	OI	OASI	Ours
Lena	22.21	21.32	22.07	24.21
Baboon	19.15	17.94	19.08	20.24
Pepper	25.22	23.26	24.00	26.05
Barbara	19.98	18.21	20.06	23.11

Fig. 7. Enlarged part of the images in fig.6

Fig.8 shows the comparison of the reconstructed images for Lena by the compared methods and our technique in the situation of consecutive block loss. Significant improvements can be found in the recovered image by the proposed method, especially on the blocks with the strong edges or complex texture. And the similar results are obtained for other test images.

Fig. 8. Reconstructed images for Lena for consecutive block loss

Conclusions

This paper proposed an adaptive patch-based inpainting algorithm for image block recovery in block-based coding image transmission. The proposed approach is based on a prior information – patch similarity within the image. By taking advantage of the information, we recover the lost pixels by copying pixel values from the source based on a similarity criterion to keep local continuity. The pixel recovery is performed in a sequential fashion in which the recovered pixels can be used in the recovery process afterwards. In order to alleviate the error propagation with sequential recovery, we proposed an adaptive combination strategy which merges different directional recovered pixels according to the confidence of the estimated recovery performance. Experimental results show that the proposed method provides significant gains in both subjective and objective measurements for image block recovery.

The method is designed mainly for the image block error recovery and we are currently exploring the different combination strategies to extend our method for more general missing regions.