A New Error Concealment Algorithm Based on Directional Texture Synthesis

In this paper, we propose a novel directional texture synthesis based error concealment algorithm to recover damaged video images. It uses the confidence level and structure information to calculate the priority of patch, which contributes to improve the ability to select the best matching block when the damaged area is very large. The JM86 model of H.264 standard is used to evaluate the algorithm. And experimental results show that our algorithm achieved a better image reconstruction results than the improved Multi-directional texture interpolation algorithm, with 1.2 to 1.4dB gain in PSNR and 0.5 percent to 1 percent gain in SSIM.


Introduction
Enacted in 2003, the H.264 standard is widely used in video conference, digital storage media, Internet and other fields [1] .Due to the application of inter/intra frame prediction technology in the video coding framework of H.264 standard, error will spread both in time and space domain.To get a better quality of image decoding without increasing redundancy and delay, error concealment should be used in the decoder.
Nowadays, error concealment methods can be classified into two common categories: spatial error concealment and temporal error concealment.Spatial approaches make use of the surrounding correctly received image data in the same frame to reconstruct the missing blocks.The simplest way is to use the adjacent pixel to interpolate, or find a suitable adjacent block and replace it.The best neighborhood matching (BNM) method [2] finds the best matching block in the same frame, making use of the similarity between blocks.The Multi-Texture Interpolation (MTI) method [3] calculates the texture directions of video image, and then a main direction and a secondary direction are selected.Admittedly, it has improved the accuracy of interpolation direction, but the reconstruction results of the damaged texture are unsatisfactory.In [4], by detecting the edge of surround blocks to get directions of edge of themselves, in order to get the interpolation direction for every pixel in the block, calculate the pertinent pixels and it's weighing determined by distance, and the method of interpolation decided by the status flag of surround blocks.All of those methods are better than MTI algorithm, but there is no good result in the area of texture just because of imprecise divide the direction of edge.In [5], an improved Multi-Texture Interpolation (IMTI) algorithm is proposed.It divides the multi-texture area of the lost macro-block adaptive into several sub-blocks, and uses different methods to conceal the error according to the type of sub-block.It performs well when the texture is plentiful.But the PSNR value would fluctuate around the threshold drastically when the picture texture is not plentiful.
In this paper, we proposed a new directional texture synthesis (DTS) error concealment algorithm in order to overcome the drawback of the algorithms which used in [3,5].The newly proposed algorithm exploits the improved Sobel operator to calculate the main texture direction and makes full use of the structure information, which contributes to improve the calculate method of the priority of patch and the effect of error concealment.In addition, we use the JM86 model to simulate and compare the effect of DTS algorithm and IMTI algorithm.

The analysis of directional texture synthesis algorithm
Suffering from poor network performance, there might be lost or wrong transmitted packets.As a result, error macro-block will be detected during the decoding process.To reconstruct the damaged texture for video image, we need to calculate the texture's main direction prior to search the best matching block.In [6], Wei uses the Sobel edge detector to obtain the spectrum of edge images, and uses the spectrum characteristics to calculate texture's main direction.However, this direction is not precise enough, and the reconstruction results for video images are unsatisfactory when the texture structure is very complex.And Sun utilizes the gradient values of the pixel which located on the boundary of the target region to calculate of the main and the secondary direction [3] .However, when the damaged area is too large, the result is not satisfactory either.
In this paper, we propose to use the improved Sobel operator to obtain the spectrum of edge images, and determine the direction of texture according to the gradient values of pixel.During decoding, the quality of the output image reconstruction on the edge of the damaged area is highly influenced by the order in which the filling process proceeds, so we must calculate the priority of each patch in order to determine the order of reconstruction.When looking for the best matching block, we need to use different size of patch for different region if the frequency of the texture is not the same.Otherwise, it will lead to unsatisfactory results for the large frequency region, because of the patch is too large, or the patch is too small, or the covered process of small frequency region is too cumbersome that leads to rapidly increase in computing amount.Thereby, we change the patch selection method in this paper.Because of the self-similar of texture, we should start from the center of the patch to find the best matching block for each block and replace it along the texture's main direction of the image.But, when comparing the block obtained by error concealment with the block with no error, the effect of the former will be relatively worse as a matching block.Thus, we set the confidence level as one of weight to search for the matching block.The flowchart of the proposed algorithm is shown in Fig. 1.
Fig . 1 The processes of DTS error concealment algorithm

310
Information Technology for Manufacturing Systems III

The design of algorithm modules
The calculation of video image's main texture direction.In order to calculate the main texture direction more accurately, this paper adopts the Sobel operator to extract the image's edge.However, it has an opposite direction compared with the commonly coordinate system, and it will cause trouble to the calculation of angle and slope.Therefore, the operator of y-direction has been amended, as shown in eq. 1, and the operator of x-direction as shown in eq. 2. The gradient vector of pixel in an adjacent space with a size of 3×3 is composed by G x and G y which belong to the x-direction and y-direction respectively.The gradient modulus and gradient direction to current pixel can be derived by eq. 3 and eq. 4.
-1 -2 -1 , ( ) In order to characterize the texture direction precisely, θ(x,y) is evenly divided into 32 parts.And then the dexter 16 direction is selected.The selection process is as follows: A) Set a counter for each direction; B) Calculate gradient direction and gradient modulus for each pixel in adjacent space, the counter will summate the gradient modulus on the corresponding direction; C) The direction associated with counter which has the maximum value is considered as video image's main texture direction.The calculation of patch priority.In order to determine the order to replace each patch, we define its priority P(p) as the products of the following two terms: ( ) ( ) ( ) where |Ψ p | is the area of Ψ p with default value is 9×9, α is a normalization factor with typical value 255 for gray-level image, n p is a unit vector orthogonal to the front δΩ in the point p, and ⊥ denotes the orthogonal operator.The priority P(p) is computed for every border patch, with distinct patches for each pixel on the boundary of the target region.The function C(q) is initialed as follows: Advanced Engineering Forum Vols.6-7 ( ) 0 C q q = ∀ ∈ Ω , and ( ) 1 as is shown in Fig. 2.
Fig. 2 Symbol Map Having found the proper source patch Ψ p ' , the value of each missing pixel, replaced by the corresponding pixel in the obtained source patch, p q ′ ′ ∈ Ψ , and the confidence level should be updated to give a lower priority to the pixels which already filled a part of the target region.Therefore the confidence level is updated as: where β is a weight factor lower than 1 for the previous confidence level C(q).Because the change in gradient ( ) I p ∇ of the pixel on the boundary of the target region reflects the change of image frequency, the higher the value, the faster change the frequency, and more structure information and texture information the image contained.Under this situation, D(p) is even more important, so we determinate the value of γ according to D(p) in this paper.In [6], it will obtain better results for complex texture image when γ = 2.And letting γ = 1 to deal with the image which has simple structure [7] .After several trials, we decide to calculate the value of γ by Table 1.
Table 1 The formula to calculate the value of γ ( ) The calculation of patch size.As is shown in the previous section, the change in the gradient of pixel will reflect the change of image frequency, while the spatial frequency reflects the amount of information contained by image.The smaller the frequency, the smoother the image, therefore the template should be larger; rather than vice versa, the template should be smaller.The size of template has four options: 3×3, 5×5, 7×7, 9×9.
The selection of the best matching block for each patch and replacing.Generally, we assume that the probability distribution of color values are independent for each other [6] to pixel in adjacent space, so we can calculate the difference between target block and candidate block, and select the best matching block.After repeated trials, we found that the performance of the patch obtained via error concealment operation might be relatively worse than the patch with no error concealment operation.Thus, we use the confidence level of each pixel as a weight factor when it comes to evaluate d(Ψ p ,Ψ s ).Information Technology for Manufacturing Systems III ˆârg min ( , ) where Ψ p is the target block, p Ψ is the best match block, φ is the search area in the φ along the texture direction, β 1 and β 2 are the confidence level of the pixel p and q respectively, and m×m is the size of patch.After the best matching block is obtained, we use it to replace the corresponding patch.

Simulation results
We use JM86 as the test platform.The algorithms are tested on three QCIF video sequences: Suzie and Pamphlet.Because this paper only focus on the error concealment algorithm which used in intra, all the test sequences are set as III Frame structure and encoded in QCIF format at 30Hz frame rate.Each frame is divided into four slices.The quantization parameter is set at 28.After coding, RTP format packet stream would be generated.In order to show the effectiveness of DTS algorithm, a comparison between the performance of the proposed algorithm and the IMTI algorithm is made.The number of packets is randomly dropped according to the error pattern.In this simulation, the packet loss rate is set as 15%, 28% and 35% respectively.SSIM and PSNR are used as measures of the objective criteria of image quality.Fig. 3 and Fig. 4 show the result of visual quality of reconstructed video images.Fig. 3 (a) and (b) show the original and damaged frame respectively, Fig. 3 (c) and (d) show the result generated by IMTI algorithm and DTS algorithm respectively.We can find that the DTS algorithm obtained better visual quality, as well as better complete boundary and smoothness.
Table 2 shows the PSNR and SSIM performance of the reconstructed sequences with packet loss rate is at 28%.We can see that DTS algorithm is always superior to the IMTI algorithm, especially in the test case that β=0.95.Thus, in the subsequent test, we set β=0.95.
Table 3 shows the PSNR and SSIM performances of the reconstructed sequences with different packet loss rates.One can see that DTS algorithm outperforms IMTI algorithm for all sequences, with 1.2~1.4dBgain in PSNR and about 0.7% gain in SSIM.

Conclusion
In this paper, we propose a new error concealment algorithm based on improved directional texture synthesis, and use the improved Sobel operator to calculate the main texture direction for video image.In addition, a new method is used to calculate the priority of patches.We treat the confidence level of the pixel obtained from concealment with the confidence level of correctly received pixels differently, in order to make concealment for the damaged block to find the best matching block.The results of experiments show that the DTS algorithm has effectively solved the problem of selecting the best matching block from the reference blocks and it has a fine reconstruction quality for both different packet loss rates and video sequences.

5 )
where C(p) and D(p) are the confidence term and the data term respectively, and γ is a weight factor for D(p).The definition of C(p) and D(p) are as follows: Fig.3 The 129th frame of Suzie

Table 2
PSNR(dB)and SSIM performance of the reconstructed sequences with different β

Table 3
The PSNR(dB) and SSIM performance of the reconstructed sequences with different packet loss rates