KR101568590B1 - Image deinterlacing system the method using region-based back propagation artificial neural network - Google Patents

Abstract

The present invention relates to an image deinterlacing system and a method thereof wherein excellent performance is provided regardless of linear or non-linear relation between adjacent fields, and balance between time consumption and visual quality is maintained in a deinterlacing work. The image deinterlacing system using a region-based back propagation artificial neural network according to the present invention includes: a modeling unit which trains achitectures and parameters of an edge back propagation artificial neural network (BP-ANN) and a smooth BP-ANN by dividing a region of a target pixel into an edge data set and a sooth data set; and a deinterlacing unit which deinterlaces an image by using the BP-ANN trained in the modeling unit.

Description

[0001] The present invention relates to an image deinterlacing system and an image deinterlacing method using the deinterleaving artificial neural network,

The present invention relates to a domain-based BP-ANN technique for deinterlacing, and more particularly to a domain-based BP-ANN technique for deinterlacing that provides good performance regardless of the linear or nonlinear relationship between adjacent fields, To an image deinterlacing system and a method thereof.

Interlaced scanning has long been widely applied to TV broadcasting standards including the National Television Systems Committee, Phase Alternation by Line, Sequential Color with Memory, and HDTV (high definition television) formats. When compared to progressive scan, the interlaced scan uses the same bandwidth, and the received frame rate is twice. However, these interlaced scanning formats have disadvantages such as jagged patterns, edge flickering, and line twitters [1]. This disadvantage lowers the visual quality of the video. Furthermore, interlaced scanning has recently been developed and is unsuitable for display devices such as LCD-TV, HDTV, and PC, which require advanced formats.

Many techniques have been proposed to solve this problem. These techniques are generally divided into two categories: intra-field techniques and inter-field techniques. The interfield technique uses spatial-temporal interpolation to fill missing lines. For this reason, the motion vector obtained by searching neighboring frames is mainly used. Compared with the intra method, this technique provides performance improvement but requires a high computational complexity. Moreover, inadequate motion estimation can lead to serious degradation of serious image and corresponding parts.

Intra-field de-interlacing technology is widely applied to real-time applications that require management of memory and computing power. Intra-field deinterlacing has been proposed in various linear algorithms such as line average (LA), edge-based line average (ELA), and efficient ELA. This conventional technique is to calculate a linear relationship between an existing line and a missing line, which is regarded as a fixed parameter filter by interpolating missing pixels using linear spatial averaging for some estimation directions And estimates missing pixels.

However, the relationship between missing pixels and existing pixels is mostly nonlinear, and nonlinear techniques provide better results for steep vertical fixed edges [2]. The best results can be obtained when the applied technique meets the requirements corresponding to the specific area.

Recently, the de-interlacing technique of NN-neural network has been proposed as a technique to reveal the hidden regular pattern between the missing line and the existing line [3, 4]. Plaziac's results show that the PB algorithm produces better results in linear and median filters, even under noisy conditions. In addition, in neural networks, the activation function is replaced by a polynomial function to reduce processing time and hardware complexity in hardware design [5]. However, the BP-NN architecture for de-interlacing has not been discussed in the prior art. Modular NN was used to perform deinterlacing algorithms in both inter and intra modes, which led to performance improvements [4]. However, the deinterlacing process using NN requires a simple deinterlacing process because of its high computational complexity.

[1] G. De Haan, "Television display processing: past & future," in Proc. IEEE ICCE'07, pp. 1-2, IEEE, Las Vegas (2007). [2] S. Shanbhag and D. Agarwal, "Deinterlacing technology: an overview," Int. J. Adv. Eng. Sci. Technol. 11 (1), 129-131 (2011). [3] N. Plaziac, "Image interpolation using neural networks," IEEE Trans. Image Process. 8 (11), 1647-1651 (1999). [4] H. Choi and C. Lee, "Motion adaptive deinterlacing with modular neural networks," IEEE Trans. Circuits Syst. Video Technol. 21 (6), 844-848 (2011). [5] G. Seo, H. Choi, and C. Lee, "Efficient implementation of neural network deinterlacing," Proc. SPIE 7245, 724519 (2009).

Disclosure of Invention Technical Problem [8] The present invention has been proposed in order to solve the above problems, and it is an object of the present invention to provide an image deinterlacing system using an area-based reverse transfer artificial neural network and a method of generating an optimal input model and simplifying a deinterlacing process. .

The image deinterlacing system using the area-based back-propagation artificial neural network divides an area to which a target pixel belongs into an edge dataset and a smooth data set, and performs a back propagation artificial (BP-ANN) neural network), a modeling unit for training with the architecture and parameters of the smooth BP-ANN, and a de-interlacing unit for de-interlacing the image using the BP-ANN trained in the modeling unit.

Also, in the image deinterlacing system according to the present invention, the modeling unit may include a training image calculation unit for classifying the smooth region and the edge region to which the target pixel belongs and binarizing the training image, a training image calculation unit for randomly selecting training points based on the binarized image and pixels All pixels of the training data set presented in the BP-ANN are data set normalization units linearly normalizing from 0-255 to 0-1 and smoothness BP-ANN and edge BP-ANN architectures and parameters And a training execution unit using the training execution unit.

Further, in the image de-interlacing system according to the present invention, the de-interlacing unit may include a BP-ANN applying unit for training an input image by applying the BP-ANN trained by the modeling unit, and a performance evaluating unit And a confirmation unit.

The image de-interlacing method according to the present invention comprises the steps of: (a) dividing an area to which a target pixel belongs into an edge dataset and a smooth dataset using a modeling unit, and performing a back propagation artificial neural network and the smooth BP-ANN, and (b) deinterlacing the image using the BP-ANN trained in step (a) using a de-interlacing unit .

As described above, the image-based deinterlacing system using the area-based back-propagation artificial neural network according to the present invention can classify the linear / non-linear characteristics of the missing pixels by dividing the input pattern into the edge area and the smooth area in the adjacent field, It is possible to simplify the structure with the optimum structure of NN.

1 is a conceptual diagram showing a topology of a BP-ANN having a general nm-1 structure.
2 is a diagram showing a pixel configuration according to a region classification in the present invention;
3 is a block diagram showing the overall configuration of a video deinterlacing system according to the present invention;
4 is a flowchart illustrating an overall flow of a video deinterlacing method according to the present invention.
5 illustrates pixels of a region-based BP-ANN de-interlacing of various inputs in the present invention.
6 is an error-epoch graph of the BP-ANN in the present invention.
7 is a subjective quality comparison image between the conventional technique and the image deinterlacing method according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image deinterlacing system and an image deinterlacing system using an area-based reverse transfer artificial neural network according to the present invention will be described in detail.

1. Architecture of BP-NN

BP algorithm is the most studied NN algorithm because of self-adaptation, self-learning, robustness and generalizability. A BP-ANN with a three-layer topology can access nonlinear functions with some precision. The BP-ANN structure of n-m-1 type is as shown in Fig.

In BP-ANN, there is input n and one output, and hidden layer has neuron m. The output of the hidden layer is I _j , the threshold of the hidden layer is θ _j , and the transfer function of the hidden layer is f ₁ . On the other hand, the threshold of the output layer is θ _k, and the transfer function of the output layer is f ₂ . The weight from the input layer to the hidden layer is W _ij , and the weight of the output layer from the hidden layer is W _jk . Equation 1 below shows the relationship between input and output of hidden neurons. The output of the entire network is y, the desired output is y ', and the error function is defined as e = y'-y.

BP-ANN's learning processing continuously adjusts the weights and thresholds to minimize network errors. The transfer function for the hidden layer in the present invention is a "tansig" as shown in the equation (3), of the output layer is "purelin" as shown in Equation 4, where P _j, Q, I _j, y are the input and output respectively, Transfer function.

The conventional Levenberg-Marquardt optimization algorithm is used to update the weights and thresholds to speed up the convergence and input-output mapping of the BP-ANN. The result is that the network captures the inherent mappings from the input / output pattern pairs collected during the training process, and at the same time the simulation can be trained to model unknown systems from functions that can be created for new or untrained data sets of data .

2. BP-ANN system based on domain for deinterlacing and its method

When calculating missing pixels, other areas of missing pixels have different patterns. In the present invention, the interpolated pixel x is divided into the smoothing area S and the edge area E, which is expressed by the following equation (5).

Here, T is a threshold value, and F (x) is a function described by the following equation (6).

는 벡터 X의 평균값이며, x_d는 도 2에 도시된 바와 같이, 보간된 픽셀 주변의 6개의 존재하는 픽셀 중 하나이다. BP-ANN의 훈련 전에 그것이 속한 영역을 결정하기 위해 보간된 픽셀은 수학식 5에 따라 분류해야 한다. 따라서 에지 BP-ANN는 E 영역에 배정되는 반면 스무스 BP-ANN는 S 영역에 배정된다. here,

Is the mean value of the vector X, and x _d is one of the six existing pixels around the interpolated pixel, as shown in FIG. Prior to the training of the BP-ANN, the interpolated pixels should be classified according to equation (5) to determine the area to which it belongs. Thus, the edge BP-ANN is assigned to the E-domain while the smooth BP-ANN is assigned to the S-domain.

As shown in FIG. 3, the image de-interlacing system according to the present invention divides an area to which a target pixel belongs into an edge dataset and a smooth dataset, and generates a back propagation artificial neural (BP-ANN) and a deinterlacing unit 20 for deinterlacing the image using the BP-ANN trained in the modeling unit 10. The deinterlacing unit 20 includes a deinterlacing unit 20,

The modeling unit 10 according to the present invention includes a training image calculation unit 11 for binarizing a training image by dividing a smoothing region and an edge region to which a target pixel belongs and applying Equations 5 and 6, A training point selector 12 for arbitrarily selecting training points based on the image and pixels, all pixels of the training data set presented to the BP-ANN are data set normalization units linearly normalizing from 0-255 to 0-1 13) and the training execution unit 14 using the architectures and parameters of the smooth BP-ANN and the edge BP-ANN.

The deinterlacing unit 20 according to the present invention includes a BP-ANN application unit 21 for training an input image by applying BP-ANN trained by the modeling unit, and a Peak Signal-to- and a performance verification unit 22 for evaluating performance by calculating a noise ratio.

FIG. 4 is a flowchart illustrating an overall process of an image deinterlacing method using an area-based reverse transfer artificial neural network according to the present invention. Referring to FIG. 4, an edge data set (edge dataset) (A) training with the architecture and parameters of a back-propagation artificial neural network (BP-ANN) and a smooth BP-ANN, by using a de-interlacing unit 10 and a smooth data set, And (b) deinterlacing the image using the BP-ANN trained in step (a).

In the step (a) of the present invention, the training image calculation unit 11 may be used to classify the smoothing area and the edge area to which the target pixel belongs by applying equations (5) and (6) (S20) of arbitrarily selecting a training point based on the binarized image and the pixel using the training point selection unit (12), and using the data set normalization unit (13) All pixels of the training data set presented to ANN are linearly normalized from 0-255 to 0-1 (S30), and the smoothing BP-ANN and edge BP-ANN And training (S40) using the architectures and the parameters.

In step (b), the step (b) includes the steps of training the input image by applying the BP-ANN trained by the modeling unit using the BP-ANN application unit 21 (S50) , Combining the edge points and smoothing points generated in the trained BP-ANN (S55), and evaluating the performance (S60) by calculating the PSNR of the image using the performance verifying section do.

In an embodiment of the present invention, the MATLAB NNs toolbox was selected to perform de-interlacing using region-based BP-ANN. The parameter values for the training algorithm (S40) are as follows. epoch = 500, learning rate = 0.05, maximum number of iterations = 20, and error tolerance = 10 ^-10 . On the other hand, the Levenberg-Marquardt BP technique was chosen to update the weights and biases of the network.

3. Experimental Results

The following three experiments were conducted to evaluate the performance of the deinterlacing algorithm according to the present invention. LA, CASA, LABI, LCID, MCAD and MELA techniques were used to compare the performance of the present invention. The LA method is a simple linear method of interpolating missing pixels as an average of existing pixels at the top and bottom. LA was used because of its simplicity and lack of motion artifacts. The other five algorithms proposed over the last five years showed better performance than LA. To demonstrate the performance of the algorithm according to the present invention, simulations were performed using an Intel Core 2 Duo CPU E8500 processor and MATLAB at 3.16 GHz.

3.1 Experiment 1: optimal number of hidden layer neurons and training set

The determination of the number of neurons in the hidden layer is crucial in choosing an architecture for domain-based BP-ANN. Too much neurons result in underfitting, while too many neurons in the hidden layer may result in some problems such as overfitting and central processing unit (CPU) time consumption.

 The region-based BP-ANN used in the first experiment has six input neurons equal to the input pixels defined in Fig. The number of hidden layer neurons is selected as 4, 8, 12, 16, and 20, and each training data set has 50,000 pixels in E and S regions. The PSNR and CPU time of the simulation are as shown in Table 1.

As shown in Table 1, more hidden neurons provide a better visual quality (PSNR) of the sample image, leading to more computation time (CPU-time). According to the rule of thumb that determines the optimal number of neurons in a hidden layer, the number of hidden neurons should be less than twice the input layer size. Therefore we use 12 hidden neurons in the next experiment on the training data set.

In order to determine the size of the training data set, a different number of different training sets of 1000, 5000, 10000, and 50000 were collected for each area. The PSNR values of the different training data set sizes are as shown in Table 2. The average PSNR value in Table 2 shows the proper size of the training data set between 10000 and 50000.

3.2 Experiment 2: Optimal input model of domain-based BP-ANN

In Experiment 2, the number of hidden layer neurons was directly proportional to the number of input neurons, and the training set size of each network was set to 50000. In fact, the optimal number of hidden layer neurons and the training set is a completely dependent solution, so no exact definition is required. Regularly proven BP-ANNs using untrained datasets can be evaluated whether the network is over-buffeted or not. If the untrained data set falls to performance, it means that the training is less likely, and going back to the previous iteration will create a simulation approaching the correct output.

One of the training error-epoch graphs shown in Figure 6 uses a training set to train the BP-ANN. The Y-axis error means the square error, and the optimal performance is reached after 30 times of test validation. The architectural coordination and training processes should be alternated until valid performance is achieved. The region-based BP-ANN learning process is executed using the steps shown in Fig. First, the position is arbitrarily selected from the test image. 5 shows the next three patterns of the selected pixel. : The light gray pixels are in one, the dark gray pixels are the inputs of the BP-ANN, and the crossing pixels are the outputs of the BP-ANN. Second, the selected location is divided into S area and E area using equations (5) and (6), and the corresponding inputs and outputs of the training set are collected in each area. Third, the two BP-ANNs train roughly the intersection pixels of the original image. When fusion occurs in two BP-ANNs, the region-based BP-ANN performs interpolation of images in the present invention. Table 3 shows the results of region-based BP-ANN using different input models.

3.3 Experiment 3: Generic deinterlacing model using region-based BP-ANN (objective and subjective performance comparison)

Based on these experiments, an optimal balance between consumption time and PSNR is achieved in the area-based BP-NN de-interlacing algorithm of the 8-16-1 structure and 50000 training data sets. To evaluate the present invention, a training image including eight baboon, man, girl, goldhill, zelda, toys, airplane, and Barbara is used and a test image including Lena, finger, peppers and boat is used.

Table 4 shows the comparison results between the PSNR of the algorithm according to the present invention and the previously tested benchmarks tested.

The method according to the present invention and the LA, CASA, LABI, LCID, MCAD, and MELA algorithms were evaluated. The method according to the present invention improves the image quality because it is 0.5938 (LA), 0.4837 (CASA), 0.6382 (LABI), 0.5812 (LCID), 0.1424 (MCAD), and 0.4837 I can conclude. Table 5 shows the CPU time of each de-interlaced image.

The deinterlacing method according to the present invention requires a minimum CPU time when compared with all test benchmarks except LA, LCID, and MELA techniques. The method according to the present invention reduced the average CPU time to 64.25, 90.19 and 93.88%, respectively, when compared to CASA, LABI and MCAD.

Figure 7 shows the subjective result of the Boat image. From these comparative images, it can be seen that the algorithm according to the present invention can obtain the highest subjective quality among all the test benchmarks. The algorithm according to the invention is a clean image with only very little visual artifact. Furthermore, as shown in FIG. 7, it can be seen that the algorithm according to the present invention has improved edge preservation and edge sharpness after de-interlacing. The algorithm according to the present invention thus provides the best subjective performance in the period of PSNR and at the same time provides optimal visual performance.

4. Conclusion

As described above, in the image deinterlacing system using the area-based reverse transfer artificial neural network and the method thereof, the target output pixel is divided into the edge area E and the smoothing area S for more effective simulation. One BP-ANN is assigned to each area. We constructed effective structure and model for deinterlacing, and performed three experiments applying different parameters. Good performance is cost time and memory, and the balance between time consumption and visual quality can be confirmed from the experimental results. Considering balancing between CPU time and PSNR, the optimal structure of the BP-ANN for de-interlacing was 8-16-1 of the input mode shown in FIG. 5B. The technique according to the present invention has the potential to be widely applied, and it can be confirmed that the method according to the present invention has a better result than the conventional deinterlacing method.

The ANN de-interlacing method applied to the present invention shows good performance irrespective of the linear or non-linear relationship between adjacent fields. In order to distinguish the linear / nonlinear characteristics of the missing pixels in the adjacent field, the input pattern is divided into the edge area and the smooth area, and the optimal structure of the NN is simplified. We also trained ANNs in different input pixel modes. It can be confirmed that the present invention provides the best performance among the techniques studied to date.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the invention is defined by the appended claims. .

10: Modeling unit
11: Training image calculation unit
12: Training point selector
13: Data set normalization unit
14: Training execution department
20: Deinterlacing section
21: Application of BP-ANN
22: Performance Verification Unit
100: Video deinterlacing system

Claims (8)

A modeling unit that trains an area to which a target pixel belongs with an edge dataset and a smooth dataset and trains the architecture and parameters of a back propagation artificial neural network (BP-ANN) and a smooth BP-ANN And
And a de-interlacing unit for de-interlacing the image using the BP-ANN trained by the modeling unit,
The modeling unit,
A training image calculation unit for binarizing the training image by distinguishing the smoothing region and the edge region to which the target pixel belongs;
A training point selection unit for arbitrarily selecting training points based on the binarized images and pixels;
All pixels of the training data set presented in the BP-ANN are data set normalization units linearly normalizing from 0-255 to 0-1 and
And a training execution unit using architectures and parameters of the smooth BP-ANN and the edge BP-ANN.
delete The method according to claim 1,
Wherein the training image calculation unit binarizes the image by applying Equations (5) and (6).
&Quot; (5) "

&Quot; (6) "

Here, the interpolated pixel is x, the smooth region is S, the edge region is E,

Is the mean value of the vector X and x _d is one of the six existing pixels around the top and bottom of the interpolated pixel.
The method according to claim 1,
The de-
A BP-ANN application unit for training an input image by applying the BP-ANN trained in the modeling unit;
And a performance verifying unit for evaluating the performance by calculating a PSNR of the image.
(a) A modeling unit is used to divide an area to which a target pixel belongs into an edge dataset and a smooth dataset, and a back propagation artificial neural network (BP-ANN) and a smoothing BP-ANN Steps in training with the architecture and parameters and
(b) deinterlacing the image using the BP-ANN trained in step (a) using a de-interlacing unit,
The step (a)
(a-1) binarizing the training image by separating the smoothing area and the edge area to which the target pixel belongs, using the training image calculating part;
(a-2) arbitrarily selecting a training point based on the binarized image and pixels using a training point selection unit;
(a-3) linearly normalizing all pixels of the training data set presented in the BP-ANN from 0-255 to 0-1 using the data set normalizer and
(a-4) training using smooth BP-ANN and edge BP-ANN architectures and parameters using a training execution unit.
delete 6. The method of claim 5,
The step (b)
(b-1) training the input image by applying the BP-ANN trained in the modeling unit using the BP-ANN application unit, and
(b-2) combining edge points and smoothing points generated in the trained BP-ANN.
8. The method of claim 7,
After the step (b-2)
(b-3) evaluating the performance by calculating the PSNR of the image using the performance verifying unit.

Images