CN100477789C - A method of inter-frame image enhancement based on Joint Motion Photographic Experts Group 2000 - Google Patents

Abstract

The invention discloses a method of intensifying video frequency image. It includes the following procedures: set up image reference pins that are used in intensifying the compensation for the video frequency image pins whose data is lost; divide the pins of loss of data into data blocks that are MxN pixel(both M and N are integer); form the motion estimation between reference pins and pins of loss of data, so it's easy to search the data block that is matching the data block whose data is lost in reference pins; make use of the above pins to intensify the lost data blocks in the pins of loss of data. The method can use the relevance between image pins to compensate the loss of image data caused by network transmission congestion, and upgrade the image quality.

Description

A method of inter-frame image enhancement based on Joint Motion Photographic Experts Group 2000

technical field

The present invention relates to a method for inter-frame enhancement of video images, in particular to a method for inter-frame enhancement of video images compressed based on Motion JPEG 2000, which can recover as much as possible at the receiving end Lost video image detail.

Background technique

With the popularization of multimedia applications, the development of digital video technology and the increase of image transmission on the network, such as the use of videophone, remote browsing and retrieval of images, global multi-user virtual environment sharing, etc., the research on image processing technology has become more and more important. more and more important.

Image processing includes image compression, image enhancement and restoration, image denoising, and image description, classification, and recognition. For the process of image acquisition, storage and transmission, it is urgent to develop image compression technology rapidly. The data volume of a color (24bit/pixel) digital image with medium resolution (640*480) is about 7.37Mbit. The huge data volume brings great difficulties to the image storage and transmission. Therefore, images must be encoded before transmission.

The traditional JPEG compression technology has been unable to meet people's requirements for multimedia image data, so a more powerful and efficient image compression standard is needed. In this context, JPEG 2000 came into being.

Motion JPEG 2000 (Motion JPEG 2000) is an intra-frame coding method for moving images. It does not use inter-frame correlation, but uses the compression method provided by JPEG 2000 to remove the intra-frame redundancy of each frame in the video sequence frame by frame.

As time goes by, people have higher and higher requirements for image compression processing. In addition, video transmission based on the Internet and wireless channels has become more and more common. In practical applications, some important images, such as satellite remote sensing images and medical images, need to be transmitted through the Internet. These image transmissions require high image quality.

However, video transmission over the Internet or wireless channels is often affected by network congestion due to the limitation of channel width. When the network is congested, the router selectively truncates some data packets representing detailed information, which will result in a reduction in the image quality of the truncated frames.

In JPEG 2000, the spatial domain signal is subjected to discrete wavelet transform (DWT) after pre-processing such as color space change, DC removal, and normalization to remove the correlation between data. Fig. 1 shows a schematic diagram of one-dimensional discrete wavelet transform decomposition. The basic filtering method of one-dimensional DWT is to decompose the original signal into two non-aliasing sub-bands in the frequency domain through high-frequency and low-frequency filter banks, and keep the number of signals unchanged by down-sampling. The remaining low-frequency signals after a DWT are often still relevant. Further k (k is an integer) sub-band decomposition can be performed on it until the correlation between signals can be ignored.

The two-dimensional DWT is based on the one-dimensional row DWT and downsampling, and then performs a DWT on the column signal, and performs downsampling to ensure that the amount of data after the transformation is the same as before the transformation. After a two-dimensional DWT transformation, the original image is divided into four subbands. The only LL sub-band (row low frequency and column low frequency) can still be decomposed, and a hierarchical structure is formed after 3-5 levels. This hierarchical structure enables JPEG 2000 to support multi-resolution stream structures.

作为量化索引值，其中sign()表示一个变量的符号，

表示向下取整。这种量化方式的优点是可以根据信噪比要求，提供一种渐进的码率模式，即可以先用较小的步长对信号进行量化。然后根据用户的不同需求以较宽的步长(一般是原始步长的2的整数次幂倍)进行反量化。此时，只需丢弃量化索引值的一些最低有效位(LSB)即可。The wavelet coefficients after DWT need to be quantized. A special feature of the quantizer of JPEG 2000 is the introduction of the concept of "Dead Zone". For each subband quantizer, the "DeadZone" is twice as wide as the other steps. This means that if a quantizer with a step size _Δb is used in subband b, one coefficient y(i,j) of this subband can be given by

As a quantized index value, where sign() represents the sign of a variable,

Indicates rounding down. The advantage of this quantization method is that it can provide a progressive code rate mode according to the requirements of the signal-to-noise ratio, that is, the signal can be quantized with a smaller step size first. Then dequantize with a wider step size (generally an integer power of 2 of the original step size) according to different needs of users. At this point, it is only necessary to discard some least significant bits (LSBs) of the quantization index value.

Then it is necessary to perform Embedded Block Coding with Optimal Truncation (EBCOT) on the quantized wavelet coefficients. EBCOT is a bit-plane-based coding method. According to the order of bit planes from high to low, data in the same bit plane is extracted for encoding. This provides a progressive encoding of image quality from coarse to fine. At the same time, EBCOT also provides support for multiple truncation points, that is, in order to adapt to a certain compression rate, the code stream can be truncated at any truncation point of the JPEG 2000 code stream. Equivalent to discarding all insignificant coefficients and the less significant bits of all significant coefficients in the current bit-plane.

The code stream after EBCOT is sent to the arithmetic encoder, packaged into a Motion JPEG 2000 (MJP2) code stream, and sent to the decoder. The decoder at the decoding end performs arithmetic decoding, inverse EBCOT, inverse quantization, inverse discrete wavelet transform, and post-processing in the reverse order of the above, to obtain decoded images of each frame.

Progressively scalable decoding schemes are equally important for video transmission. On the Internet or wireless channels, due to the limitation of channel bandwidth, video transmission is often affected by network congestion, resulting in picture pauses when watching online. If the MJP2 code stream is transmitted in progressive mode, when encountering network congestion, the router can actively cut off the code stream of each frame to reduce the amount of data and ensure the smooth flow of the network. At this time, the user can still watch continuous pictures, but the detailed information about the image is lost, which will inevitably lead to a decline in image quality.

Therefore, there is a need for a method for recovering the detailed information of the lost image as much as possible at the decoding end.

Contents of the invention

The purpose of the present invention is to provide a method for enhancing the received video image to improve the image quality, which can utilize the inter-frame correlation of each frame of the moving image received by the decoding end to restore the frame loss caused by information loss to a certain extent details, thereby enhancing image quality.

In order to achieve the purpose of the present invention, a video image enhancement method is provided, comprising the steps of: establishing an image reference frame for enhancing and compensating a video image frame with data loss; dividing the data loss frame into data with a size of M×N pixels block; using the inverse discrete wavelet transform to convert the frequency domain signal of the reference frame into a spatial domain signal;

Moving the obtained spatial domain signal by one pixel in the direction of x-axis, y-axis and x=y, and performing discrete wavelet transform on the obtained image signals in three moving directions to obtain the shifted frequency domain of the reference frame signal; do motion estimation between the shifted reference frame and the data loss frame, so as to search for a data block in the reference frame that matches the data block in the data loss frame; use the match found in the reference frame Block Augmentation Data blocks lost in lost frames.

The moving image enhancement method of the present invention can use the correlation between image frames to compensate and enhance image data loss caused by network transmission congestion, etc., and improve image quality.

Description of drawings

Through the following detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, the purpose, features and advantages of the above and other aspects of the present invention will become more obvious and easier to understand. It should be noted that these embodiments are only to illustrate the present invention , rather than limit the technical solution of the present invention to this, wherein:

Figure 1 is a schematic diagram illustrating multi-level discrete wavelet transform;

2 is a flowchart illustrating a method for enhancing video images according to an embodiment of the present invention;

Fig. 3 shows a schematic diagram of the peak signal-to-noise ratio gain of the image after the image is enhanced by the image enhancement method according to the present invention.

Detailed ways

An embodiment of a method for enhancing an image by utilizing inter-frame correlation of an image according to the present invention will be described in detail below with reference to the accompanying drawings.

First, the JPEG 2000 standard and discrete wavelet transform are introduced. As an upgraded version of JPEG, JPEG 2000 has the following characteristics:

1. High compression ratio

The compression performance of JPEG 2000 is about 20% higher than that of JPEG, that is to say, under the same compression rate, the distortion of JPEG 2000 is about 20% smaller than that of JPEG. At the same time, the system using JPEG 2000 has good stability, smooth operation, good anti-interference and easy operation.

2. Support both lossy and lossless compression

JPEG only supports lossy compression, while JPEG 2000 can support lossless compression. In practical applications, important images such as satellite remote sensing images, medical images, and photos of cultural relics are very suitable for JPEG 2000 compression.

3. Realized progressive transmission (progressive transmission)

This is an extremely important feature of JPEG 2000. It can first transmit the outline of the image, and then gradually transmit the data to continuously improve the image quality, so that the image can be displayed from hazy to clear, instead of slowly moving from top to bottom like the current JPEG. It is shown that this has great significance in network transmission.

4. Support "Region Of Interest (ROI)"

The user can arbitrarily specify the compression quality of the region of interest on the image, and can also choose the specified part to be decompressed first, so as to make the key point stand out. The advantage of this method is that it combines the receiver's subjective demand for compression and realizes interactive compression.

The biggest difference between JPEG 2000 and traditional JPEG is that it abandons the block coding method mainly based on discrete cosine transform adopted by JPEG, and instead adopts the multi-resolution coding method mainly based on wavelet transform.

Wavelet transform is a modern spectrum analysis tool. It can not only examine the frequency domain characteristics of local time domain processes, but also examine the time domain characteristics of local frequency domain processes. Therefore, even for non-stationary processes, it is handy to deal with. It can transform the image into a series of wavelet coefficients, which can be efficiently compressed and stored. In addition, the rough edges of the wavelet can better represent the image, because it eliminates the blockiness common to DCT compression.

The locality of wavelet in space and frequency domain is locality in statistical sense. The locality mentioned here means that the image space range actually involved by a transformation coefficient is local. Therefore, in order to fully restore a certain part of the image, it is not necessary that all the codes are accurately preserved, only that a part of the codes corresponding to it has no error. Therefore, region-of-interest compression can be achieved. At the same time, JPEG 2000 supports shaping wavelet, so it can realize lossless compression.

Wavelet is based on three main basic theories, which are pyramid coding, filter bank theory, and subband coding. Wavelet transform combines these three technologies. Wavelet transform can decompose signals composed of different frequencies that are intertwined together into signals of different frequencies, so it can be effectively applied to encoding, decoding, edge detection, data compression, and linearization of nonlinear problems. Good analysis of local time domain and frequency domain signals, making up for the lack of Fourier transform.

The principle of enhancing the video image based on the JPEG 2000 code stream of the present invention is illustrated below.

Motion JPEG 2000 is a new international standard for encoding moving pictures. Uses the same intra-coding technique as JPEG 2000. The bitstream is prepackaged frame by frame. When network congestion occurs, routers selectively discard certain unimportant parts and transmit only important parts for image reconstruction. The so-called unimportant data packets refer to data packets that have less impact on human vision and have a higher or clearer image resolution. The loss of high-frequency information will ease the burden of network transmission, but it will reduce the quality of the image to a certain extent.

Since Motion JPEG 2000 uses intra-frame coding technology, the inter-frame correlation between the frames of the encoded motion picture still exists, and the information lost during the transmission of a certain frame image will not affect other frames. Therefore, the lost data can be recovered to a certain extent by using the relevant frames that have not suffered the image data loss at the decoding end, thereby enhancing the image quality.

According to an embodiment of the present invention, for example, in the process of receiving an image through the Internet, it is assumed that image data loss occurs in the current frame, while the reference frame remains relatively intact. Motion estimation is first made between the current frame and a reference frame. It is assumed that each frame of data received is decoded to obtain a frame sequence I ⁿ . According to the progressive transmission principle of JPEG 2000, the more data received, the better the image quality will be restored. Therefore, if it is detected at the receiving end that the amount of data in a certain frame is significantly smaller than the amount of data in other frames, data loss may occur in that frame. In this case, image enhancement needs to be performed on the received current frame.

The enhancement process of performing discrete wavelet transform on video images will be described below with reference to FIG. 2 .

The process of image enhancement needs to use a reference frame, which should be a received frame without data loss. In order to perform image enhancement, a buffer needs to be set at the receiving end to store reference frames for enhancing image frames with data loss, that is, a frame of image signals without data loss is stored in the buffer. During the receiving process, it is judged whether the newly received image frame suffers from data loss. If the new frame does not suffer data loss, the buffer can be flushed with a new frame.

For ease of discussion, in the following examples it is assumed that the image data of the received current frame, eg ^In , is lost. It should be understood that the frame whose image data is truncated during network transmission is not limited to the current frame. For example, the previous frame In ^-1 frame stored in the buffer without data loss can be used as the reference frame. Use x ⁿ (i, j) to represent the spatial domain signal of the nth frame where data loss occurs. After multi-level DWT transformation, the wavelet coefficients in sub-band b are expressed as y _b ⁿ (i, j), b∈{kLL , kHL, kLH, kHH}, where kHL (k is a positive integer) represents the result of performing high-pass filtering on the row signal and performing low-pass filtering on the column signal in the decomposition of the k-th layer.

For the convenience of description, it is assumed that each frame of image undergoes only one level of discrete wavelet transform (DWT). Since the transmission is a moving image, in order to perform image enhancement on an image frame subject to data loss, it is first necessary to find a matching data block between the reference frame and the current frame. As shown in Figure 2, in the LL subband, the I ^nth frame frequency domain signal is divided into data blocks of M*N (M and N are positive integers) pixels of fixed size, and the data blocks are divided into I ^n-1 data blocks one by one. The motion search and matching are performed in the LL sub-bands and the three LL sub-bands after motion offset (the specific method will be described in detail later). If a match is obtained, the data of the I nth frame is enhanced with the data block of the I ^{n -1th} frame. If no match is found, the data block of the reference frame is discarded.

Due to the motion of the image, there is a certain offset between the relevant data blocks between the reference frame and the lost current frame. In this case, in order to find a matching data block for image enhancement, it is necessary to shift the pixels of the image frame. In this embodiment, the process of searching for relevant data blocks is performed by moving the data blocks of the reference frame.

In order to search for the corresponding data block, an inverse discrete wavelet transform (IDWT) is performed on the frequency domain signal of the I ^n-1th frame, so as to obtain the spatial domain signal x ^n-1 (i, j) of the I ^n-1th frame. Next, move the spatial domain signal x ^n-1 (i, j) by one pixel along the x-axis, y-axis and x=y direction to obtain three images x ^n-1 (i-1, j), x ^{n- 1} (i, j-1), and x ^n-1 (i-1, j-1). Perform DWT transformation on the three images respectively to obtain three shifted frequency domain signals of the In ^-1th frame. Denote the transformed signals as y _b(1) ^n-1 (i, j), y _b(2) ^n-1 (i, j), and y _b(3) ^n-1 (i, j), and ${\mathrm{the\; y}}_{b\left(0\right)}^{\mathrm{no}-1}(i,j)={\mathrm{the\; y}}_b^{\mathrm{no}-1}(i,j).$ The reason for such movement is that the image needs to be down-sampled after DWT transformation. If a data block moves an odd number of pixels between two frames, the samples retained between the two frames after down-sampling are complementary sets. Move one pixel each in the x, y and x=y directions, plus the image itself includes the odd and even combinations of all movements of the data block in both directions of the plane.

Divide the received frequency domain signal y _LL ⁿ (i, j) in the LL subband of the current frame into data blocks of fixed size, for example, 8*8. Then one by one in the four LL subbands y _LL(0) ^n-1 (i, j), y _LL(1) ^n-1 (i, j), y _LL(2) ^n-1 (i, j) of the reference frame j), y _LL(3) ^n-1 (i, j) to search for the data block most similar to each data block of the current frame.

The search process can use a three-step method, and the maximum absolute distance (MAD) rule can be used to determine whether two data blocks are similar, that is, the absolute value of the difference between the current frame and the frequency domain signal of the corresponding subband of the shifted reference frame is calculated. The sum of the values, i.e., computes $\underset{(i,j)\∈\mathrm{block}}{\mathrm{\Σ}}|{\mathrm{the\; y}}_{\mathrm{LL}}^{\mathrm{no}}(i,j)-{\mathrm{the\; y}}_{\mathrm{LL}\left(k\right)}^{\mathrm{no}-1}(i+\mathrm{di},j+\mathrm{dj})|,$ Find (di, dj, k) that minimizes this. This minimum value is compared with a preset threshold. If the minimum value is greater than the set threshold, it indicates that the two data blocks do not match. If the minimum value is less than or equal to the set threshold, the (di, dj, k) is recorded as a motion vector. This motion compensation method is called low-band-shift method (Low-Band-Shift), which is characterized in that accurate motion vectors can still be obtained when high-frequency sub-band information is lost.

此时可以得到下面的公式(1)表示的未经损失的y_LL ⁿ(i，j)的分布范围：After the motion vector (di, dj, k) corresponding to a certain wavelet coefficient y _LL ⁿ (i, j) of the current frame is found in the reference frame, its corresponding wavelet coefficient in the reference frame is y _{LL(k )} ^n-1 (i+di, j+dj). Thereafter, the lost data in the current frame can be enhanced. As an example, for example, if y _LL ⁿ (i, j) loses the p least significant bits of the current frame during transmission, what is actually received at the receiver is

At this time, the distribution range of y _LL ⁿ (i, j) without loss expressed by the following formula (1) can be obtained:

$\left\{\begin{array}{cc}{\stackrel{<span\; class="notranslate">\; \&Center\; Dot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>{\stackrel{<span\; class="notranslate">\; \&Center\; Dot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; p</span>}}& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; sign</span>}<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; \&Center\; Dot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>\\ {\stackrel{<span\; class="notranslate">\; \&Center\; Dot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>{\stackrel{<span\; class="notranslate">\; \&CenterDot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; sign</span>}<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; \&CenterDot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; p</span>}}& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; sign</span>}<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; \&CenterDot;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; LL</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

的增强结果。其具体操作如下：Use the value closest to y _LL(k ^n-1 (i+di, j+dj) within the range listed in the above formula as the wavelet coefficient of the current frame subject to loss

enhanced results. Its specific operation is as follows:

相同，即：

则可以令

作为

的增强结果。也就是说，用y_LL(k ^n-1(i+di，j+dj)的最后p个最低有效位来补偿受到损失的当前帧的

If the wavelet coefficients y _LL(k ^n-1 (i+di, j+dj) in the reference frame are all the same as

same, ie:

then you can make

as

enhanced results. That is, use the last p least significant bits of y _LL(k ^n-1 (i+di, j+dj) to compensate for the loss of the current frame

If the wavelet coefficient y _LL(k ^n-1 (i+di, j+dj) in the reference frame is still consistent with If there is a difference, compare the two endpoints of the interval determined by the inequality in formula (1). Select a point closer to y _LL(k ^n-1 (i+di, j+dj) from these two endpoints as the enhancement result for the current frame.

In the worst case, y _LL ⁿ (i, j) of the current frame loses all its valid bits (for example, the data in the entire subband is lost), in this case, directly make the current frame of ${\stackrel{\&Center\; Dot;\&Center\; Dot;}{\mathrm{the\; y}}}_{\mathrm{LL}}^{\mathrm{no}}(i,j)={\mathrm{the\; y}}_{\mathrm{LL}(k}^{\mathrm{no}-1}(i+\mathrm{di},j+\mathrm{dj}).$

According to the nature of the discrete wavelet transform, for the three high-frequency sub-bands HL, LH and HH of the image, there is no need to go to the previous frame for motion estimation and search for corresponding data blocks. If a wavelet coefficient y _LL ⁿ (i, j) corresponds to the motion vector (di, dj, k), then for all subbands b, the motion vectors corresponding to y _b ⁿ (i, j) are (di , dj, k). In this case, the coefficient y _b(k) ^n-1 (i+di, j+dj) in the reference frame can be used as a reference to enhance the coefficient y _b ⁿ (i, j) of the current frame of the missing data .

After the wavelet coefficients of the current frame of the lost data are enhanced, the enhanced wavelet coefficients are subjected to inverse discrete wavelet transform and post-processing such as DC restoration, color space inverse transformation, etc., and the video can be restored at the receiving end. The reconstructed image after the image has been enhanced.

For the sake of simplicity, the first-level DWT is taken as an example to illustrate the method of compensating and enhancing the video image by using the inter-frame correlation of the image. Generally, a frame of image needs to undergo 3-5 levels of DWT transformation before encoding. The above algorithm can be extended to m-level DWT transformation. The process is as follows:

1. According to the method of one-stage DWT transformation, the LL _m , HL _m , LH _m , HH _m (m is an integer) subbands of ^In are enhanced.

2. Inverse DWT transform is performed on the four enhanced sub-bands to obtain the low-frequency sub-band LL _m-1 before the m-th DWT transform.

3. LL _m-1 is an enhanced sub-band, and on this basis, compensation and enhancement can be performed on the other three sub-bands HL _m-1 , LH _m-1 , and HH _m-1 of level m-1. Its enhancement method is similar to the process explained earlier. In addition, it should be pointed out that the motion vector (di, dj, k) _m of the previous stage can be used to speed up the search for the motion vector of the present stage.

4. Thus, in the manner as described above, image frames subject to data loss are enhanced step by step in order of resolution from low to high, until the entire processing process ends.

Fig. 3 shows the peak signal-to-noise ratio gain of the image after the image is enhanced by the image enhancement method according to the present invention. This example tests the MPEG-2 standard test image "moving object and calendar". The parameters used are as follows:

Video sequence: moving objects and calendars, select 300 frames of images, 352*288 pixels/frame, and the images adopt progressive scanning.

Wavelet base: use the 9/7 real-type filter bank provided by JPEG 2000 Part 1; motion compensation uses low-band shift (Low-Band-Shift).

The compression ratio is 2.0bpp. The odd frames are truncated, and the truncated code stream is equivalent to 1.0bpp. After enhancing the truncated odd frames by using the method of the present invention, an average peak signal-to-noise ratio of 3.12dB can be obtained. The abscissa in FIG. 3 represents the frame number, and the ordinate represents the peak signal-to-noise ratio. The curve formed by the blank squares in the figure represents the peak signal-to-noise ratio after image enhancement, and the curve formed by the black squares represents the peak signal-to-noise ratio of the original image. It can be clearly seen from Figure 3 that the quality of the enhanced image is significantly improved compared with the image before processing.

Having thus far described the invention in connection with specific embodiments, it should be noted that the description herein is for illustrative purposes only and that changes and variations may be made without departing from the spirit or scope of the following claims.

Claims (8)

1. video image enhancing method comprises step:

Foundation is used for the video frame image that data are lost is strengthened the image reference frame of compensation;

The loss of data frame is divided into size is the data block of M * N pixel, wherein M and N are positive integer;

Utilize inverse discrete wavelet transform to convert the frequency-region signal of reference frame to the spatial domain signal;

At the x axle, the direction of y axle and x=y moves a pixel with resulting described spatial domain signal, and the picture signal of resulting three moving directions is carried out wavelet transform to obtain the frequency-region signal of reference frame through displacement;

In the estimation of between reference frame and the loss of data frame of displacement, taking exercises, so that in reference frame, occur the data block that the data block of loss of data is mated in search and the loss of data frame;

Be used in the matched data piece that searches in the reference frame and strengthen the data block of losing in the loss of data frame.

2. video image enhancing method according to claim 1 is characterized in that further being included in and sets up after the described image reference frame, with the step of described reference frame storing in memory.

3. video image enhancing method according to claim 1 and 2 is characterized in that described step of taking exercises estimation between reference frame and loss of data frame is that the frequency-region signal of reference frame and the frequency-region signal of loss of data frame are carried out motion search and coupling.

4. video image enhancing method according to claim 1 is characterized in that at the video image transmitting terminal carrying out intraframe coding through the moving image frame of one-level wavelet transform at least.

5. video image enhancing method according to claim 1, it is characterized in that further comprising the absolute value that calculates through the difference between the frequency-region signal of the frequency-region signal of the mobile image of the reference frame of wavelet transform and loss of data frame, and with the minimum value that obtains and the step of pre-set threshold comparison.

6. video image enhancing method according to claim 5 is characterized in that not matching if described minimum value, then shows two data blocks of reference frame and loss of data frame greater than preset threshold; If this minimum value is less than or equal to preset threshold, then writes down the motion vector of described reference frame displacement and strengthen the data block of described loss of data frame with wavelet coefficient corresponding in the described reference frame with this motion vector.

7. video image enhancing method according to claim 6, it is characterized in that if having only p least significant bit in the wavelet coefficient in the reference frame different with the least significant bit of the wavelet coefficient of loss of data frame, then strengthen the least significant bit of the wavelet coefficient of loss of data frame with the least significant bit in the wavelet coefficient of reference frame, otherwise in the dynamic range that allows with the wavelet coefficient of loss of data frame and the immediate numerical value of reference frame strengthen the wavelet coefficient of described loss of data frame.

8. according to claim 1,2,4, the described video image enhancing method of in 5 to 7 any one, it is characterized in that further being included in the wavelet coefficient after the enhancing of the described loss of data frame of obliterated data is carried out inverse discrete wavelet transform and direct current restores, the color space inverse transformation is to recover the step of video image through the reconstructed image after strengthening.

Images