JPH10271509A - Image encoder and image decoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the reproducing image of excellent quality by obtaining a picture element value by motion compensation prediction from a picture element on the outside of a second area and obtaining the picture element value by interpolation from peripheral picture elements for the picture element for which the motion compensation prediction is not performed inside a first area. SOLUTION: A motion vector preparation part 101 for synthesis prepares a second motion vector by using the shape data of the first area and the second area and a first motion vector. In this case, the first motion vector indicates a motion vector used for predicting a first reference frame from a second reference frame at the time of encoding a low-order layer frame. A motion compensation prediction part 103 predicts the picture element value inside the first area from the second reference frame by using the second motion vector by second motion compensation prediction. A controller 104 connects a switch 105 to the side of the first reference frame when the picture element value on a synthesis frame is on the outside of the first area. The controller 104 also controls an interpolation part 106 and performs the interpolation of an area incapable of correctly predicting the picture element value in the prediction part 103.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention belongs to the field of digital image processing, and more particularly to moving picture coding for efficiently coding image data and moving picture decoding for coded data created by this moving picture coding apparatus. The present invention relates to a decoding device.

[0002]

2. Description of the Related Art In image coding, a method of superimposing different moving image sequences has been studied.

[0003] Document "Image coding using hierarchical representation and multiple templates" (IEICE IE 94-159, pp. 99)
In -106 (1995)), a new sequence is created by superimposing a moving image sequence as a background and a moving image sequence of a component moving image as a foreground (for example, a human image or a video of a fish cut out by the chroma key technique). The method is described.

[0004] In addition, the document "Time hierarchical coding based on image contents"("TemporalScalability")
based on image content ", I
SO / IEC JTC1 / SC29 / WG11 MPE
G95 / 211 (1995)) describes a technique for creating a new sequence by superimposing a moving image sequence of a component moving image having a high frame rate on a moving image sequence having a low frame rate.

In this method, as shown in FIG. 18, the entire image is motion-compensated and predictively coded at a low frame rate in the lower layer, and a selected area (dotted portion) of a frame not coded in the lower layer is processed in the upper layer. Is subjected to motion compensation prediction coding. Reference frames used for motion compensation prediction of an upper layer are a decoded frame of a lower layer and a decoded frame of an upper layer. It is assumed that a part that attracts the viewer's attention, such as a person part, is selected as the selected area.

In this method, if only the lower layer is decoded, the entire image is reproduced at a low frame rate. On the other hand, if both the lower layer and the upper layer are decoded, only the selected area is reproduced at a high frame rate. At this time, for a frame in which the lower layer decoded image does not exist, 2
The lower layer is synthesized using the decoded lower layer frames. Then, the decoded image (selected area) of the upper layer is superimposed on the synthesized lower layer frame to obtain a reproduced image.

Next, a method of synthesizing the lower layer in the conventional method will be described. FIG. 16 is a block diagram of a conventional system.

[0008] Delay section 1601 delays the input lower layer decoded image and outputs a second reference frame. Second
The reference frame is an undelayed lower layer decoded image (first
A reference frame) is input to the switch 1603.

The controller 1602 receives shape data of a selected area (hereinafter referred to as a first area) on a first reference frame, and checks whether or not a pixel on the synthesized frame is within the first area. If the pixel on the composite frame is in the first area, the controller connects the switch to the left side and sets the pixel value of the second reference frame as the pixel value of the composite frame. Otherwise, the controller connects the switch to the right and sets the pixel value of the first reference frame as the pixel value of the composite frame.

[0010] Superimposing section 1604 superimposes the decoded image of the upper layer on the composite frame. Assuming that the weighted image obtained from the decoded shape data of the upper layer is W, the synthesized lower layer frame is B, and the upper layer frame is E, the superimposed frame S is obtained by the following equation.

S (x, y) = [1-W (x, y)] B (x, y) + E (x, y) W (x, y) (1) where (x, y) Is coordinates representing a pixel position in space. The shape data is a binary image having a value of 1 inside the selected region and a value of 0 outside the selected region, and a weighted image W (x, y) can be obtained by applying a low-pass filter to the image a plurality of times. . That is, the weighted image W (x, y) takes a value of 1 inside the selected area, 0 outside the selected area, and 0 to 1 at the boundary of the selected area.

For example, frame a and frame d in FIG.
Are two frames used for lower layer synthesis, and the upper layer decoded image in frame b is superimposed. At this time, the first reference frame in FIG.
The second reference frame is frame a.

[0013] The controller connects the switch to the left if the pixel on the composite frame is in the first area, and connects the switch to the right if not. In this way, the pixel values outside the selection areas of the two lower layer frames are excluded, except for the area (intersection area) where the first area and the selection area (second area) on the second reference frame a overlap. , Copied to the composite frame.

In the intersection area, the pixel values in the selected area of the frame d are copied. However, since the decoded image of the upper layer (the decoded image of the frame b) is superimposed on the synthesized frame by the superimposing unit, the selection is not performed. If the motion of the area is not large, the intersection area is almost hidden by the decoded image of the upper layer.

[0015]

In the prior art, when an area other than the selected area changes with time, a discontinuous boundary appears in the synthesized image even in an area other than the above-mentioned intersection area, and the image quality is deteriorated. There's a problem. Next, this problem will be described with reference to FIG.

In FIG. 17, two lower layer frames (first
In the reference frame (second reference frame), the selected area (the halftone dot person) swings his upper body to the right, and the car moves to the left in the background. When creating a composite frame using these frames, pixel values in the first region are copied from the second reference frame, and pixel values outside the first region are copied from the first reference frame.

Since the background vehicle is moving, the vehicle is separated into two in the composite frame as shown in the drawing, and an edge appears at the boundary of the first area. In this example, the specific object moves outside the selected area. However, when the entire background moves due to panning of the camera or the like, the entire background of the composite frame is separated into two parts, and the boundary of the first area is divided into two parts. The pixel value changes discontinuously. Such a discontinuous change in the pixel value results in a very large visual deterioration.

An object of the present invention is to solve the above problems and to provide an encoding device and a decoding device that do not degrade the quality of a synthesized frame.

[0019]

To achieve the above object, the present invention provides: (1) lower layer coding for coding a moving picture sequence at a low frame rate; Performs upper layer coding to encode the region at a higher frame rate, decodes the lower layer only the lower layer of the lower frame rate, and decodes up to the upper layer by decoding the lower layer and the upper layer. Layer coding apparatus and decoding apparatus that superimpose the upper layer with the upper layer to obtain a higher frame rate than the lower layer for the specific area, wherein there is no lower layer frame corresponding to the same frame position as the upper layer during decoding Then, the nonexistent lower layer frame is synthesized using the two decoded lower layer frames, and What is the two one first reference frame lower layer frames in the lower layer frame, and the other second
A reference frame, the specific area on the first reference frame is a first area, the specific area on the second reference frame is a second area, and in a composite frame, pixels outside the first area are: A pixel value is copied from a pixel at the same position in the first reference frame, and for a pixel in the first region, a pixel value is obtained by motion compensation prediction from a pixel outside the second region in the second reference frame, For a pixel in which motion compensation prediction is not performed in one area, a pixel value is obtained by interpolation from peripheral pixels for which a pixel value has already been obtained.

(2) In (1), the lower layer frame is encoded by the first motion compensation prediction means, and the motion compensation prediction in the synthesis is performed by the second motion compensation prediction means. The first used in
Is used for the second motion compensation prediction to perform the synthesis.

(3) In (2), the first reference frame is encoded from the second reference frame by the first motion compensation prediction, and a second motion vector is obtained for the first region. In obtaining the second motion vector, a second motion vector is obtained based on a predetermined prediction equation from the first motion vector existing around and the second motion vector already obtained, and the second motion vector is obtained. Perform motion compensation prediction.

(4) In (3), a second reference frame is encoded from the first reference frame by the first motion compensation prediction, and the second motion vector is inverted.
A motion vector (third motion vector) from the second reference frame to the first reference frame is obtained, the second motion compensation prediction is performed using the third motion vector, and the second motion compensation prediction is performed. When one pixel on the composite frame is
When referring to a plurality of pixels on the reference frame, the average value of the plurality of pixel values is used as a predicted pixel value.

(5) In (3), the first reference frame is encoded from a lower layer frame other than the second reference frame (another lower layer frame) by the first motion compensation prediction, and the second reference frame is encoded. A motion vector (fourth motion vector) from the second reference frame to the first reference frame is obtained from the vector by a predetermined interpolation formula and extrapolation formula,
The second motion compensation prediction is performed using the fourth motion vector.

(6) In (3) or (4), as the first motion vectors existing in the periphery, those located in the second area and the first area are excluded.

(7) In (5), as the first motion vectors existing around, those located in the specific area and the first area on the another lower layer frame are excluded.

(8) (1), (2), (3),
(4), (5), (6) or (7), using the three decoded lower layer frames instead of using the two decoded lower layer frames, and combining the non-existent lower layer frames. In performing the above, the three lower layer frames are respectively referred to as a first reference frame, a second reference frame, and a third reference frame, and the specific regions on each reference frame are referred to as a first region, a second region, and a third region, respectively. Region, and in the composite frame, for pixels outside the first region, copy pixel values from pixels at the same position in the first reference frame, and for pixels in the first region, copy the pixel values in the second reference frame. Obtaining a pixel value by motion compensation prediction from pixels outside the second area;
Adaptively switching between a case where a pixel value is obtained from pixels outside the third region in the third reference frame by motion compensation prediction and a case where a pixel value is obtained by a weighted average of pixel values obtained by the two motion compensation predictions. A pixel value is obtained by interpolation, and for a pixel in which the motion compensation prediction is not performed in the first area, a pixel value is obtained by interpolation from peripheral pixels for which a pixel value has already been obtained.

(9) (1), (2), (3),
In (4), (5), (6), (7) or (8),
An encoding device obtains a motion vector used for the second motion compensation, encodes the motion vector, decodes the motion vector, and uses it for a second motion compensation prediction performed at the time of the synthesis.

[0028]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, the concept of the present invention will be described with reference to FIG. The first reference frame and the second reference frame in FIG. 9 are the same as those described in FIG. FIG.
Similarly to the above, pixels outside the first area on the composite frame are copied from the first reference frame.

For the pixels in the first area on the composite frame, the pixel values are obtained from the second reference frame by motion compensation prediction. That is, considering the movement of the automobile shown in FIG.
The rear area of the vehicle in the reference frame is copied into the first area on the composite frame. The shaded region in FIG. 9 is a region in which the pixel value cannot be correctly predicted by the motion compensation prediction in the first region.

In such an area, the pixel value is not copied, and the pixel value is obtained by interpolation from surrounding pixels. Even if the background of the selected area is moving as described above,
The lower layer frame can be synthesized so that distortion due to discontinuous pixel value changes does not occur.

Next, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing the first embodiment.
The delay unit 102, the switch 105, and the superposition unit 107 are shown in FIG.
Works the same as.

The synthesizing motion vector creating unit 101 in FIG.
A motion vector for synthesis (second motion vector) is created using the shape data of the first area and the second area and the first motion vector. Here, the first motion vector indicates a motion vector used to predict the first reference frame from the second reference frame when encoding the lower layer frame. Hereinafter, the motion compensation performed in lower layer coding is referred to as first motion compensation, and the motion compensation performed in lower layer combining is referred to as second motion compensation.

In FIG. 1, the motion compensation prediction unit 103 predicts the pixel value in the first area from the second reference frame by the second motion compensation prediction using the second motion vector. The controller 104 controls the switch to be connected to the first reference frame if the pixel value on the composite frame is outside the first area, and to the opposite side if not. The controller also controls the interpolation unit 106 to perform interpolation on an area where the pixel value cannot be correctly predicted by the motion compensation prediction unit.

Hereinafter, the first embodiment will be described in detail with reference to FIGS. In FIG. 10, it is assumed that the circular selection area moves to the right and the background moves to the left. The first reference frame is divided into blocks during lower layer coding, and each block is coded by motion compensation prediction using a first motion vector.

In the synthesis motion vector creation unit 101 shown in FIG. 1, a second motion vector is obtained for a block including the first area. At this time, the block is scanned, for example, from the upper left to the lower right in the raster scan order, and it is checked whether the block includes the first area. In FIG. 10, first, block a
, A second motion vector is determined, and then block b
, A second motion vector is determined, and the same processing is continued up to the lower right block of the frame.

FIG. 5 is a block diagram of a synthesizing motion vector creating unit used in the first embodiment. The candidate motion vector determination unit 501 determines whether the first motion vector
Or the second motion vector is selected as a candidate motion vector. For example, as shown in FIG. 14, the motion vectors MV1, MV2, MV3 on the three blocks on the upper, left, and upper right of the target block are set as candidate motion vectors.

The three first motion vectors shown in FIG. 10 are candidate motion vectors for the block a. For block b, the first and second motion vectors at the upper and upper right and the second motion vector at the left are candidate motion vectors. In some cases, no block exists at the block position of the candidate vector at the image end. Further, there is a case where the first motion vector does not exist because the mode is set to the intra mode during lower layer coding. In these cases, the candidate motion vector is set to a specific value (for example, 0).

The motion vector predictor 502 in FIG. 5 obtains a second motion vector from a candidate motion vector by a predetermined prediction formula. For example, a median is obtained for each component of the candidate motion vector, and is set as a second motion vector. Alternatively, an average may be obtained for each component of the candidate motion vector, and the average may be used as the second motion vector.

The motion compensation prediction unit 103 shown in FIG. 1 performs motion compensation prediction on a portion located in the first area on the synthesized frame using the second motion vector. For example, the first area part included in the block a in FIG. 10 is different from the area c in the second reference frame from the area c on the second reference frame.
Each is predicted from the region d on the reference frame.

In FIG. 10, for simplicity, the regions c and d
Does not overlap with the second region, but if it does, the pixel value corresponding to the overlap region is filled by interpolation as described later.

The interpolation unit 106 shown in FIG. 1 obtains, by interpolation, the pixel value of the pixel in the first area where the second motion compensation prediction is not performed under the control of the controller. That is, when the pixel position associated with the second motion vector is included in the second region from the pixel position of the first region,
This pixel is set as an interpolation target pixel.

This processing is performed for one frame,
Obtain the interpolation target area. Next, the pixel value of the interpolation target area is obtained by interpolation from peripheral pixel values. For example, an average value of peripheral pixel values is obtained, and the interpolation area is uniformly filled with the average value. Alternatively, a method of gradually interpolating pixel values from the periphery toward the inside of the interpolation area may be used.

The above-described candidate vector determination unit 5 of FIG.
In 01, it is desirable to exclude those on the first area as the first motion vectors existing around the target block. This is because, for example, the first motion vector on the first area represents the motion of the selected area, and is not appropriate as a candidate motion vector for predicting the second motion vector. When the background motion is small, the first motion vector on the second area may not correctly represent the background motion. Therefore, the first motion vector on the second area is also a candidate motion vector. May be excluded.

Further, the interpolation unit 106 shown in FIG. 1 is a block located inside the first area instead of the interpolation target area.
If the first motion vector does not exist in the vicinity of the first motion vector, all the pixels in the block may be used as the interpolation target pixels to obtain the interpolation target area. This is because when there is no background block around the block, the second motion vector cannot be obtained with high accuracy, and it may be better to use interpolation.

As described above, in the first embodiment, the second
A background motion vector (second motion vector) is predicted from a first motion vector from the reference frame to the first reference frame, and a combined frame is created based on the second motion vector.

Next, a second embodiment of the present invention will be described. In the second embodiment, as shown in FIG. 11, prediction from a first reference frame to a second reference frame is performed by first motion compensation prediction of lower layer coding. FIG. 2 shows a block diagram of the second embodiment. The delay unit 202 and the superposition unit 207 perform the same operation as in FIG.

The synthetic motion vector creating unit 201 shown in FIG. 2 uses the shape data of the first and second areas and the first motion vector to generate a motion vector (third motion vector) used for the second motion compensation.
Is obtained. FIG. 6 shows a block diagram thereof. Candidate motion vector determination unit 601, motion vector prediction unit 6
02 obtains the second motion vector by performing the same operation as in FIG. Next, the motion vector inversion unit 603 inverts the second motion vector to obtain a third motion vector.

The motion compensation prediction unit 203 shown in FIG. 2 performs motion compensation prediction from pixels of the second reference frame to pixels in the first area of the first reference frame based on the third motion vector. As shown in FIG. 11, since the direction of prediction is opposite to that of normal motion compensation, predicted blocks generally overlap each other on a synthesized frame.

With respect to the pixels in such an overlapping portion, the pixel values of each prediction block are accumulated, and the average value thereof is used as the pixel value. In addition, pixels that are not predicted occur in the synthesized frame due to overlapping prediction blocks, and this is interpolated by the interpolation unit 206 of FIG. 2 described later.

The controller 204 of FIG. 2 controls the switch 205 to be connected to the first reference frame if the pixel value on the composite frame is outside the first area, and to the opposite side if not. The controller also controls the interpolation unit 206 to perform interpolation on an area where the pixel value cannot be correctly predicted by the motion compensation prediction unit. The interpolation unit 206 obtains, by interpolation, the pixel value of the pixel in the first area where the second motion compensation prediction is not performed under the control of the controller.

As in the first embodiment, a pixel whose pixel position associated with the third motion vector is included in the second area is selected as the interpolation pixel. Further, pixels that cannot be associated with the second reference image by the third motion vector due to the overlap of the prediction blocks also become the interpolation area.
The interpolation method is the same as in the first embodiment.

As described above, in the second embodiment, the first
A motion vector of the background is predicted from the first motion vector from the reference frame to the second reference frame, and is inverted to obtain a third motion vector from the second reference frame to the first reference frame. A composite frame is created based on the motion vector of.

Next, a third embodiment of the present invention will be described. In the third embodiment, as shown in FIG. 12, in the first motion compensation prediction of lower layer coding, 2 bits used for prediction are used.
The prediction from the frame different from the one lower layer frame (another lower layer frame) to the first reference frame is performed. FIG. 3 shows a block diagram of the third embodiment. Delay section 302, motion compensation prediction section 303, controller 30
4, the switch 305, and the superimposing unit 307 operate in the same manner as in FIG.

The synthesis motion vector creating unit 301 shown in FIG. 3 uses the first area, the selected area (other area) on another lower layer frame, and the first motion vector to generate a motion vector used for the second motion compensation. A vector (a fourth motion vector) is obtained. Here, the first motion vector indicates prediction from another lower layer frame to the first reference image.

FIG. 7 is a block diagram of the synthesizing motion vector creating section. The candidate motion vector determination unit 701 and the motion vector prediction unit 702 obtain the second motion vector by performing the same operation as in FIG. Next, the motion vector extrapolation unit 703 extrapolates the second motion vector to obtain a fourth motion vector indicating prediction from the second reference image to the first reference image.

FIG. 15A shows an example of the extrapolation process. Here, each frame is viewed from the side. The second motion vector indicated by the solid line is the ratio t of the frame intervals in FIG.
1: A fourth motion vector indicated by a dotted line is obtained based on t2. In the present embodiment, extrapolation processing as shown in FIG. 15A is performed, but the extrapolation processing is as shown in FIGS. 15B to 15D depending on the positional relationship of each frame. However, FIG.
In 5 (d), interpolation is used instead of extrapolation.

As described above, in the third embodiment, the motion vector of the background is predicted from the first motion vector from another lower layer frame to the first reference frame, and extrapolated to predict the second motion vector. A fourth motion vector from the frame to the first reference frame is obtained, and a combined frame is created based on the fourth motion vector.

Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, as shown in FIG. 13, in the first motion compensation prediction, a first reference frame is predicted from two frames, a second reference frame and a third reference frame. FIG. 4 shows a block diagram of the fourth embodiment. The superimposition unit 408 operates in the same manner as in FIG.

The synthesis motion vector creation unit 401 shown in FIG. 4 obtains a second motion vector from the first motion vector and the already obtained second motion vector by a predetermined prediction formula. In the present embodiment, the first motion vector of each block may be one (forward prediction vector or backward prediction vector) or two (forward prediction vector and backward prediction vector).

For example, all the blocks around the block a in FIG. 13 (three blocks shown in FIG. 14) have only forward prediction vectors. In such a case, a forward prediction vector is also obtained as the second motion vector. As for the block b in FIG.
(First motion vector and second motion vector) and two backward vectors (first motion vector), the second motion vector of block b is used for forward prediction and backward prediction. Find two motion vectors.

The method of obtaining each of the second motion vectors is as follows.
This is the same as the embodiment. Generally, the number of forward vectors and backward vectors is checked for peripheral blocks, and the larger prediction direction is used for the second motion compensation. When the number of vectors is the same, bidirectional prediction is used for the second motion compensation.

The motion compensation prediction unit 404 in FIG. 4 obtains the pixel value of the first region of the first reference frame from the second and third reference frames by using the second motion vector by bidirectional motion compensation prediction. . The first reference frame is the first
And the second reference frame is the output from the second delay unit 403.

For example, the block a in FIG. 13 is forward predicted from the area c of the second reference frame, and the block b is bidirectional predicted from the area d of the second reference frame and the area e of the third reference frame. However, in the case of bidirectional prediction, when one of the prediction blocks includes a pixel in the selected area, unidirectional prediction is performed without using the pixel of the prediction block.

For example, in FIG. 13, although the block b is predicted from the region d and the region e, the portion of the region d that overlaps the second region is not subjected to the bidirectional prediction, and the backward prediction from the region e is used.

The controller 405 in FIG. 4 controls the switch 406 as in the first embodiment, and switches the pixel value of the first reference frame and the pixel value from the motion compensation prediction unit. The controller also controls the interpolation unit 407 using the shape data of the first, second, and third regions and the second motion vector, and the interpolation unit calculates a pixel value of a pixel that is not predicted by the second motion compensation prediction. Interpolate.

As described above, in the fourth embodiment, the background motion vector is predicted from the first motion vector used for the bidirectional prediction of the lower layer, and the first reference is used from the second and third reference frames. A combined frame is created by performing bidirectional prediction on the frame. Here, the case where the lower layer frame is bidirectional is described. However, even when the lower layer frame is unidirectionally predicted, the bidirectional prediction can be used in the lower layer synthesis.

For example, in the encoding of the lower layer, the forward prediction from the second reference frame to the first reference frame and the first prediction
When forward prediction from a reference frame to a third reference frame is used, a second motion vector is obtained from the second reference frame by the method of the first embodiment, and the second motion vector is obtained from the third reference frame. If the third motion vector is obtained by the method of the embodiment, a combined frame can be created from the second reference frame and the third reference frame using bidirectional prediction.

Next, a fifth embodiment of the present invention will be described. In the fifth embodiment, a part of a motion vector used to create a composite frame is obtained by an encoding device,
Encode. FIG. 8 shows a block diagram of the fifth embodiment.

On the encoding apparatus side in FIG. 8, first, a candidate motion vector determination unit 801 and a motion vector prediction unit 802 obtain a second motion vector by the same method as in the first embodiment. Next, using the second motion vector as an initial vector, the motion vector search unit 803 performs a vector search by block matching to obtain a more accurate motion vector (corrected motion vector) for the background portion of the first area.

However, the blocks for obtaining the corrected motion vector are limited to blocks including both the first area and other areas. At this time, only pixels corresponding to areas outside the first area in the block are used for vector matching. The corrected motion vector is fed back to the candidate motion vector determination unit and used for the next motion vector prediction.

The motion vector encoding unit 804 of FIG. 8 encodes the corrected motion vector obtained by the motion vector search unit 803. In the encoding of the corrected motion vector, only the difference from the corresponding second motion vector (used as the initial vector) is encoded.

On the decoding apparatus side in FIG. 8, the candidate motion vector determination unit 805 and the motion vector prediction unit 806 determine the second motion vector in the same manner as in the first embodiment. These operations are the same as those on the encoding device side.

Next, the corrected moving vector decoding unit 807 decodes the corrected moving vector from the second moving vector and the encoded data of the corrected moving vector. The decoded correction vector is supplied to the candidate motion vector determination unit 805, and is used for the next motion vector prediction.

The controller 808 in FIG.
9 if the current block includes only the first region, the switch is connected to the motion vector predictor, and if the current block includes the first region and other regions, the switch is connected to the opposite side, The second motion vector or the corrected motion vector is output to the motion compensation prediction unit. The motion compensation prediction unit (not shown) performs the second operation by the same operation as that used in the first embodiment.
A background pixel in the first region of the first reference frame is predicted from the reference frame.

As described above, in the fifth embodiment, the second motion vector is corrected for the block corresponding to the boundary of the selected area, and a higher-quality composite frame is obtained.

[0076]

【The invention's effect】

(1) According to the moving picture coding apparatus and the moving picture decoding apparatus of the present invention, when combining an uncoded lower layer from a decoded lower layer, the background of the selected area of the upper layer moves. In this case, it is possible to create a good synthesized frame and to reproduce a high quality image when only the selected area is reproduced at a high frame rate by decoding up to the upper layer.

(2) When the motion compensation prediction for lower layer synthesis is performed using the motion compensation prediction of the pixel value of the lower layer, the former motion compensation prediction can be performed efficiently.

(3) If the motion compensation prediction for lower layer synthesis and the motion compensation prediction of the pixel value of the lower layer are in the same direction, the motion vector used for the former motion compensation prediction is used for the former motion compensation prediction. In order to obtain a motion vector to be used, the former motion compensation prediction can be performed by simple processing.

(4) When the motion compensation prediction for lower layer synthesis and the motion compensation prediction of the pixel value of the lower layer are in the opposite directions, the motion vector used in the latter motion compensation prediction is inverted to the former. In order to obtain a motion vector to be used in the motion compensation prediction, it is possible to perform the former motion compensation prediction by a simple process.

(5) When the motion compensation prediction of the pixel value of the lower layer is performed using a frame different from the frame used for the synthesis of the lower layer as a reference image, interpolation and extrapolation of the motion vector are performed. A motion vector used for motion compensation prediction for lower layer synthesis is obtained, and motion compensation prediction for synthesis can be performed efficiently.

(6) A first motion vector around a target area for which motion compensation prediction for lower layer synthesis is performed in (3) or (4) (a motion vector for motion compensation prediction of the pixel value of the lower layer) In the case of excluding those located in a specific area on the lower layer used for composition, a good composite image can be obtained.

(7) Also, as the first motion vector around the target area for which the motion compensation prediction for lower layer synthesis is performed in (5), a vector located in a specific area on the another frame is excluded. In this case, it is possible to obtain a good composite image.

(8) The above (1), (2),
In (3), (4), (5), (6) or (7), when using bidirectional motion compensation prediction as motion compensation prediction for lower layer synthesis, a higher quality composite image is obtained. Becomes possible.

(9) The above (1), (2),
(3), (4), (5), (6), (7) or (8)
Then, when encoding the corrected motion vector on the encoding side,
By accurately performing synthesis in the vicinity of the selected area, it is possible to obtain a higher quality reproduced image.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an encoding device according to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating an encoding device according to a second embodiment of the present invention.

FIG. 3 is a block diagram illustrating an encoding device according to a third embodiment of the present invention.

FIG. 4 is a block diagram illustrating an encoding device according to a fourth embodiment of the present invention.

FIG. 5 is a diagram illustrating a compositing motion vector creating unit according to the first embodiment;

FIG. 6 is a diagram illustrating a synthesis motion vector creation unit according to a second embodiment.

FIG. 7 is a diagram illustrating a compositing motion vector creating unit according to a third embodiment;

FIG. 8 is a block diagram illustrating an encoding device according to a fifth embodiment of the present invention.

FIG. 9 is a diagram illustrating the concept of the present invention.

FIG. 10 is a diagram illustrating a first embodiment.

FIG. 11 is a diagram illustrating a second embodiment.

FIG. 12 is a diagram illustrating a third embodiment.

FIG. 13 is a diagram illustrating a fourth embodiment.

FIG. 14 is a diagram illustrating prediction of a motion vector according to the present invention.

FIG. 15 is a diagram illustrating extrapolation and interpolation of a motion vector according to the present invention.

FIG. 16 is a diagram illustrating a conventional lower layer combining method.

FIG. 17 is a diagram illustrating a problem of the conventional technique.

FIG. 18 is a diagram illustrating the concept of time stratification.

[Explanation of symbols]

 101 Synthetic motion vector creation unit 102 Delay unit 103 Motion compensation prediction unit 104 Controller 105 Switch 106 Interpolation unit 107 Superposition unit

Continued on the front page (72) Inventor Norio Ito 22-22 Nagaikecho, Abeno-ku, Osaka-shi, Osaka Inside Sharp Corporation

Claims (9)

[Claims] 1. A lower layer encoding method for encoding a moving image sequence at a low frame rate, and an upper layer encoding method for encoding one or more specific regions of the moving image sequence at a high frame rate. Decoding only the lower layer having a lower frame rate, and decoding up to the upper layer, decoding the lower layer and the upper layer, superimposing the lower layer and the upper layer, and setting a higher frame rate than the lower layer for the specific area. A hierarchical encoding device and a decoding device for obtaining, when decoding, when there is no lower layer frame corresponding to the same frame position as the upper layer, using the two lower layer frames thus decoded. Layer frames are synthesized, and in the synthesizing, one lower layer of the two lower layer frames is used. Frame first reference frame, the other as a second reference frame,
The specific area on the first reference frame is a first area, the specific area on the second reference frame is a second area,
In the composite frame, for pixels outside the first area, pixel values are copied from pixels at the same position in the first reference frame, and for pixels in the first area, the second area in the second reference frame is copied. A pixel value is obtained from outside pixels by motion compensation prediction, and for a pixel in which the motion compensation prediction is not performed in the first area, a pixel value is obtained by interpolation from peripheral pixels for which a pixel value has already been obtained. A moving picture encoding apparatus and a moving picture decoding apparatus, which are characterized. 2. The moving picture encoding apparatus and the moving picture decoding apparatus according to claim 1, wherein the lower layer frame is encoded by a first motion compensation prediction means, and the motion compensation prediction in the synthesis is performed by a second motion compensation prediction. By means,
A moving picture coding apparatus and a moving picture decoding apparatus, wherein the synthesis is performed by using a first motion vector used in the first motion compensation prediction for the second motion compensation prediction. 3. The moving picture encoding apparatus and the moving picture decoding apparatus according to claim 2, wherein a first reference frame is encoded from a second reference frame by the first motion compensation prediction, and In obtaining the second motion vector, and obtaining the second motion vector, a second motion vector is obtained based on a predetermined prediction formula from the surrounding first motion vector and the already obtained second motion vector. 2. A moving picture encoding apparatus and a moving picture decoding apparatus, wherein two moving vectors are obtained and the second motion compensation prediction is performed. 4. The moving picture encoding apparatus and the moving picture decoding apparatus according to claim 3, wherein a second reference frame is encoded from the first reference frame by the first motion compensation prediction, and the second motion vector is encoded. To obtain a motion vector (a third motion vector) from the second reference frame to the first reference frame, perform the second motion compensation prediction using the third motion vector, and perform the second motion compensation prediction. When one pixel on the synthesized frame refers to a plurality of pixels on the second reference frame at the time of the motion compensation prediction, an average value of the plurality of pixel values is used as a predicted pixel value. A video encoding device and a video decoding device. 5. The moving picture encoding apparatus and the moving picture decoding apparatus according to claim 3, wherein a first reference frame is converted from a lower layer frame other than a second reference frame (another lower layer frame) to the first motion. Encoding is performed by compensated prediction, and a motion vector (fourth motion vector) from the second reference frame to the first reference frame is obtained from the second motion vector by a predetermined interpolation formula and extrapolation formula. A video encoding device and a video decoding device, wherein the second motion compensation prediction is performed using the motion vector of 6. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 3 or 4, wherein the first and the second surroundings exist.
A moving vector encoding apparatus and a moving image decoding apparatus, wherein a moving vector located in the second area and the first area is excluded. 7. The moving image encoding device and the moving image decoding device according to claim 5, wherein the specific region and the first region on the another lower layer frame are used as the first moving vector existing around the moving image encoding device and the moving image decoding device. A video encoding device and a video decoding device, characterized in that the video coding device and the video decoding device are characterized by excluding those located in the video data. 8. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 1, wherein decoding is performed instead of using the two decoded lower layer frames. In synthesizing the nonexistent lower layer frames using three lower layer frames, the three lower layer frames are referred to as a first reference frame, a second reference frame, and a third reference frame, respectively. The above specific areas are respectively referred to as a first area and a second area.
In the combined frame, pixel values of pixels outside the first region are copied from pixels at the same position in the first reference frame, and pixels in the first region are copied in the first frame. A case where a pixel value is obtained by motion compensation prediction from a pixel outside the second region in the second reference frame; a case where a pixel value is obtained by motion compensation prediction from a pixel outside the third region in the third reference frame; The pixel value is obtained by adaptively switching between the case where the pixel value is obtained by the weighted average of the pixel value obtained by the compensation prediction, and the pixel value for which the motion compensation prediction is not made in the first area is already obtained. A moving image encoding device and a moving image decoding device, wherein a pixel value is obtained by interpolation from neighboring pixels that have been set. 9. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 1, 2, 3, 4, 5, 6, 7, or 8, wherein a moving picture used for the second motion compensation by the coding apparatus. Vector, the motion vector is encoded, and the decoding device decodes the motion vector,
A moving picture coding apparatus and a moving picture decoding apparatus, which are used for motion compensation prediction.