JP3339544B2 - Dissolve detection method and device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a method and an apparatus for detecting a dissolve from compression-encoded video data.

[0002]

2. Description of the Related Art The amount of video data is enormous, and the only way to know its contents is to look at all the videos in chronological order. If the video is divided based on a certain standard, it is useful for skipping the video and understanding the contents roughly.

When a video is viewed in chronological order, a point at which a shot is switched is called a cut point. Since the content of the image changes abruptly before and after the cut point, the cut point can be detected by a method of “calculating the difference between adjacent images in chronological order and treating a portion having a large difference as a cut point”. For example, a “moving image automatic division method” that calculates a change of a continuous frame from the number of pixels whose physical quantities such as luminance and color at a position (x, y) of a continuous image frame have changed and detects a cut point (see Kaihei 5-37853). Under these circumstances, it has been a problem to detect a slowly changing scene change such as dissolve in a stable manner.

[0004] The dissolve is a type of scene change in which the image content gradually changes from scene A to scene B. Fade-in and fade-out in which an image gradually disappears or changes from white can be considered as a special case where the scene A or the scene B is a single color of white or black. These scene changes are generally linear changes, and the luminance and color components gradually change.

The characteristics of the dissolve change will be described with reference to FIG. Scene A (41) in the original image sequence (FIG. 5 (1))
To the scene B (45) gradually over the T frame 42. In the process of changing from scene A to scene B, the corresponding pixel (x, y) gradually changes from the component value of scene A to the component value of scene B (FIG. 5 (2)). . The component value of each pixel in the dissolve is
It is represented by the following equation.

[0006]

(Equation 1) Here, I (x, y) _A and I (x, y) _B are blocks I (x, y) in an image in scenes A and B, respectively.
y) means the component, T is the total number of frames in the dissolve generation section, and t is the frame number counted from the first frame where the dissolve has started. 43 and 44 are luminosity at t ₁ th frame. In the dissolve, the above-described slowly changing frame continues continuously over the entire screen. When encoding is performed, such a gradual change is subjected to motion prediction. Therefore, in a P picture sequence of encoded data, difference information between a motion vector due to the motion prediction and blocks on a reference frame is recorded. ing. Therefore,
Motion estimation is performed, and only difference information between blocks on the reference frame is recorded and transmitted. Thus, for example, in the case of encoding scheme as P Bikuchiya is continuous, DC component of the difference information 46 contained in the t ₁ frame of blocks of P picture in the change dissolve (Fig. 5 (3))
Is

[0007]

(Equation 2) Becomes The feature amount reflecting such a phenomenon is calculated by the slowly changing region calculation unit. It should be noted that even in the case of a scene change that changes slowly, a wipe in which a part of the content is continuously replaced between the scene A and the scene B without going through the process of fusing the video is treated separately from the dissolve.

In detecting a change in dissolve, it is difficult to distinguish whether a change in an image component such as luminance is caused by a movement or a change caused by a gradual change in a dissolve or the like. There is a problem that a scene in which a subject moves is erroneously detected as a dissolve. Therefore, the above problem is solved by obtaining a motion vector or an optical flow.
tic partitioning of full-
motion video "(Hong Jian
Zhang, Atreyi Kankanhalli, S
tephenW. Smolial, Multimedi
a Systems 1993: 10-28).

In order to detect a dissolve from a compression-encoded video such as the MPEG system, the only way to detect the dissolve is to decode the data once to recover an uncompressed digital video and use the conventional technique. However, the decoding process has a problem that it takes a long time to calculate. Further, the above-described dissolve detection technique has a problem that it takes a very long time to calculate a motion vector. It is inefficient to require an enormous amount of time for both the decoding process and the process of calculating the motion vector. Therefore, a technique for detecting a dissolve without requiring a decoding process is required, but such a technique does not exist at present.

[0010]

In the case where a dissolve is detected from encoded and compressed data, after performing a decoding process,
Although it is necessary to perform the dissolve detection processing according to the conventional method, there is a problem that both the decoding processing and the dissolve detection processing require time.

SUMMARY OF THE INVENTION An object of the present invention is to solve the problems of the prior art, and to perform a dissolve without performing a decoding process from a compressed coded video and without requiring a time-consuming process such as calculating a motion vector. Is to provide a dissolve detection method and apparatus for detecting a high speed.

[0012]

According to the present invention, there is provided a dissolve detection method comprising the steps of: converting an image data sequence compressed by an encoding method including an inter-frame or inter-field predictive encoding method; Of information , dynamic
Of the luminance difference DC component in the block to which
Sign (+, 0, -) and change trend calculating step of calculating as the change tendency, over a plurality of adjacent frames, slow changes change trend gradual change blocks continue to the constant change has occurred The method includes a slowly changing region extracting step of extracting a region as a region, and a determining step of determining that a dissolve is present when a ratio of the slowly changing region to the screen is larger than a threshold value.

Further, the dissolve detection apparatus according to the present invention converts the image data string compressed by the compression coding method including the inter-frame or inter-field predictive coding method from the coding information included in the image data string . Motion prediction
The sign (+,
A change tendency calculating unit that calculates 0,-) as a change tendency;
Over a plurality of adjacent frames, strange change trend is constant
Region extracting unit that extracts a block that keeps changing as a slowly changing region in which a gradual change has occurred, and determines that a dissolve exists when the ratio of the slowly changing region to the screen is larger than a threshold value It has a determination unit to perform.

According to the present invention, a change tendency of an image sequence is calculated from coding information included in an image data sequence compressed by an inter-frame or inter-field predictive coding method. When the ratio of the area showing the same change tendency over the frame to the whole screen is larger than the threshold value, it is determined that the dissolve exists. As described above, in the present invention, since the feature information is directly extracted from the encoded data without decoding the data, the dissolve can be detected at high speed.

[0015]

According to another embodiment of the present invention, in the step of calculating the change tendency, among the encoded information included in the image data sequence, the luminance of the block to which the motion prediction has been applied.
When calculating the change tendency based on the sign (+, 0,-) of the DC component of the difference, the sign of the difference information is changed in consideration of which direction the motion prediction is from, the past or the future. , The change tendency is calculated.

[0017] Other dissolve detection method of the present invention, frame
Compressed code including inter-frame or inter-field predictive coding
From the image data sequence compressed by the
Of the encoding information included in the data sequence, two consecutive
A block at a corresponding position in the frame,
Luminance difference DC component in block to which motion prediction was applied
Trend calculation stage for calculating the absolute value of the difference between
The luminance difference over the floor and a plurality of adjacent frames.
A block whose absolute value of the DC component difference is smaller than a predetermined threshold value
The region as a gradual change area where a gradual change has occurred.
Extracting a gradual change area, and assigning the gradual change area to the screen.
When the dissolve is present when
The determining step is provided. Other dissolve detection of the present invention
The output device uses an inter-frame or inter-field prediction coding method.
Image data compressed by the compression coding method including the formula
From the sequence, among the encoding information included in the image data sequence,
The block existing at the corresponding position in two consecutive frames
And the brightness of the block on which the motion
Calculates the absolute value of the difference of the DC component
The change tendency calculation unit and, over a plurality of adjacent frames,
The absolute value of the difference between the luminance difference DC components is smaller than a predetermined threshold value.
Smaller blocks and gradual change areas with gradual changes
Slowly changing area extraction unit for extracting
Is greater than the threshold, the dissolve
It has a determination unit that determines that it exists.

According to the present invention, a change tendency is calculated by referring to difference information to be allocated in a region (non-intra block in MPEG) to which motion prediction is applied, which is included in an inter-frame or inter-field predictive coded image. . Inter-frame or inter-field predictive coding is
An area similar to the encoding target area is searched from images at different times, and motion vector information indicating the position of the similar area and difference information between the encoding target block and the reference area are stored. Since the motion vector is calculated in advance by the motion prediction, the difference information is equal to a change in the image content after removing the motion of the camera and the motion of the subject. That is, it is possible to handle a change in the image content without decoding the encoded image and then removing the movement of the camera or the like.

From such difference information, it is calculated how the image shows a change tendency, and an area showing a certain change tendency is calculated as a slowly changing area. Since the dissolve has a characteristic that changes gradually over a plurality of frames, the dissolve can be detected from the magnitude relationship of the ratio of the gradual change area to the screen.

[0020]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a flowchart of a dissolve detection method according to an embodiment of the present invention.

An inter-frame forward predictive coded image sequence 2 is extracted from an input compressed image sequence 1 and input to a change tendency calculating step 3. In the change tendency calculation step 3, the change tendency 4 of each image sequence is calculated. The change tendency 4 is input to the slowly changing region extraction stage 5. In the slowly changing region extracting step 5, a feature amount for measuring a change in image content of each image sequence over a long time is extracted as a slowly changing region 6 of the image sequence. Examples of the change tendency calculating step 3 and the slowly changing area extracting step 5 will be described later. The gradual change area 6 is input to the determination step 7, and a final dissolve 8 is obtained based on the ratio of the gradual change area 6 to the screen.

In the change tendency calculating step 3, a symbol is used to indicate how the characteristic amounts such as brightness, hue, texture, edge, and the like, which characterize the image, change. For example, the change tendency of the brightness can be quantified by the following simple method. With reference to the difference information allocated in the area (non-intrablock in MPEG) assigned to the motion prediction included in the coded information, the sign {+, 0,-} of the luminance difference DC component is represented by a symbol indicating a change tendency. And The symbol '+' indicates that the brightness increases between fields, the symbol '0' indicates that it is constant, and the symbol '-' indicates that the brightness decreases. As for the hue, the sign of the color difference DC component can be used similarly to the brightness.

In the gradual change region calculation stage 5, the change tendency is
An area that keeps a constant change over several frames is counted. For example, this is a step of calculating an area that uniformly changes over several frames characteristic of a dissolve, such as a gradually increasing brightness.

FIG. 2 is a block diagram showing the configuration of the dissolve detection device according to one embodiment of the present invention.

This dissolve detection device detects a dissolve 8 from the input compressed image data sequence 1. In the present embodiment, the input compressed image data stream 1, the image data sequence {P _t-2, which is a P-picture which has been compressed by inter-frame or inter-field predictive coding, P _t-1, P _{t,
Pt + 1} , _{Pt + 2} } are handled. The input compressed image data sequence 1 is
The image file may be an image file stored in a storage device such as a hard disk or a CD-ROM, or may be an image file connected to a network and transmitted. size,
The sample rate is arbitrary, but it must incorporate an inter-frame or inter-field predictive coding scheme. As an example, H. H.261, MPEG1, and MPEG2.

An input compressed image data sequence 1 is analyzed by an encoding information analysis unit 10 to analyze picture data, a serial number from the first picture, a picture type, and the like, and a data sequence memory 11 for storing data over several tens of frames. And the information is stored together. In the data string memory 11, in addition to supplementary information such as all or a part of the encoded data, the frame number from the first file of the image, and the feature amount obtained by the pre-processing, the data is acquired using an external device and added. External information such as a shooting time and a shooting location to be performed, and external data 9 which is user data separately input by a user may be stored as necessary.

The P picture sequence 2 (P _t ) of the image sequence read from the data sequence memory 11 is used as a change tendency calculating unit 12.
, And the change tendency 4 is calculated. The change tendency over a plurality of frames is compared with the change tendency 4, and a gradual change area 6 (DL), which is a gradual change area over a plurality of frames, is compared.
P _t ) is calculated. Obtained slow change region 6 (DLP _t )
Is input to the dissolve determination unit 14, and when the rate of the gradual change area 6 occupying the entire screen is larger than the threshold,
The final dissolve 8 is determined.

Next, detailed examples of the slowly changing region extraction unit 13 and the dissolve determination unit 14 will be described in order.

The change tendency calculating section 12 and the slowly changing area calculating section 1
3 and a first example of the dissolve determination unit 14 will be described with reference to FIG. Here, attention is paid to the +/- signs included in the difference information.

In the first example, with reference to the inter-frame prediction coded frame, an area having a sign having the same difference between the luminance and the chrominance in three consecutive frames is defined as a slowly changing area. Specific processing will be described with reference to the flowchart of FIG. The variables used here will be described. Variable sign
(I, j) holds the sign (+, 0,-) of the luminance difference d (i, j) (i, j is a variable indicating the position of the block) of the block (i, j) in the previous frame. Also, item
r (i, j) is a variable for measuring how many times the same code appears in the block (i, j). Also, c
nt is a variable for counting the number of blocks having the same sign over three consecutive frames.

First, in step 21, i and j are set to 0.
Initialize to Step 22 determines whether or not the block has been hit by motion prediction having a luminance difference. Step 23 determines whether the sign of the current luminance difference and the sign of the past luminance difference are both positive. Step 24 determines whether the sign of both the current luminance difference sign and the past luminance difference sign satisfies a negative condition. If steps 23 and 24 are satisfied, those areas are determined to be slow change areas, and the slow change areas are counted in step 25. On the other hand, if the conditions in Steps 23 and 24 are not satisfied, the past code and the current code are different, so the process jumps to Step 29, where the code information used for the presence determination process of the slowly changing area in the next frame and thereafter is Reset. In addition, since the sign of the luminance difference has also changed, i is an accumulation of codes from the past.
The value of ter (i, j) is also reset at step 30.

On the other hand, when it is determined in step 22 that the block has no luminance difference, the block (i, j) is generally an intra-block coded block. The intra-block coded block appears when the pixel greatly changes and the motion prediction is incorrect. In the coding of the dissolve change, it is assumed that there may be an intra-block coded block in the course of the change, and step 2
The processing is continued according to the conditions of 3 to 24 being satisfied.

Furthermore, depending on the encoding process, an exceptional block that is neither a block to which motion prediction has been applied nor an intra-block encoded block may appear. In such a case, it is difficult to guess what the value of the encoded information is, so that it is ignored. The code information of the luminance difference used in the process of determining the presence of the slowly changing area after the next frame stores the past code information, and the number of consecutive codes is reset. Since these exceptional blocks occur very rarely, it is assumed that all the blocks for which the motion prediction is incorrect are intra-coded blocks,
The processing may be continued. That is, “N” in step 26
The path from “o” to step 30 may be eliminated.

In the above processing, iter (i, j) stores the number of times the same code appears continuously. Next, in step 27, it is determined whether or not iter (i, j) is 3 or more. If it is 3 or more, 1 is added to cnt in step 28. At steps 31 to 34, the above processing is performed for all blocks.

As a result of the above processing, the variable cnt
Indicates the area of a region having the same sign with the same luminance difference over three consecutive frames. If the area of the slow change region indicated by the variable cnt is larger than a certain threshold,
It is determined that a dissolve exists.

In the above example, in step 22, the condition that the predicted block is used is used.
| D (i, j) | may be changed to a condition that it is smaller than a certain positive value. Also, instead of performing the processing of the block encoded in the frame (step 26), i
Various modifications can be made, such as not changing the value of ter (i, j).

The change tendency calculating section 12 and the slowly changing area extracting section 1
3 and a second example of the dissolve determination unit 14 will be described with reference to FIG. Here, attention is paid to the size of the difference information.

In the second example, an area having the same difference in luminance between two consecutive frames is referred to as a slowly changing area with reference to an inter-frame predictive coded frame.
Specific processing will be described with reference to the flowchart of FIG.
The variables used here will be described. The variables are the same as in the first example (FIG. 3), but _dt (i, j) holds the value of the luminance difference of block (i, j) in t frames. Instead of steps 23 and 24 in the first example, the processing of step 35 is performed, and _dt (i, j) and _dt-1 (i,
If the absolute value of the difference in j) is smaller than the threshold value σ, the difference value indicates the same value. In step 35, the threshold value σ is fixed, but σ may be changed adaptively. Further, the method may include a step of determining a value of σ based on a quantization step included in the encoded information.

In this example, the input video is only an image sequence (P-picture sequence) according to the inter-frame or inter-field forward predictive coding method. An image (B picture) by an inter- or inter-field bidirectional predictive encoding method, or an image (I-picture by an intra-frame or
Picture) etc. appear. In this example, I pictures and B pictures are ignored. However, since the B picture holds the difference information like the P picture, the change tendency is calculated also from the B picture, and the slowly changing area is extracted.
Processing such as application to dissolve detection is also possible.
However, in this case, since the B picture refers to both the past and future images, the difference information is determined by considering whether the difference information is one or both of the past and future images. It is necessary to include processing such as reversing the sign.

[0041]

As described above, the present invention detects a dissolve in a compression-encoded image data sequence using various types of information contained in an inter-frame or inter-field predictive encoded image. Accordingly, it is possible to detect the dissolve at high speed without decoding the data.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating a dissolve detection method according to an embodiment of the present invention.

FIG. 2 is a configuration diagram of a dissolve detection device according to an embodiment of the present invention.

FIG. 3 is a flowchart illustrating a first example of a change tendency calculation unit 12, a slowly changing region calculation unit 13, and a dissolve determination unit 14.

FIG. 4 is a flowchart showing a second example of the change tendency calculation unit 12, the gradual change region calculation unit 13, and the dissolve determination unit 14.

FIG. 5 is a schematic diagram of a dissolve change.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 input compressed image data 2 inter-frame forward predictive coded image sequence 3 change tendency calculation stage 4 change tendency 5 slow change region extraction stage 6 slow change region 7 determination stage 8 dissolve 9 external data 10 encoded information analysis unit 11 data sequence memory 12 change trend calculating unit 13 slow change region extraction section 14 dissolve determination unit 21-35 step 41 scene A 42 the number of frames _{T 43 t 1 44 I (x} , y) A + (I (x, y) A -I ( x,
y) _B ) / Tt ₁ 45 Scene B 46 Difference information for each frame

Continuation of the front page (56) References: Boon-Lock Yeo and Bad Liu, Rapid Scene Analysis on Compressed Video, IEEE TRANSACTIONS ON CIRCUITS AND VOL. 5 No. 6, p. 533-544 (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68 H04N 5/262-5/278

Claims (5)

(57) [Claims] An image data sequence compressed by a compression coding method including an inter-frame or inter-field predictive coding method, from among coding information included in the image data sequence, a luminance in a block to which a motion prediction is applied. Difference DC
Sign components (+, 0, -) and change trend calculating step of calculating as the change tendency, over a plurality of adjacent frames, the change trend gradual change blocks continue to the constant change has occurred A dissolve detection method comprising: a slowly changing region extracting step of extracting as a slowly changing region; and a determining step of determining that a dissolve is present when a ratio of the slowly changing region to the screen is larger than a threshold value. 2. In the change tendency calculating step, the sign (+, 0, 0) of the luminance difference DC component in the block to which the motion prediction has been applied is included in the encoding information included in the image data sequence.
-) a when calculating the change tendency based, taking into consideration whether the motion prediction is predicted from either direction past and future, the
The dissolve detection method according to claim 1, wherein a sign of the luminance difference DC component is changed to calculate a change tendency. 3. An image data sequence compressed by a compression coding method including an inter-frame or inter-field predictive coding method, and corresponding to two consecutive frames in coding information included in the image data sequence. A change tendency calculating step of calculating, as a change tendency, an absolute value of a difference of a luminance difference DC component in a block which is located at a position and which has been subjected to motion prediction; A step of extracting a block in which the absolute value of the component difference is smaller than a predetermined threshold value as a slowly changing area in which a gradual change has occurred; and A dissolve detection method having a determining step of determining that a dissolve is present when it is larger. 4. An image data sequence compressed by a compression coding method including an inter-frame or inter-field predictive coding method, from the coding information included in the image data sequence, the luminance in a block to which motion prediction has been applied. Difference DC
Sign components (+, 0, -) and change trend calculating unit which calculates the change trend, over a plurality of adjacent frames, the change trend gradual change blocks continue to the constant change has occurred A dissolve detection device comprising: a slowly changing region extracting unit that extracts a slowly changing region; and a determining unit that determines that a dissolve exists when a ratio of the slowly changing region to the screen is larger than a threshold. 5. An image data sequence compressed by a compression coding method including an inter-frame or inter-field predictive coding method, the coding information included in the image data sequence corresponding to two consecutive frames. A change tendency calculating unit for calculating, as a change tendency, an absolute value of a difference between the luminance difference DC components in a block which is located at a position and which has been subjected to motion prediction; A slowly changing region extracting unit that extracts a block whose absolute value of the component difference is smaller than a predetermined threshold value as a slowly changing region in which a gentle change has occurred; A dissolve detection device having a determination unit that determines that a dissolve is present when it is large.