JP4525064B2 - Motion vector detection apparatus, motion vector detection method, and computer program - Google Patents

Description

  The present invention relates to a motion vector detection device, a motion vector detection method, and a computer program. More specifically, the present invention relates to a motion vector detection device, a motion vector detection method, and a computer program that execute a motion vector detection process from moving image data.

  With the recent enhancement of information processing devices and communication terminals, high-speed communication infrastructure, and the spread of high-density recording media such as DVDs and Blu-ray discs, Storage and reproduction of moving image data using a high-density recording medium has been actively performed. Along with this situation, there is a demand for improved efficiency and high speed in data processing for moving image data, for example, encoding processing.

  Motion-compensated image coding in high-efficiency coding of moving image data, moving object detection processing, speed detection processing, etc. in a visual sensor of a traffic monitoring system or autonomous vehicle, movement of each object included in the image data Therefore, a process for detecting the direction and magnitude (speed) of the image, that is, a motion vector detection process is required.

  For example, as an example of motion compensation type image encoding processing, an MPEG (Moving Picture Coding Experts Group) method, which is an international standard method for high-efficiency video encoding, has been proposed. This MPEG method is a DCT (Discrete). This is a method of performing coding that combines (Coscine Transform) and motion compensated prediction coding. In motion-compensated predictive coding, a correlation between image signal levels in a current frame constituting moving image data and a continuous frame of the immediately preceding frame is detected, and a motion vector is obtained based on the detected correlation. Efficient encoding is achieved by performing a motion image correction process based on the motion vector.

As one of motion vector detection methods, a block matching method is known. The outline of the block matching method will be described with reference to FIG. For example, a current frame [F _t ] 20 at time (t) and a previous frame [F _t-1 ] 10 at time (t−1) shown in the figure are extracted. . One screen of the frame image is divided into small areas (hereinafter referred to as blocks) composed of a plurality of pixels, m pixels × n lines.

The current frame [F _t ] 20 is set as a reference frame, and the check block By21 of the reference frame is moved within a predetermined search area 22, and has the smallest pixel value difference from the reference block Bx11 of the previous frame [F _t-1 ] 10. That is, the test block having the best pixel value (the most correlated) is detected. It is estimated that the reference block Bx11 of the previous frame [F _t−1 ] 10 has moved to the position of the highly correlated test block detected from the current frame [F _t ] 20. Based on the vector indicating the estimated motion, a motion vector of each pixel is obtained. As described above, the block matching method is a method for obtaining a motion vector by performing a correlation determination (matching determination) process between frames in a predetermined block (m × n) unit.

  In the block matching method, a motion vector is obtained for each block. As an evaluation value representing the correlation of each block, that is, the degree of coincidence, for example, values of pixels at the same spatial position between a plurality of pixels in the reference block Bx and a plurality of pixels in the inspection block By The frame difference is obtained by subtracting the frame difference, and the sum of the absolute values of the calculated frame differences is added. Alternatively, the sum of squares of frame differences can be used.

  However, since the block matching method described above is a full search for comparing all data in the search area, the number of comparisons required for detection is very large, and there is a drawback that it takes time for motion detection.

  In addition, when a moving part and a stationary part are included in a block, it cannot be said that the movement detected in units of blocks corresponds to the movement of individual pixels in the block. Such a problem can be adjusted by setting the block size. For example, if the block is enlarged, the problem of multiple motions in the block is likely to occur in addition to an increase in the amount of calculation. On the other hand, when the block size is reduced so that a plurality of motions are not included in the block, the matching determination area is reduced, which causes a problem that the accuracy of motion detection is lowered. That is, when block matching is performed, there is a high possibility that many inspection blocks similar to the reference block, that is, many inspection blocks having a high correlation with the reference block appear. This is because those not caused by motion are included, and the accuracy of motion detection decreases. For example, when a character telop moves in a horizontal or vertical direction, the influence of a repetitive pattern tends to appear. In the case of kanji character patterns, the same character often has the same pattern when divided into small parts. Therefore, there is a problem that it is difficult to obtain an accurate movement when a plurality of movements are mixed in a block.

  The applicant of the present patent application proposes a motion vector detection method and a detection apparatus that can detect a motion vector for each pixel without increasing the amount of calculation and prevent erroneous detection in Patent Document 1, for example. ing.

The point of the motion vector detection process disclosed in Patent Document 1 is that a motion vector is not determined by calculating an evaluation value for each pixel or block, but as a first step process, a plurality of pixels are added to one of the frames. Set multiple blocks, set representative points of each block, examine the correlation between each representative point and each pixel in the search area set in the other frame, calculate the evaluation value based on the correlation information, and evaluate An evaluation value table as correlation information based on values is formed, and a plurality of candidate vectors are extracted from the evaluation value table. Next, as a process of the second step, a candidate vector that seems to be the best for each pixel is selected from the extracted candidate vectors and associated with them, and determined as a motion vector for each pixel. in this way,
Evaluation value table generation processing,
Candidate vector selection processing based on the evaluation value table,
Process for associating an optimal one of a plurality of candidate vectors as a motion vector corresponding to each pixel This is a method for obtaining a motion vector for each pixel by the above process. This method is hereinafter referred to as a candidate vector method.

  The advantage of the motion vector detection process by the candidate vector method is that the calculation amount can be reduced by extracting a limited number of candidate vectors based on the evaluation table. In addition, even in a boundary portion of a subject where erroneous detection of a motion vector is likely to occur, it may be possible to determine the best motion vector corresponding to each pixel from candidate vectors narrowed down in advance. Conventionally, a method has been adopted in which a motion vector of each pixel is calculated as a pixel difference between frames as an evaluation value, and a full search process is performed to obtain an evaluation value for all pixels in the frame. Since it is possible to determine the best motion vector corresponding to each pixel from the candidate vectors that have been narrowed down in advance, the probability that the same evaluation value will occur is reduced compared to the full search process, and erroneous detection is prevented. Is prevented.

However, this candidate vector method also has the following problems.
(A) Since the absolute value of the frame difference is integrated when the evaluation value table is formed, for example, when the pixel value data is 8 bits from 0 to 255, the absolute value data of the frame difference is 8 bits from 0 to 255. When the evaluation value table is composed of memory, a large memory capacity is required.
(B) Since the evaluation values corresponding to all the representative points have been integrated, the representative point is a pixel of a stationary background portion, or the representative point is a pixel in a flat block The calculation result of the representative point and the corresponding candidate point becomes meaningless data, and this meaningless data is accumulated in the evaluation value table, so that the candidate vector may not be extracted correctly.
(C) When assigning the motion vector corresponding to each pixel, even if the candidate vector includes a clearly inappropriate vector, the evaluation value is calculated by the block matching method for all candidate vectors. As a result, the computation is wasted.

The candidate vector schemes disclosed so far have various problems.
JP 2001-61152 A

  The present invention has been made in view of the above-described problems, and in motion vector detection processing based on the candidate vector method, efficient and accurate motion vector detection in units of one pixel without increasing the amount of calculation. It is an object of the present invention to provide a motion vector detection device, a motion vector detection method, and a computer program.

  In particular, the present invention performs a correlation determination process with pixel values of pixels of different frames specified by a candidate vector in a process of selecting and associating a motion vector corresponding to a pixel constituting moving image data from a plurality of candidate vectors. Then, as a result of this correlation determination process, by selecting only candidate vectors that specify pixels with a correlation, and executing a block matching process that applies a block that is set corresponding to the selected candidate vector, It is an object of the present invention to provide a motion vector detection apparatus, a motion vector detection method, and a computer program that can eliminate useless block matching processing and enable efficient and accurate motion vector detection.

The first aspect of the present invention is:
A motion vector detection device for detecting a motion vector from moving image data;
An evaluation value table forming unit that generates an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction unit that extracts one or more candidate vectors based on the evaluation value table;
A motion vector detection unit that executes a process of selecting a motion vector corresponding to each frame constituent pixel of moving image data from the candidate vectors and associating the motion vector,
The evaluation value table forming unit
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation unit for calculating
A correlation determination unit that calculates an evaluation value based on the correlation information calculated by the correlation calculation unit;
From the evaluation values calculated by the correlation determination unit, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. A representative point selection unit that outputs the generated information of the evaluation value table,
An evaluation value table calculation unit that integrates the evaluation values output by the representative point selection unit and generates an evaluation value table as an integration result,
The motion vector detection unit
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation The motion vector detecting apparatus is characterized in that the process of determining as a motion vector is executed.

  Furthermore, in one embodiment of the motion vector detection device of the present invention, the motion vector detection unit selects a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vector and a default still vector. The configuration is characterized in that the process of selecting and associating with each pixel is executed.

  Furthermore, in an embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit is configured to execute candidate vector extraction processing based on peak (extreme value) detection of the evaluation value table, and the evaluation value table The integrated evaluation value at the point of interest is larger than both the horizontal direction and the vertical direction, and the processing for selecting a coordinate whose change in the integrated evaluation value is convex as a peak (extreme value) is executed. Features.

Furthermore, in one embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit sets the coordinates of the target point as (i, j), and the integrated evaluation value of the target point in the evaluation value table as f (i, j j), the following formula:
A = f (i, j) × 2-f (i + 1, j) −f (i−1, j)
B = f (i, j) × 2-f (i, j + 1) −f (i, j−1)
When both A and B calculated by the above are larger than a predetermined threshold, the coordinates are identified as peak (extreme value) formation points.

  Furthermore, in one embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit further selects, as a peak (extreme value), a coordinate whose change in the integrated evaluation value is convex, and from the selected coordinate, A feature is that a preset number of candidate vectors are determined in order from the largest integrated evaluation value.

Furthermore, in an embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit is configured to execute candidate vector extraction processing based on peak (extreme value) detection of the evaluation value table, and the evaluation value table Is a pre-determined filter coefficient multiplied by a filter coefficient that amplifies the steepness of the peak (extreme value) , and is calculated as conversion data. The peak (extreme value) detection process is executed based on the above.

Furthermore, in one embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit sets the coordinates of the target point as (i, j), and the integrated evaluation value of the target point in the evaluation value table as f (i, j j) and the conversion data is g (i, j),
g (i, j) =
f (i, j−1) × Coef [1] + f (i−1, j) × Coef [3]
+ F (i, j) × Coef [4] + f (i + 1, j) × Coef [5]
+ F (i, j + 1) × Coef [7]
In the above equation, Coef [1] is a filter coefficient corresponding to the coordinates (i, j−1), Coef [3] is a filter coefficient corresponding to the coordinates (i−1, j), and Coef [4]. the filter coefficients corresponding to the coordinates (i, j), Coef [ 5] , the filter coefficients corresponding to the coordinates (i + 1, j), Coef [7] is a filter coefficient corresponding to the coordinates (i, j + 1) Yes , it is a filter coefficient that amplifies the sharpness of the peak (extreme value) .
It is characterized in that the peak (extreme value) detection process is executed based on the value of the conversion data g (i, j) calculated by the above formula.

  Furthermore, in one embodiment of the motion vector detection device of the present invention, the candidate vector extraction unit is configured to further determine a preset number of candidate vectors in order from the largest conversion data. It is characterized by.

Furthermore, the second aspect of the present invention provides
A motion vector detection method for detecting a motion vector from moving image data,
An evaluation value table forming step for generating an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction step of extracting one or more candidate vectors based on the evaluation value table;
A motion vector detection step of executing a process of selecting a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vectors and associating the motion vector,
The evaluation value table forming step includes:
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation step for calculating
A correlation determination step of calculating an evaluation value based on the correlation information calculated in the correlation calculation step;
From the evaluation values calculated in the correlation determination step, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. Representative point selection step to output as the generation information of the evaluation value table,
An evaluation value table calculating step for integrating the evaluation values output in the representative point selection step and generating an evaluation value table as an integration result;
The motion vector detection step includes
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation The motion vector detection method is characterized in that it is a step of executing a process of determining as a motion vector.

  Furthermore, in one embodiment of the motion vector detection method of the present invention, the motion vector detection step includes selecting a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vector and a default still vector. It is a step of executing a process of selecting and associating with each pixel.

  Furthermore, in an embodiment of the motion vector detection method of the present invention, the candidate vector extraction step is a step of executing a candidate vector extraction process based on peak (extreme value) detection of the evaluation value table, and the evaluation value table Including a step of executing a process of selecting, as a peak (extreme value), a coordinate in which the integrated evaluation value at the attention point is greater than both the horizontal and vertical neighbors and the change in the integrated evaluation value is convex. Features.

Further, in one embodiment of the motion vector detection method of the present invention, the candidate vector extraction step includes the coordinates of the target point as (i, j), and the integrated evaluation value of the target point in the evaluation value table as f (i, j j), the following formula:
A = f (i, j) × 2-f (i + 1, j) −f (i−1, j)
B = f (i, j) × 2-f (i, j + 1) −f (i, j−1)
A step of identifying the coordinates as peak (extreme value) formation points when both A and B calculated by the above are larger than a predetermined threshold value.

  Furthermore, in one embodiment of the motion vector detection method of the present invention, the candidate vector extraction step further selects a coordinate having a convex change in the integrated evaluation value as a peak (extreme value), and from the selected coordinate, The method includes a step of determining a preset number of candidate vectors in order from the largest integrated evaluation value.

Furthermore, in an embodiment of the motion vector detection method of the present invention, the candidate vector extraction step is a step of executing a candidate vector extraction process based on peak (extreme value) detection of the evaluation value table, and the evaluation value table Is a pre-determined filter coefficient multiplied by a filter coefficient that amplifies the steepness of the peak (extreme value) , and is calculated as conversion data. And a step of executing a peak (extreme value) detection process based on the above.

Further, in one embodiment of the motion vector detection method of the present invention, the candidate vector extraction step includes the coordinates of the target point as (i, j), and the integrated evaluation value of the target point in the evaluation value table as f (i, j j) and the conversion data is g (i, j),
g (i, j) =
f (i, j−1) × Coef [1] + f (i−1, j) × Coef [3]
+ F (i, j) × Coef [4] + f (i + 1, j) × Coef [5]
+ F (i, j + 1) × Coef [7]
In the above equation, Coef [1] is a filter coefficient corresponding to the coordinates (i, j−1), Coef [3] is a filter coefficient corresponding to the coordinates (i−1, j), and Coef [4]. the filter coefficients corresponding to the coordinates (i, j), Coef [ 5] , the filter coefficients corresponding to the coordinates (i + 1, j), Coef [7] is a filter coefficient corresponding to the coordinates (i, j + 1) Yes , it is a filter coefficient that amplifies the sharpness of the peak (extreme value) .
It is a step of executing a peak (extreme value) detection process based on the value of the conversion data g (i, j) calculated by the above formula.

  Furthermore, in one embodiment of the motion vector detection method of the present invention, the candidate vector extraction step further includes a step of determining a preset number of candidate vectors in order from the largest conversion data. It is characterized by.

Furthermore, the third aspect of the present invention provides
A computer program that causes a motion vector detection device to execute a motion vector detection process for detecting a motion vector from moving image data.
An evaluation value table forming step for generating an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction step of extracting one or more candidate vectors based on the evaluation value table;
A motion vector detection step of executing a process of selecting a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vector and associating the motion vector ;
And execute
In the evaluation value table forming step,
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation step for calculating
A correlation determination step of calculating an evaluation value based on the correlation information calculated in the correlation calculation step;
From the evaluation values calculated in the correlation determination step, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. Representative point selection step to output as the generation information of the evaluation value table,
The evaluation value output in the representative point selection step is integrated, and an evaluation value table calculation step for generating an evaluation value table as an integration result is executed,
In the motion vector detection step ,
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation A computer program is characterized in that the step of determining as a motion vector is executed .

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a computer system capable of executing various program codes, such as a CD, FD, or MO. It is a computer program that can be provided by a recording medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, in the process of selecting and associating a motion vector corresponding to each pixel constituting moving image data from a plurality of candidate vectors, correlation determination with pixel values of pixels of different frames specified by the candidate vector As a result of this correlation determination process, only candidate vectors that specify pixels with a correlation are selected, and a block matching process that applies a block set corresponding to the selected candidate vector is executed. Accordingly, it is possible to reduce useless block matching processing, and it is possible to perform a motion vector detection apparatus and a motion vector detection that enable efficient and accurate motion vector detection.

  Furthermore, according to the configuration of the present invention, in the candidate vector extraction process, the integrated evaluation value at the target point in the evaluation value table is larger than both the horizontal direction neighbor and the vertical direction neighbor, and the change in the integrated evaluation value is convex. A process of selecting coordinates as peaks (extreme values), or by multiplying the integrated evaluation values in the evaluation point table and the integrated evaluation values in the vicinity of the attention point by a predetermined filter coefficient and calculating the conversion data as conversion data, Since the peak (extreme value) detection process is executed based on the data, the candidate vector extraction process from the evaluation value table can be executed accurately and efficiently.

  Furthermore, according to the configuration of the present invention, in the candidate vector extraction process, the integrated evaluation value of the coordinates where the change of the integrated evaluation value at the attention point of the evaluation value table is convex, or the attention point and the attention point of the evaluation value table A result obtained by multiplying a neighboring integrated evaluation value by a predetermined filter coefficient and integrating the result is calculated as conversion data, and a predetermined number of candidate vectors are determined in order from the largest conversion data. Therefore, a motion vector detection device capable of efficiently selecting a limited number of candidate vectors is realized.

  Furthermore, according to the configuration of the present invention, in the process of selecting and associating the motion vector corresponding to each pixel constituting the moving image data from a plurality of candidate vectors, the motion vector corresponding to each frame constituent pixel of the moving image data is evaluated. Since the candidate vector acquired from the value table and the default static vector are selected and associated with each pixel, there is no need to detect the static vector from the evaluation value table, and there is a peak with movement (extreme value) ) Can be generated in a prominent form, and a motion vector detection device and a motion vector detection capable of accurate motion vector detection are possible.

Hereinafter, a motion vector detection device, a motion vector detection method, and a computer program according to the present invention will be described in detail with reference to the drawings. The description will be made in the order of the following items.
1. 1. Outline of the present invention 2. Overall configuration of motion vector detection device and outline of motion vector detection processing procedure 3. Details of evaluation value table formation processing 4. Representative point selection process in evaluation value table formation process Specific example of evaluation value table 6. 6. Details of candidate vector extraction processing Details of motion vector decision processing

[1. Outline of the present invention]
Hereinafter, in the motion vector detection process described below, a representative point matching method is applied. The representative point matching method is disclosed in Japanese Patent No. 2083999, which was previously filed by the applicant of this patent and has been patented. In other words, the motion vector detection process described below is a process example using the candidate vector method (disclosed in Japanese Patent Application Laid-Open No. 2001-61152) described in the background section and using the representative point matching method. is there.

  In the following description, an example in which one frame constituting moving image data is set as one screen and a motion vector detection process in a frame is performed by a mutual verification process between the respective screens (frames) will be described. The present invention is not limited to such a processing example, and can be applied to, for example, a case where one field divided into one frame is treated as one screen and motion vector detection processing is performed in units of one field.

  The processing example described below will be described mainly as a processing example for moving image data based on a television signal. However, the present invention is also applicable to various moving image data other than a television signal. . Also, when a video signal is a processing target, either an interlace signal or a non-interlace signal may be used.

The features of the motion vector detection processing in the motion vector detection device and the motion vector detection method of the present invention are roughly divided into the following three.
1. When creating an evaluation value table on one screen by the representative point matching method, a frequency distribution type evaluation that integrates determination results obtained by adaptively determining the absolute value of the frame difference between the representative point and the reference pixel for each representative point with a variable threshold. Configure as a value table. With this configuration, the possible value of the evaluation value can be set to a bit length less than 8 bits (for example, 1 bit), and the memory capacity can be reduced.
2. When forming the frequency distribution type evaluation value table, the representative point is selected and determined so that meaningless data is not accumulated in the evaluation value table. With this configuration, the credibility of the evaluation value table is improved, and candidate vectors can be extracted correctly.
3. When assigning motion vectors corresponding to each pixel, pixel matching and block matching are used together. With this configuration, when it is determined that block matching is unnecessary in pixel matching, block matching processing can be omitted, and the number of computations can be reduced.

  First, referring to FIG. 2 to FIG. 4, an evaluation value table creation process based on the representative point matching method applied in the motion vector detection process of the present embodiment, a candidate vector extraction process based on the evaluation value table, and an extracted candidate vector The outline of the motion vector setting process corresponding to each pixel based on the above will be described.

For example, a current frame [F _t ] 80 at time (t) and a previous frame [F _t-1 ] 70 at time (t−1) shown in FIG. .

For example, the current frame [F _t ] 80 is a reference frame, the previous frame [F _t−1 ] 70 is divided into a plurality of blocks of m pixels × n lines, and a representative point Ry representing each block is set. The representative points of each block are, for example,
a. The pixel value at the center of the block,
b. The average pixel value of all the pixels in the block,
c. An intermediate value of the pixel values of all the pixels in the block,
Etc., pixel values representing blocks are associated with each other.

In the representative point matching method, a predetermined search area 81 is set in the current frame [F _t ] 80 in correspondence with the representative point Ry 71 of the block set in the previous frame [F _t−1 ] 70, and the set search area 81 is set. The pixel value of each pixel included in the pixel is compared with the pixel value of the representative point Ry71. The search area 81 is set as a pixel area of p pixels × q lines, for example.

That is, the representative point pixel value of any one of the above a to c and the pixel value of each pixel in the search area 81 are compared and verified to calculate an evaluation value (for example, frame difference or determination result). The evaluation value is calculated for each shift (each pixel position) in the search area 81. A search area is set in the current frame [F _t ] 80 corresponding to each representative point of the block set in the previous frame [F _t−1 ] 70, and the representative point pixel value and each pixel in the corresponding search area are set. Evaluation values based on the comparison of the pixel values are acquired and integrated for all representative points in one screen. Therefore, an evaluation value table having the same size as the search area is formed.

Note that the search area corresponding to each representative point may be set so as to partially overlap the adjacent search area as shown in FIG. In the example shown in FIG. 3A, the search area is set to an area of p × q (pixels or lines). For example, the representative point 71a of the block set in the previous frame [F _t−1 ] 70 is set. The corresponding search area 81a and the search area 81b corresponding to the representative point 71b of the block set in the previous frame [F _t-1 ] 70 are set as an overlapping area.

  In this way, the search area is set in association with each representative point, and each representative point is compared with the pixels in the search area set in correspondence with the representative point to obtain a comparison value. Based on this, for example, the higher the degree of correlation (the higher the degree of matching of pixel values), the higher the evaluation value is set, and the evaluation values corresponding to the constituent pixels of each search area are set.

The evaluation values in each search area are integrated as shown in FIG. 3B, and as a result, an evaluation value table 90 as shown in FIG. 3C is generated. The evaluation value table is set to the representative points Ry1 to Ryn of each block set to, for example, n blocks set in the previous frame [F _t-1 ] 70 and the current frame [F _t ] 80 which is a reference frame. An evaluation value based on comparison with each pixel in the search area corresponding to each of the representative points Ry1 to n, for example, an integrated value of difference absolute values is calculated and formed as an evaluation value table having the same size as the search area.

  In the prior art, 8-bit data as a difference value of pixel values (for example, 8 bits of 0 to 255) is integrated. However, in the present invention, for example, 1 bit based on the difference data is used as described later. The evaluation value is calculated and the evaluation value of 1-bit data is integrated. Details of this processing will be described later.

  In the evaluation value table 90, a peak (extreme value) occurs when the correlation between the pixel value and the representative point at each shift position (i, j) in the search area is high. The peak appearing in the evaluation value table corresponds to the movement of the display object on the screen of the moving image data.

  For example, when the entire screen (frame) moves in the same manner, the evaluation value table having the same size as the search area (p × q) has 1 in the position corresponding to the end point of the vector having the movement direction and distance. Two peaks appear. If there are two objects with different movements in the screen (frame), two vectors having different movement directions and distances in the evaluation value table having the same size as the search area (p × q). Two peaks appear at two positions corresponding to the end point of. When there is a stationary part, a peak corresponding to the stationary part also appears.

Based on such peaks appearing in the evaluation value table, motion vector candidates (candidate vectors) in the previous frame [F _t−1 ] 70 and the current frame [F _t ] 80 which is a reference frame are obtained.

  After extracting a plurality of candidate vectors based on the peak appearing in the evaluation value table, for each pixel of the frame, select the most suitable candidate vector from the extracted candidate vectors, and use it as a motion vector corresponding to each pixel. Set.

  A motion vector setting process corresponding to each pixel to be executed based on the extraction candidate vector will be described with reference to FIG.

In FIG. 4, the central pixel 91 indicates one pixel of the previous frame [F _t−1 ]. This pixel has, for example, a luminance value (α). Also, it is assumed that a plurality of candidate vectors have already been extracted based on the peaks that have appeared in the evaluation value table, and these candidate vectors are candidate vectors A, B, and C shown in the figure. It is determined that one pixel 91 of the previous frame [F _t−1 ] moves according to any of these candidate vectors and is displayed at a position corresponding to the pixel of the current frame [F _t ].

In FIG. 4, a pixel a95, a pixel b96, and a pixel c97 are current frames at respective pixel positions estimated as destinations based on candidate vectors A, B, and C from one pixel 91 of the previous frame [F _t−1 ]. The pixel of [F _t ] is shown. The correlation between the pixel value of the block including these three pixels and the pixel value of the block including the pixel 91 is determined by the block matching process, the pair having the highest correspondence is selected, and the candidate set in the selected pair Let the vector be the motion vector of the pixel 91.

  A similar process, that is, an optimal vector selection process from candidate vectors, is performed for all pixels included in one screen (frame), and a motion vector corresponding to each pixel is set.

  As described above, the representative point matching can set a representative point representing each block, calculate an evaluation value for only the set representative point, and set a candidate vector. By calculating the evaluation value of only the representative points, the amount of calculation required for calculating the evaluation value can be reduced, and high-speed processing is possible.

  In the following embodiments of the motion vector detection apparatus and motion vector detection method of the present invention to be described, a motion vector detection configuration involving candidate vector extraction processing to which the above-described representative point matching is applied will be described.

[2. Overall configuration of motion vector detection device and outline of motion vector detection processing procedure]
FIG. 5 shows the configuration of an embodiment of the motion vector detection apparatus of the present invention for executing the motion vector detection process, and the processing sequence of the motion vector detection process is shown in the flowchart of FIG.

  As shown in FIG. 5, the motion vector detection apparatus includes an evaluation value table formation unit 101, a candidate vector extraction unit 102, a motion vector detection unit 103, and a control unit (controller) 104. The evaluation value table forming unit 101 inputs image data to be subjected to motion vector detection processing via an input terminal, and generates an evaluation value table. The input image is image data obtained by raster scanning, for example. The image data includes, for example, a luminance signal in a component signal of a digital color video signal.

  The input image data is supplied to the evaluation value table forming unit 101, and an evaluation value table having the same size as the search area is formed based on the representative point matching method described above. This is the process of step S101 in the flow of FIG.

  The candidate vector extraction unit 102 extracts a plurality of motion vectors as candidate vectors in one screen from the evaluation value table generated by the evaluation value table forming unit 101. That is, as described above, a plurality of candidate vectors are extracted based on the peaks that appear in the evaluation value table. This is the process of step S102 in the flow of FIG.

  The motion vector determination unit 103 determines, for each pixel in all frames, the correlation between pixels associated with the candidate vector by block matching or the like for each of the plurality of candidate vectors extracted by the candidate vector extraction unit 102. Then, a candidate vector that connects the blocks having the highest correlation is set as a motion vector corresponding to the pixel. This is the process of step S103 in the flow of FIG. This process is the process described above with reference to FIG.

  The motion vector determination unit 103 performs an optimal vector selection process from candidate vectors for all pixels included in one screen (frame), determines a motion vector corresponding to each pixel, and outputs the determined motion vector To do.

  The control unit 104 performs processing timing control in the evaluation value table forming unit 101, candidate vector extraction unit 102, motion vector detection unit 103, storage of intermediate data in a memory, output processing control, and the like.

  The above is the flow of the generation of the evaluation value table based on the representative point matching and the motion vector detection process to which the candidate vector method is applied, which is executed in the motion vector detection device of the present invention.

[3. Details of evaluation value table formation process]
Next, details of the evaluation value table forming unit in the motion vector detection device of the present invention will be described. A detailed configuration of the evaluation value table forming unit is shown in FIG. FIG. 7B shows the configuration of an evaluation value table forming unit set in a conventional apparatus (disclosed in Japanese Patent Laid-Open No. 2001-61152) for comparison.

  As shown in FIG. 7A, the evaluation value table forming unit 200 in the motion vector detection device of the present invention includes a correlation calculation unit 210, a variable threshold value generation unit 220, a correlation determination unit 230, and an evaluation value table calculation unit 250. .

  The correlation calculation unit 210 includes a representative point memory 211, a subtraction circuit 212 that calculates pixel value difference data, and an absolute value calculation unit 213. The variable threshold value generation unit 220 includes a feature amount extraction unit 221, a feature amount memory 222, and a threshold value generation unit 223. The correlation determination unit 230 includes a comparison unit 230 that performs a comparison process between the frame difference absolute value data that is the output value from the correlation calculation unit 210 and the threshold data that is the output from the variable threshold value generation unit 220. For example, an evaluation value of 1-bit data is calculated as a point and one evaluation value for each pixel, and is output to the evaluation value table calculation unit 250. The evaluation value table calculation unit 250 accumulates evaluation values made of, for example, 1-bit data in the evaluation value accumulation unit 251, generates an evaluation value table, and stores it in the evaluation value table memory 252.

  The evaluation value table forming unit of the conventional type (disclosed in Japanese Patent Laid-Open No. 2001-61152) shown in FIG. 7B has a correlation calculation unit, and accumulates output (8 bits) data of the correlation calculation unit. The evaluation value table is generated and stored in the evaluation value table memory.

  First, the processing of the correlation calculation unit, which is a common component of the evaluation value table forming unit shown in FIGS. 7A and 7B, will be described.

  The image data is input to the correlation calculation unit 210 via the input terminal, for example, in units of frames. The image data input to the correlation calculation unit 210 is supplied to the subtraction circuit 212 and the representative point memory 211.

The image data stored in the representative point memory 211 is, for example, predetermined representative point data generated from image data input in units of frames. For example, one representative point is set in a block set by dividing the screen described above with reference to FIGS. 2 and 3, for example, an m × n pixel block. The representative points are
a. The pixel value at the center of the block,
b. The average pixel value of all the pixels in the block,
c. An intermediate value of the pixel values of all the pixels in the block,
The pixel value data representing the block is associated.

  Specifically, for example, image data (pixel value data) of pixel positions spatially thinned out from the input frame image is controlled by a signal from the control unit (controller) 104 (see FIG. 5). And is stored in the representative point memory 211 as representative point data.

  The representative point data of the previous frame read from the representative point memory 211 and the image data of the current frame are supplied to the subtraction circuit 212.

  The subtraction circuit 211 calculates a pixel value difference, for example, a frame difference (correlation calculation result) between the pixel value of the representative point of the previous frame and the pixel in the search area corresponding to each representative point set in the image data of the current frame. Calculate and output to the absolute value calculator 213.

  The absolute value calculation unit 213 inputs the representative point data of the previous frame input from the subtraction circuit 211 and the frame difference (correlation calculation result) based on the image data of the current frame, and calculates the absolute value of the frame difference.

  The conventional evaluation value table forming unit shown in FIG. 7B is configured to generate an evaluation value table based on the frame difference absolute value (8 bits) that is the output value of the correlation calculation unit. An integration process of frame difference absolute values (8 bits) set for each pixel is executed to generate an evaluation value table. The pixel value has, for example, a luminance level of 256 gradations including 0 to 255 as a luminance value, and 8 bits are applied as pixel value data indicating the 256 gradations.

  Accordingly, the correlation calculation unit of the conventional evaluation value table forming unit calculates the 8-bit evaluation value by subtracting the 8-bit representative point pixel value data and the 8-bit pixel value of each pixel in the search area. Data was applied to the integration process.

  As described above, by integrating the correlation calculation results for all the representative points in one screen, an evaluation value table is generated, and candidate vectors are extracted based on peaks (extreme values) appearing in the generated evaluation value table. Executed.

  Therefore, in order to extract candidate vectors, it is only necessary to identify peaks (extreme values) that appear in the evaluation value table. As described above, the conventional evaluation value table forming unit calculates an 8-bit evaluation value and applies it to the integration process as it is. The information required for the evaluation value table is peak information, and it is not always necessary to integrate 8-bit evaluation values for identifying peaks (extreme values). It is not necessary to integrate 8-bit data. There is a problem that the amount of calculation in the value integration unit 251 is excessive and the required memory capacity is increased.

  Therefore, the evaluation value table forming unit 200 shown in FIG. 7A in the present embodiment inputs the 8-bit frame difference absolute value obtained as an output from the correlation calculation unit 210 to the correlation determination unit 230, and performs correlation determination. For example, the unit 230 degenerates to 1-bit data, and outputs the 1-bit data as an evaluation value to the evaluation value integration unit 251 of the evaluation value table calculation unit 250. The evaluation value integration unit 251 performs integration processing of 1-bit data. The evaluation value table is generated. With this configuration, the amount of calculation processing for integration in the evaluation value integration unit 251 is greatly reduced, and an efficient and high-speed evaluation value table can be generated. Further, the memory capacity as a storage area for the evaluation value data and the evaluation value table can be reduced. As a result, the hardware scale can be reduced without degrading performance.

  The absolute value of the frame difference from the correlation calculation unit 210 is supplied to the correlation determination unit 230 of the evaluation value table forming unit 200 of the present invention shown in FIG. This is the difference data between the representative point pixel value data of 8 bits and the search area comparison pixel value of 8 bits, similar to the correlation evaluation unit of the conventional evaluation value table forming unit 200, and is input as 8-bit data. The In this example, the pixel is described as having 256 pixel value levels from 0 to 255. The correlation determination unit 230 compares the input 8-bit data with the variable threshold value input from the variable threshold value generation unit 220 in the comparison unit 231 and outputs an evaluation value of 1 bit [0] or [1]. .

This 1-bit evaluation value is, for example,
The bit value [1] has a correlation between the pixel value of the representative point and the pixel value serving as the comparison reference for the search area. The bit value [0] is the pixel value serving as the comparison point for the representative point and the search area. It indicates either no correlation with.

  Based on this output, the evaluation value integration unit of the evaluation value table calculation unit 250 performs a process of integrating the number of bit values [1]. As a result, a frequency distribution type evaluation value table indicating whether or not there are many correlated results is generated. The frequency distribution type evaluation value table increments the frequency of the deviation when there is a correlation between the representative point and the reference point in each deviation (i, j) in the search area, and displays all representatives of one screen. The point correlation determination results are integrated. In other words, the frequency f (i, j) of each deviation (i, j) in the evaluation value table indicates the number of times that the correlation condition between the representative point and the reference point is satisfied. Then, a peak (extreme value) corresponding to the movement of the display object on the screen of the moving image data appears.

  Therefore, the integration of 1-bit data indicating only the presence or absence of the correlation is performed without performing the integration process of the 8-bit data representing the difference between the pixel value data of the representative point and the 8-bit pixel value serving as the comparison reference of the search area Also in the evaluation value table generated by the above, a peak (extreme value) corresponding to the movement of the display object on the moving image data screen appears, and the candidate vector can be selected.

  In the correlation determination unit 230 in the evaluation value table calculation unit 250 of the present invention shown in FIG. 7A, the variable threshold value is input from the variable threshold value generation unit 220 and the frame difference absolute as the difference data input from the correlation calculation unit 210 is displayed. Compare with the value.

  The reason why the variable threshold is applied without using the fixed threshold is to enable more accurate motion detection corresponding to the actual motion level.

  An image having a large level variation in space, that is, an image having a large level change (pixel value variation) in the vicinity of a certain pixel of interest, for example, a plurality of vertical stripe pattern images, etc. In the case of an image, the frame difference becomes very large even if there is a slight movement in the image in the next frame. Therefore, the frame difference output from the correlation calculation unit 210 is a large value. On the other hand, in the case of an image with a small level fluctuation in space, for example, an image with a generally flat luminance distribution such as a cheek part of a human face, a greenish meadow, or a cloud floating in the sky In other words, the frame difference is small even if there is movement in the image in the next frame. Therefore, the frame difference output from the correlation calculation unit 210 is a small value.

  If the frame difference data output from the correlation calculation unit 210 is uniformly distinguished as “with correlation” or “without correlation” using a fixed threshold, an image having a flat luminance distribution as a whole is obtained. In some cases, even if there is a motion, it is determined that there is no correlation, and it may be determined that the image data has no motion.

  Therefore, in the apparatus of the present invention, the temporal variation of the pixel of interest is normalized with the spatial variation, and the presence or absence of motion is determined based on this normalized output. For this processing, the variable threshold value generation unit 220 generates a variable threshold value using data obtained by normalizing the temporal variation of the pixel of interest with the variation in space, and applies the variable threshold value. Compare with the output frame difference data.

  That is, by applying different threshold values according to image data, the comparison unit 231 of the correlation determination unit 230 performs a comparison process with the frame difference absolute value as difference data input from the correlation calculation unit 210 [1]. Then, the evaluation value of [0] without correlation is generated, and the evaluation value output as the frequency in the frequency distribution evaluation value table is obtained.

  By performing the correlation determination based on the variable threshold, compared with the determination of the presence or absence of movement made based only on the temporal direction fluctuation (corresponding to the determination applying a fixed threshold), it corresponds to the actual degree of movement, More accurate motion detection can be performed.

  In other words, by performing correlation determination using a variable threshold, even if the fluctuation in the time direction is large, if the fluctuation in level in the frame, that is, the fluctuation in space is relatively large, the motion detection sensitivity is increased. descend. Even if the fluctuation in the time direction is small, if the level fluctuation in the frame, that is, the fluctuation in the space is relatively small, the motion detection sensitivity is improved, and the accurate motion determination according to the image data is performed. That is, correlation determination processing is possible.

  The process of the variable threshold value generation unit 220 will be described. The variable threshold value generation unit 220 extracts a feature amount that represents a variation in space, and sets a different threshold value corresponding to the feature amount. The feature amount extraction unit 221 executes extraction of the feature amount representing the variation in the space.

  The feature quantity extraction unit 221 receives the output value of the subtraction circuit 212 of the correlation calculation unit 210, that is, the frame difference data, and uses the dynamic range of the neighborhood area (3 × 3) of the representative point as a feature quantity representing the variation in space. (DR) is calculated.

  Here, an example in which the dynamic range (DR) of the region near the representative point (3 × 3) is applied as the feature amount representing the variation in space will be described. As the feature amount representing the variation in space, for example, It is possible to use the dynamic range of the three pixels adjacent to the representative point, the difference sum, the average deviation sum of the neighboring regions, and the like.

  The dynamic range (DR) of the representative point vicinity region (3 × 3) can be calculated by using the correlation calculation result of the representative point matching. A dynamic range (DR) calculation process of the representative point vicinity region (3 × 3) using the correlation calculation result of the representative point matching will be described with reference to FIG.

The FIG. 8 (A), the current frame _[F t], the current frame _{[F t]} 1 previous previous frame _{[F t-1],} 2 previous frame _{[F t} of the current frame _{[F t] -2} ] is shown.

In the neighborhood (3 × 3) of the representative point Ry in the previous frame [F _t−1 ] immediately before the current frame [F _t ], for example, the pixel at the representative point Ry is defined as the center pixel (i, j). Then, it is configured by nine pixels (i−1, j−1) to (i + 1, j + 1) shown in FIG. The dynamic range (DR) of the representative point Ry vicinity region (3 × 3) is a difference between the maximum value and the minimum value of the pixel values of these nine pixels (i−1, j−1) to (i + 1, j + 1). The value represented.

The feature amount extraction unit 221 of the variable threshold value generation unit 220 sets the dynamic range of the neighborhood (3 × 3) of a certain representative point Ry of the previous frame [F _t−1 ] to [F _t−2 ] the previous frame [F _t−1 ]. Calculation is based on the frame difference of the correlation calculation at the representative point Py. A configuration example of the feature amount extraction unit 221 that executes this processing is shown in FIG.

The feature amount extraction unit 221 receives, as frame differences, difference data between pixel points in a search area and representative points of consecutive frames. It is assumed that the difference data of the representative point Py of [F _t-2 ] two frames before shown in FIG. 8 and the region (3 × 3) in the vicinity of the representative point Ry of the previous frame [F _t-1 ] are input. .

  As the difference data, the representative point Py and the difference data of each of the nine pixels (i−1, j−1) to (i + 1, j + 1) are sequentially input, and the maximum value detecting unit 301 and the minimum value detecting unit 303 represent the difference data. The maximum value and the minimum value of the difference data of the point Py and each of the nine pixels (i−1, j−1) to (i + 1, j + 1) are obtained. First, the first input difference data passes through the maximum value detection unit 301 and the minimum value detection unit 303 and is stored in the registers 302 and 304. Next, the difference data to be input, the maximum value detection unit 301, and the minimum value detection unit In step 303, the maximum difference value is stored in the register 302, and the minimum difference value is stored in the register 304. By repeatedly executing this, the maximum value and the minimum value of the representative point Py and the difference data of each of the nine pixels (i−1, j−1) to (i + 1, j + 1) are obtained.

  The representative point Py obtained as a result and the maximum value and the minimum value of the difference data of each of the nine pixels (i−1, j−1) to (i + 1, j + 1) are subtracted by the subtraction circuit 305, and this result is representative. It is output via the latch 306 as a dynamic range (DR) of the vicinity region (3 × 3) of the point Ry, that is, nine pixels (i−1, j−1) to (i + 1, j + 1).

As described above, the dynamic range of the neighborhood (3 × 3) of the representative point Ry is a value represented by the difference between the maximum value and the minimum value of the nine pixel values shown in FIG. is there. The frame difference between the representative point Py two frames before [F _t−2 ] and the nine points is only the change in the DC component of the nine points, so the difference between the maximum and minimum values of the nine frame differences is also The same dynamic range as that described above.

An example in which specific pixel values are set will be described. It is assumed that nine pixel values of the previous frame [F _t−1 ] have pixel values (luminance levels) of 10, 11, 7, 9, 9, 12, 15, 11, and 8. It is assumed that the pixel value of the representative point Ry of [F _t-2 ] two frames before is [11]. In this case, the frame differences between the representative point Ry and the nine pixel values of the previous frame [F _t−1 ] are respectively
10-11 = -1
11-11 = 0
7-11 = -4
9-11 = -2
9-11 = -2
12-11 = + 1
15-11 = + 4
11-11 = 0
8-11 = -3
Based on these nine pieces of difference data, the value (DR) obtained by the feature quantity extraction unit 221 shown in FIG. 9 is obtained by subtracting the minimum value [−4] from the maximum value [+4] of the result of Equation 9 above. Which is [8]. On the other hand, the dynamic range obtained directly from the nine pixel values {10, 11, 7, 9, 9, 12, 15, 11, 8} of the previous frame [F _t-1 ] is 15 (MAX) -7 ( MIN) = 8, and all the values are [8], and all have the same value.

In this way, using the correlation result in the representative point matching process executed by the correlation calculation unit 210 for the previous frame [F _t-2 ] and the previous frame [F _t-1 ], that is, using the frame difference data, The dynamic range in the vicinity of the representative point of the previous frame [F _t-1 ] can be calculated, and this dynamic range (DR) is applied as a feature amount of the variation in space.

  The variable threshold value generation processing procedure executed in the variable threshold value generation unit 221 will be described with reference to the flow shown in FIG.

  First, in step S151, the frame difference data calculated by the representative point matching process executed in the correlation calculation unit 210 is captured. In step S152, the feature amount extraction unit 221 calculates a dynamic range (DR) near the representative point based on the frame difference data according to the above-described process.

  In step S153, the calculated dynamic range (DR) is stored in the feature amount memory 222 in association with the representative point. In step S154, the dynamic range (DR) data corresponding to the representative point is read from the feature amount memory 222. In step S155, the read dynamic range (DR) is converted into a threshold value by the threshold value generation unit 223. In step S156, the converted variable threshold value is output to the correlation determination unit 230.

  As described above, the feature amount extraction unit 221 of the variable threshold value generation unit 220 in FIG. 7A calculates the dynamic range near the feature point based on the frame difference data input from the correlation calculation unit 210 by the above-described processing. Then, this is stored in the feature amount memory 222 as a feature amount representing the variation in the space, and the threshold value generation unit 223 sets this feature amount as a threshold value and outputs it to the correlation determination unit 230.

  The threshold value generation process in the threshold value generation unit 223, that is, the threshold value generation process based on the dynamic range data in the vicinity of the representative point as the feature amount of the variation in space will be described with reference to FIG.

  The dynamic range (DR) data calculated based on the frame difference data in the feature amount extraction unit 221 is input to the threshold generation unit 223. This data is 8-bit pixel value difference data and 8-bit data.

  The bit decomposition unit 351 of the threshold generation unit 223 decomposes the input 8-bit dynamic range (DR) from the MSB in units of 1 bit, and extracts only the upper 6 bits. The upper bits are D7, D6, D5, D4, D3, and D2, respectively.

  The OR circuit 352 receives 3 bits D7, D6, and D5 from the MSB, performs an OR operation, and outputs 1 bit. If the bit value [1] is included in any of the 3 bits D7, D6, D5, [1] is output.

  Further, the output of the OR circuit 352 and the output D4 of the bit decomposition unit 351 are input to the OR circuit 353, and the result of the OR operation is set as the output value. Further, the output of the OR circuit 353 and the output D3 of the bit decomposing unit 351 are input to the OR circuit 354, and the result of the OR operation is output. Furthermore, the output of the OR circuit 354 and the output D2 of the bit decomposing unit 351 are input to the OR circuit 355, and the result of the OR operation is used as the output value.

  A total of 3 bits d2, d1, and d0 of the output of the OR circuit 353, the output of the OR circuit 354, and the output of the OR circuit 355 are input to the bit synthesis unit 356, and the bit synthesis unit 356 starts with d2, d1 from the higher order. , D0 3-bit data [000] to [111] are converted into decimal numbers, and values 0 to 7 are output as threshold values. As described above, the threshold generation unit 223 calculates the threshold by the bit information compression processing of the constituent bits of the dynamic range data near the representative point as the feature amount of the variation in the space near the target pixel.

  FIG. 12 is a graph in which the input / output relationship of the threshold value generation unit 223 is plotted, and a variable threshold value provided with a threshold upper limiter is generated. In FIG. 12, the horizontal axis represents the dynamic range (DR) calculated based on the frame difference input to the variable threshold value generation unit, the vertical axis represents the dynamic range (DR), and the correlation determination unit 230 Shows the variable threshold value to be output.

  For example, when the dynamic range (DR) is 0 to 3, the variable threshold value output to the correlation determination unit 230 is 0. When the dynamic range (DR) is 4 to 7, the variable threshold value output to the correlation determination unit 230 is 1. The same applies to the following. When the dynamic range (DR) is 28 or more, the variable threshold value output to the correlation determination unit 230 is 7, and a variable threshold value provided with a threshold upper limiter is generated.

  Even if the dynamic range (DR) is small, that is, a flat image, the threshold value is set to be relatively small. Therefore, in the correlation determination in the correlation determination unit 230, it is determined that there is no correlation in a relatively accurate manner. Become.

  Correlation determination section 230 receives frame difference absolute value data, which is an output value from correlation calculation section 210, and further receives the above-described variable threshold value from variable threshold value generation section 220 and performs comparison processing in comparison section 231. To do. If the frame difference absolute value data is smaller than the variable threshold, the correlated [1] is output to the evaluation value table calculation unit 250 as 1-bit data, and if it is larger than the variable threshold, no correlation [0] is output. The result is output to the evaluation value table calculation unit 250.

  The evaluation value table calculation unit 250 integrates the input 1-bit data in the evaluation value integration unit 251 to generate an evaluation value table, and stores the evaluation value table in the evaluation value table memory 252.

  The evaluation value accumulation unit 251 performs a process of accumulating the number (frequency) of bit values [1] as the evaluation value from the correlation determination unit 230. As a result, a frequency distribution type evaluation value table indicating whether or not there are many correlated results is generated. The frequency distribution type evaluation value table increments the frequency of the deviation when there is a correlation between the representative point and the reference point in each deviation (i, j) in the search area, and displays all representatives of one screen. The point correlation determination results are integrated. In other words, the frequency f (i, j) of each deviation (i, j) in the evaluation value table indicates the number of times that the correlation condition between the representative point and the reference point is satisfied. Then, a peak (extreme value) corresponding to the movement of the display object on the screen of the moving image data appears.

  Therefore, the integration of 1-bit data indicating only the presence or absence of the correlation is performed without performing the integration process of the 8-bit data representing the difference between the pixel value data of the representative point and the 8-bit pixel value serving as the comparison reference of the search area An evaluation value table generated by the above, that is, a frequency distribution type evaluation value table is generated.

  FIG. 13 is a flowchart for explaining the processing sequence of the evaluation value table forming unit in the present embodiment. Each step of the flowchart will be described.

  First, in step S201, the representative point matching process is executed based on the input image data to calculate difference data, and in step S202, the absolute value of the difference data is calculated. These processes are executed in the correlation calculation unit 210 of the evaluation value table forming unit 200 in FIG.

  In step S203, the correlation calculation unit 210 generates a variable threshold based on the frame difference data generated in step S201. This process is a process executed in the variable threshold value generation unit 220 of the evaluation value table forming unit 200 in FIG.

  In step S204, “there is correlation” or “no correlation” is obtained by comparing the absolute value of the difference data generated by the correlation calculation unit 210 in step S202 with the variable threshold generated by the variable threshold generation unit 220 in step S203. 1-bit data is generated as an evaluation value to be indicated. This process is a process executed in the correlation determining unit 230 of the evaluation value table forming unit 200 in FIG.

  Next, in step S205, 1-bit data as an evaluation value indicating the presence or absence of correlation generated by the correlation determination unit 230 is integrated to generate a frequency distribution evaluation value table. This process is a process executed in the evaluation value table calculation unit 250 of the evaluation value table forming unit 200 in FIG.

  As described above, the evaluation value table forming unit according to the present embodiment acquires the dynamic range that can be acquired from the frame difference data calculated in the correlation calculation unit as the feature amount indicating the variation in the space, and based on the acquired feature amount. Thus, a variable threshold value is generated, and the presence or absence of correlation is determined by the correlation determination unit based on the variable threshold value, so that accurate motion detection is possible even in a flat image. In addition, since the evaluation value output from the correlation determination unit is reduced to 1-bit data, the amount of integration calculation in the evaluation value integration unit is significantly reduced. With these configurations, it is possible to perform more accurate motion correlation determination corresponding to the actual motion level while reducing the hardware scale, and dramatically increase the credibility of extreme values in the evaluation value table It becomes.

[4. Representative point selection process in evaluation value table formation process]
The basic configuration of the evaluation value table forming unit 101 (see FIG. 5) in the motion vector detection device of the present invention has been described above. The evaluation value table generated by the evaluation value table forming unit 101 is an evaluation value table for extracting candidate vectors. As described above, the candidate vector is determined based on the peak (extreme value) appearing in the evaluation value table.

  Increasing the credibility of the peak (extreme value) in the evaluation value table is important for extracting effective candidate vectors. The above-described evaluation value table forming unit adopts a variable threshold method, normalizes the temporal variation of the representative points with the variation in space, and determines the presence or absence of movement based on this normalized output, thereby enabling more accurate movement. It is possible to form a frequency distribution type evaluation value table corresponding to.

  However, in the actual representative point matching, since the correlation determination between the point (one pixel) and the point (one pixel) is performed, there are cases where the determination is difficult. Hereinafter, a configuration example in which representative point selection processing is performed as processing for excluding correlation determination of representative points that are difficult to determine will be described. By performing the representative point selection process, it is possible to further improve the credibility of the evaluation value table. Hereinafter, three examples of the configuration example for performing the representative point selection process will be described.

(4.1. Representative point selection processing example 1)
First, as a representative point selection processing example 1, a configuration example in which representative points of a portion (hereinafter referred to as a stationary portion) in which the variation in the space near the representative point is small, that is, the dynamic range of the representative point neighboring region is small (hereinafter referred to as a stationary portion) is excluded. This will be described with reference to FIG.

  In the case of a representative point that exists in a stationary part where the fluctuation in the space near the representative point is small (the dynamic range is small), even if there is movement, the time fluctuation (frame difference) is also small, so It is difficult to distinguish from minute time fluctuations due to noise components. Therefore, the representative points are selected based on the feature amount of the variation in the space near the representative point.

  FIG. 14 shows a configuration of an evaluation value table forming unit having a configuration for executing a process of selecting a representative point based on a feature amount of variation in the space near the representative point.

  In the evaluation value table forming unit 200 shown in FIG. 14, the correlation calculation unit 210, the variable threshold value generation unit 220, the correlation determination unit 230, and the evaluation value table calculation unit 250 have the same configurations as those in the above-described embodiment, and a description thereof will be omitted. .

  The evaluation value table forming unit 200 illustrated in FIG. 14 includes a representative point selection unit 240. The representative point selection unit 240 includes a reference value memory 241, a comparison unit 242, and a gate 243. The representative point selection unit 240 receives the output (1-bit evaluation value) from the correlation determination unit 230, executes evaluation value selection, and the in-space variation in the region near the representative point is small (dynamic range is small). A process of excluding an evaluation value corresponding to a representative point existing in a normal part from an integration target of the evaluation value table is executed. The representative point selection unit 240 selects and outputs only the effective evaluation values corresponding to the representative points other than the representative points existing in the stationary part where the variation in the space in the vicinity of the representative points is small (dynamic range is small).

  A feature amount representing a variation in the space of the representative point vicinity region generated by the variable threshold value generation unit 220, for example, a dynamic range is supplied to the comparison unit 242 in the representative point selection unit 240 and stored in the reference value memory 241 in advance. If it is smaller than the reference value, it is determined that the evaluation value corresponds to a representative point that exists in a stationary part where the variation in the space near the representative point is small (dynamic range is small), and the reliability of the correlation determination result Is output from the comparison unit 242 to the gate 243 and the gate 243 is set to OFF, so that the evaluation value is output, that is, the correlation determination unit 230 has a correlation. Even if it is determined, the evaluation value [1] is not output and is not counted as the integrated frequency in the frequency distribution type evaluation value table.

  On the other hand, if the feature amount representing the variation in the space in the vicinity of the representative point generated by the variable threshold value generation unit 220, for example, the dynamic range is larger than a predetermined reference value stored in the reference value memory 241, the correlation The reliability of the determination result is regarded as high, and the output of the evaluation value, that is, the correlation determination unit 230 is output by outputting the [H] signal from the comparison unit 242 to the gate 243 and turning the gate 243 on [ON]. When it is determined that there is a correlation, the evaluation value [1] is output and counted as the frequency integrated in the frequency distribution type evaluation value table.

  As described above, in the steady-state part having a small feature amount representing the variation in the space in the vicinity of the representative point generated by the variable threshold value generation unit 220, for example, the correlation determination result based on the comparison with the variable threshold value in the correlation determination unit 230 The correlation determination result in such an area is not counted as the frequency to be counted in the evaluation value integration table, so that the correlation determination result with low reliability is affected on the evaluation value integration table. Thus, the credibility of the evaluation value table can be improved.

  In addition to the dynamic range, a feature such as a standard deviation, a high-frequency component, an orthogonal transformation higher-order component, a difference from an average value, or the like may be applied as a feature amount representing the variation in the space near the representative point.

(4.2. Representative point selection processing example 2)
Next, as a representative point selection processing example 2, a configuration example in which representative points existing in a still region are excluded will be described with reference to FIG.

  Although the peak of the evaluation value table is used for selection of candidate vectors, it is not necessary to detect the static region as the peak of the evaluation value table. In the case of moving image data, in many cases, it is assumed that a still region exists, and a still vector is set as a default candidate vector as one of candidate vectors. Therefore, it is not necessary to make a peak corresponding to the static vector appear as a peak in the evaluation value table.

  If a peak corresponding to the still region is generated in the evaluation value table, there is a possibility that the peak corresponding to the candidate vector indicating that there is another motion is not noticeable. Therefore, the correlation determination result of the representative points existing in the still region is a frequency that causes a decrease in the reliability of the evaluation value table.

  The configuration of the evaluation value table forming unit 200 of this embodiment is shown in FIG. The evaluation value table forming unit 200 of this embodiment is provided with a representative point selection unit 260 for preventing the correlation determination result of the representative points existing in the still region from being output from the correlation determination unit 230. The correlation calculation unit 210, the variable threshold value generation unit 220, the correlation determination unit 230, and the evaluation value table calculation unit 250 have the same configuration as that of the above-described embodiment, and a description thereof will be omitted.

  The representative point selection unit 260 uses the result of the representative point matching process executed by the correlation calculation unit 210 to exclude the representative point correlation determination result existing in the still region, and performs the still determination of the representative point. . The representative point selection unit 260 includes a reference value memory 261, a comparison unit 262, a flag memory 263, and a gate 264.

  The comparison unit 262 inputs the result of the representative point matching process executed in the correlation calculation unit 210, compares the result with a preset reference value stored in the reference value memory 261, and the representative point is in the still region. It is determined whether or not.

A specific example of the stillness determination process executed by the comparison unit 262 will be described with reference to FIG. The stationary determination of the representative point Ry of the previous frame [F _t-1 ] will be described with reference to FIG. Static determination of the representative point Ry in the previous frame _{[F t-1]} is a representative point Ry in the previous frame _{[F t-1],} to determine a frame difference between Qy point of the current frame _{[F t-1]} Instead, the determination is made based on the frame difference between the representative point Ry of the previous frame [F _t-1 ] and the point Py of the previous frame [F _t-2 ]. This is because when the frame difference between Ry and Qy is used, the stillness of the representative point Ry can be determined only when the Qy data of the current frame is supplied, but the frame difference between Ry and Py is used. This is because when all the data of the previous frame [F _t−1 ] is supplied, the stillness determination result of the representative point existing in the previous frame [F _t−1 ] is acquired.

The absolute value of the frame difference between the representative point Ry of the previous frame [F _t-1 ] and the point Py of the previous frame [F _t-2 ] is the correlation at the representative point Py of the frame [F _t-2 ]. Since it is calculated by the calculation unit 210, it can be used.

The comparison unit 262 inputs the absolute value of the frame difference between the representative point Ry of the previous frame [F _t-1 ] calculated by the correlation calculation unit 210 and the point Py of the previous frame [F _t-2 ], If this frame difference absolute value is smaller than a preset reference value stored in the reference value memory 261, it is assumed that the representative point Ry of the previous frame [F _t-1 ] is in the still region, and the still flag (0 ) Is stored in the flag memory 263.

On the other hand, the frame difference absolute value between the representative point Ry of the previous frame [F _t-1 ] calculated by the correlation calculation unit 210 and the point Py of the previous frame [F _t-2 ] is stored in the reference value memory 261. If the stored reference value is equal to or greater than the preset reference value, the representative point Ry of the previous frame [F _t−1 ] is regarded as not in the still region, and the motion flag (1) is stored in the flag memory 263. These flags are stored in the flag memory 243 by timing control by the controller (the controller 104 in FIG. 5).

  Further, the representative point stationary determination flag is read from the flag memory 263 at an appropriate timing by the controller, and when the read data is (0), an [L] signal is output to the gate 264 and the gate 264 is turned off [OFF]. Therefore, even if the correlation determination unit 230 determines that there is a correlation, the evaluation value [1] is not output and is counted as the integrated frequency in the frequency distribution evaluation value table. do not do.

  On the other hand, when the data read from the flag memory 263 is (1), the evaluation value is output, that is, the correlation is generated by outputting the [H] signal to the gate 264 and turning the gate 264 on [ON]. When the determination unit 230 determines that there is a correlation, the evaluation value [1] is output and counted as the frequency integrated in the frequency distribution evaluation value table.

  As described above, based on the absolute value of the frame difference as a result of the representative point matching process executed in the correlation calculation unit 210, it is determined whether or not the representative point exists in the still region, and the representative point exists in the still region. The correlation determination result is not output from the correlation determination unit 230, so that the correlation-related evaluation value [1] is not counted as a frequency in the frequency distribution-type evaluation value table. An evaluation value [1] having a correlation with respect to a representative point other than is selected and output as an effective evaluation value, and counted as a frequency in the frequency distribution type evaluation value table. As a result, it is possible to generate a highly reliable evaluation value table in which the peak corresponding to the motion is increased.

(4.3. Representative point selection processing example 3)
Next, as a representative point selection processing example 3, a configuration example in which representative points with a large number of candidates are excluded will be described with reference to FIG.

  The correlation calculation at a certain representative point is performed on the pixels in the search area corresponding to the representative point, but the pixel (number of candidates) determined to have a correlation with the representative point in the search area is, for example, within the search area. If there are a large number of pixels in the search area, for example, if the number of pixels (number of candidates) determined to have a correlation is ½ or more of the total number of pixels in the search area, the correlation determination result at that representative point is low in reliability. I can say that. Therefore, in the representative point selection processing example 3, the correlation determination result for the representative points having a large number of candidates is excluded from the generated data of the evaluation value table.

  The configuration of the evaluation value table forming unit 200 of this embodiment is shown in FIG. The evaluation value table forming unit 200 of the present embodiment is provided with a representative point selection unit 270 for preventing the correlation determination result for the representative points having a large number of candidates from being output from the correlation determination unit 230. The correlation calculation unit 210, the variable threshold value generation unit 220, the correlation determination unit 230, and the evaluation value table calculation unit 250 have the same configuration as that of the above-described embodiment, and a description thereof will be omitted.

  The detailed configuration of the candidate number determination unit 271 configured in the representative point selection unit 270 is shown in FIG. The candidate number determination unit 271 stores the 1-bit data as the determination result, which is the output of the correlation determination unit 230, in correspondence with each representative point and stores it in the buffer memory 381. At the same time, the counter 382 determines “1”. Count the number, that is, the number of candidates that are correlated. When the counting of the counter 382 corresponding to one representative point is completed, the total number is input to the comparison unit 384 and compared with a preset reference value stored in the reference value memory 383.

  If the number determined as [1] with respect to the representative point counted by the counter 382, that is, the number of candidates that are correlated, is larger than the reference value, it is determined that the correlation determination result at the representative point is low in reliability, and the gate By outputting the [L] signal to 385 and setting the gate 385 to OFF [OFF], even when the evaluation value is output, that is, even when the correlation determination unit 230 determines that there is a correlation, the evaluation value [ 1] is not output and is not counted as the integrated frequency in the frequency distribution evaluation value table.

  On the other hand, when the number determined as [1] with respect to the representative point counted by the counter 382, that is, the number of candidates with correlation is equal to or less than the reference value, the correlation determination result at the representative point is a reliable effective evaluation value. When the output of the evaluation value, that is, the correlation determination unit 230 determines that there is a correlation by outputting the [H] signal to the gate 385 and setting the gate 385 to ON [ON]. Evaluation value [1] is output and counted as the frequency integrated in the frequency distribution type evaluation value table.

The basic configuration and the three representative point selection processing configuration examples have been described above for the configuration of the evaluation value table forming unit in the motion vector detection processing configuration of the motion vector detection device of the present invention. In addition, although the three representative point selection processing configuration examples have been described individually, in an actual process, it is possible to execute all three representative point selection processes in combination, and execute them in combination. By
a. A representative point in a stationary part where the variation in the space in the vicinity of the representative point is small b. Representative points in the stationary region c. Representative points with a large number of candidates Correlation determination results corresponding to these representative points determined to be low reliability can be excluded from the frequencies accumulated in the frequency distribution type evaluation value table, and the correlation determination results with high reliability It is possible to form a highly reliable frequency distribution type evaluation value table in which only the evaluation results consisting of

[5. Specific example of evaluation value table]
An example of a frequency distribution type evaluation value table generated when the motion vector detection apparatus having the above-described evaluation value table forming unit configuration is applied and processing on actual moving image data is executed will be described.

  As specific moving image data, data in which a moving object (A) 401 and a moving object (B) 402 exist in the foreground of a static background 400 as shown in FIG. 18 is applied.

  The moving object (A) 401 is an object that moves in the horizontal right direction, and the correct motion vector corresponding to the pixel in the display area of this object is (Vx, Vy) = (9, 0). That is, this is a pixel area in which a horizontal right motion vector is to be set. The moving object (B) 402 is an object moving in the horizontal left direction, and (Vx, Vy) = (− 9, 0) is a correct motion vector corresponding to the pixel in the display area of this object. That is, this is a pixel region in which a horizontal left motion vector is to be set.

  For this moving image data, a conventional method, for example, an evaluation value table generation process disclosed in Japanese Patent Application Laid-Open No. 2001-61152, that is, as shown in FIG. FIG. 19 shows an evaluation value table generated by applying the bit difference frame data as it is and integrating the bit difference frame data.

  On the other hand, FIG. 20 shows a frequency distribution type evaluation value table generated in accordance with the processing of the evaluation value table forming unit according to the present invention described with reference to FIG. 7A and FIGS. 14, 15, and 16. The table shown in FIG. 20 is a frequency distribution type evaluation value table formed by using a variable threshold and representative point selection together.

The frequency distribution type evaluation value table shown in FIG. 20 includes, as shown in FIG. 7A, 8-bit difference frame data generated by the correlation calculation unit 210 by the representative point matching process as 1-bit data in the correlation determination unit 230. This is a table based on the result of integrating these, and as described with reference to FIGS. 14, 15, and 16,
a. A representative point in a stationary part where the variation in the space in the vicinity of the representative point is small b. Representative points in the stationary region c. A representative point with a large number of candidates is a table generated by performing a process of excluding the correlation determination result corresponding to the representative point determined to have low reliability from the frequency accumulated in the frequency distribution evaluation value table.

  In FIG. 19, it corresponds to the peak (extreme value) 411 corresponding to the motion vector (9, 0) corresponding to the moving object (A) and the motion vector (−9, 0) corresponding to the moving object (B). Three peaks (extreme values), that is, a peak (extreme value) 412 and a peak (extreme value) 413 corresponding to a motion vector (0, 0) indicating stillness corresponding to the still region of the background, appear. Since the background still area occupies the most part on the screen, the peak (extreme value) 413 corresponding to the motion vector (0, 0) indicating the stillness corresponding to the background still area shows the largest peak and moves. A peak (extreme value) 411 corresponding to the motion vector (9, 0) corresponding to the object (A) and a peak (extreme value) 412 corresponding to the motion vector (−9, 0) corresponding to the moving object (B). The result shows a peak (extreme value) that is not conspicuous.

On the other hand, in the table shown in FIG. 20, which is a table generated by applying the configuration of the present invention, the peak 421 corresponding to the motion vector (9, 0) corresponding to the moving object (A) and the moving object (B). A peak 422 corresponding to the corresponding motion vector (−9, 0) indicates a large peak (extreme value) as compared to the table of FIG. 19, and a motion vector (0, 0) indicating stillness corresponding to the background still region. The peak (extreme value) 423 indicating 0) is an inconspicuous result. As explained with reference to FIG. 14, FIG. 15 and FIG.
a. A representative point in a stationary part where the variation in the space in the vicinity of the representative point is small b. Representative points in the stationary region c. Representative points with a large number of candidates This is based on the results generated by performing the process of excluding the correlation determination results corresponding to the representative points determined to have low reliability from the frequencies accumulated in the frequency distribution evaluation value table. .

  In order to make the peak easy to understand, FIGS. 21 and 22 show two-dimensionally expressed evaluation value tables. FIG. 21 is two-dimensional data of an evaluation value table obtained as a result of integrating the conventional 8-bit difference data shown in FIG. FIG. 22 shows two-dimensional data of a frequency distribution type evaluation value table generated according to the configuration of the present invention shown in FIG.

  In FIG. 21, it corresponds to the peak (extreme value) 411 corresponding to the motion vector (9, 0) corresponding to the moving object (A) and the motion vector (−9, 0) corresponding to the moving object (B). A peak (extreme value) 412 and a peak (extreme value) 413 corresponding to a motion vector (0, 0) corresponding to a still region corresponding to the background appear, and three peaks (extreme values) appear, indicating a motion indicating stillness. The peak (extreme value) 413 corresponding to the vehicle (0, 0) shows the largest peak, and the peak (extreme value) 411, 412 of the motion vector (9, 0) or (-9, 0) corresponding to the moving object. The evaluation value table has a small steepness and is easy to be erroneously detected.

  On the other hand, in FIG. 22, a peak 421 corresponding to the motion vector (9, 0) corresponding to the moving object (A) and a peak 422 corresponding to the motion vector (−9, 0) corresponding to the moving object (B). However, it is a table that has a large peak (extreme value), a large steepness, and a reliable detection. The peak (extreme value) 423 indicating the motion vector (0, 0) indicating stillness corresponding to the still region of the background is small. This is an inevitable phenomenon because the representative point is selected by stillness determination. In the case where the selection of the representative point by stillness determination is performed, it is not necessary to detect from the evaluation value table by including the still vector as a necessary vector in the candidate vector, and there is no problem.

[6. Details of candidate vector extraction process]
As described above, the position of the peak (extreme value) appearing in the evaluation value table corresponds to the movement existing on one screen, and the candidate vector is determined based on the position of the peak (extreme value) appearing in the evaluation value table. To be acquired. For example, in the frequency distribution evaluation value table shown in FIGS. 20 and 22, the motion vector (9, 0) and the motion vector (−9, 0) can be extracted as candidate vectors.

  It can be said that the higher the peak (extreme value) steepness in the evaluation value table, the higher the credibility, and the higher the frequency, the higher the credibility. Hereinafter, a method for extracting candidate vectors from the evaluation value table in consideration of these will be described.

  The candidate vector extraction unit 102 in the motion vector detection apparatus shown in FIG. 5 executes the candidate vector extraction process from the evaluation value table. The candidate vector extraction unit 102 receives the frequency distribution type evaluation value table generated by the evaluation value table forming unit 101 using the method described above, and executes peak (extreme value) detection from the received frequency distribution type evaluation value table. The candidate vector is extracted based on the coordinates of the detected peak (extreme value).

  As described above, the evaluation value table has a size corresponding to the search area. If the search area range is set to -H to (H-1) in the horizontal direction and -V to (V-1) in the vertical direction, the size of the evaluation value table is 2H in the horizontal direction and 2V in the vertical direction. It becomes the size of. A conceptual diagram of the evaluation value table in this case is shown in FIG. FIG. 23 is a plan view of the evaluation value table. Horizontal: 2H and vertical: 2V correspond to the size of the search area.

  The candidate vector extraction unit 102 shown in FIG. 5 receives the evaluation value table generated by the evaluation value table forming unit 101, and in order to check the peak (extreme value) from the received evaluation value table, the evaluation value table shown in FIG. The vicinity of the attention point 501 at 500 is referred to. The attention point 501 has coordinates (i, j).

  For example, when there is a peak at the center position of the search area, it corresponds to the still vector (0, 0). The candidate vector extraction unit 102 performs processing for determining whether or not there is a peak at each coordinate (i, j) point, and extracts a candidate vector corresponding to the coordinate (i, j) determined to have a peak. .

  When there is a peak at the center of the search area, for example, at coordinates (0, 0), the still vector (0, 0) is extracted as a candidate vector. However, in the frequency distribution type evaluation value table of the present invention, the candidate vector corresponding to the static vector (0, 0) is not a target to be obtained from the evaluation value table, and therefore the peak (pole) corresponding to the static vector (0, 0) is used. It is not essential to detect (value).

  The candidate vector extraction unit 102 executes processing for detecting a peak in the evaluation value table 500. In this case, if the information area of the memory storing the evaluation value table information is set to the same size as the search area, the peak position However, when it is located at the top, bottom, left, or right ends of the evaluation value table, it may be difficult to accurately detect the peak position. Therefore, it is preferable that the size of the evaluation value table memory is set slightly wider than the search area.

  A block diagram showing the configuration of the candidate vector extraction unit 102 is shown in FIG. The candidate vector extraction unit 102 includes a data conversion unit 511, a conversion data memory 512, and a frequency order sort processing unit 513. First, in the data conversion unit 511, the data f (i, j) in the evaluation value table is converted in consideration of the extreme value data, and the conversion data g (i, j) is created. Data f (i, j) indicates the frequency of coordinates (i, j). The converted data g (i, j) is data f (i, j) in consideration of the extreme value position or the steepness of the extreme value position and the extreme value with respect to the data f (i, j) indicating the frequency. This is a result of executing an operation based on. Details of these processes will be described later. The conversion data g (i, j) is stored in the conversion data memory 512.

  The conversion data stored in the conversion data memory 512 is then converted into data corresponding to a preset number of candidate vectors in order from a large value of the conversion data g (i, j) by the frequency order sort processing unit 513. These are taken out in order and set as candidate vectors.

  Details of the conversion process based on the data f (i, j) indicating the frequency of the extreme value acquired from the evaluation value table executed in the data conversion unit 511 will be described. Two methods for calculating the conversion data g (i, j) based on the data f (i, j) indicating the frequency of the extreme value will be described.

(6.1. Peak (extreme value) position detection method)
This is a method that places importance on only the position of the peak (extreme value) without taking into account the steepness of the peak (extreme value). The processing flow is shown in FIG. The outline of the peak (extreme value) position detection method is that the change in the slope of the target point on both sides in the horizontal direction is convex (the second derivative component exceeds 0), and the target point is adjacent to the target point in the vertical direction. If the change in slope is convex, the point of interest is the peak (extreme value), so the frequency f (i, j) is set as g (i, j) as it is, and otherwise the peak Since it is not (extreme value), the frequency conversion is reset and data conversion processing is performed so that g (i, j) = 0. A finite number set in advance is extracted as a candidate vector from the larger one of the converted data g (i, j), which is identified as converted data, that is, identified as a peak and set to a value other than 0.

  Each step of the flow will be described. In step S401, as an initial setting, a processing target is set to i = 0, j = 0, that is, the position of one coordinate point in the evaluation value table corresponding to the search area shown in FIG.

In step S402, j <2V, and in step S403, it is determined whether i <2H, that is, whether the processing target is an internal coordinate of the evaluation value table. If it is an internal coordinate of the evaluation value table, in step S404,
A = f (i, j) × 2-f (i + 1, j) −f (i−1, j)
B = f (i, j) × 2-f (i, j + 1) −f (i, j−1)
Is calculated.

  In A, the frequency f (i, j) of the evaluation value table at the point of interest (i, j) forms a peak in the horizontal direction. That is, it is an index value indicating whether or not the change in slope on both sides in the horizontal direction is convex, and B is the frequency f (i, j) of the evaluation value table at the point of interest (i, j) that peaks in the vertical direction. Is forming. That is, it is an index value as to whether or not the change in inclination on both sides in the vertical direction of the attention point is convex.

  In step S405, it is determined whether A is larger than a preset threshold value (TH). If it is determined that the value is large, the change in the slope of the attention point f (i, j) on both sides in the horizontal direction is convex, that is, the frequency f (i, j) of the evaluation value table at the attention point (i, j). ) Is determined to form a peak in the horizontal direction.

  In step S406, it is determined whether or not B is larger than a preset threshold value (TH). If it is determined that the value is large, the change in slope of the attention point f (i, j) on both sides in the vertical direction is convex, that is, the frequency f (i, j) of the evaluation value table at the attention point (i, j). ) Form a peak in the vertical direction.

When both determinations in steps S405 and S406 are determined as Yes, the frequency f (i, j) of the evaluation value table at the point of interest (i, j) forms a peak (extreme value) in both the horizontal direction and the vertical direction. In this case, in step S407,
g (i, j) = f (i, j)
That is, the frequency f (i, j) of the evaluation value table at the point of interest (i, j) is set as conversion data.

If either of the determinations in steps S405 and S406 is determined to be No, the frequency f (i, j) of the evaluation value table at the point of interest (i, j) is a peak (extreme value) in both the horizontal direction and the vertical direction. ) In this case, in this case, in step S408,
g (i, j) = 0
That is, the conversion data = 0 is set.

  The update process of the value (i) is executed in step S409, and the update process of the value (j) is executed in step S411, so that the frequency data f (i, j) of all coordinates constituting the evaluation value table is obtained. Verification is performed, and only the frequency f (i, j) in the evaluation value table forms a peak (extreme value) in both the horizontal direction and the vertical direction.

  The data set as the conversion data g = 0 is not recognized as a peak (extreme value), and the frequency order sort processing unit 513 sets the value of the conversion data g (i, j) in which a value other than 0 is set. It arranges from the larger one and extracts a finite number set in advance as a candidate vector.

(6.2. Filter processing method)
Next, a filter processing method will be described as another method for calculating the conversion data g (i, j) from the frequency data f (i, j) executed in the data conversion unit 511.

  This is a method that also takes into account the steepness of the peak (extreme value), and is a method of converting the data of the evaluation value table by filtering. An example of the filter coefficient applied in this method is shown in FIG.

  26A shows a filter coefficient setting example when the general form is used, FIG. 26B shows the case where the processing unit is a frame, and FIG. 26C shows the case where the processing unit is a field. The second derivative component is added to the frequency of the attention point. FIG. 27 shows a data conversion processing flow according to this filter processing method.

  Each step of the flow will be described. In step S501, as an initial setting, a processing target is set to i = 0, j = 0, that is, the position of the upper left coordinate of the evaluation value table corresponding to the search area shown in FIG.

In step S502, j <2V, and in step S503, it is determined whether i <2H, i.e., whether the processing target is an internal coordinate of the evaluation value table. If it is an internal coordinate of the evaluation value table, in step S504,
g (i, j) =
f (i, j−1) × Coef [1] + f (i−1, j) × Coef [3]
+ F (i, j) × Coef [4] + f (i + 1, j) × Coef [5]
+ F (i, j + 1) × Coef [7]
Is calculated.

  In this way, the frequency data f (i, j) of the attention point (i, j), the frequency data of the two adjacent positions in the horizontal direction, and the frequency data of the two adjacent positions in the vertical direction are respectively multiplied by the filter coefficients, The integration result is assumed to be converted data g (i, j) of the target coordinate (i, j).

  By this processing, the result of amplification of the peak (extreme value) steepness can be set as the conversion data g (i, j) of the coordinate of interest (i, j).

  The update process of the value (i) is executed in step S505, and the update process of the value (j) is executed in step S511, whereby the frequency data f (i, j) of all coordinates constituting the evaluation value table is obtained. Verification is performed, and converted data g (i, j) of each coordinate is calculated by an expression to which a filter coefficient is applied.

  The frequency order sorting processing unit 513 arranges the converted data g (i, j) of each coordinate calculated by the formula to which the filter coefficient is applied, and extracts a finite number set in advance as a candidate vector.

[7. Details of motion vector determination process]
The motion vector detection unit 103 illustrated in FIG. 5 is a process in which the candidate vector extraction unit 102 selects and associates an optimum vector for each pixel from a plurality of candidate vectors extracted from the evaluation value table by any of the above-described methods. To determine the motion vector for each pixel.

  FIG. 28A shows the configuration of the motion vector detection unit 103 in the motion vector detection device of the present invention. For comparison, FIG. 28B shows a block diagram showing a configuration of motion vector detection processing of the conventional method. FIG. 29 is a diagram for explaining an outline of processing of the motion vector detection unit 103 in the motion vector detection device of the present invention.

  In the motion vector detection unit, an optimal vector is selected and associated with each pixel from a plurality of candidate vectors, and a motion vector for each pixel is determined. In this process, basically, evaluation by block matching with the moving position based on each candidate vector is performed, and a candidate vector indicating the pixel data having the highest correlation is selected. That is, a motion vector to be associated with each pixel is determined based on matching evaluation for a plurality of candidate vectors.

  As described above, the static vector (0, 0) is set as the default candidate vector as one of the candidate vectors. Unlike the candidate vector selected based on the peak detection of the frequency distribution type evaluation value table, the static vector (0, 0) is set in advance as a default candidate vector. In the motion vector detection unit 103, a candidate vector selected based on peak detection in the frequency distribution type evaluation value table and a static vector (0, 0) set as a default candidate vector are associated with each pixel. A process of associating motion vectors is executed.

For example, assuming that the preset number of candidate vectors is 3, the above-described frequency distribution type evaluation value table is formed between the previous frame [F _t−1 ] and the current frame [F _t ], and then the evaluation value table is displayed. As a result of extracting candidate vectors based on the above, it is assumed that three candidate vectors V1, V2, and V3 as shown in FIG. 29 are obtained.

  Although the static vector (0, 0) should be originally considered, as an example, a processing example when a motion vector is selected from three candidate vectors V1, V2, and V3 will be described here.

A process of determining the best motion vector from among three candidate vectors for the target pixel with P (0) as the target pixel of the previous frame [F _t−1 ] will be described. The pixels of the current frame [F _t ] to which the target pixel P (0) of the previous frame [F _t−1 ] has moved by the candidate vector are respectively A (0), B (0), C ( 0).

Assume that adjacent pixels around the target pixel P (0) include a block composed of eight pixels P (1),..., P (8) as shown in the figure. Although not shown in the vicinity of the pixel position of the current frame [F _t ] specified by the three candidate vectors V1, V2, and V3, A (0), B (0), and C (0), A (1),..., A (8), B (1),..., B (8), C (1),.

In block matching in the motion vector detection unit, a block region including peripheral pixels of the target pixel is set, a block including the target pixel P (0) in the previous frame [F _t−1 ], and candidate vectors (V1, V2). , V3), the pixel position of the current frame [F _t ] and the correlation of the pixel values of the block including A (0), B (0), C (0) are determined. Here, the block matching size is, for example, 3 pixels × 3 pixels for easy understanding.

In the conventional motion vector detection method shown in FIG. 28B, pixel value data of the current frame [F _t ] is stored in the current frame memory 631, and the previous frame [F _t−1 ] is stored in the previous frame memory 632. The pixel value data is stored, and the pixel value data of the current frame [F _t ] and the pixel value data of the previous frame [F _t−1 ] are read from each of these memories, and the block matching processing unit 633 reads them in advance. Based on the acquired candidate vector data, a block including each pixel is set, a block matching process is executed, and a candidate vector corresponding to the block with the highest correlation is set as a motion vector of the pixel. .

  When there are candidate vectors V1, V2, and V3, the block matching processing unit 633 calculates the following (Expression 1), (Expression 2), and (Expression 3) as evaluation values in the block matching process for each pixel.

In the above formula, (Formula 1) is a block including a pixel of interest in the previous frame [F _t−1 ] P (0) and the pixel position of the current frame [F _t ] specified by the candidate vector V1, A ( 0) is a block matching evaluation value of a block including (0), and (Expression 2) is a block including a target pixel of P (0) in the previous frame [F _t−1 ] and a current frame [ F _t ] is the block matching evaluation value of the block including the pixel position, B (0), and (Equation 3) is a block including the target pixel P (0) in the previous frame [F _t−1 ] and a candidate. This is a block matching evaluation value of a block including the pixel position of the current frame [F _t ] designated by the vector V3 and C (0).

  This is a process of selecting the minimum value of these evaluation values and using the candidate vector corresponding to the expression that outputs the minimum value as the motion vector of the pixel P (0).

The motion vector detection unit 103 according to the present invention shown in FIG. 28A stores the pixel value data of the current frame [F _t ] in the current frame memory 611 and stores the previous frame [F _{t− 1} ] pixel value data is stored, the pixel value data of the current frame [F _t ] and the pixel value data of the previous frame [F _t−1 ] are read from each of these memories, and the block matching processing unit 615 is read out. , A block including each pixel is set based on the candidate vector data and the block matching process is executed, but the candidate vector that does not need to be matched before the block matching process in the block matching processing unit 615 is executed. Is removed by the temporary determination unit 614 and only candidate vectors that need to be matched are selected. The selected candidate vector is output from the candidate vector information storage unit 613 storing the candidate vector information to the block matching processing unit 615, and the block matching process is executed only for the selected candidate vector to correspond to the block with the highest correlation. A process of setting the candidate vector as the motion vector of the pixel is executed.

In the method of the present invention, candidate vector extraction processing is executed from the evaluation value table formed by representative point matching, and the provisional determination unit 614 performs the target pixel P (0) of the previous frame [F _t−1 ]. ) And the pixel position of the current frame [F _t ] specified by the candidate vector (V1, V2, V3), A (0), B (0), C (0), and the difference is calculated in advance. If it is larger than the predetermined threshold, it is determined that the motion vector corresponding to the pixel is inappropriate, and it is determined that it is not necessary to perform block matching determination based on a block including surrounding pixels.

  That is, when the pixel value difference calculated in each of the following expressions is larger than the threshold value, the block matching based on the candidate vector corresponding to the expression is not executed.

The provisional determination unit 614 calculates the above (Expression 4), (Expression 5), and (Expression 6). (Expression 4) is the absolute value of the pixel value difference between the pixel of interest of the previous frame [F _t-1 ] P (0) and the pixel A (0) of the current frame [F _t ] specified by the candidate vector V1. (Equation 5) represents the pixel value difference between the target pixel of the previous frame [F _t-1 ] P (0) and the pixel B (0) of the current frame [F _t ] specified by the candidate vector V2. (Equation 6) shows the pixel of interest of the previous frame [F _t−1 ] as P (0) and the pixel C (0) of the current frame [F _t ] specified by the candidate vector V3. The absolute value of the pixel value difference is shown.

  The provisional determination unit 614 compares the calculation results Ea ′, Eb ′, and Ec ′ of the above (Expression 4), (Expression 5), and (Expression 6) with predetermined threshold values. Candidate vectors corresponding to the expression are excluded from the block matching processing targets. Determination information based on the above formula by the provisional determination unit 614 is input to the candidate vector information storage unit 613, and a candidate vector whose pixel value difference obtained by the above formula is larger than a threshold from the candidate vector information storage unit 613 is a block. It is not output to the matching processing unit 615, and block matching processing corresponding to the candidate vector is not executed.

For example, when the threshold value is set as the pixel value difference [32], the target pixel of the previous frame [F _t−1 ] is P (0) and the current frame [F _t ] specified by the candidate vectors V1, V2, and V3. If the pixel value difference absolute value of the pixel A (0), B (0), C (0) exceeds 32, the candidate vector is not output from the candidate vector information storage unit 613 to the block matching processing unit 615, The block matching process corresponding to the candidate vector is not executed.

  By providing the provisional determination unit 614, it is possible to determine the best motion vector while reducing the number of operations in the block matching process.

  In the above description, the processing corresponding to the still vector (0, 0) is omitted, but the pixel difference value corresponding to the still vector (0, 0) is also similarly determined in the temporary determination unit 614. It is calculated and determined whether or not to select as an execution target of the block matching process. If the pixel difference value corresponding to the still vector (0, 0) is smaller than the threshold value, the block matching process is set as the target. Based on the matching process, the candidate vector or still vector corresponding to the block with the highest correlation is set as the motion vector corresponding to the pixel from the result of the block matching process corresponding to the still vector and other candidate vectors. .

  FIG. 30 is a flowchart illustrating a processing procedure in the motion vector determination unit. Each step will be described.

  First, in step S <b> 601, target pixel data is captured from the previous frame memory 612 to the temporary determination unit 614. In step S602, pixel data corresponding to the candidate vector is fetched from the current frame memory 611 to the temporary determination unit 614. In this process, pixel data corresponding to the still vector is also captured as a processing target.

  Next, in step S603, the temporary determination unit 614 determines a candidate vector that does not need to be subjected to the block matching process, based on the above-described equations (Equations 4 to 6). In this process, pixel data corresponding to a still vector is also processed, and pixel value difference data of a pixel selected corresponding to the still vector is also calculated.

  In step S <b> 604, information on candidate vectors determined to be necessary to execute block matching processing in the temporary determination unit 614 is fetched from the candidate vector information storage unit 613 into the block matching processing unit 615.

  In step S <b> 605, the pixel-of-interest data is captured from the previous frame memory 612 to the block matching processing unit 615. In step S606, the block matching processing unit 615 executes the block matching process only for candidate vectors that the temporary determination unit 614 determines to require the block matching process (including a default still vector in some cases). In step S607, a candidate vector corresponding to the block with the highest correlation is selected as a result of the block matching process, and a process of setting the motion vector of the pixel is executed.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims section described at the beginning should be considered.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, in the process of selecting and associating a motion vector corresponding to each pixel constituting moving image data from a plurality of candidate vectors, pixels of different frames specified by the candidate vector A correlation determination process with the pixel value of the image is executed, and as a result of the correlation determination process, only candidate vectors that specify pixels with a correlation are selected, and a block set corresponding to the selected candidate vector is applied. By executing the block matching process, it is possible to reduce the useless block matching process, and it is possible to perform a motion vector detection apparatus and a motion vector detection that enable efficient and accurate motion vector detection. Therefore, for example, by applying the present invention to an image processing apparatus that executes encoding processing of moving image data, an efficient and accurate motion vector detection process can be performed.

  Furthermore, according to the configuration of the present invention, in the candidate vector extraction process, the integrated evaluation value at the target point in the evaluation value table is larger than both the horizontal direction neighbor and the vertical direction neighbor, and the change in the integrated evaluation value is convex. A process of selecting coordinates as peaks (extreme values), or by multiplying the integrated evaluation values in the evaluation point table and the integrated evaluation values in the vicinity of the attention point by a predetermined filter coefficient and calculating the conversion data as conversion data, Since the peak (extreme value) detection process is executed based on the data, the candidate vector extraction process from the evaluation value table can be executed accurately and efficiently. Therefore, for example, by applying the present invention to an image processing apparatus that executes encoding processing of moving image data, an efficient and accurate motion vector detection process can be performed.

  Furthermore, according to the configuration of the present invention, in the candidate vector extraction process, the integrated evaluation value of the coordinates where the change of the integrated evaluation value at the attention point of the evaluation value table is convex, or the attention point and the attention point of the evaluation value table A result obtained by multiplying a neighboring integrated evaluation value by a predetermined filter coefficient and integrating the result is calculated as conversion data, and a predetermined number of candidate vectors are determined in order from the largest conversion data. Therefore, a motion vector detection device capable of efficiently selecting a limited number of candidate vectors is realized. Therefore, for example, by applying the present invention to an image processing apparatus that executes encoding processing of moving image data, an efficient and accurate motion vector detection process can be performed.

  Furthermore, according to the configuration of the present invention, in the process of selecting and associating the motion vector corresponding to each pixel constituting the moving image data from a plurality of candidate vectors, the motion vector corresponding to each frame constituent pixel of the moving image data is evaluated. Since the candidate vector acquired from the value table and the default static vector are selected and associated with each pixel, there is no need to detect the static vector from the evaluation value table, and there is a peak with movement (extreme value) ) Can be generated in a prominent form, and a motion vector detection device and a motion vector detection capable of accurate motion vector detection are possible. Therefore, for example, by applying the present invention to an image processing apparatus that executes encoding processing of moving image data, an efficient and accurate motion vector detection process can be performed.

It is a figure explaining the outline | summary of a block matching method. Explains the outline of the evaluation value table creation process by the representative point matching method applied in the motion vector detection process, the candidate vector extraction process based on the evaluation value table, and the motion vector setting process corresponding to each pixel based on the extracted candidate vector (No. 1) Explains the outline of the evaluation value table creation process by the representative point matching method applied in the motion vector detection process, the candidate vector extraction process based on the evaluation value table, and the motion vector setting process corresponding to each pixel based on the extracted candidate vector (No. 2) Explains the outline of the evaluation value table creation process by the representative point matching method applied in the motion vector detection process, the candidate vector extraction process based on the evaluation value table, and the motion vector setting process corresponding to each pixel based on the extracted candidate vector (No. 3) It is a figure which shows one Example structure of the motion vector detection apparatus of this invention which performs a motion vector detection process. It is a flowchart explaining the process sequence of the motion vector detection apparatus of this invention which performs a motion vector detection process. It is a figure which shows the detailed structure of the evaluation value table formation part of the motion vector detection apparatus of this invention, and a conventional structure. It is a figure explaining the dynamic range (DR) calculation process of the representative point vicinity area | region (3x3) using the correlation calculation result of representative point matching. It is a figure which shows the structural example of a feature-value extraction part. It is a flowchart explaining the variable threshold value generation process procedure performed in a variable threshold value generation part. It is a figure explaining the threshold value generation process based on the dynamic range data of the representative point vicinity as a feature-value of the variation in space in a threshold value generation part. It is a figure which shows the graph which plotted the input-output relationship of the threshold value production | generation part. It is a figure which shows the flowchart explaining the process sequence of an evaluation value table formation part. It is a figure explaining the structure of the evaluation value table formation part with the structure which excludes the representative point of the part (stationary part) where the dynamic range of a representative point vicinity area is small. It is a figure explaining the structure of the evaluation value table formation part which has the structure which excludes the representative point which exists in a still area. It is a figure explaining the structure of the evaluation value table formation part which has the structure which excludes the representative point with many candidates. It is a figure which shows the detailed structure of the candidate number determination part comprised in a representative point selection part. It is a figure which shows the example of the moving image data used as an evaluation value table production | generation process target. It is a figure which shows the example of an evaluation value table produced | generated by applying the 8-bit difference frame data output from a correlation calculating part as it is, and integrating | accumulating. It is a figure which shows the example of a frequency distribution type | mold evaluation value table produced | generated according to the process of the evaluation value table formation part which concerns on this invention. It is the two-dimensional data of the evaluation value table obtained as a result of integrating the conventional 8-bit difference data shown in FIG. 21 is two-dimensional data of a frequency distribution type evaluation value table generated according to the configuration of the present invention shown in FIG. 20. It is a figure which shows the conceptual diagram (plan view) of the evaluation value table which has a magnitude | size whose horizontal is 2H and vertical is 2V. It is a block diagram which shows the structure of a candidate vector extraction part. It is a figure which shows the peak (extreme value) position detection processing flow by the peak (extreme value) position detection system which attached importance only to the position of a peak (extreme value), without taking into consideration the steepness of a peak (extreme value). It is a figure which shows the example of a filter coefficient setting applied to the filter processing system as a peak (extreme value) position detection process which considered the steepness of the peak (extreme value). It is a figure which shows the peak (extreme value) position detection processing flow by the filter processing system as a peak (extreme value) position detection process which considered the steepness of the peak (extreme value). It is a figure which shows the structure of the motion vector detection part in the motion vector detection apparatus of this invention, and a conventional structure. It is a figure explaining the process outline | summary of the motion vector detection part in the motion vector detection apparatus of this invention. It is a figure which shows the flowchart explaining the process sequence in a motion vector determination part.

Explanation of symbols

10 Previous frame 11 Inspection block By
12 Search area 20 Current frame 21 Reference block Bx
70 Front frame 71 Representative point Ry
80 Current Frame 81 Search Area 90 Evaluation Value Table 91 Previous Frame Pixel 95, 96, 97 Current Frame Pixel 101 Evaluation Value Table Formation Unit 102 Candidate Vector Extraction Unit 103 Motion Vector Detection Unit 104 Control Unit (Controller)
200 Evaluation Value Table Forming Unit 210 Correlation Calculation Unit 211 Representative Point Memory 212 Subtraction Circuit 213 Absolute Value Calculation Unit 220 Variable Threshold Generation Unit 221 Feature Amount Extraction Unit 222 Feature Amount Memory 223 Threshold Generation Unit 230 Correlation Determination Unit 231 Comparison Unit 240 Representative Point Selection unit 241 Reference value memory 242 Comparison unit 243 Gate 250 Evaluation value table calculation unit 251 Evaluation value integration unit 252 Evaluation value table memory 260 Representative point selection unit 261 Reference value memory 262 Comparison unit 263 Flag memory 264 Gate 270 Representative point selection unit 271 Candidate number determination unit 301 Maximum value detection unit 302 Register 303 Minimum value detection unit 304 Register 305 Subtraction circuit 306 Latch 351 Bit decomposition unit 352 to 355 OR circuit 356 Bit synthesis unit 381 Buffer memory 382 Counter 83 the reference value memory 384 comparing unit 385 the gate 400 static background 401 moving object (A)
402 Moving object (B)
411-413 peak (extreme value)
421 to 423 peak (extreme value)
500 Evaluation Value Table 501 Point of Interest 511 Data Conversion Unit 512 Conversion Data Memory 513 Frequency Order Sort Processing Unit 611 Current Frame Memory 612 Previous Frame Memory 613 Candidate Vector Information Storage Unit 614 Temporary Determination Unit 615 Block Matching Processing Unit 631 Current Frame Memory 632 Previous Frame memory 635 Block matching processor

Claims (17)

A motion vector detection device for detecting a motion vector from moving image data;
An evaluation value table forming unit that generates an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction unit that extracts one or more candidate vectors based on the evaluation value table;
A motion vector detection unit that executes a process of selecting a motion vector corresponding to each frame constituent pixel of moving image data from the candidate vectors and associating the motion vector,
The evaluation value table forming unit
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation unit for calculating
A correlation determination unit that calculates an evaluation value based on the correlation information calculated by the correlation calculation unit;
From the evaluation values calculated by the correlation determination unit, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. A representative point selection unit that outputs the generated information of the evaluation value table,
An evaluation value table calculation unit that integrates the evaluation values output by the representative point selection unit and generates an evaluation value table as an integration result,
The motion vector detection unit
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation A motion vector detection apparatus, characterized in that a process for determining a motion vector is executed. The motion vector detection unit
Claim a motion vector corresponding to the frame each constituent pixel of the moving image data, characterized in that said candidate vectors, is configured to perform processing for associating to each pixel selected from among the static vector default settings 1 The motion vector detection device described in 1. The candidate vector extraction unit
It is a configuration for executing candidate vector extraction processing based on peak (extreme value) detection in the evaluation value table,
A configuration for executing processing for selecting, as a peak (extreme value), a coordinate whose integrated evaluation value at a target point in the evaluation value table is larger than both the horizontal direction neighbor and the vertical direction neighbor and the change in the integrated evaluation value is convex. The motion vector detection apparatus according to claim 1, wherein The candidate vector extraction unit
When the coordinates of the point of interest are (i, j) and the integrated evaluation value of the point of interest in the evaluation value table is f (i, j),
A = f (i, j) × 2-f (i + 1, j) −f (i−1, j)
B = f (i, j) × 2-f (i, j + 1) −f (i, j−1)
2. The motion vector detection according to claim 1, wherein the coordinates are identified as a peak (extreme value) formation point when both A and B calculated by the above are greater than a predetermined threshold value. apparatus. The candidate vector extraction unit further includes:
A coordinate in which the change in the integrated evaluation value is convex is selected as a peak (extreme value), and a preset number of candidate vectors are determined in order from the selected coordinate in descending order of the integrated evaluation value. The motion vector detection device according to claim 3, wherein the motion vector detection device is provided. The candidate vector extraction unit
It is a configuration for executing candidate vector extraction processing based on peak (extreme value) detection in the evaluation value table,
The integrated evaluation values in the evaluation value table at the attention point and in the vicinity of the attention point are predetermined filter coefficients , and the result of multiplication by multiplying the filter coefficient that amplifies the steepness of the peak (extreme value) is calculated as conversion data. The motion vector detection device according to claim 1, wherein a peak (extreme value) detection process is executed based on the converted data. The candidate vector extraction unit
When the coordinate of the point of interest is (i, j), the integrated evaluation value of the point of interest in the evaluation value table is f (i, j), and the conversion data is g (i, j),
g (i, j) =
f (i, j−1) × Coef [1] + f (i−1, j) × Coef [3]
+ F (i, j) × Coef [4] + f (i + 1, j) × Coef [5]
+ F (i, j + 1) × Coef [7]
In the above equation, Coef [1] is a filter coefficient corresponding to the coordinates (i, j−1), Coef [3] is a filter coefficient corresponding to the coordinates (i−1, j), and Coef [4]. the filter coefficients corresponding to the coordinates (i, j), Coef [ 5] , the filter coefficients corresponding to the coordinates (i + 1, j), Coef [7] is a filter coefficient corresponding to the coordinates (i, j + 1) Yes , it is a filter coefficient that amplifies the sharpness of the peak (extreme value) .
2. The motion vector detection device according to claim 1, wherein a peak (extreme value) detection process is executed based on the value of the conversion data g (i, j) calculated by the above equation. The candidate vector extraction unit further includes:
8. The motion vector detection apparatus according to claim 6, wherein a predetermined number of candidate vectors are determined in order from the largest conversion data. A motion vector detection method for detecting a motion vector from moving image data,
An evaluation value table forming step for generating an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction step of extracting one or more candidate vectors based on the evaluation value table;
A motion vector detection step of executing a process of selecting a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vectors and associating the motion vector,
The evaluation value table forming step includes:
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation step for calculating
A correlation determination step of calculating an evaluation value based on the correlation information calculated in the correlation calculation step;
From the evaluation values calculated in the correlation determination step, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. Representative point selection step to output as the generation information of the evaluation value table,
An evaluation value table calculating step for integrating the evaluation values output in the representative point selection step and generating an evaluation value table as an integration result;
The motion vector detection step includes
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation A motion vector detection method, characterized in that it is a step of executing a process of determining as a motion vector. The motion vector detection step includes
10. The step of executing a process of selecting a motion vector corresponding to each frame constituent pixel of moving image data from the candidate vector and a default still vector and associating with each pixel. The motion vector detection method described in 1. The candidate vector extraction step includes:
Performing a candidate vector extraction process based on peak (extreme value) detection in the evaluation value table;
Executing a process of selecting, as a peak (extreme value), a coordinate in which the integrated evaluation value at the target point in the evaluation value table is larger than both the horizontal and vertical neighbors and the change in the integrated evaluation value is convex. The motion vector detection method according to claim 9, further comprising: The candidate vector extraction step includes:
When the coordinates of the point of interest are (i, j) and the integrated evaluation value of the point of interest in the evaluation value table is f (i, j),
A = f (i, j) × 2-f (i + 1, j) −f (i−1, j)
B = f (i, j) × 2-f (i, j + 1) −f (i, j−1)
10. The motion vector detection according to claim 9, further comprising the step of identifying the coordinates as peak (extreme value) forming points when A and B calculated by the above are both greater than a predetermined threshold value. Method. The candidate vector extraction step further includes:
Selecting a coordinate having a convex change in the integrated evaluation value as a peak (extreme value), and determining a preset number of candidate vectors from the selected coordinate in order from the largest integrated evaluation value; The motion vector detection method according to claim 11 or 12, further comprising: The candidate vector extraction step includes:
Performing a candidate vector extraction process based on peak (extreme value) detection in the evaluation value table;
The integrated evaluation values in the evaluation value table at the attention point and in the vicinity of the attention point are predetermined filter coefficients , and the result of multiplication by multiplying the filter coefficient that amplifies the steepness of the peak (extreme value) is calculated as conversion data. 10. The motion vector detection method according to claim 9, further comprising a step of executing a peak (extreme value) detection process based on the converted data. The candidate vector extraction step includes:
When the coordinate of the point of interest is (i, j), the integrated evaluation value of the point of interest in the evaluation value table is f (i, j), and the conversion data is g (i, j),
g (i, j) =
f (i, j−1) × Coef [1] + f (i−1, j) × Coef [3]
+ F (i, j) × Coef [4] + f (i + 1, j) × Coef [5]
+ F (i, j + 1) × Coef [7]
In the above equation, Coef [1] is a filter coefficient corresponding to the coordinates (i, j−1), Coef [3] is a filter coefficient corresponding to the coordinates (i−1, j), and Coef [4]. the filter coefficients corresponding to the coordinates (i, j), Coef [ 5] , the filter coefficients corresponding to the coordinates (i + 1, j), Coef [7] is a filter coefficient corresponding to the coordinates (i, j + 1) Yes , it is a filter coefficient that amplifies the sharpness of the peak (extreme value) .
10. The motion vector detection method according to claim 9, wherein the peak (extreme value) detection process is performed based on the value of the conversion data g (i, j) calculated by the above equation. The candidate vector extraction step further includes:
16. The motion vector detection method according to claim 14, further comprising a step of determining a preset number of candidate vectors in order from the largest conversion data. A computer program that causes a motion vector detection device to execute a motion vector detection process for detecting a motion vector from moving image data.
An evaluation value table forming step for generating an evaluation value table based on pixel value correlation information between different frames on the time axis;
A candidate vector extraction step of extracting one or more candidate vectors based on the evaluation value table;
A motion vector detection step of executing a process of selecting a motion vector corresponding to each frame constituent pixel of the moving image data from the candidate vector and associating the motion vector ;
And execute
In the evaluation value table forming step,
Correlation information between frames based on a representative point matching process that determines the correlation between the representative point of the block set in one frame constituting the moving image data and the pixel value of the constituent pixels of the search area set in the other different frame A correlation calculation step for calculating
A correlation determination step of calculating an evaluation value based on the correlation information calculated in the correlation calculation step;
From the evaluation values calculated in the correlation determination step, only effective evaluation values excluding evaluation values based on correlation determination results of representative points whose number of pixel candidates determined to have a correlation is larger than a predetermined threshold are selected. Representative point selection step to output as the generation information of the evaluation value table,
The evaluation value output in the representative point selection step is integrated, and an evaluation value table calculation step for generating an evaluation value table as an integration result is executed,
In the motion vector detection step ,
A candidate for executing a correlation determination process between a pixel value of a pixel for setting a motion vector and a pixel value of a pixel in a different frame specified by the candidate vector, and specifying a pixel having a correlation as a result of the correlation determination process Select only the vector and execute the block matching process that applies the block set corresponding to the selected candidate vector. As a result of the block matching process, the candidate vector corresponding to the block with the highest correlation A computer program for executing a step of determining as a motion vector of a computer program.

Images