JP7077115B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to a technique for detecting and tracking the movement of a subject or the like in an image.

In an image pickup device such as a digital still camera or a digital video camera, for example, by detecting the amount of movement between frame images of a captured image and aligning each of a plurality of frame images, an image due to camera shake or the like can be obtained. May correct blur. As a method of detecting the amount of movement between frame images, for example, a method of estimating the amount of movement from a captured frame image is known. Various methods for estimating the amount of motion using a frame image have been conventionally proposed, and a typical example thereof is motion vector detection by template matching. In template matching, first, one of two temporally adjacent frame images in a video is used as an original image, and the other is used as a reference image. Then, a rectangular area of a predetermined size arranged on the original image is used as a template block, and a correlation with the distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the movement destination of the template block, and the direction and movement amount to the movement destination when the position of the template block on the original image is used as a reference is used as the motion vector. Detected.

Further, as a technique for improving the detection rate of a motion vector, for example, a technique of extracting feature points from an image and arranging template blocks based on the extracted feature points to perform template matching is known. However, when feature point extraction is performed on the entire image, the distribution of feature points may become uneven. When the motion vector obtained from the non-uniform distribution of the feature points is used for the blur correction of the image, the region where the distribution of the feature points is concentrated may be the main blur correction. On the other hand, in Patent Document 1, an image is divided into a plurality of grids, feature values representing the size of features are calculated for each pixel in each grid, and the pixel with the largest feature value is the feature point of the corresponding grid. Disclosed is a technique for making the distribution of feature points substantially uniform by extracting as.

Further, as a technique for improving the detection rate of a motion vector, for example, a technique for tracking a feature point is known. For example, Patent Document 2 discloses a technique for realizing tracking of feature points by sequentially detecting motion vectors of feature points extracted from an image over a plurality of consecutive frame images. However, when tracking feature points, for example, the number of feature points may disappear or decrease, resulting in tracking failure. When the feature points of the tracking target disappear or the number decreases in this way, it is necessary to set (supplement) another feature point as the tracking target. In addition, for example, Patent Document 3 discloses a technique for limiting the range of tracking from the moving direction of the tracking target and the like.

Japanese Unexamined Patent Publication No. 2008-192060 JP-A-2007-334625 Japanese Unexamined Patent Publication No. 2012-73997

By the way, as a method for further improving the motion vector detection accuracy, for example, two processing areas are set in one image, and a large number of motion vectors are detected by a motion vector detection unit corresponding to each of the two processing areas. There is also. However, when the feature point tracking is performed by applying this method, for example, when the subject moves across the boundary of the adjacent processing area, the feature point to be tracked disappears. In this case, for example, even if the moving direction of the tracking target is known as in Patent Document 3, the feature points cannot be tracked, and the tracking performance is deteriorated.

Therefore, it is an object of the present invention to improve the detection performance of the motion vector, prevent the vanishing point disappearing at the time of tracking the feature point, and improve the tracking performance.

The image processing apparatus of the present invention is provided with a subject detecting means for detecting a subject from a captured image, a setting means for setting a plurality of processing areas for the image, and a corresponding processing area for each of the processing areas. Processing in which feature points are extracted for each extraction frame area in which the corresponding processing area is divided into a plurality of areas, motion vector detection is performed using the feature points, and the feature points are tracked based on the feature points and the motion vector. When the subject moves across the boundary of the processing area, the means and the control means for controlling the setting when the processing means corresponding to at least the moving destination processing area of the moving subject extracts a new feature point. And, characterized by having.

According to the present invention, it is possible to improve the detection performance of the motion vector, prevent the vanishing point disappearing at the time of tracking the feature point, and improve the tracking performance.

It is a figure which shows the schematic structure of the digital camera which is an example of an image processing apparatus. It is a figure used for the schematic structure and operation explanation of the vector detection part. It is a figure which shows the relationship of a grid, a feature point, and a matching area. It is a flowchart of the process from motion vector detection to feature point tracking. It is a figure which shows the schematic structure of the feature point calculation part. It is a graph used for the explanation of the determination process of the accuracy determination unit. It is a flowchart of a tracking destination determination process. It is a figure used for the outline explanation of the feature point tracking process. It is a figure used for the explanation of the movement of a processing area and a subject, and the tracking of a feature point. It is a figure which showed the example which set the upper and lower two processing areas with respect to the photographed image. It is a figure used for the explanation of the grid arrangement control of 1st Embodiment. It is a flowchart of grid arrangement control in 1st Embodiment. It is a figure used for the explanation of the moving example of the subject position of 1st Embodiment. It is a figure used for the explanation of the grid arrangement of the 2nd Embodiment. It is a flowchart of the feature amount adjustment control in 2nd Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram showing a schematic configuration of a digital camera which is an application example of the image processing apparatus according to the embodiment of the present invention.
In FIG. 1, the imaging optical unit 101 includes a lens, a diaphragm, and the like. At the time of photographing, the image pickup optical unit 101 performs focus adjustment and exposure adjustment, and forms an optical image on the image pickup element 102.
The image pickup device 102 has a photoelectric conversion function for converting an optical image into an electric signal (analog image signal), and is composed of a CCD, a CMOS sensor, or the like.
The A / D conversion unit 103 converts the analog image signal from the image pickup device 102 into a digital image signal.

The DRAM (memory) 107 is a memory for storing data and the like, and stores a predetermined number of still images, moving images for a predetermined time, data such as audio, constants for operation of the CPU 112, expanded programs, and the like. It has enough storage capacity. In the case of the present embodiment, the DRAM 107 can store at least the original image and the reference image in the template matching process described later. When motion vector detection, which will be described later, is performed, the original image and the reference image are adjacent frame images on the time axis. For example, when the reference image is the current frame image (current frame), the original image is relative to the reference image. It is the previous frame image (previous frame) in time.
The memory control unit 106 writes or reads data from the DRAM 107 in response to an instruction from the CPU 112 or the data transfer unit 105.
The ROM 109 is a memory that can be electrically erased and recorded, and an EEPROM or the like is used. The ROM 109 stores constants, programs, etc. for the operation of the CPU 112.
The non-volatile memory control unit 108 writes or reads data from the ROM 109 (non-volatile memory) in response to an instruction from the CPU 112.

The CPU 112 is composed of a microcomputer or the like that controls the entire image processing device (entire digital camera), gives an operation instruction to each unit, and executes various control processes. The CPU 112 controls the image processing unit 104, the data transfer unit 105, the memory control unit 106, the non-volatile memory control unit 108, the display control unit 110, the operation unit 113, and the image sensor 102 via the bus 114. The CPU 112 realizes control and processing of each part by executing a program stored in the ROM 109.

The bus 114 is a system bus, and the bus 115 is an image data bus.
The display unit 111 includes a liquid crystal monitor and the like, and is controlled by the display control unit 110 to display various images, a user interface screen, and the like.
The operation unit 113 includes switches, buttons, and the like operated by the user, and is used for operations such as power on / off, shutter on / off, and the like.
The gyro sensor 116 detects the posture (horizontal position, vertical position, etc.) and movement direction of the camera, and notifies the CPU 112. By receiving a detection signal such as movement from the gyro sensor 116, the CPU 112 can also calculate the camera shake direction and the shake amount due to camera shake and the like.

The image processing unit 104 is composed of a circuit, a processor, a buffer memory, and the like that perform various image processing. Further, in the case of the present embodiment, the image processing unit 104 processes a plurality of processing areas set for one captured image by a motion vector detection unit provided corresponding to each, and performs motion vector detection and features. It has a function to perform point tracking. Here, when subject tracking or the like is performed based on feature points and motion vectors, if a large number of motion vectors can be detected with high accuracy and a large number of feature points can be tracked, the tracking accuracy of the subject can be improved. On the other hand, in real-time processing of moving images and the like, the number of motion vectors that can be detected by one motion vector detection unit is limited. Therefore, in the present embodiment, a plurality of processing areas are set for one captured image, and each processing area is processed by the motion vector detection unit corresponding to each, so that a large number of processing areas can be improved while improving the motion vector detection accuracy. It is feasible to track the feature points of the above with high accuracy. Therefore, the image processing unit 104 of the present embodiment includes two vector detection units, a subject detection unit 123 and a first vector detection unit 121 and a second vector detection unit 122. The subject detection unit 123 detects the feature amount and the movement amount of the face portion and the object portion of the subject from the captured image, and detects the main subject based on them. The first vector detection unit 121 and the second vector detection unit 122 have the same function and independently perform motion vector detection. In the case of the present embodiment, two processing regions, a first region and a second region, are set for one image, the first vector detection unit 121 sets the first region, and the second vector detection unit 122 sets the second processing region. Process the second area. That is, the image processing unit 104 can detect a large number of feature points and a large number of motion vectors with high accuracy by processing the first region and the second region in parallel, and based on these, the image processing unit 104 can detect a large number of feature points and a large number of motion vectors. By tracking the feature points, the tracking accuracy is improved. A detailed description of the setting of the processing area, the extraction of feature points, the detection of motion vectors, and the tracking of feature points in the image processing unit 104 of the present embodiment will be described later.

The data transfer unit 105 is composed of a plurality of DMACs (Direct Memory Access Controllers) that transfer data.
The image processing unit 104 and the data transfer unit 105 may have a hardware configuration or may be realized by a software configuration executed by a microcomputer or the like. When the image processing unit 104 and the data transfer unit 105 are configured by a microcomputer, the processing of the image processing unit 104 and the data transfer unit 105 is realized by executing a program stored in the ROM 109.

Hereinafter, the configuration of the first and second vector detection units 121 and 122 of the image processing unit 104 and the operation of the data transfer unit 105 will be outlined with reference to FIGS. 2 to 9. After that, a detailed configuration and operation according to the present embodiment will be described with reference to the respective drawings after FIG. Since the first and second vector detection units 121 and 122 have the same function, they are described below only as the vector detection unit unless it is necessary to distinguish them.

FIG. 2 is a diagram used for explaining the schematic configuration and operation of the vector detection unit (121, 122). The vector detection unit is controlled by the CPU 112, and also accesses the memory via the data transfer unit 105 to execute the motion vector detection process and the feature point tracking process. Therefore, in addition to the functional configurations related to motion vector detection and feature point tracking, FIG. 2 also shows the functional configurations of the CPU 112 and the functional configurations of the data transfer unit 105 and the memory related thereto. In FIG. 2, the main configurations related to motion vector detection and feature point tracking are a matching image generation unit 201, a feature point calculation unit 202, a matching processing unit 203, an accuracy determination unit 204, and a tracking destination determination unit 205. Further, in FIG. 2, the functional configuration included in the CPU 112 is the processing area setting unit 260, and the functional configurations included in the data transfer unit 105 are RDDMACa221, RDDMACb222, and WRDMAC231. RDDMACa221 and RDDMACb222 represent a DMAC function when reading data to the DRAM 107, and WRDMAC231 represents a DMAC function when writing data to the DRAM 107. The memory includes the DRAM 107 and the SRAM 206 (not shown in FIG. 1).

FIG. 3 shows an arrangement and feature points of each grid when the vector detection unit reads image data for each grid from the DRAM 107 via the data transfer unit 105 and extracts feature points, and a template matching area and a search area. It is a figure which shows the relationship.
At the time of motion vector detection by template matching, the data transfer unit 105 reads out image data from the DRAM 107 for each grid obtained by dividing the frame image into a plurality of frames, and transfers the image data to the vector detection unit. The size of each grid is preset, and the number of grids divided from the frame image is preset in the horizontal and vertical directions. The grid obtained by dividing the frame image into a plurality of frames includes a grid as an extraction frame area used for feature point extraction and template matching, and a grid as a non-extraction frame area used for template matching although feature point extraction is not performed. There is. In the example of FIG. 3, the white grid 302 in the figure is a grid as an extraction frame area where feature point extraction is performed. A peripheral grid 301 is arranged around a grid group consisting of each grid 302 from which feature point extraction is performed. The peripheral grid 301 is a grid as a non-extracted frame area used in template matching, although feature point extraction is not performed.

The vector detection unit calculates a feature value representing the size of the feature of the image in the grid 302 of the extraction frame area for each pixel, and acquires the pixel having the largest feature value as the feature point 303 of the corresponding grid 302. As a result, one feature point 303 is acquired for each grid 302. Further, the feature amount of the feature point 303 of the grid 302 corresponds to the feature value of the pixel made as the feature point 303. Then, the vector detection unit determines a rectangular area having a preset size as the template area 304 with the feature point 303 as the center. The vector detection unit stores the data of the feature points 303 and the template area 304 obtained for each of these grids 302 in the DRAM 107. The template area 304 stored in the DRAM 107 is used as the template area 304 of the original image (previous frame) in the template matching. Then, the vector detection unit performs a search using the template area 304 obtained from the grid 302 at the corresponding position in the previous frame with respect to the grid 302 of the reference image (current frame) in the template matching.

At the time of this search, the vector detection unit sets a rectangular area having a preset size larger than the template area 304 around the feature point 303 as the search area 305 in the grid 302 of the current frame (reference image). .. The vector detection unit uses the image of the template area 304 and scans the search area 305 in order to perform template matching. Since the scanning method for template matching is a known technique, detailed description thereof will be omitted. At the time of template matching, the vector detection unit performs a correlation calculation process for obtaining a correlation between the pixel value in the template area 304 and the pixel value in the rectangular area having the same size as the template area 304 in the search area 305. Then, the vector detection unit detects the position of the rectangular area having the highest correlation with the template area 304 of the grid 302 of the previous frame in the search area 305 of the grid 302 of the current frame as the movement destination of the template area 304. .. Further, the vector detection unit uses the position of the template area 304 in the grid 302 of the previous frame as a reference position, and the direction of the movement destination of the template area 304 in the search area 305 of the grid 302 of the current frame with respect to the reference position. Find the amount of movement. The vector detection unit detects the direction and the amount of movement thus obtained as vector values in the grid 302.

In FIG. 4, processing from grid division by each DMAC (221, 222, 231) function of the data transfer unit 105 shown in FIG. 2 to calculation of the tracking destination feature point based on the feature point and the motion vector in the vector detection unit. The flowchart of is shown. Hereinafter, the configuration of the vector detection unit and the data transfer operation of the data transfer unit 105 shown in FIG. 2 will be described in more detail with reference to the flowchart of FIG. The processing of this flowchart may be executed by a hardware configuration, may be realized by a software configuration based on a program executed by a microcomputer or the like, and a part may be realized by a hardware configuration and the rest may be realized by a software configuration. You may. The program executed by the microcomputer or the like may be stored in the ROM 109, for example, may be acquired from a recording medium such as an external memory (not shown), or may be acquired via a network (not shown) or the like. .. In the following description, steps S400 to S409 of each process are abbreviated as S400 to S409. It is assumed that these things are the same in other flowcharts described later.

First, in S400, the CPU 112 integrates the detection information from the gyro sensor 116 and the information from the subject detection unit 123. Then, as the processing area setting in the processing area setting unit 260, the CPU 112 sets a processing area in which the motion vector detection processing and the feature point tracking are independently performed by the first and second vector detection units 121 and 122, respectively. The details of the processing area setting method will be described in the items of the first embodiment and the second embodiment described later.

Next, in S401, the RDDMACa221 of the data transfer unit 105 reads out the input image data 241 of the current frame to be the motion vector detection target from the DRAM 107 via the bus 115. At this time, the RDDMACa221 of the data transfer unit 105 reads out image data for each grid unit of the grid 302 and the peripheral grid 301 shown in FIG. The input image data 241 is data after various image processings have already been performed by the image processing unit 104. Then, the RDDMACa221 outputs the input image data 241 read for each grid unit to the matching image generation unit 201 and the feature point calculation unit 202 of the vector detection unit. Further, the RDDMACa221 sends the grid coordinate information 252 indicating the coordinate position of the read grid to the tracking destination determination unit 205, which will be described later. After S401, the process proceeds to the processing of S402 performed by the matching image generation unit 201 and the processing of S404 performed by the feature point calculation unit 202.

Upon shifting to S402, the matching image generation unit 201 generates matching image data 242 used for template matching for motion vector detection, and outputs the matching image data 242 to the WRDMAC231 of the data transfer unit 105. Specifically, the matching image generation unit 201 is a bandpass filter circuit, and removes high-frequency components and low-frequency components of the image signal, which are unnecessary for the template matching process, to generate matching image data 242. After S402, the process proceeds to the process of S403 performed by WRDMAC231 of the data transfer unit 105.

Upon shifting to S403, the WRDMAC 231 writes the matching image data 242 sent from the matching image generation unit 201 to the DRAM 107 via the bus 115. The matching image data 242 is data generated from the current frame, and is image data referred to at the time of template matching. Further, the DRAM 107 also stores the matching image data 243 generated from the frame image immediately before on the time axis with respect to the current frame. The matching image data 243 is data generated in the previous frame and is original image data in template matching. After S403, the process proceeds to the process of S405 performed by the RDDMACb222 of the data transfer unit 105.

The processing of S404 performed by the feature point calculation unit 202 is performed in parallel with the processing of S402 and the subsequent S403 described above. In S404, the feature point calculation unit 202 calculates a new feature point for the current frame.

FIG. 5 shows a configuration example of the feature point calculation unit 202. As shown in FIG. 5, the feature point calculation unit 202 includes a feature filter unit 501, a feature evaluation unit 502, and a feature point determination unit 503.
The feature filter unit 501 is composed of a plurality of filters such as a bandpass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. The feature filter unit 501 removes unnecessary high-frequency components and low-frequency components from the image data by a band path filter, and then performs horizontal differential filter processing and vertical differential filter processing. Further, the feature filter unit 501 performs a smoothing filter process on the data subjected to the differential filter process in the horizontal direction and the vertical direction. The data subjected to the differential filter processing and the smoothing filter processing in each of the horizontal and vertical directions is sent to the feature evaluation unit 502.

From the image of the grid processed by the feature filter unit 501, the feature evaluation unit 502 includes pixels of the intersections of the two edges and pixels of the points of the curved portion where the curvature is maximized. The value of the pixel whose differential value is large in multiple directions is calculated as a feature value. If the intersection of the two edges or the point of the curved portion having the maximum curvature does not exist in the grid, the feature value is not calculated. Hereinafter, as an example of the method of detecting the feature points from the grid, the method of Sh and Tomasi will be described as an example. Since Shi and Tomasi is a known method, only an outline thereof will be described here.

The feature evaluation unit 502 creates an autocorrelation matrix H from the values of pixels that have been subjected to differential filter processing in the horizontal and vertical directions by the feature filter unit 501. The equation of the autocorrelation matrix H is expressed by equation (1).

In the equation (1), Ix is the value of the pixel after the horizontal differential filter is applied, Iy is the value of the pixel after the vertical differential filter is applied, and the autocorrelation matrix H convolves the Gaussian filter G. Is sought after. Then, the feature evaluation unit 502 obtains the feature value by the feature evaluation formula of Sh and Tomasi represented by the formula (2). The feature evaluation formula of Sh and Tomasi shown in the formula (2) indicates that the smaller eigenvalue of the eigenvalues λ1 and λ2 of the autocorrelation matrix H in the formula (1) is used as the feature value.

Shi and Tomasi = min (λ1, λ2) Equation (2)

The feature value data obtained for each grid by the feature evaluation unit 502 is sent to the feature point determination unit 503.
The feature point determination unit 503 determines the pixel having the largest value among the feature values calculated for each pixel by the feature evaluation unit 502 from the grid as the feature points of the corresponding grid. If the feature value is not calculated by the feature evaluation unit 502, the feature point determination unit 503 does not acquire the feature points of the grid. Then, the coordinate information of the feature points acquired in the grid is stored in the memory provided in the feature point determination unit 503. In the present embodiment, the coordinates of the feature points are represented by relative coordinates (PX, PY) when the coordinates (x, y) at the upper left end of the grid are set to the origin (0,0). The coordinates of the feature points may be expressed by coordinates representing pixel positions in the frame image in addition to the relative coordinates (PX, PY). Further, the memory included in the feature point determination unit 503 has a capacity capable of storing the coordinate information of the feature points determined in each grid of the previous frame and the coordinate information of the feature points determined in each grid of the current frame. have. Of the coordinate information of the feature points stored in the feature point determination unit 503, the coordinate information 251 of the feature points in the grid of the previous frame is output to the RDDMACb222 when the template matching process is started by the matching processing unit 203 described later. To. However, in the first template matching process, since the feature point coordinates of the tracking destination cannot be calculated from the previous frame, all the processes using the new feature point coordinates are performed.

The explanation is returned to the flowchart of FIG. After S404, the process proceeds to the process of S405 performed by the RDDMACb222 of the data transfer unit 105.
In S405, the RDDMACb222 reads the image data of the template area 304 and the search area 305 from the DRAM 107. Specifically, the RDDMACb222 reads out the image data 254 of the template area 304 centered on the feature point 303 from the matching image data 243 based on the coordinate information 251 calculated in the previous frame and the grid coordinate information 252. Further, the RDDMACb222 reads out the image data 253 of the search area 305 centered on the feature point 303 from the matching image data 242 based on the coordinate information 251 calculated in the previous frame and the grid coordinate information 252. Then, the RDDMACb222 outputs the image data 253 of the search area 305 and the image data 254 of the template area 304 to the matching processing unit 203. After S405, the process proceeds to the processing of S406 performed by the matching processing unit 203 of the vector detection unit.

In S406, the matching processing unit 203 uses the image data 253 in the search area and the image data 254 in the template area to perform a correlation calculation for calculating a correlation value for each pixel of the image, and calculates a vector value from the correlation value. do. Specifically, the matching processing unit 203 obtains the sum of the difference absolute values (Sum of Absolute Difference, hereinafter abbreviated as SAD) of each pixel as the correlation value. Equation (3) shows an arithmetic expression when the matching processing unit 203 obtains the correlation value S_SAD by the operation of SAD.

In the equation (3), f (i, j) represents a pixel value at the coordinates (i, j) in the template area, and g (i, j) is a rectangle to be calculated for the correlation value in the search area. Represents each pixel value in the area. The rectangular area for which the correlation value is calculated in the search area is an area having the same size as the template area, and is an area scanned in order from the upper left end portion in the search area during template matching. Hereinafter, the rectangular area for which the correlation value is calculated in the search area is referred to as a “correlation value calculation area”. The matching processing unit 203 calculates the absolute value of the difference between the pixel value f (i, j) in the template area and the pixel value g (i, j) in the correlation value calculation area, and obtains the total value to obtain the correlation. Find the value S_SAD. The pixel value used at this time is, for example, a luminance value. The smaller the value of this correlation value S_SAD, the smaller the difference in luminance value between the images in both regions, that is, the textures of the image in the template area and the image in the correlation value calculation area in the search area are similar. Represents that. Here, an example of obtaining the correlation value by SAD has been given, but the present invention is not limited to this, and other correlation values such as the sum of squared differences (SSD) and the normalized cross-correlation (NCC) may be used.

Then, the matching processing unit 203 obtains the coordinate position where the correlation value becomes the minimum value. The coordinate position where the correlation value becomes the minimum value is considered to be the position of the rectangular area having the highest correlation with the template area of the grid of the previous frame in the search area of the grid of the current frame. Further, the position of the rectangular area having the highest correlation with the template area in the search area represents the destination indicating which position the template area of the previous frame is in the current frame. The matching processing unit 203 uses the position of the template area in the grid of the previous frame as a reference position, and sets the direction and the amount of movement of the above-mentioned movement destination as a vector value in the grid with respect to the reference position. Then, the matching processing unit 203 outputs the vector value and the correlation value information 255 to the accuracy determination unit 204.

As the process of S407, the accuracy determination unit 204 determines whether or not the image of the template area of the grid of the previous frame or the search area of the grid of the current frame is an image suitable for motion vector detection. Specifically, in S407, the accuracy determination unit 204 obtains the maximum value, the minimum value, the average value, and the minimum value of the correlation value information 255 for each pixel calculated in S406. Then, the accuracy determination unit 204 performs a low contrast determination, a maximum value protrusion determination of the pixel value, and a repeat pattern determination based on the maximum value, the minimum value, the average value, and the minimum value of the correlation values.

6 (a) to 6 (d) show a low contrast determination based on the maximum value, a minimum value, an average value, and a minimum value of the correlation value, a maximum value protrusion determination of the pixel value, a repeat pattern determination, and a pixel value. The relationship between is shown as a graph. However, the smaller the correlation value, the higher the similarity between the images. Therefore, the maximum value of the pixel values in FIGS. 6 (a) to 6 (d) represents the minimum value in the correlation value, and the minimum value of the pixel values. Represents the maximum value in the correlation value, and the maximum value of the pixel value represents the minimum value in the correlation value.

In the low contrast determination, the accuracy determination unit 204 has a low contrast in the correlation value calculation area when the difference between the maximum value and the minimum value of the correlation value in the correlation value calculation area is smaller than the preset threshold value. Judge that there is. Further, the accuracy determination unit 204 determines how outstanding the minimum value of the correlation value in the correlation value calculation region is in the maximum value protrusion determination of the pixel value. For example, when the value obtained by dividing the difference between the maximum value and the average value of the pixel values and the difference between the maximum value and the minimum value of the pixel values is smaller than the preset threshold value, the accuracy determination unit 204 determines the correlation value. It is determined that the peak in the calculation area is low. On the other hand, when the value obtained by dividing the difference between the maximum value and the average value of the pixel values and the difference between the maximum value and the minimum value of the pixel values is larger than the threshold value, the accuracy determination unit 204 is high in the correlation value calculation area. Judged as a peak. Further, the accuracy determination unit 204 determines that the repetition pattern is a repetition pattern when the difference between the minimum value and the minimum value of the pixel values in the correlation value calculation area is smaller than the preset threshold value. do.

FIG. 6A gives an example in which the accuracy determination unit 204 has good low contrast determination, maximum pixel value protrusion determination, and repeated pattern determination. On the other hand, FIG. 6B shows an example in which low contrast is determined in the low contrast determination, and the maximum pixel value is shown in FIG. 6B as compared with the example in FIG. 6A. The difference between the value and the minimum value is small. FIG. 6 (c) shows an example in which a low peak is determined in the determination of the maximum value protrusion of the pixel value, and FIG. 6 (c) shows the maximum pixel value as compared with the example of FIG. 6 (a). The division value between the difference between the value and the average value and the difference between the maximum value and the minimum value of the pixel value is small. FIG. 6 (d) shows an example in which the repeat pattern is determined in the repeat pattern determination. Compared with the example of FIG. 6 (a), FIG. 6 (d) shows the maximum value and the maximum value of the pixel values. The difference is small.

The accuracy determination unit 204 performs the low contrast determination, the maximum value protrusion determination, and the repeat pattern determination as described above for each grid. Then, when the accuracy determination unit 204 determines that none of the low contrast, maximum value protrusion, and repetition pattern is applicable, the images in the template area and the search area are determined to be images suitable for motion vector detection. .. On the other hand, when the accuracy determination unit 204 determines that the contrast is low, the maximum value is projected, or the repetition pattern is used, the accuracy determination unit 204 determines that the images in the template area and the search area are not suitable for motion vector detection. ..

Then, in S408, the CPU 112 sends the determination information and the vector information 256 representing the determination results of the low contrast, low peak, and repetition pattern to the SRAM 206 for writing.
Next, as the process of S409, the tracking destination determination unit 205 performs a process of determining a tracking destination feature point to be used for the template matching process of the next frame, which will be described later.

FIG. 7 is a flowchart showing the flow of the tracking destination feature point determination process in the tracking destination determination unit 205. Hereinafter, an outline of the process of determining the tracking destination feature point will be described with reference to the flowchart of FIG. 7.
In S701, the tracking destination determination unit 205 calculates the coordinates of the tracking destination feature point of the next frame based on the vector information of the current frame input from the SRAM 206 and the coordinates of the tracking destination feature point of the current frame. That is, the tracking destination determination unit 205 uses the coordinates of the tracking destination feature point of the current frame as the vector start point, and sets the coordinates of the destination vector end point moved by the amount corresponding to the direction and size represented by the vector value of the vector information. , Determined as the coordinates of the tracking destination feature point of the next frame.

Next, in S702, the tracking destination determination unit 205 determines whether or not the determination information input from the SRAM 206 is an OK determination. That is, the tracking destination determination unit 205 determines whether or not the tracking destination feature point calculated in S701 is an effective tracking destination feature point that can be used for the template matching process of the next frame. As described above, the determination information input from the SRAM 206 is determined to be low contrast in the low contrast determination, low peak in the maximum pixel value protrusion determination, or a repeat pattern in the repeat pattern determination. If it is determined, the determination is NG. On the other hand, if none of them is applicable, the judgment is OK. Then, in S702, the tracking destination determination unit 205 transitions to S704 when the determination information is NG determination, and transitions to S703 when the determination information is OK determination.

When transitioning to S703, the tracking destination determination unit 205 determines whether or not the tracking destination feature point coordinates calculated in S701 are within the angle of view of the frame. Specifically, the tracking destination determination unit 205 determines that the template area centered on the tracking destination feature point coordinates is within the frame angle of view (OK determination) when the template area is contained inside the peripheral grid, and determines S705. On the other hand, if it does not fit, it is determined to be NG and the transition to S704 is performed.

When the transition to S704 occurs, the tracking destination determination unit 205 discards the tracking destination feature point coordinates corresponding to the positions determined to be NG in S702 or S703 and replaces them with new feature point coordinates. Specifically, the tracking destination determination unit 205 is a new frame of the current frame calculated at a position corresponding to the grid of the extraction frame area of the position where the discarded tracking destination feature point is first calculated as a new feature point. Performs the process of replacing with feature point coordinates. After this S704, the process of the tracking destination determination unit 205 transitions to S705.

Upon transition to S705, the tracking destination determination unit 205 determines whether or not all the tracking destination feature point coordinates corresponding to the number of grids in the extraction frame area have been calculated. Then, the tracking destination determination unit 205 ends the processing of the flowchart of FIG. 7 when it is determined that all the tracking destination feature point coordinates have been calculated, and on the other hand, when it is determined that all the tracking destination feature point coordinates have not been calculated, the processing is performed in S701. Return and perform the above-mentioned processing after S701.

The above is a rough flow from grid division in the flowchart of FIG. 4 to motion vector detection and tracking destination feature point determination. Although the flowchart of FIG. 4 shows the processing for one frame, the vector detection unit and the data transfer unit 105 perform the same grid division for each frame, motion vector detection, and tracking destination feature point determination. Perform processing.

Next, with reference to FIGS. 8 and 9 (a) and 9 (b), the outline of the feature point tracking and the feature point tracking when the processing area is set for the purpose of improving the detection accuracy of the motion vector. I will explain the problems that arise in.
FIG. 8 is a diagram showing an outline of the feature point tracking process, and shows the grid 1404 of the extraction frame area, the template area 1403, the search area 1406, and the peripheral grid 1405, which are the same as those of FIG. 3 described above. At the time of motion vector detection, as shown in FIG. 8, feature points 1401 are calculated in the grid 1404 of the extraction frame area, and template matching is performed by the template area 1403 to calculate the vector value 1407. Then, in the processing of the next frame, the feature point 1401 is set as the vector start point, the point separated by the direction and the magnitude according to the vector value 1407 is set as the tracking destination feature point 1402, and the template matching centering on the tracking destination feature point 1402 is performed. Will be done. After that, the same feature point tracking as described above is performed over a plurality of frames.

Here, in order to improve the detection accuracy of the motion vector, a plurality of processing areas are set for one image, and feature points are extracted and motion is extracted by a vector detection unit individually provided corresponding to each of the plurality of processing areas. It is assumed that the vector is detected and the feature points are tracked.
In FIG. 9A, one image is divided into two processing regions of the first and second regions 1502 and 1501, the first region 1502 is processed by the first vector detection unit, and the second region 1501 It is a figure which showed the extraction feature point and the tracking destination feature point at the time of processing in the 2nd vector detection unit. In this way, when processing is performed by the first and second vector detection units, the grid of the extraction frame area in which the feature point extraction is performed and the feature point extraction are provided in the first and second regions 1502 and 1501, respectively. Peripheral grids that are not implemented are set. Further, an overlapping region 1500 is provided in the first region 1502 and the second region 1501. Then, as shown in FIG. 9A, when the motion vector is, for example, a vector to the right, the tracking destination feature point moves to the right. However, when the tracking destination feature points 1510 to 1514 at the right end of the first region 1502 are traced, the region on the right side thereof becomes the second region 1501 outside the first region 1502. In this case, the feature point cannot be tracked by the first motion vector detection unit, and the feature point disappears, resulting in deterioration of the feature point tracking performance.

In FIG. 9B, similarly to FIG. 9A, two processing regions of the first and second regions 1502 and 1501 are set in one image, and as shown by, for example, arrow 1503 in the image. It shows an example in which the subject 1504 moving to the right is shown. That is, in the example of FIG. 9B, the subject 1504 moves from the first region 1502 to the second region 1501 across the boundary between the first region 1502 and the second region 1501. And. Further, it is assumed that each excellent feature point 1520 to 1527 capable of detecting the subject 1504 is extracted or tracked from the first region 1502 by the first vector detection unit. However, since each feature point 1520 to 1527 moves to the right as the subject 1504 moves, it is traced to the vicinity of the overlapping region 1500 and then disappears from the first region 1502. Even if there are excellent feature points that can detect the movement of the subject 1504 in this way, for example, when the subject 1504 straddles the boundary between the first region 1502 and the second region 1501, the feature points disappear. As a result, the feature point tracking performance will deteriorate.

Therefore, in the present embodiment, when the feature points disappear due to the subject moving across the boundary of the processing area, the vector detection unit corresponding to at least the processing area of the moving source of the subject extracts new feature points. It controls the setting of the time and makes it easy to set (supplement) a new tracking destination feature point.

<First Embodiment>
Hereinafter, when the feature point disappears due to the subject moving across the boundary of the processing area, the image of the first embodiment is subjected to the processing for facilitating the setting (supplement) of the new tracking destination feature point. The processing apparatus will be described with reference to FIGS. 10 to 13. In the above-mentioned examples of FIGS. 9 (a) and 9 (b), an example in which one image is divided into the left-right direction (horizontal direction) and the processing area is set is given, but in the following description, the image is moved in the vertical direction (vertical direction). An example of setting the processing area separately for each direction) will be given. The configuration and schematic processing of the image processing apparatus according to the first embodiment are the same as those described with reference to FIGS. 1 to 7 above.

FIG. 10 is a diagram showing an example of a case where a processing area is set so as to divide one image in the vertical direction in the image processing apparatus according to the first embodiment. In the image processing apparatus of the first embodiment, as shown in FIG. 10, the processing area setting unit 260 sets a processing area divided in the vertical direction with respect to the captured image 805 obtained by photographing. In the example of FIG. 10, in order to make it possible to detect a large number of motion vectors of the subject 804 and its surroundings, the first region 801 on the upper side and the lower side of the image region of the subject 804 and its surroundings in the captured image 805. An example in which two processing areas of the second area 802 are set is shown. Further, in FIG. 10, the subject 804 moves upward as shown by the arrow 803 and straddles the boundary between the first region 801 and the second region 802 (hereinafter, referred to as a processing region boundary). An example of moving to is shown.

11 (a) and 11 (b) are diagrams showing the setting of the processing area for the captured image and the arrangement example of the grid set in each processing area. The image processing apparatus of the first embodiment sets the first region 901 and the second region 902 for the captured image 900 as shown in FIG. 11 (a), and similarly, in FIG. 11 (b). As shown, the first region 911 and the second region 912 are set for the captured image 910. Further, the image processing apparatus sets a grid of the extraction frame area in which the feature point extraction is performed and a peripheral grid in which the feature point extraction is not performed in the first and second regions, and further, the first region. An overlapping region is provided at the processing region boundary between the region and the second region as described above. The example of FIG. 11A shows a state in which the subject 903 exists in the second region 902 and does not yet straddle the processing region boundary between the first region 901 and the second region 902. On the other hand, in the example of FIG. 11B, the subject 913 existing in the second region 912 in FIG. 11A straddles the processing region boundary between the first region 911 and the second region 912. , Represents a state of movement to the position of the subject 913.

As shown in FIG. 11B, when the subject 913 moves across the boundary of the processing area, the image processing apparatus of the present embodiment predicts the position where the subject appears after the movement, and the predicted appearance position is reached. The grid layout setting is controlled so that the grid of the extraction frame area is densely arranged.

FIG. 12 is a flowchart showing the flow of processing until the motion vector is detected by predicting the subject position after the movement and controlling the grid arrangement when the subject moves to the position straddling the boundary of the processing area. S1001 to S1005 in FIG. 12 are processing steps until the subject position after movement is predicted and the grid arrangement is controlled, and these processes are performed by the CPU 112 (including the processing area setting unit 260) in S400 of FIG. Will be done. The motion vector detection process of S1007 in FIG. 12 corresponds to the process after S401 in FIG.

In S1001, the CPU 112 shifts the processing mode of the image processing unit 104 to a mode in which the captured image is divided into a plurality of processing areas and each is processed by the vector detection unit. In the case of the present embodiment, the image processing unit 104 sets the first and second regions for the captured image, the first region is processed by the first vector detection unit 121, and the second region is the second region. Transition to the mode to be processed by the vector detection unit 122 of 2.

Next, in S1002, the processing area setting unit 260 of the CPU 112 calculates the coordinates representing the current position of the subject based on the detection information of the subject by the subject detection unit 123. As the coordinates of the subject, it is assumed that the coordinates of the tracking feature points on the subject position are used. If there are multiple tracking feature points, the coordinates of the tracking feature point closest to the processing area boundary are used. In addition, the coordinates of the tracking feature points on the subject position may be the position of the center of gravity of the subject, the intermediate position between the positions of the plurality of tracking feature points, the coordinates of the tracking feature point having the largest feature amount, and the like. FIG. 13 is a diagram showing how the subject shown in FIGS. 10, 11 (a), and 11 (b) is moving. FIG. 13 shows the subjects 1101 to 1103 in each frame in the middle of moving in order from the second region 1112 to the first region 1111 (from bottom to top). In the example of FIG. 13, the subject 1102 represents the subject of the current frame, the subject 1101 is the subject of the frame 1V (vertical synchronization) before the current frame, and the subject 1101 is the position predicted by the frame 1V after the current frame. Represents the subject.

Next, in S1003, the processing area setting unit 260 is located at the subject position based on the motion vectors calculated by the first vector detection unit 121 and the second vector detection unit 122 from the current frame and the frame 1V before, respectively. The movement amount 1104 of the feature point is calculated. When the motion vector is calculated from a plurality of tracking destination feature points, the movement amount of the subject may be obtained by any of the average value, the mode value, and the median value of each motion vector. In the present embodiment, the calculated movement amount 1104 of the subject is used as the predicted movement amount 1105 after 1V. In addition, the processing area setting unit 260 may store the subject movement amount of the past frame and calculate the predicted movement amount 1105 in consideration of the past movement amount.

Next, in S1004, the processing area setting unit 260 determines whether or not the subject crosses the processing area boundary in the next frame based on the subject position calculated in S1002 and the predicted movement amount of the subject calculated in S1003. In the case of the present embodiment, since the processing area is set in the vertical direction, the processing area setting unit 260 determines the processing area boundary straddle, the vertical coordinate Ly of the processing area boundary, the vertical coordinate Py of the subject, and the vertical direction. Judgment is made by the following equations (4) and (5) using the predicted movement amount My of.

Ly <(Py + My) equation (4)
Ly> (Py + My) Equation (5)

In the equation (4), when the subject moves from the bottom to the top and crosses the processing region boundary (when the subject 1103 moves from the second region 1112 to the first region 1111 as shown in FIG. 13). It is a judgment formula. On the other hand, the formula (5) is a determination formula when the subject moves from the top to the bottom and straddles the processing area (when the subject moves from the first area to the second area). Then, the processing area setting unit 260 transitions to S1005 when it is determined in S1004 that the subject crosses the processing area boundary, and transitions to S1006 when it is determined that the subject does not cross the processing area boundary.

After transitioning to S1005, as shown in FIG. 11B, the processing area setting unit 260 sets the first area 911 to which the subject 913 moves across the processing area boundary and the subject 913 before moving. The grid arrangement is controlled for the second region 912 of the movement source of the above. The processing area setting unit 260 reduces the size of the grid of the extraction frame area where the feature point extraction is performed and makes it dense with respect to the position where the subject 913 is predicted to move in the first area 911 of the movement destination. Set the grid layout so that the others are peripheral grids. Here, assuming that the predicted appearance position Oy that appears after the subject moves in the vertical direction (vertical direction) is Oy, the predicted appearance position Oy can be calculated by the following equations (6) and (7).

Oy = Ly-Py + My equation (6)
Oy = Py + My-Ly equation (7)

The formula (6) is a calculation formula when the subject moves from the bottom to the top and straddles the processing area boundary. On the other hand, the formula (7) is a calculation formula when the subject moves from the top to the bottom and straddles the processing area boundary.

As described above, the image processing apparatus of the present embodiment finely arranges the grid of the extraction frame region in which the feature point extraction is performed, based on the predicted appearance position of the subject. Here, when the feature point of the tracking target disappears in the processing area of the movement destination due to the subject straddling the boundary of the processing area, it is necessary to set (supplement) a new feature point as the feature point of the tracking target. It becomes. In the present embodiment, the feature points to be tracked are set in the processing area of the movement destination by setting the grid of the extraction frame area where the feature point extraction is performed at the predicted appearance position of the subject to be finely and densely arranged. The setting control for compensation is performed so that it is easy to be compensated (easily compensated). Then, in the grid of the extraction frame area finely arranged at the predicted appearance position of the subject, a new feature point is set (that is, a tracking target) by the replacement process using the new feature point described in S704 of FIG. 7 described above. The feature points of are compensated). As described above, in the present embodiment, the compensation of the disappeared feature points is a new feature obtained from the grid of the extraction frame area densely arranged corresponding to the subject area which is predicted to appear at the moving destination across the boundary of the processing area. It is performed by the process of replacing with the feature points in the point cloud.

On the other hand, the processing area setting unit 260 arranges the second area 912, which is the movement source of the subject 913, in a grid arrangement so as to avoid the subject 913 straddling the boundary of the processing area (excluding the feature point extraction process). For example, as shown in FIG. 13, for the area of the subject 913 that straddles the processing area boundary in the second area 912, a peripheral grid in which feature point extraction is not performed is set. As a result, in the grid corresponding to the position of the subject 913 leaving the second region 912, the replacement process by the new feature point in S704 is not performed. As described above, in the present embodiment, the processing area of the moving source of the subject is arranged in a grid so as to avoid the subject straddling the boundary of the processing area, so that the feature points of the tracking target are extracted from the grid of the moving source. Compensation is controlled so that it is not set.

The explanation is returned to the flowchart of FIG.
When it is determined in S1004 that the subject 913 does not cross the processing region boundary and the transition to S1006 occurs, the processing region setting unit 260 receives both the first and second regions 901 and 902 as in the example of FIG. 11A. , Make the grid arrangement evenly over the entire processing area. As a result, the replacement process using the new feature points described in S704 is performed by the feature points in the new feature point group obtained from the entire screen of the corresponding processing area.

After the above-mentioned S1005 or S1006, the CPU 112 shifts the processing of the image processing unit 104 to S1007, and executes the processing of S401 to S409 in FIG. That is, in S1007, the image processing unit 104 processes the first region set as described above by the first vector detection unit 121, and processes the second region by the second vector detection unit 122. do.

As described above, in the image processing apparatus of the first embodiment, when the subject moves across the boundary of the processing area, feature points are extracted in the area where the subject is predicted to appear in the processing area to which the subject is moved. Arrange the grids that carry out the above densely. Further, the image processing device arranges a grid in which the feature point extraction is not performed in the area where the subject moves, and removes the grid from which the feature point extraction is performed from the subject area. As a result, according to the image processing apparatus of the first embodiment, even if the excellent feature points detected in the area of the subject disappear at the boundary of the processing area, the feature points of the new tracking target in the processing area of the moving destination are used. The probability that tracking can be continued increases, and it becomes possible to improve the feature point tracking performance.

In the flowchart of FIG. 12 described above, an example is given in which only the frame at the moment when the subject crosses the processing area boundary is branched to S1005, but when the subject crosses the processing area boundary over a plurality of V periods, the processing area boundary is crossed. It may be controlled to branch to S1005 for a plurality of V periods. Further, the grid arrangement may be finely adjusted so that the subject straddles the boundary of the processing area.

In the first embodiment, an example of setting the processing area in the vertical direction has been described, but the same grid layout control can be performed when the processing area is set in the horizontal direction. Further, in the present embodiment, the direction for setting the processing area is set based on at least one of the position information of the photographing device (camera) for capturing the image and the movement information of the subject. For example, when the position information of the camera or the movement information of the subject changes, the direction for setting the processing area may be switched based on at least one of the position information of the camera and the movement information of the subject after the change. Then, after switching the direction for setting the processing area based on the position information of the camera and the movement information of the subject, the process may shift to the grid arrangement control processing shown in FIG.

<Second embodiment>
Hereinafter, the processing in the image processing apparatus according to the second embodiment will be described with reference to FIGS. 14 and 15. The configuration and main processing of the image processing apparatus according to the second embodiment are the same as those described with reference to FIGS. 1 to 7 above. In the second embodiment, the processing area setting performed in S400 of FIG. 4, the new feature point calculation of the current frame performed by the feature point calculation unit 202 in S404, and the tracking destination feature point coordinates in S704 of FIG. 7 are new features. The process of replacing with the coordinates of points will be described.

FIG. 14 shows a grid arrangement in which the processing areas of the first area 1201 and the second area 1202 are set for the captured image 1200 in the image processing device of the second embodiment, and the subject 1203 straddles the processing area boundary. It is a figure which showed the example of control. In the example of FIG. 14, similarly to the above, in the first region 1201 and the second region 1202, a grid on which feature point extraction is performed and a peripheral grid on which feature point extraction is not performed are set, and overlapping regions are formed. It will be provided. The example of FIG. 14 shows an example in which the subject 1203 enters the first region 1201 from the second region 1202 across the processing region boundary.

Also in the second embodiment, when the subject moves across the boundary of the processing area, the position where the subject appears in the destination processing area is predicted, and the feature point to be tracked with respect to the predicted appearance position is Compensation setting control is performed so that it is easy to set (easy to be compensated). Further, also in the second embodiment, compensation control is performed so that the processing area of the movement source is not set (compensated) as a feature point of the tracking target from the subject area. However, in the case of the second embodiment, unlike the first embodiment described above, the grid arrangement is not changed depending on whether or not the subject straddles the boundary of the processing area.

In the second embodiment, the features of the feature points calculated in the grid including the subject area in the processing area where the subject enters across the boundary of the processing area and the grid including the subject area in the processing area where the subject leaves. The compensation setting control is performed so as to adjust the amount. In the case of the example of FIG. 14, the grid 1204 in the first area 1201 is a grid including the subject 1203 that enters across the processing area boundary, and the grid 1205 in the second area 1202 straddles the processing area boundary. It is a grid including the subject 1203 that goes out. The image processing apparatus of the second embodiment adjusts the feature amount of the feature points calculated in the grid 1204 and the grid 1205 when the subject 1203 straddles the processing area boundary.

FIG. 15 is a flowchart showing a flow of adjustment control of feature points in the image processing apparatus of the second embodiment. Hereinafter, the adjustment control of the feature points required in each grid in the second embodiment will be described with reference to the flowchart of FIG. 15 and S404 of FIG.
Since S1301 to S1304 are substantially the same processing as S1001 to S1004 in FIG. 12, the description thereof will be omitted. In S1304, the CPU 112 transitions to S1305 when it is determined that the subject crosses the processing area boundary, and transitions to S1306 when it is determined that the subject does not cross the processing area boundary.

Moving to S1305, as shown in FIG. 14, the CPU 112 includes the grid 1204 including the subject 1203 in the first area 1201 (moving destination of the subject) and the subject 1203 in the second area 1202 (moving source). Stores information with grid 1205. The information to be stored may be grid numbers, grid coordinates, or both. The CPU 112 calculates the predicted appearance position Oy in which the subject moves in the vertical direction (vertical direction) and appears at the destination across the boundary of the processing area by the above-mentioned equation (6) or equation (7), and the predicted appearance position is calculated. Stores information on the grid having the coordinates corresponding to and the surrounding grid. Further, the CPU 112 also stores information on the grid having the coordinates corresponding to the position of the subject straddling the boundary of the processing area and the grid around the processing area of the moving source of the subject.

After S1305, the CPU 112 shifts the processing of the image processing unit 104 to S1306, and executes the processing of S401 to S409 in FIG. That is, in S1307, the image processing unit 104 processes the first region by the first vector detection unit 121 and the second region by the second vector detection unit 122. In the case of the second embodiment, the processes other than S404 and S409 are the same as the processes described in the first embodiment, and therefore S404 and S409 will be described here.

In the second embodiment, among the processes executed in S404, up to the feature point calculation process of each grid by the feature point calculation unit 202 is the same as described in the first embodiment. Further, the timing at which the feature point coordinate information is output to the RDDMACb222 in the matching processing unit 203 is the same as in the example of the first embodiment.

In the case of the second embodiment, in S404, the CPU 112 stores the feature amount Pc of the new feature point in the SRAM 206 together with the coordinates (PX, PY) of the new feature point calculated by the feature point calculation unit 202. Therefore, in the case of the second embodiment, the SRAM 206 has a capacity capable of storing the feature points of the previous frame and the feature points of the current frame.

Then, the CPU 112 adjusts the feature amount Pc of the new feature point to generate the feature amount Pc'based on the grid information stored for the grid including the subject in S1305. Explaining with reference to the example of FIG. 14, the CPU 112 adjusts the grid 1204 including the subject 1203 in the first area 1201 of the movement destination straddling the processing area boundary by the calculation formula of the following formula (8). The later feature amount Pc'is obtained.

Pc'= Pc × Gu formula (8)

In the formula (8), Gu is an adjustment value, which is a gain value set in advance for increasing the feature amount. In the case of this embodiment, Gu is set to a value of Gu> 1.0. That is, for the grid including the moving destination subject that straddles the boundary of the processing area, the adjustment processing that preferentially increases the feature amount of the feature point is performed. The adjusted feature amount Pc'in the processing area of the moving destination may be obtained by adding Gu as an offset value.

On the other hand, the CPU 112 obtains the adjusted feature amount Pc'by the calculation formula of the following formula (9) for the grid 1205 in which the subject 1203 is included in the second area 1202 of the movement source when crossing the processing area boundary. Ask.

Pc'= Pc × Gl equation (9)

In the formula (9), Gl is an adjustment value, which is a gain value set in advance for reducing the feature amount. In the case of this embodiment, Gl is set to a value of Gl <1.0. That is, for the grid including the moving source subject when straddling the processing area boundary, adjustment processing is performed so as to reduce the feature amount. The adjusted feature amount Pc'in the processing area of the movement source may be obtained by subtracting Gl as an offset value.

Further, in the second embodiment, when the transition to S409 occurs, the tracking destination determination unit 205 performs a process of determining the tracking destination feature point. In the case of the second embodiment, among the processes of the flowchart of FIG. 7 executed by the tracking destination determination unit 205, the process different from the first embodiment is only S704, and therefore the process of S704 will be described below.

The tracking destination determination unit 205 of the second embodiment discards the tracking destination feature point coordinates corresponding to the positions determined to be NG in S702 or S703 in S704 and replaces them with new feature point coordinates. Specifically, the tracking destination determination unit 205 performs a process of replacing with the coordinates of the current frame of the new feature point having the largest feature amount among the new feature points calculated in S404 and stored in the SRAM 206. Therefore, when the adjustment process for preferentially increasing the feature amount of the feature point is performed on the grid including the moving destination subject that straddles the boundary of the processing area, the feature point for which the feature amount is greatly adjusted is a new feature point. The process of replacing with the coordinates of the current frame of is easy to be performed. That is, the feature points of the grid including the subject of the movement destination straddling the boundary of the processing area are easily selected as the tracking destination feature points. On the other hand, when adjustment processing is performed to reduce the feature amount of the grid including the moving source subject when crossing the processing area boundary, the feature point adjusted to have the feature amount smaller is the current frame of the new feature point. The process of replacing with the coordinates of is not performed. That is, the feature points of the grid including the moving source subject when crossing the processing area boundary are not selected as the tracking destination feature points.

As described above, in the second embodiment, it is predicted whether or not the subject moves and crosses the processing area boundary, and if it straddles, the subject is included in the processing area of the destination after straddling. The feature points to be tracked are preferentially supplemented for the grid to be tracked. Further, in the second embodiment, in the processing area of the moving source, the feature points of the tracking target are not supplemented to the grid including the subject. As a result, according to the second embodiment, the probability that the excellent feature points detected in the subject can be tracked even if they disappear at the boundary of the processing area is increased, and the feature point tracking performance can be improved. In the second embodiment, the example of setting the processing area in the vertical direction (vertical direction) has been described, but when the processing area is set in the horizontal direction (horizontal direction), control is performed by the same concept. Just do it.

Further, in the first and second embodiments described above, an example in which two motion vector detection units are provided has been described, but the present invention can also be applied when three or more motion vector detection units are provided. be. For example, in the case of three, three processing areas are set in the vertical direction or the horizontal direction with respect to the captured image. Then, the same effect as that of the above-described embodiment can be obtained by performing the same grid arrangement control and adjustment of the feature amount as described above according to whether or not the subject moves across the boundary of the adjacent processing area. can. Further, when setting a plurality of processing areas for one captured image, the processing areas may be set not only in the vertical direction or the horizontal direction but also in the vertical and horizontal directions. In this case, the number of motion vector detection units is also provided according to the processing areas set in the vertical and horizontal directions.
Further, the processes described in the first and second embodiments described above may be performed not only individually but also in combination.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
Further, in the present embodiment, a digital camera is mentioned as an example of an image processing device, but various devices such as a tablet terminal, a smartphone, an electronic game machine, a drive recorder, a navigation device, and a personal computer capable of processing captured images are used. May be good. In this case, those arrangements acquire an image taken by a camera function mounted on the device itself, or an image taken by a digital camera or the like, and the image data is obtained from each of the above-described embodiments. Execute the process. Further, the processing according to the embodiment in these devices may be realized by executing a computer program on an internal microcomputer or the like. The computer program for realizing the processing according to the embodiment is provided via a recording medium, various networks, or a communication line.

The above-mentioned embodiments are merely examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

104: Image processing unit, 105: Data transfer unit, 106: Memory control unit, 107: DRAM (memory), 108: Non-volatile memory control unit, 109: ROM (nonvolatile memory), 112: CPU, 121: First Vector detection unit 113, 122: second vector detection unit, 123: subject detection unit

Claims (13)

Subject detection means that detects the subject from the captured image,
A setting means for setting a plurality of processing areas for the image, and
Feature points are extracted for each extraction frame area that is provided corresponding to each of the processing areas and the corresponding processing areas are divided into a plurality of areas, motion vector detection is performed using the feature points, and the feature points and motion vectors are detected. A processing means for tracking the feature points based on
When the subject moves across the boundary of the processing area, the control means for controlling the setting when the processing means corresponding to at least the moving destination processing area of the moving subject extracts a new feature point, and the control means.
An image processing device characterized by having. The processing means is
An extraction means for extracting feature points for each extraction frame area in which the corresponding processing areas are divided into a plurality of areas.
One of two adjacent frames on the time axis is used as the original image, and the other is used as the reference image. The pixel value in the rectangular area of the reference image and the rectangular area of the original image centered on the feature point. A vector detection means that detects a motion vector by performing a correlation operation with the pixel value of
It has a tracking destination determining means for calculating a feature point to be tracked in an image of the next frame based on the motion vector and the feature point.
The image processing apparatus according to claim 1, wherein the control means controls settings related to extraction of feature points in the extraction means. The control means is
When the subject moves across the boundary of the processing area, the subject appears in the processing area of the movement destination across the boundary based on the position of the subject in the processing area of the movement source and the motion vector. Predict the position,
The first or second aspect of the present invention, wherein the processing means corresponding to the processing area of the moving destination is controlled to set the extraction frame area to be densely arranged according to the predicted position. Image processing device. The control means is
Based on the position of the subject in the processing area of the movement source when the subject moves across the boundary of the processing area and the motion vector, the position where the subject appears at the movement destination across the boundary is predicted. death,
Claims 1 to 3 are characterized in that the setting is controlled so as to adjust the feature amount of the feature point of the extraction frame region according to the predicted position with respect to the processing means corresponding to the processing region of the movement destination. The image processing apparatus according to any one of the above items. The control means is characterized in that the processing means corresponding to the processing area of the moving destination is made to make the adjustment so as to increase the feature amount of the feature point of the extraction frame area according to the predicted position. The image processing apparatus according to claim 4. The control means according to claim 1, wherein when the subject moves across the boundary of the processing area, the control means also controls the setting of the processing means corresponding to the processing area of the moving source of the moving subject. 5. The image processing apparatus according to any one of 5. The image processing according to claim 6, wherein the control means controls the processing means corresponding to the processing area of the movement source to set the extraction frame area to be removed from the position of the subject. Device. The control means controls the setting of the processing means corresponding to the processing area of the moving source so as to adjust the feature amount of the feature points extracted in the extraction frame area according to the position of the subject. The image processing apparatus according to claim 6 or 7. The control means causes the processing means corresponding to the processing area of the moving source to make the adjustment so as to reduce the feature amount of the feature points extracted in the extraction frame area according to the position of the subject. The image processing apparatus according to claim 8. The processing means corresponding to the processing area to which the subject is moved is new based on the control of the setting by the control means when the feature point disappears when the subject moves across the boundary of the processing area. The image processing apparatus according to any one of claims 1 to 9, wherein the feature points extracted from the above are supplemented. The setting means according to any one of claims 1 to 10, wherein the processing area is set based on at least one of the position of the photographing apparatus for capturing the image and the movement of the subject. Image processing device. An image processing method executed by an image processing device.
The subject detection process that detects the subject from the captured image,
A setting process for setting a plurality of processing areas for the image, and
Feature points are extracted for each extraction frame area that is provided corresponding to each of the processing areas and the corresponding processing areas are divided into a plurality of areas, motion vector detection is performed using the feature points, and the feature points and motion vectors are detected. A processing process for tracking the feature points based on
When the subject moves across the boundary of the processing area, a control step for controlling the setting when extracting a new feature point in the processing step corresponding to at least the processing area of the moving subject and the control step.
An image processing method characterized by having. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 11.

Images