JP6068896B2 - Image processing apparatus and program - Google Patents

Description

  The present invention relates to an image processing apparatus and a program.

  Conventionally, various techniques for detecting an edge of an image have been proposed for image segmentation.

Wei-Ying Ma, et al. “EdgeFlow: A Technique for Boundary Detection and Image Segmentation” IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL9, NO.8, AUGUST 2000

  However, in the conventional technology, for example, the edge may be detected from a part that the user does not recognize as an edge, or the edge may be missing in a part that the user recognizes as an edge, and the edge detection accuracy is still improved. There was room for.

An image processing apparatus as an example of the present invention includes an acquisition unit that acquires an input image, a vector generation unit, an edge detection unit, a setting unit, and a division unit. The vector generation unit obtains, in each direction, a random field indicating the direction of image change and an energy field indicating the intensity of image change for the target pixel of the input image. The vector generation unit generates an edge flow vector of the target pixel using the random field and the energy field. The edge detection unit combines edge flow vectors generated at each position of the input image, and detects edge pixels of the input image from positions where the combined edge flow vectors face each other in the image. The setting unit sets a first range and a second range having a larger probability field than the first range in the circumferential direction of the target pixel. The dividing unit divides the second range into a plurality of regions. The vector generation unit generates an edge flow vector in each of the plurality of regions.
An image processing apparatus as an example of the present invention includes an acquisition unit that acquires an input image, a vector generation unit, an edge detection unit, and a determination unit. The vector generation unit obtains, in each direction, a random field indicating the direction of image change and an energy field indicating the intensity of image change for the target pixel of the input image. The vector generation unit generates an edge flow vector of the target pixel using the random field and the energy field. The edge detection unit combines edge flow vectors generated at each position of the input image, and detects edge pixels of the input image from positions where the combined edge flow vectors face each other in the image. The determination unit determines a pixel at a first distance from the target pixel and a pixel at a second distance different from the first distance from the target pixel. The vector generation unit generates a first edge flow vector obtained using a pixel at a first distance from the target pixel and a second edge flow vector obtained using a pixel at a second distance from the target pixel. To do.

  According to the present invention, the edge detection accuracy can be further improved.

The figure which shows the structural example of the image processing apparatus in one Embodiment. (A): a diagram showing an example of an input image, (b): a diagram showing an example of region information, (c): a diagram showing an example of an output image The flowchart which shows the example of the area division | segmentation process in one real form (A), (b): The figure which shows the integration process example of a partial area | region Flow chart showing an example of a subroutine for edge detection processing Illustration of random field and energy field Diagram showing an example of edge flow vector generation (A)-(d): Outline diagram of edge flow vector synthesis processing (A)-(c): Explanatory drawing of composition processing of edge flow vector (A): A diagram showing how to obtain the vertical edge V (x, y), (b): a diagram showing how to obtain the horizontal edge H (x, y). (A), (b): The figure which shows the example of the interpolation of an edge pixel (A), (b): Explanatory drawing of an example of zero clip processing in characteristic processing 1 The figure which shows the production | generation example of the edge flow vector in the 1st process of the characteristic process 2 The figure which shows the operation example in a 1st process The figure which shows the production | generation example of the edge flow vector in the 2nd process of the characteristic process 2 The figure which shows the operation example in a 2nd process (A), (b): explanatory diagram of interpolation processing in characteristic processing 3 (A), (b): The figure which shows the 1st example which calculates | requires the extension direction of an edge pixel from the end point of an edge pixel (A)-(e): The figure which shows the example of a template (A), (b): The figure which shows the 2nd example which calculates | requires the extension direction of an edge pixel from the end point of an edge pixel The figure which shows the example of a search of the connection destination of the interpolation edge in the characteristic process 4 (A), (b): Explanatory drawing of processing example in characteristic processing 5 The figure which shows an example of image processing (deletion of attention area) The figure which shows an example (movement of attention area) of image processing The figure which shows an example of image processing (filter processing to an attention area) Diagram showing an example of tag information assignment The figure which shows an example of the extraction process of recognition object The figure which shows the search example of the corresponding point at the time of a stereo image decompression | restoration The figure which shows the example of the similarity determination of an image Diagram showing an example of image compression processing Diagram showing an example of motion vector calculation The figure which shows the structural example of the imaging device in other embodiment.

<Example of Configuration of Image Processing Apparatus in One Embodiment>
FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to an embodiment. In one embodiment, as an example of an image processing apparatus, a computer that performs region division processing of an input image by edge detection processing using an edge flow vector will be described.

  The image processing apparatus 11 includes an input / output I / F 12, an image storage unit 13, a band pass processing unit 14, a vector generation unit 15, an edge detection unit 16, an integration processing unit 17, and a region information storage unit 18. , An image processing unit 19, a CPU 20, and a program storage unit 21.

  An input device 22 (keyboard, pointing device, etc.), an output device 23 (monitor, printer, etc.), and a data reading unit 24 are connected to the input / output I / F 12 of the image processing apparatus 11. The data reading unit 24 is used when reading input image data or a program to be subjected to area division processing from the outside. The data reading unit 24 communicates with, for example, a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device in accordance with a known communication standard. It is a communication device (USB interface, LAN module, wireless LAN module, etc.).

  The input / output I / F 12 receives data from the input device 22 and the data reading unit 24 and outputs image data to the output device 23.

  The image storage unit 13 is a storage medium that stores input image data. FIG. 2A shows an example of the input image. The input image data of the image storage unit 13 is output to the band pass processing unit 14 and the image processing unit 19 under the control of the CPU 20.

  The bandpass processing unit 14 performs bandpass filter processing on the input image. The output of the bandpass processing unit 14 is connected to the vector generation unit 15.

  The vector generation unit 15 generates an edge flow vector of the target pixel of the input image. The output of the vector generation unit 15 is connected to the edge detection unit 16.

  The edge detection unit 16 combines edge flow vectors generated at each position of the input image, and detects edge pixels of the input image from positions where the combined edge flow vectors face each other in the image. Then, the edge detection unit 16 extracts a closed space (partial region) surrounded by edge pixels from the input image. Note that the output of the edge detection unit 16 is connected to the integration processing unit 17.

  The integration processing unit 17 integrates adjacent regions having similar image characteristics among a plurality of partial regions extracted from the input image. Then, the integration processing unit 17 outputs an edge image (region information) indicating the region division result of the input image. FIG. 2B is a diagram illustrating an example of region information corresponding to FIG.

  The area information storage unit 18 is a storage medium that stores the area information. The area information stored in the area information storage unit 18 is output to the image processing unit 19 under the control of the CPU 20.

  The image processing unit 19 generates an output image by superimposing a boundary line based on the region information on the input image. In addition, the image processing unit 19 uses the region information to perform various types of image processing for each partial region of the input image (for example, an image with region-specific blur processing, region-specific tag information addition, image division, subject extraction). Synthesis etc.) can also be applied. FIG. 2C is a diagram illustrating an example of an output image corresponding to FIGS. 2A and 2B.

  The CPU 20 is a processor that comprehensively controls the operation of the image processing apparatus 11 by executing a program.

  The program storage unit 21 stores a program executed by the CPU 20 and various data necessary for executing the program. For example, the program storage unit 21 is a storage medium such as a hard disk or a nonvolatile semiconductor memory.

  Here, the functional blocks in the image processing apparatus 11 shown in FIG. 1 can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and realized by a program loaded in the memory in terms of software. For example, the bandpass processing unit 14, the vector generation unit 15, the edge detection unit 16, the integration processing unit 17, and the image processing unit 19 may be an ASIC (Application Specific Integrated Circuit) or a program module processed by the CPU 20. It may be. The image storage unit 13 and the area information storage unit 18 may be a plurality of storage areas secured on the same memory.

<Description of area division processing>
FIG. 3 is a flowchart illustrating an example of region division processing in one embodiment. The process of the flowchart of FIG. 3 is started when the CPU 20 executes a program when an instruction from the user is received.

  Step # 101: The image processing apparatus 11 acquires an input image via the data reading unit 24. The input image data is recorded in the image storage unit 13 under the control of the CPU 20.

  Step # 102: The band-pass processing unit 14 removes noise components from the input image using a band-pass filter. For example, the # 102 band pass processing unit 14 performs low pass filter processing on the input image.

  Step # 103: The vector generation unit 15 and the edge detection unit 16 execute an edge detection process of the input image. The operation of edge detection processing in one embodiment will be described later.

  Step # 104: The integration processing unit 17 executes integration processing of partial areas. Then, the integration processing unit 17 outputs area information indicating the result of area division of the input image.

  Here, the integration process in # 104 is performed using a known area integration algorithm. For example, the integration processing unit 17 calculates the similarity between a certain partial area and its surrounding partial areas for a plurality of partial areas extracted from the input image. This similarity is calculated using, for example, features such as luminance and color difference of the image. Then, the integration processing unit 17 integrates the calculated adjacent areas with high similarity into one area. By repeating such integration processing, adjacent regions having similar image characteristics are grouped into one of the partial regions subdivided by the edge detection processing, so that a region division result close to the user's sense can be obtained.

  FIG. 4 is a diagram illustrating an example of partial area integration processing. FIG. 4A shows a state in which edge detection processing is performed on an image obtained by photographing rose flowers. In FIG. 4A, each rose petal is extracted as an independent partial region. On the other hand, FIG. 4B shows a state where partial region integration processing has been performed on the image of FIG. In FIG. 4B, partial regions of rose petals having similar colors are integrated, and the entire rose is extracted as one region.

  Step # 105: The image processing unit 19 outputs the result of the area division processing.

  For example, the image processing unit 19 generates an output image by superimposing a boundary line based on the region information on the input image. Then, the CPU 20 displays the output image on the output device 23. Further, the image processing unit 19 may selectively perform image processing for each partial region of the input image using the region information. An example of image processing in # 105 will be described later. Above, description of the flowchart of FIG. 3 is complete | finished.

<Description of edge detection processing>
Next, an example of a subroutine of edge detection processing (# 103 in FIG. 3) will be described.

  In the following description, for convenience of understanding, an outline of general edge detection processing using edge flow vectors will be described first with reference to the flowchart of FIG. Then, the characteristic process in one embodiment is explained in full detail.

  Step # 201: The vector generation unit 15 specifies a pixel of interest for which an edge flow vector is obtained from the input image. As an example, the vector generation unit 15 in # 201 sequentially designates all pixels as target pixels in order from left to right one line at a time starting from the upper left corner of the input image.

  Step # 202: The vector generation unit 15 obtains a random field indicating the direction of image change and an energy field indicating the intensity of image change in each direction of the target pixel (# 201).

  FIG. 6 is an explanatory diagram of a random field and an energy field. The vector generation unit 15 samples a plurality of peripheral pixels (sample points) at equal intervals on the circumference of a certain distance from the target pixel. In the following example, the distance from the target pixel s (x, y) to the peripheral pixel is defined as r. Further, eight peripheral pixels are set at 45 ° intervals with the target pixel s (x, y) as the center. The angle of the peripheral pixel located on the right side of the target pixel s (x, y) is defined as θ0, and the angles of the peripheral pixels counterclockwise from θ0 are defined as θ1 to θ7.

  At this time, a certain peripheral pixel F (s, r, θi) can be expressed as a set of an energy field E (s, θi) and a random field P (s, r, θi) (F (s, r, θi)). = (P (s, r, θi), E (s, θi))). However, i is an integer of 0-7.

  The energy field E (s, θi) indicates the strength of change in the image (the magnitude of the edge energy). It should be noted that the energy fields E of peripheral pixels at positions rotated by 180 ° about the target pixel are equal (that is, E (s, θ) = E (s, θ + π)).

  The random field P (s, r, θi) is a value of the probability of determining the energy direction (θ or θ + π) from the pixels separated by the distance r. Note that the value of the random field P is normalized between neighboring pixels at positions rotated by 180 ° around the pixel of interest, and P (s, r, θ) + P (s, r, θ + π) = 1. is there.

  As an example, the vector generation unit 15 in # 202 may obtain a combination of the energy field E (s, θi) and the random field P (s, r, θi) in each peripheral pixel by the following procedure. Note that when there are eight peripheral pixels, eight energy fields E (s, θi) and eight random fields P (s, r, θi) are obtained.

  Here, the pixel value at the coordinates (x, y) is defined as I (x, y), and the Gaussian at the coordinates (x, y) is defined as G (x, y). Further, a first-order partial differential of G (x, y) in the x direction is defined as GD (x, y) (GD (x, y) = dG (x, y) / dx). A function obtained by rotating the derivative GD (x, y) by θ is defined as GD (x, y, θ) = GD (x ′, y ′). However, x ′ = x cos θ + y sin θ, and y ′ = − x sin θ + y cos θ.

At this time, the value of the energy field E (s, θi) can be obtained by Expression (1).
E (s, θi) = | I (x, y) * GD (x, y, θi) | (1)
Further, when Err (s, r, θi) = | I (x + rcos (θi), y + rsin (θi)) * G (x, y) −I (x, y) * G (x, y) | The value of the field P (s, r, θi) can be obtained by equation (2).
P (s, r, θi) = Err (s, r, θi) / (Err (s, r, θi) + Err (s, r, θi + π)) (2)
Step # 203: The vector generation unit 15 generates an edge flow vector of the target pixel using the random field and the energy field.

  FIG. 7 is a diagram illustrating an example of generation of edge flow vectors. First, the vector generation unit 15 obtains each sum of random fields within a range of a predetermined angle (for example, 180 °) while changing surrounding pixels to be sampled. Then, the vector generation unit 15 obtains a range in which the total sum of the random fields is the largest (direction Θ (s) in which the image changes greatly). Further, the vector generation unit 15 obtains an edge flow vector from the value of the energy field in the direction Θ (s) in which the image changes greatly.

As an example, the vector generation unit 15 in # 203 may obtain the direction Θ (s) in which the image change is large using Expression (3).
Θ (s) = argmax_ {θ} Σ_ {A} P (s, r, θ ′)
However, A = {θ ′ | θ ≦ θ ′ <θ + π} (3)
In addition, the vector generation unit 15 in # 203 may obtain the edge flow vector FL by Expression (4).
FL = Σ_ {B} E (s, θ ′) exp (jθ ′)
However, B = {θ ′ | Θ (s) ≦ θ ′ <Θ (s) + π} (4)
Step # 204: The vector generation unit 15 determines whether or not the generation of the edge flow vector is completed for all the target pixels. 8A is an example of an input image, and FIG. 8B is a diagram showing an initial state of the edge flow vector in FIG. 8A. In addition, FIG. 8 quotes FIG. 4 of a nonpatent literature 1. FIG.

  If the above requirement is satisfied (YES side), the process proceeds to # 205. On the other hand, if the above requirement is not satisfied (NO side), the vector generating unit 15 returns to # 201, designates the next pixel as the target pixel, and repeats the above operation.

  Step # 205: The edge detection unit 16 synthesizes the edge flow vector generated at each position of the input image by propagating it in the direction of each vector. Then, the edge detection unit 16 repeats the above synthesis process until there is no edge flow vector to be synthesized and it is settled. FIG. 8C is a diagram illustrating a state in which the edge flow vectors are synthesized and settled from FIG. 8B.

  Here, the synthesis process of the edge flow vector will be described with reference to FIG. FIG. 9A shows the initial state of the edge flow vector in the vicinity of the edge extending in the vertical direction of the image. In the synthesis process of # 205, the edge flow vector is propagated in the direction of each vector. At this time, the edge flow vector does not move at a position where the vectors face each other. In the first synthesis process, the first edge flow vector located in the vicinity of the edge does not move, and the second edge flow vector and the third edge flow vector move in the direction of the edge. Then, the first edge flow vector and the second edge flow vector are synthesized (see FIG. 9B). Thereafter, in the second synthesis process, the first edge flow vector after synthesis does not move, and the third edge flow vector moves in the direction of the edge. Then, the first edge flow vector and the third edge flow vector are synthesized (see FIG. 9C). By such a synthesis process, power is concentrated on a vector near the edge.

  Step # 206: The edge detection unit 16 detects edge pixels of the input image using the combined edge flow vector. For example, the edge detection unit 16 in # 206 may detect edge pixels by the following processes (A1) to (A3).

  (A1) The edge detection unit 16 obtains edge sizes in the vertical direction and the horizontal direction, respectively, for positions where the combined edge flow vectors face each other.

FIG. 10A is a diagram illustrating how to obtain the vertical edge V (x, y), and FIG. 10B is a diagram illustrating how to obtain the horizontal edge H (x, y). is there. For example, when the horizontal component of the edge flow vector FL (x, y) is defined as a (x, y) and the vertical component is defined as b (x, y), the edge detection unit 16 uses the expression (5) to determine the vertical direction. The edge V (x, y) may be obtained, and the horizontal edge H (x, y) may be obtained from the equation (6).
V (x, y) = | b (x, y) −b (x, y + 1) | (5)
H (x, y) = | a (x, y) −a (x + 1, y) | (6)
(A2) The edge detection unit 16 obtains the edge strength at each position using the V (x, y) and H (x, y). For example, the edge detection unit 16 may obtain the edge strength (edge Norm (x, y)) from the Euclidean norm of V (x, y) and H (x, y) by Expression (7).
edgeNorm (x, y) = √ (V (x, y) ^ 2 + H (x, y) ^ 2) (7)
(A3) The edge detection unit 16 compares the value of the edge strength (edgeNorm (x, y)) at each position with the threshold edgeTH1. Then, the edge detection unit 16 detects, as an edge pixel, a pixel whose edge intensity value exceeds the threshold edgeTH1 (edgeNorm (x, y)> edgeTH1).

  The edge detection unit 16 generates a flag edgeBIN (x, y) indicating whether each pixel is an edge pixel. The edge detection unit 16 sets the value of edgeBIN (x, y) to “1” at the position of the edge pixel, and sets the value of edgeBIN (x, y) to “0” at the position other than the edge pixel. The edge detection unit 16 can obtain an edge image in which the position of the edge pixel is binarized by referring to the value of edgeBIN (x, y). FIG. 8D shows an example in which an edge image is superimposed on the input image (FIG. 8A).

  Step # 207: The edge detection unit 16 performs known thinning processing and spike noise removal processing on the edge image, respectively. Note that the process of # 207 may be omitted as appropriate.

  Step # 208: The edge detection unit 16 connects edge pixels of the edge image, and extracts a closed space (partial region) surrounded by the edge pixels from the input image. At this time, the edge detection unit 16 interpolates edges that do not form a closed space as necessary so as to be connected to neighboring edge pixels. Here, FIG. 11A is a diagram illustrating an example before interpolation of edge pixels, and FIG. 11B is a diagram illustrating an example after interpolation of edge pixels. The portion where the edge is missing in FIG. 11A (indicated by a broken line) is interpolated in FIG. 11B.

  Then, after the process of # 208, the CPU 20 returns to the process of # 104 in FIG. Above, description of the flowchart of FIG. 5 is complete | finished. Next, the characteristic processing in one embodiment will be described in detail. It is not always necessary to perform all the characteristic processes described later, and one or a plurality of characteristic processes may be selectively performed.

<Characteristic Process 1 in One Embodiment>
For example, as shown in FIG. 12A, in a gradation portion where the gradation of the image changes gradually, an edge flow vector with a small power capturing a slight gradation change is generated. Due to the above edge flow vector, an unnatural edge may be detected in the middle of gradation.

  As a countermeasure for the above, in # 202, the vector generation unit 15 sets the values of the random field and energy field in the predetermined direction to different values when the magnitude of the random field and energy field in the predetermined direction is less than the threshold value. Replace with and process. For example, the vector generation unit 15 forcibly replaces the value of the random field and energy field in the predetermined direction with zero when the magnitude of the random field and energy field in the predetermined direction is less than the threshold (random field). And zero clipping of energy fields).

  For example, the vector generation unit 15 in one embodiment includes a value of Err (s, r, θ) in a predetermined direction and a value of Err (s, r, θ + π) in the opposite direction to the predetermined direction. Are both less than the threshold TH, the values of the random and energy fields in these directions are forced to zero. That is, when Err (s, r, θ) <0 and Err (s, r, θ + π) <0, E (s, θ) = E (s, θ + π) = 0 and P (s, r, θ) = P (s, r, θ + π) = 0. At this time, F (s, r, θ) = F (s, r, θ + π) = 0.

  By performing the above-described zero clip processing, an edge flow vector is not generated in the middle of gradation as shown in FIG. According to the characteristic processing 1, since the possibility that an edge is detected from a portion that the user does not recognize as an edge is reduced, the edge detection accuracy is improved, and the edge detection result is made closer to the user's recognition. Can do.

<Characteristic process 2 in one embodiment>
In edge detection processing using a general edge flow vector, one edge flow vector is generated for one pixel of interest. In the above case, since the edge flow vector is generated only in the direction in which the image change is strongest, the direction in which the image change is strongest when there are multiple locations where the edge should be detected in the vicinity of the target pixel. Otherwise, the boundary is missing.

  As a countermeasure against the above, in # 202 and # 203, the vector generation unit 15 generates a plurality of edge flow vectors for one target pixel. For example, the vector generation unit 15 generates a plurality of edge flow vectors from one target pixel by executing at least one of the following first process and second process. Of course, the first process and the second process may be executed in combination.

(Description of the first process)
FIG. 13 is a diagram illustrating an example of generating an edge flow vector in the first process. In the first process, the process until obtaining the energy field E (s, θi) and the random field P (s, r, θi) in each peripheral pixel is the same as that in # 202. The vector generation unit 15 in the first process divides the range of the direction Θ (s) in which the image changes greatly into a plurality of regions in the circumferential direction, and generates an edge flow vector in each region. In the following description, an example in which the direction Θ (s) in which the image changes greatly is divided into three in the first process will be described.

For example, the vector generation unit 15 in the first process obtains the direction Θ (s) in which the image change is large using Expression (8).
Θ (s) = argmax_ {θ} Σ_ {A} P (s, r, θ ′)
However, A = {θ ′ | θ ≦ θ ′ <θ + 1.5π} (8)
In addition, the vector generation unit 15 in the first process may obtain the edge flow vectors FL1 to FL3 from Expressions (9) to (11), respectively.
FL1 = Σ_ {B1} E (s, θ ′) exp (jθ ′)
However, B1 = {θ ′ | Θ (s) ≦ θ ′ <Θ (s) + 2β} (9)
FL2 = Σ_ {B2} E (s, θ ′) exp (jθ ′)
However, B2 = {θ ′ | Θ (s) + 2β ≦ θ ′ <Θ (s) + 4β} (10)
FL3 = Σ_ {B3} E (s, θ ′) exp (jθ ′)
However, B3 = {θ ′ | Θ (s) + 4β ≦ θ ′ <Θ (s) + 6β} (11)
That is, in the examples of the above formulas (8) to (11), the vector generation unit 15 has a range in which the sum of the random fields is the largest in increments of 270 ° (direction Θ (s) in which the image changes greatly) Ask for. Then, the vector generation unit 15 divides the range of the direction Θ (s) where the image changes greatly into three in the circumferential direction, and generates an edge flow vector from the value of the energy field in each region. In addition, the area | region to divide | segment does not need to be spatially continuous and may be separated in the circumferential direction.

  FIG. 14 is a diagram illustrating an operation example in the first process. In the first process, a plurality of flow vectors are generated for one pixel of interest. When there are a plurality of boundaries around the pixel of interest, different boundary edges are detected by a plurality of flow vectors. Therefore, in the first process described above, the possibility of missing edges during the edge detection process is reduced, so that the edge detection accuracy is improved and the edge detection result can be made closer to the user's recognition.

  The number of divisions in the direction Θ (s) where the image changes greatly is not limited to three, and the number of divisions may be changed as appropriate. Further, the range of the region A in the above formula (8) is not limited to 270 °, and can be arbitrarily set.

(Explanation of the second process)
FIG. 15 is a diagram illustrating an example of generating an edge flow vector in the second process. The vector generation unit 15 in the second process obtains a plurality of random fields and energy fields using a plurality of different sample points. For example, the vector generation unit 15 obtains a plurality of sets of random fields and energy fields by differentiating the distance between the target pixel and the surrounding pixels. And the vector production | generation part 15 produces | generates an edge flow vector from the random field and energy field of a different distance, respectively.

  For example, the vector generation unit 15 in the second process includes eight first peripheral pixels separated from the target pixel s (x, y) by r1 at 45 ° intervals around the target pixel s (x, y), and the target pixel The eight second peripheral pixels that are r2 away from the pixel s (x, y) are sampled. In the example of FIG. 15, r1> r2.

  The vector generation unit 15 obtains an energy field E (s, θi) and a random field P (s, r1, θi) for each of the eight first peripheral pixels, and uses the above random field and energy field to obtain an edge flow vector. FL1 is generated. Similarly, the vector generation unit 15 obtains an energy field E (s, θi) and a random field P (s, r2, θi) for each of the eight second peripheral pixels, and uses the above random field and energy field. An edge flow vector FL2 is generated. In addition, what is necessary is just to obtain | require said random field, energy field, and edge flow vector FL1, FL2 by said Formula (1)-Formula (4).

  FIG. 16 is a diagram illustrating an operation example in the second process. In general, when the distance between the pixel of interest and the sample point is changed, the direction of the image change obtained from the difference between the two is often different. For example, as shown in FIG. 16, consider a case where two L-shaped boundaries having different linear distances exist around the pixel of interest. At this time, if the distance between the target pixel and the sample point is short, an edge flow vector is generated toward a boundary relatively closer to the target pixel. Similarly, if the distance between the target pixel and the sample point is long, an edge flow vector is generated toward the boundary that is relatively far from the target pixel. Different boundary edges are detected by generating a plurality of edge flow vectors from random and random fields. Therefore, in the second processing described above, the possibility of missing edges during the edge detection processing is reduced, so that the edge detection accuracy can be improved and the result of edge detection can be made closer to the user's recognition.

  Note that three or more types of distances from the target pixel to the surrounding pixels may be set, and the edge flow vector may be obtained for each.

<Characteristic process 3 in one embodiment>
In the above edge pixel detection (# 206), the edge intensity is detected by threshold processing to detect the edge pixel. Therefore, while it is possible to suppress noise in the edge image, there are cases where pixels that should be detected as boundaries are missing.

  As the above countermeasure, in # 206, the edge detection unit 16 performs an interpolation process based on the edge strength of the pixel and the presence / absence of a nearby edge pixel.

  17A and 17B are explanatory diagrams of the interpolation process in the characteristic process 3. FIG. 17A shows a state before the interpolation process, and FIG. 17B shows a state after the interpolation process. The edge pixel has edgeBIN (x, y) = 1, and the non-edge pixel has edgeBIN (x, y) = 0. In addition, among the non-edge pixels, pixels whose edge intensity is equal to or lower than the first threshold edgeTH1 and equal to or higher than the second threshold edgeTH2 (edgeTH2 ≦ edgeNorm (x, y) ≦ edgeTH1) are shown in black in the drawing. Note that the second threshold edgeTH2 is lower than the first threshold edgeTH1, and is set to a value that can detect a relatively small edge.

  In the additional processing example 3, the edge detection unit 16 replaces edgeBIN (x, y) of the pixel adjacent to the edge pixel from “0” to “1” among the non-edge pixels whose edge strength is equal to or greater than the second threshold value. To do. As a result, non-edge pixels whose edge strength is equal to or greater than the second threshold are changed to edge pixels. As described above, by treating a pixel adjacent to an edge pixel and having a certain degree of edge strength as an edge pixel, the possibility of missing edges at successive boundaries is reduced, and the edge detection accuracy is improved.

<Characteristic Process 4 in One Embodiment>
When the edge pixels are detected and then interpolated by connecting the edge pixels to form a closed space (# 208), if the interpolation is performed so as to connect the shortest distance from the end point of the edge, the edge of the image becomes unnatural. May be connected.

  As the above countermeasure, in # 208, the edge detection unit 16 obtains the extension direction of the edge pixel from the end point of the edge. Then, the edge detection unit 16 determines the position of the interpolation edge connected to the end point to form a closed space based on the extension direction.

(How to find the extension direction of edge pixels)
As a first example, the edge detection unit 16 may obtain an edge extension direction from a change between a pixel position of an end point and a pixel position of an edge pixel extending from the end point and separated by a predetermined distance. FIG. 18 is a diagram illustrating an example of obtaining the extension direction of the edge pixel from the end point of the edge pixel. In the example of FIG. 18, the edge detection unit 16 obtains the edge extension direction from the change in the pixel position of the edge pixel that is two pixels away from the end point. In FIG. 18, the rightward movement in the horizontal direction (x direction) is represented by a positive value, and the upward movement in the vertical direction (y direction) is represented by a positive value.

  Here, in the case of FIG. 18A, the amount of movement from the edge pixel that is two pixels away from the end point to the end point is two pixels in the x direction and −1 pixel in the y direction. In the case of FIG. 18A, the extending direction of the edge is atan (1 / −2) = 333 °.

  In the case of FIG. 18B, the amount of movement from the edge pixel 2 pixels away from the end point to the end point is 0 pixel in the x direction and −2 pixel in the y direction. In the case of FIG. 18B, the extending direction of the edge is atan (2/0) = 270 °.

  As a second example, the edge detection unit 16 may obtain the extension direction of the edge pixel by a matching process using a template indicating a correspondence relationship between the pattern of the edge pixel and the extension direction.

  FIGS. 19A to 19E show examples of templates whose edge extending directions (angles) are 360 °, 333 °, 315 °, 297 °, and 270 °, respectively. The template of FIG. 19 has a size of 3 × 3 pixels and shows a connection pattern of edge pixels for two pixels from the end point. In the template of FIG. 19, “1” indicates an edge pixel, and “0” indicates a non-edge pixel. In addition, the position of the end point in each template is shown by hatching around it. Each template is stored in, for example, the program storage unit 21 in a state where information on the extension direction (angle) of the edge is associated in advance.

  FIG. 20 is a diagram illustrating how to obtain the extension direction using a template. The edge detection unit 16 performs matching between the edge image and the template, and searches for a template that matches the edge pixels of two pixels from the end point. Then, the edge detection unit 16 may obtain the edge extension direction (angle) from the template that matches the edge image.

  For example, in the case of FIG. 20A, the pattern of edge pixels for two pixels from the end point matches the template of FIG. Therefore, in the case of FIG. 20A, the extending direction of the edge is 333 °. For example, in the case of FIG. 20B, the pattern of edge pixels for two pixels from the end point matches the template of FIG. Therefore, in the case of FIG. 20B, the extending direction of the edge is 270 °.

(Method for determining the position of the interpolation edge)
Then, when searching for the connection destination of the interpolation edge from the end point of the edge, the edge detection unit 16 preferentially searches for the edge pixel located in the extension direction of the edge. FIG. 21 is a diagram illustrating a search example of the connection destination of the interpolation edge. For example, the edge detection unit 16 may perform the following processes (B1) to (B3).

  (B1) First, the edge detection unit 16 searches for an edge pixel located on a straight line along the extending direction of the edge (see FIG. 21A). If an edge pixel exists on the straight line, the edge detection unit 16 determines the detected edge pixel as a connection destination of the interpolation edge.

  (B2) Next, when the connection destination of the interpolation edge cannot be detected in (B1) above, the edge detection unit 16 searches for an edge pixel located within a predetermined angle range from the end point with the straight line in (B1) as the center. (See FIG. 21 (b)). When an edge pixel exists within the above range, the edge detection unit 16 determines the detected edge pixel as a connection destination of the interpolation edge. When a plurality of edge pixels are detected from the above range, the edge detection unit 16 determines whether the angle formed by the straight line between the edge extension direction and the straight line connecting the end point and the edge pixel is the smallest or from the end point. The edge pixel having the smallest linear distance may be determined as the connection destination of the interpolation edge.

  (B3) Further, when the connection destination of the interpolation edge cannot be detected in (B2), the edge detection unit 16 selects edge pixels in a wider angle range than (B2) with the straight line (B1) as the center. Search (see FIG. 21C). If an edge pixel exists within the above range, the edge detection unit 16 may determine the detected edge pixel as a connection destination of the interpolation edge.

  According to the above processing, when forming a closed space, an edge pixel closer to the edge extension direction is more easily determined as a connection destination of an interpolation edge. Therefore, since the edges of the image are connected in a relatively natural state when forming the closed space, the final edge detection accuracy can be improved.

<Characteristic Process 5 in One Embodiment>
When a closed space is formed after detecting edge pixels (# 208), it is preferable to delete edges that do not have other edges in the vicinity, while edges that detect one boundary intermittently are connected. Are preferably interpolated.

  Therefore, as shown in FIG. 22, in # 208, the edge detection unit 16 regards the edge detection unit 16 as noise when the edge length is less than the threshold and there is no other edge pixel within the predetermined range. To delete edge pixels. Further, when the edge length is less than the threshold value and there is another edge pixel within a predetermined range, the edge detection unit 16 connects the edge pixel and the other edge pixel to perform interpolation.

  The above processing avoids cases where noise edges are unnaturally connected to other edges, or conversely, edges that should be detected are removed excessively. Can be improved.

<Example of image processing using region information>
Next, image processing (# 105) using region information will be described in detail. For example, the CPU 20 in # 105 can cause the image processing unit 19 to execute the following image processing (1) to (8).

(1: Generation of original image for coloring)
For example, the CPU 20 may output the area information (edge image) as an original image for coloring. Note that the image processing unit 19 may generate an original image for coloring by performing processing such as line width normalization and spike noise removal on the edge image as necessary.

(2: Retouching the attention area)
The image processing unit 19 may retouch the attention area of the input image.

  As an example, the image processing unit 19 may perform a process of deleting a subject in a region of interest and interpolating a portion that has been lost due to the deletion (see FIG. 23). Here, the image may be simply interpolated with the pixel values of the surrounding area. For example, Norihiko Kawai et al., "Image restoration by energy minimization based on pattern similarity" IEICE Technical Report , PRMU2006-106, Dec.2006 may apply the disclosed method. As a result of the above processing, the subject of the input image can be selectively deleted in the post-processing step using the result of area division, so that an image preferable for the user can be obtained.

  As another example, the image processing unit 19 may perform a process of moving the position of the subject in the attention area (see FIG. 24). In this case, the image processing unit 19 moves the position of the subject in the attention area in the image according to the user's operation. Then, the image processing unit 19 synthesizes the subject in the attention area at the position of the movement destination, and interpolates the missing portion of the image (the movement source of the attention area). With the above processing, the position of the subject on the input image can be adjusted in the post-processing step using the result of the region division, so that an image preferable for the user can be obtained.

  As another example, the image processing unit 19 may selectively perform a filtering process on the subject in the attention area. FIG. 25 shows an example in which a blur filter is applied to a subject (person) in the attention area. Note that the filter applied to the region of interest may be, for example, a noise removal filter, an edge enhancement filter, a mosaic filter, or the like. With the above processing, a desired image effect can be given to an arbitrary portion on the input image in the post-processing step using the result of region division, so that an image preferable for the user can be obtained.

(3: Attaching tag information to the attention area)
The image processing unit 19 may add tag information (metadata) to the attention area designated by the user (see FIG. 26). The tag information includes, for example, text information describing attributes of the attention area (person name, mountain, building, etc.) and user comments, area image size, image data amount, color information, and position of the area on the input image. This is a concept including image information such as. Of the tag information, text information is input by a user, for example, and image information is generated by, for example, the image processing apparatus 11. The tag information can be used as, for example, an image search tool or a user's memo.

  In one embodiment, since a region of interest can be easily specified using the result of region division, the user can efficiently add tag information to a desired subject region.

(4: Recognition target extraction process)
The image processing unit 19 may execute a process of extracting a predetermined recognition target from the input image using the result of area division. In this case, a classifier that determines whether or not the image is a recognition target based on the characteristics of the image is prepared in the image processing unit 19 in advance. The classifier includes, for example, a known model recognition unit such as a discrimination model using a neural network or a support vector machine. Then, the image processing unit 19 inputs each area of the input image to the classifier, and determines whether each area is a recognition target. Then, the image processing unit 19 superimposes a display indicating an area determined as a recognition target on the input image and causes the display device to display the display. The information on the presence / absence of the recognition target in the image and the position of the recognition target can also be used, for example, for scene recognition of the image and search for similar images.

  FIG. 27 is a diagram illustrating an example of recognition target extraction processing. FIG. 27 shows an example in which an area corresponding to a signboard is extracted from an input image using an identifier that recognizes the signboard, and an image of the extraction result is displayed. The recognition target by the classifier is not limited to the above and can be changed as appropriate. For example, the discriminator may recognize characters, barcodes, or a specific type of subject (such as a wire mesh) as a recognition target.

  In one embodiment, the recognition target can be efficiently extracted by determining the recognition target for each region using the result of region division.

(5: Search for corresponding points at the time of stereoscopic image restoration)
When restoring a stereoscopic image from a plurality of images with different shooting positions, the image processing unit 19 may search for corresponding points between the images using the result of region division.

  FIG. 28 is a diagram illustrating a search example of corresponding points at the time of restoring a stereoscopic image. In general, when a stereoscopic image is restored, corresponding points are obtained between images at different shooting positions, and the three-dimensional shape of the subject is estimated from the change in position between the corresponding points. In the example of FIG. 28, the image processing unit 19 searches for the same object region between two images by a known method such as matching of color component histograms. Then, the image processing unit 19 searches for corresponding points from the same object region between the two images. In one embodiment, the corresponding points are searched from the range of the corresponding object using the result of the region division, so that the corresponding points can be searched more efficiently than in the case where the corresponding points are searched from the entire image.

(6: Image similarity determination)
Further, the image processing unit 19 may determine the similarity between two images using the result of area division.

  FIG. 29 is a diagram illustrating an example of image similarity determination. Here, an input image that is a criterion for similarity determination is referred to as a reference image, and an input image for which similarity is determined is referred to as a processing target image. It is assumed that the reference image and the processing target image are each subjected to area division processing.

  In the example of FIG. 29, first, the image processing unit 19 executes matching processing for matching the attention area of the reference image with each area of the processing target image. In this matching process, the image processing unit 19 obtains an evaluation value indicating the level of correlation between the two areas to be compared. When the attention area of the reference image matches the area of the processing target image, the image processing unit 19 calculates a relatively high evaluation value. For example, for the matching processing between the above-described regions, for example, a known method such as matching using a corner as a feature amount or matching using a color component histogram can be applied.

  Here, the image processing unit 19 weights the evaluation value according to the size of the region or the difference in the images. For example, since the region having a larger area has a greater influence on the similarity of the image, the image processing unit 19 increases the weight coefficient of the evaluation value as the area of the matching region is larger. In addition, when the position of the attention area in the reference image and the position of the corresponding area in the processing target image are deviated on the image, the image processing unit 19 reflects the positional deviation of the object in the similarity of the image. The weighting coefficient of the evaluation value is reduced in accordance with the size of the region position deviation.

  Further, the image processing unit 19 may perform geometric transformation such as enlargement, reduction, and rotation on one of the areas to be compared during the matching process. When the region matching is successful by the above geometric transformation, the image processing unit 19 evaluates according to the degree of the geometric transformation in order to reflect the difference in the size and orientation of the object in the similarity of the image. Decrease the value weighting factor.

  The image processing unit 19 sequentially designates all areas of the reference image as attention areas, and obtains evaluation values calculated by matching processing with the area of the processing target image. Then, the image processing unit 19 determines that the processing target image is similar to the reference image when a calculation result based on the obtained evaluation value (for example, a value using an integrated value of evaluation values calculated for each region as a parameter) is equal to or larger than a threshold value. When the calculation result based on the obtained evaluation value is less than the threshold value, it is determined that the processing target image is not similar to the reference image.

  In the example of FIG. 29, an image similar to the reference image such as the processing target image 1 has a high evaluation value calculation result because many areas match the reference image. On the other hand, an image that is significantly different from the reference image, such as the processing target image 2, has a low value in the evaluation value calculation result because most of the regions do not match the reference image. Therefore, in one embodiment, comprehensive similarity with the reference image can be determined based on the calculation result of the evaluation value.

(7: Image compression)
Further, the image processing unit 19 may change the compression rate of the image for each divided region using the result of the region division. For example, the image processing unit 19 may decrease the compression ratio of the image in the attention area and increase the compression ratio of the image other than the attention area (see FIG. 30). In the above case, it is possible to efficiently reduce the data amount of the entire image while maintaining the information of the main subject that the user pays attention to. Note that the compression rate of the image may be changed according to the designation of the user, the average value of the attention degree in the region of ROI (Region of Interest), or the amount of high frequency components in the region.

(8: Motion vector calculation)
Further, the image processing unit 19 may obtain the motion vector of the subject in units of areas divided at the time of moving image shooting using the result of area division (see FIG. 31). In the above case, the number of times of matching can be reduced by performing matching in a region unit having a size larger than that of a normal block. In addition, by performing matching in consideration of the area area and shape characteristics, it is possible to reduce the possibility of erroneous detection even when there is a large movement between frames.

<Description of other embodiments>
FIG. 32 is a diagram illustrating a configuration example of an imaging apparatus according to another embodiment. The other embodiment is an example in which the image processing apparatus 11 of the above-described one embodiment is mounted on the electronic camera 31, and the image processing apparatus acquires an input image from the imaging unit 33 of the electronic camera 31.

  The electronic camera 31 includes a photographic lens 32, an imaging unit 33, an image processing engine 34, a first memory 35 and a second memory 36, a recording I / F 37, and an operation unit 38. Here, the imaging unit 33, the first memory 35, the second memory 36, the recording I / F 37, and the operation unit 38 are each connected to the image processing engine 34.

  The imaging unit 33 is a module that captures (captures) an image of a subject formed by the photographing lens 32. For example, the imaging unit 33 includes an imaging device that performs photoelectric conversion, an analog front-end circuit that performs analog signal processing, and a digital front-end circuit that performs A / D conversion and digital signal processing.

  The image processing engine 34 is a processor that comprehensively controls the operation of the electronic camera 31. For example, the image processing engine 34 causes the imaging unit 33 to capture an image in response to a user's shooting instruction input in the shooting mode. In addition, the image processing engine 34 is configured to execute the program by the bandpass processing unit 14, the vector generation unit 15, the edge detection unit 16, the integration processing unit 17, the image processing unit 19, and the CPU 20 in the image processing apparatus 11 of the embodiment. Works as.

  The first memory 35 is a memory that temporarily stores image data and the like, and is, for example, an SDRAM that is a volatile storage medium. The first memory 35 operates as the image storage unit 13 and the region information storage unit 18 in the image processing apparatus 11 of one embodiment.

  The second memory 36 is a memory for storing a program executed by the image processing engine 34 and is a non-volatile memory such as a flash memory, for example. The second memory 36 operates as the program storage unit 21 in the image processing apparatus 11 of one embodiment.

  The recording I / F 37 has a connector for connecting a nonvolatile storage medium 39. The recording I / F 37 writes / reads image data to / from the storage medium 39 connected to the connector. The storage medium 39 is, for example, a hard disk or a memory card incorporating a semiconductor memory. In FIG. 32, a memory card is illustrated as an example of the storage medium 39.

  The operation unit 38 has a plurality of switches that accept user operations. The operation unit 38 includes, for example, a release button for receiving a recording still image shooting instruction.

  Hereinafter, an operation example of the electronic camera 31 in another embodiment will be briefly described. As an example, the area division process in another embodiment is executed as a pre-process for subject detection and AF area detection for autofocus (AF).

  In the electronic camera 31 of another embodiment, the imaging unit 33 captures an image. Thereby, an input image corresponding to the process of # 101 in FIG. 3 is acquired. Note that the input image may be a still image acquired in accordance with a user's shooting instruction, or may be an image for observation (through image) acquired at predetermined time intervals in the shooting mode. .

  Then, the image processing engine 34 executes the processes of # 102 to # 105 in FIG. 3 and divides the input image into regions. The image processing engine 34 performs the processes of # 201 to # 208 in FIG. 5 and at least one of the characteristic processes 1 to 5 in the edge detection process of # 103. In such other embodiments, substantially the same effects as those of the above-described one embodiment can be obtained.

<Supplementary items of the embodiment>
The characteristic processes 3 to 5 described above are not limited to the edge detection process using the edge flow vector, and may be combined with other known edge detection processes using a differential filter, for example.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Image processing apparatus, 12 ... Input / output I / F, 13 ... Image storage part, 14 ... Band pass processing part, 15 ... Vector generation part, 16 ... Edge detection part, 17 ... Integration processing part, 18 ... Area information storage , 19 ... Image processing unit, 20 ... CPU, 21 ... Program storage unit, 22 ... Input device, 23 ... Output device, 24 ... Data reading unit

Claims (14)

An acquisition unit for acquiring an input image;
For each target pixel of the input image, a random field indicating the direction of image change and an energy field indicating the intensity of the image change are obtained in each direction, and the edge of the target pixel is obtained using the random field and the energy field. A vector generation unit for generating a flow vector;
An edge detection unit that synthesizes the edge flow vectors generated at each position of the input image, and detects edge pixels of the input image from a position where the combined edge flow vectors oppose each other in the image;
A setting unit for setting a first range in the circumferential direction of the target pixel and a second range having a larger random field than the first range;
A dividing unit that divides the second range into a plurality of regions,
The vector generation unit is an image processing device that generates the edge flow vector in each of the plurality of regions . An acquisition unit for acquiring an input image;
For each target pixel of the input image, a random field indicating the direction of image change and an energy field indicating the intensity of the image change are obtained in each direction, and the edge of the target pixel is obtained using the random field and the energy field. A vector generation unit for generating a flow vector;
An edge detection unit that synthesizes the edge flow vectors generated at each position of the input image, and detects edge pixels of the input image from a position where the combined edge flow vectors oppose each other in the image;
A determination unit that determines a pixel at a first distance from the target pixel and a pixel at a second distance different from the first distance from the target pixel;
The vector generation unit includes a first edge flow vector obtained using the pixel at the first distance from the target pixel, and a second edge flow vector obtained using the pixel at the second distance from the target pixel. An image processing apparatus that generates The image processing apparatus according to claim 1.
A determination unit that determines a pixel at a first distance from the target pixel and a pixel at a second distance different from the first distance from the target pixel;
The vector generation unit includes a first edge flow vector obtained using the pixel at the first distance from the target pixel, and a second edge flow vector obtained using the pixel at the second distance from the target pixel. An image processing apparatus that generates The image processing apparatus according to any one of claims 1 to 3 ,
An image processing apparatus further comprising an image processing unit that performs image processing on an image divided into regions using the edge detection result of the edge detection unit. The image processing apparatus according to claim 4 .
The image processing unit is an image processing device that generates an original image for coloring using the image obtained by the region division. The image processing apparatus according to claim 4 .
The image processing unit is an image processing device that performs at least one of deletion of a subject in a region of interest, movement of a subject in a region of interest, and local filter processing to a region of interest for the region-divided image. The image processing apparatus according to claim 4 .
The image processing unit is an image processing device that assigns tag information to a region of interest of the region-divided image. The image processing apparatus according to claim 4 .
The image processing unit is an image processing apparatus that extracts a predetermined recognition target using the region-divided image. The image processing apparatus according to claim 4 .
The image processing unit searches for corresponding points when a stereoscopic image is restored using the region-divided image. The image processing apparatus according to claim 4 .
The image processing unit is an image processing apparatus that determines a similarity of the processing target image with respect to the reference image by matching each region-divided reference image and processing target image in units of regions. The image processing apparatus according to claim 4 .
The image processing unit is an image processing apparatus that performs image compression by changing a compression rate in units of regions with respect to the region-divided images. The image processing apparatus according to claim 4 .
The image processing unit is an image processing apparatus that obtains a motion vector in units of areas using the image obtained by dividing the area. Processing to acquire the input image;
For the target pixel of the input image, a random field indicating the direction of the image change and an energy field indicating the intensity of the image change are obtained, and an edge flow vector of the target pixel is determined using the random field and the energy field. A vector generation process to generate;
Combining the edge flow vectors generated at each position of the input image, and detecting edge pixels of the input image from positions where the combined edge flow vectors oppose each other in the image; and
A setting process for setting a first range in the circumferential direction of the target pixel and a second range having a larger random field than the first range;
Dividing the second range into a plurality of regions, and causing a computer to execute the division processing,
In the vector generation process, a program for generating the edge flow vector in each of the plurality of regions . Processing to acquire the input image;
For the target pixel of the input image, a random field indicating the direction of the image change and an energy field indicating the intensity of the image change are obtained, and an edge flow vector of the target pixel is determined using the random field and the energy field. A vector generation process to generate;
Combining the edge flow vectors generated at each position of the input image, and detecting edge pixels of the input image from positions where the combined edge flow vectors oppose each other in the image; and
A determination process for determining a pixel at a first distance from the target pixel and a pixel at a second distance different from the first distance from the target pixel;
In the vector generation process, a first edge flow vector obtained using the pixel at the first distance from the target pixel and a second edge flow vector obtained using the pixel at the second distance from the target pixel. And a program that generates .