CN112991386A - Optical flow tracking device - Google Patents
Optical flow tracking device Download PDFInfo
- Publication number
- CN112991386A CN112991386A CN202110194087.0A CN202110194087A CN112991386A CN 112991386 A CN112991386 A CN 112991386A CN 202110194087 A CN202110194087 A CN 202110194087A CN 112991386 A CN112991386 A CN 112991386A
- Authority
- CN
- China
- Prior art keywords
- sub
- target image
- tracking
- target
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000003287 optical effect Effects 0.000 title claims abstract description 130
- 238000000034 method Methods 0.000 claims abstract description 97
- 230000008569 process Effects 0.000 claims description 87
- 238000004364 calculation method Methods 0.000 claims description 63
- 238000012545 processing Methods 0.000 claims description 21
- 125000004122 cyclic group Chemical group 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 12
- 238000012795 verification Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 3
- 230000004069 differentiation Effects 0.000 claims description 3
- 230000005055 memory storage Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 25
- 230000006870 function Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 108010001267 Protein Subunits Proteins 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an optical flow tracking device. For saving time in optical flow tracking. In the embodiment of the application, a CPU is adopted to obtain a first target image and a second target image, wherein the first image comprises pre-marked target feature points, and then the first target image and the second target image are processed by adopting an FPGA optical flow tracking method to obtain feature points corresponding to the target feature points in the second target image.
Description
Technical Field
The application relates to the technical field of computer vision, in particular to an optical flow tracking device.
Background
At present, the application of the pyramid-based optical flow tracking technology is more and more extensive, and the following two problems are caused by performing pyramid optical flow tracking by using a Central Processing Unit (CPU): firstly, more computational resources are occupied, secondly, the operation principle of the CPU is instruction operation, and if the CPU is occupied by other programs when pyramid optical flow tracking is performed, delay is caused and further more time cost is increased.
Disclosure of Invention
The application aims to provide an optical flow tracking device which is used for saving the time of optical flow tracking.
In a first aspect, an embodiment of the present application provides an apparatus for optical flow tracking, the apparatus including:
the system comprises a Central Processing Unit (CPU) and a Field Programmable Gate Array (FPGA), wherein the FPGA comprises a memory unit and a tracking unit, and the following steps are as follows:
the CPU is configured to acquire a first target image and a second target image, and respectively perform scaling processing with different scaling ratios on the first target image and the second target image to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image; each sub-image corresponding to the first target image comprises a pre-marked target feature point of the same tracking target, each sub-image of the first target image and each sub-image of the second target image have a one-to-one correspondence relationship, and the mutually corresponding sub-images form a sub-image group; n is a positive integer greater than 1;
the memory unit is configured to acquire and store the sub-images in each sub-image group;
the tracking unit is configured to sequentially process the sub-images in each sub-image group stored in the memory unit by adopting an optical flow tracking method based on the target feature points of the tracking target, so as to obtain feature points, corresponding to the target feature points, of the tracking target in the second target image.
In one embodiment, the central processor CPU is configured to: acquiring the first target image and the second target image, and respectively carrying out scaling processing with different scaling ratios on the first target image and the second target image to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image;
the CPU is further configured to fill the boundaries of the n sub-images of the first target image and the n sub-images of the second target image according to a preset expansion ratio.
In one embodiment, the memory cell includes a first memory subunit and a second memory subunit, wherein:
the second memory subunit is configured to acquire n preprocessed sub-images of the first target image and n preprocessed sub-images of the second target image, and transfer the n preprocessed sub-images of the first target image to the first memory subunit;
the first memory subunit is configured to acquire and store n sub-images of the first target image from the second memory subunit.
In one embodiment, the tracking unit comprises a corner point screening sub-unit and an optical flow calculation sub-unit that is performed n times in a loop, wherein:
the optical flow calculation subunit is configured to perform forward tracking in sequence based on a first input parameter and a corresponding sub-image group to obtain a feature point, corresponding to a target feature point, of the second target image, where for a first-time loop process, the first input parameter is a target feature point of a sub-image of a first target image in the corresponding sub-image group, and for a non-first-time loop process, the first input parameter is an output result of a previous loop process and a target feature point of a sub-image of the first target image in the sub-image group corresponding to the optical flow calculation subunit in the current loop;
the optical flow calculating subunit is further configured to perform reverse tracking in sequence based on a second input parameter and a corresponding sub-image group to obtain a verification feature point of the first target image, where for a first cycle process, the second input parameter is a feature point, corresponding to a target feature point, of a sub-image of a second target image in each corresponding sub-image group, and for a cycle process not in the first cycle process, the second input parameter is an output result of the previous cycle process and a feature point, corresponding to a target feature point, of a sub-image of a second target image in the sub-image group corresponding to the optical flow calculating subunit during the current cycle;
the corner screening subunit is configured to output feature points, corresponding to the target feature points, of the second target image and a mark indicating successful tracking when a difference value between the target feature points of the first target image and the verification feature points of the first target image is smaller than a preset value; and when the difference value between the target characteristic point of the first target image and the verification characteristic point of the first target image is greater than or equal to the preset value, outputting a sign of tracking failure.
In one embodiment, for each cyclic process, the output result of the optical flow calculation subunit is a sum of optical flow vectors and target feature points of sub-images of the first target image within the sub-image group corresponding to the optical flow calculation subunit, where the optical flow calculation subunit, when determining the optical flow vectors, is configured to:
acquiring a third input parameter, and respectively obtaining target gray value data of the first target image and the second target image according to the third input parameter; the target gray-scale value data includes any one of: the first gray value data of the image of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image in a first preset window taking the third input parameter as the center, and the second gray value data of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image; if the optical flow calculation subunit is in a first loop process, the third input parameter of the optical flow calculation subunit is a target feature point of a sub-graph corresponding to the first target image and a coordinate of a corresponding position of the target feature point on a sub-graph corresponding to the second target image; if the optical flow calculation subunit is in a non-first cyclic process, the input parameters of the optical flow calculation subunit are the target feature points of the subgraph corresponding to the first target image and the output result of the last cyclic process;
performing scharr filtering operation on the target gray value data to obtain gray value gradients of a subgraph corresponding to the first target image and a subgraph corresponding to the second target image;
constructing a covariance matrix according to the gray value gradient;
performing weighting operation on the subgraph corresponding to the first target image, the subgraph corresponding to the second target image and the third input parameter to obtain a weighted pixel value;
obtaining a variance matrix according to the covariance matrix and the weighted pixel values;
and obtaining the optical flow vector according to the covariance matrix and the differentiation matrix.
In one embodiment, the FPGA further comprises a control unit, wherein:
the control unit is in communication connection with the memory unit and the tracking unit respectively, and is configured to acquire the feature points of the first target image from the memory unit to the tracking unit and control the processing sequence of each sub-image group by the tracking unit.
In one embodiment, the control unit comprises: left and right frame forward tracking subunit, left and right frame backward tracking subunit, front and back frame forward tracking subunit, front and back frame backward tracking subunit, wherein:
the left and right frame forward tracking subunit is configured to control the tracking unit to perform left and right frame forward tracking on the sub image group;
the left and right frame backward tracking subunit is configured to control the tracking unit to perform left and right frame backward tracking on the sub image group with the tracking result of the left and right frame forward tracking subunit as input after the left and right frame forward tracking subunit controls the tracking unit to perform left and right frame forward tracking on the sub image group;
the front-back frame forward tracking subunit is configured to control the tracking unit to perform front-back frame forward tracking on the sub image group;
the front-back frame backward tracking sub-unit is configured to control the tracking unit to perform front-back frame backward tracking on the sub-image group by taking a tracking result of the front-back frame forward tracking sub-unit as an input after the front-back frame forward tracking sub-unit controls the tracking unit to perform front-back frame forward tracking on the sub-image group.
In one embodiment, the apparatus further comprises a double data memory cell, wherein:
the double data memory storage unit is configured to store the first target image and the second target image and send the first target image and the second target image to the CPU.
In an embodiment, after the tracking unit obtains the feature point of the tracking target in the second target image corresponding to the target feature point, the second memory subunit is further configured to mark the feature point of the second image corresponding to the target feature point, and transfer n sub-images of the marked second target image to the first memory subunit.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1a is a schematic diagram of an image of an optical flow tracking device according to an embodiment of the present application with time;
FIG. 1b is a schematic diagram of the optical flow tracking apparatus provided in the embodiment of the present application, in which two images are down-sampled in the same scale;
FIG. 1c is a schematic device diagram of an optical flow tracking device according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a CPU structure of an optical flow tracking apparatus according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a pyramid hierarchy of an optical flow tracking apparatus according to an embodiment of the present application;
FIG. 4 is a schematic diagram illustrating an optical flow tracking device according to an embodiment of the present application expanding 4 sub-images of a first target image according to a predetermined scale;
FIG. 5a is a diagram of a tracking unit structure of an optical flow tracking device according to an embodiment of the present application;
FIG. 5b is a schematic diagram illustrating an input/output of an optical flow calculating subunit of the optical flow tracking apparatus according to the embodiment of the present application executing a first loop process;
FIG. 5c is a schematic diagram illustrating an input and an output of the optical-flow calculating subunit of the optical-flow tracking apparatus according to the embodiment of the present application executing a second loop process;
FIG. 5d is a schematic diagram illustrating an input and an output of the optical-flow calculating subunit of the optical-flow tracking apparatus according to the embodiment of the present application executing a third round-robin process;
FIG. 5e is a schematic diagram illustrating an input and an output of the optical-flow calculating subunit of the optical-flow tracking apparatus according to the embodiment of the present application executing a fourth round process;
FIG. 5f is a schematic input/output diagram of the optical flow computing subunit of the optical flow tracking apparatus according to the embodiment of the present application executing a first loop process;
FIG. 5g is a schematic diagram illustrating an input and an output of the optical flow calculating subunit of the optical flow tracking apparatus according to the embodiment of the present application executing a second loop process;
FIG. 5h is a schematic input/output diagram of an optical flow computing subunit of the optical flow tracking apparatus according to the embodiment of the present application executing a third round-robin process;
FIG. 5i is a schematic diagram illustrating an input and an output of the optical-flow calculating subunit of the optical-flow tracking apparatus according to the embodiment of the present application executing a fourth round process;
FIG. 6 is a schematic diagram of a control module of an optical flow tracking apparatus according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
It is noted that the terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The inventor researches and discovers that computer vision technology is widely applied along with the development of artificial intelligence and computer technology in recent years, wherein tracking moving objects becomes an important link in image processing. Optical flow tracking technology is used for recognizing and tracking moving objects in videos, automatically driving vision parts of automobiles and automatically navigating, positioning and mapping by robots, cross-platform computer vision and machine learning software library (OPENCV) functions are called to complete calculation of pyramid optical flow tracking in related technologies, but calculation of the library functions occupies a large amount of calculation resources and time resources.
The inventor researches and finds that the pyramid-based optical flow tracking technology is applied more and more widely at present, and the pyramid optical flow tracking by using the CPU causes the following two problems: firstly, more computational resources are occupied, secondly, the operation principle of the CPU is instruction operation, and if the CPU is occupied by other programs when pyramid optical flow tracking is performed, delay is caused and further more time cost is increased.
The inventor researches and discovers that optical flow tracking is a mode for converting image brightness change into motion information description in computer vision processing. The optical flow calculation is a basic calculation mode for obtaining optical flow by combining gray scale change and a two-dimensional velocity field, and two assumptions are proposed on the basis of moving optical characteristics of a target object in an image:
1. the gray scale of a moving object (with constant brightness) remains constant for a short interval.
2. Temporal continuity or motion is "small motion". The velocity vector field in a given domain varies slowly.
In an actual scene, the position of the same object in two adjacent frames of images is different due to the motion of the object or the motion of the camera. When the scene brightness does not change greatly in this time period, it is assumed that the pixel values of the same object in the two images are unchanged as shown in fig. 1 a. Moving the feature point A in image0 to the middle position in the next frame image1, requiring small motion of another assumed condition object satisfying optical flow tracking, scaling down the two images by the same proportion as in FIG. 1b, and then calculating optical flow from the top to solve the final optical flow layer by layer iteratively.
The pixel value at (x, y) in image0 may be written as I (x, y, t) according to the image in fig. 1a over time. Wherein x is an abscissa, y is an ordinate, and t is time; the following formula can be derived here from the assumption 1:
i (x, y, t) ═ x + dx, y + dy, t + dt; formula (1)
Performing Taylor expansion on the right end of the formula (1) to obtain:
wherein epsilon represents a second-order infinite small term which can be ignored, and then the formula (2) is substituted into the formula (1) and then is divided by dt to obtain:
and u and v are velocity vectors of the optical flow along the x axis and the y axis respectively, and the following are obtained:
order to
Respectively representing the partial derivatives of the gray levels of pixel points in the image along the X, Y and T directions.
In summary, equation (3) can be written as:
Ixu+Iyv+It0; formula (5)
Where Ix, Iy, It can be obtained from the image data, and (u, v) is the optical flow vector.
In view of the above, the present application proposes an optical flow tracking device to solve the above problems. The inventive concept of the present application can be summarized as follows: the method comprises the steps of acquiring a first target image and a second target image by a CPU (Central processing Unit), wherein the first image comprises pre-marked target feature points, and then processing the first target image and the second target image by a Field Programmable Gate Array (FPGA) by adopting an optical flow tracking method to obtain feature points corresponding to the target feature points in the second target image.
FIG. 1c is a schematic diagram of an optical flow tracking device in an embodiment of the present application, including: a CPU101, an FPGA 102; the FPGA is provided with a memory unit 201 and a tracking unit 202; wherein:
the CPU101 is configured to acquire a first target image and a second target image, and respectively perform scaling processing with different scaling ratios on the first target image and the second target image to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image; each sub-image corresponding to the first target image comprises pre-marked target feature points of the same tracking target, each sub-image of the first target image and each sub-image of the second target image have one-to-one correspondence, and the sub-images corresponding to each other form a sub-image group; n is a positive integer greater than 1;
in one embodiment, as shown in fig. 2, a pyramid layering unit 203, an image edge-binding unit 204 are arranged on the CPU; wherein:
the pyramid layering unit 203 is configured to acquire a first target image and a second target image, and perform scaling processing with different scaling ratios on the first target image and the second target image respectively to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image;
in one embodiment, a sub-image corresponding to the target image may be obtained according to a scaling, for example: as shown in fig. 3, the first target image is divided into 4 sub-images at four different scales, and the second target image is divided into 4 sub-images at the same scale as the first target image.
In some embodiments, after the first target image is scaled, the feature points on the corresponding sub-image may fall outside the sub-image, and to solve this problem, in this embodiment of the application, the image edge-covering unit 204 is configured to fill the boundaries of the n sub-images of the first target image and the n sub-images of the second target image according to a preset expansion ratio.
In one embodiment, the feature points on the corresponding sub-images are corner points obtained after corner point detection, and it is to be understood that corner point detection algorithms in the related art are all applicable to the present application.
For example: taking 4 sub-images of the first target image in fig. 3 as an example, as shown in fig. 4, 1-4 in the figure are 4 sub-images obtained by scaling the first target image differently, and the 4 sub-images of the first target image are expanded according to a preset scale.
A memory unit 201 configured to acquire and store the sub-images in each sub-image group;
in one embodiment, the memory cell 201 has a first memory sub-unit and a second memory sub-unit; wherein:
the second memory subunit is configured to acquire n sub-images of the preprocessed first target image and n sub-images of the preprocessed second target image, and transfer the n sub-images of the first target image to the first memory subunit;
and the first memory subunit is configured to acquire and store the n sub-images of the first target image from the second memory subunit.
In the embodiment of the application, the first memory subunit is adopted to directly obtain the target image from the second memory subunit, so that the sharing of the hardware memory is realized.
And the tracking unit 202 is configured to sequentially process the sub-images in each sub-image group stored in the memory unit by adopting an optical flow tracking method based on the target feature point of the tracking target, so as to obtain a feature point, corresponding to the target feature point, of the tracking target in the second target image.
In one embodiment, as shown in fig. 5a, the tracking unit includes a corner point filtering sub-unit 301 and an optical flow calculating sub-unit 302 that performs the loop n times, and for the sake of understanding, the optical flow calculating sub-unit performs the loop 4 times in the tracking unit as an example. In this embodiment, the optical flow calculating subunit 302 is configured to perform forward tracking in sequence based on a first input parameter and a corresponding sub image group to obtain a feature point, corresponding to a target feature point, of a second target image, where for a first cycle, the first input parameter is a target feature point of a sub image of a first target image in the corresponding sub image group, and for a cycle not in the first cycle, the first input parameter is an output result of the previous cycle and a target feature point of a sub image of the first target image in the sub image group corresponding to the optical flow calculating subunit during the current cycle.
For the sake of understanding, the input and output of the optical flow calculation subunit during forward tracking will be described in detail below by taking an example in which the optical flow calculation subunit is executed 4 times in a loop, where:
the optical flow meter sub-unit performs a first loop process configured to perform optical flow calculation based on the target feature points of the sub-images of the first target image within the corresponding sub-image group and the corresponding sub-image group. For example: in the first loop process, the first target image in the sub-image group is 1, the second target image is 1', and the target feature point located on 1 is a. As shown in fig. 5b, the optical flow calculation is performed according to 1, 1' and a point, and an intermediate feature point a1 of the first loop process, that is, an output result of the first loop process of the optical flow calculation subunit is obtained.
The optical flow meter sub-unit performs a second round-robin process configured to perform optical flow calculations based on the first input parameters and the corresponding set of sub-images. The first input parameter in the second circulation process is the output result of the first circulation process and the target characteristic point of the sub-image of the first target image in the sub-image group corresponding to the optical flow counting operator unit when the second circulation process is executed; for example: when the second loop process is executed, the first target image in the sub-image group is 2, the second target image is 2', and the target feature point located on 2 is a. As shown in fig. 5c, the optical flow calculation is performed according to 2, 2' and A, A1, and the intermediate feature point a2 of the second loop process, i.e. the output result of the second loop process, is obtained.
The optical flow meter sub-unit, when executing the third round of the loop process, is configured to perform optical flow calculation based on the first input parameter and the corresponding sub-image group. The first input parameter of the third circulation process is the output result of the second circulation process and the target feature point of the sub-image of the first target image in the sub-image group corresponding to the third circulation process; for example: in the third cycle, the first target image in the sub-image group is 3, the second target image is 3', and the target feature point located on 3 is a. And as shown in fig. 5d, performing optical flow calculation according to 3, 3' and A, A2 to obtain an intermediate feature point A3 of the third cyclic process, namely, an output result of the third cyclic process.
The optical flow meter sub-unit, when executing the fourth round of the loop process, is configured to perform optical flow calculations based on the first input parameters and the corresponding sub-image groups. The first input parameter of the fourth cyclic process is the output result of the third cyclic process and the target feature point of the sub-image of the first target image in the sub-image group corresponding to the fourth cyclic process; for example: in the fourth loop, the first target image in the sub-image group is 4, the second target image is 4', and the target feature point located on 4 is a. As shown in FIG. 5e, the optical flow calculation is performed according to 4, 4' and A, A3 to obtain the intermediate feature point A4 of the fourth round process.
Since the fourth loop process is the last loop of the optical flow calculation subunit, the intermediate feature point a4 in the fourth loop process is the feature point B of the second target image corresponding to the target feature point.
The feature point B of the second target image corresponding to the target feature point, which is obtained according to the target feature point a of the first target image, is not necessarily accurate, so in order to improve the accuracy, after the point B is obtained by performing forward tracking, reverse tracking is performed to verify whether the point B is accurate, that is, whether the error between the verification feature point of the first image obtained according to the point B and the target feature point is within a preset range. Therefore, the optical flow calculation subunit 302 is further configured to perform reverse tracking in sequence based on a second input parameter and the corresponding sub image group to obtain verification feature points of the first target image for the tracking target, where for the optical flow calculation subunit first loop process, the second input parameter is a feature point, corresponding to the target feature point, of a sub image of the second target image in each corresponding sub image group, and for the optical flow calculation subunit not first loop process, the second input parameter is an output result of the last loop process and a feature point, corresponding to the target feature point, of a sub image of the second target image in the sub image group corresponding to the optical flow calculation subunit.
For the sake of understanding, the following description will be made in detail by taking 4 optical flow calculation subunits as an example, where:
the optical flow meter sub-unit, when performing the first loop process, is configured to perform optical flow calculations based on feature points corresponding to the target feature points of the sub-images of the second target image within the corresponding sub-image group and the corresponding sub-image group, for example: as shown in fig. 5f, during the first loop of the optical flow calculation subunit, the first target image in the sub-image group is 1, the second target image is 1 ', and the target feature point located on 1' is B. And (4) carrying out optical flow calculation according to the points 1, 1' and B to obtain an intermediate characteristic point B1 of the first circulation process of the optical flow calculation subunit.
The optical flow meter sub-unit, when executing the second loop process, is configured to perform optical flow calculation based on the second input parameter and the corresponding sub-image group. The second input parameter of the second circulation process is the output result of the first circulation process and the feature point corresponding to the target feature point of the sub-image of the second target image in the sub-image group corresponding to the second circulation process; for example: as shown in fig. 5g, during the second cycle, the first target image in the sub-image group is 2, the second target image is 2 ', and the feature point corresponding to the target feature point on 2' is B. Then optical flow calculations are performed based on 2, 2' and B, B1 to obtain the intermediate feature point B2 of the second round of the process.
The optical flow meter sub-unit, when executing the third round of the loop process, is configured to perform optical flow calculation based on the second input parameter and the corresponding sub-image group. The second input parameter of the third circulation process is the output result of the second circulation process and the feature point corresponding to the target feature point of the sub-image of the second target image in the sub-image group corresponding to the third circulation process; for example: as shown in fig. 5h, during the third cycle, the first target image in the sub-image group is 3, the second target image is 3 ', and the feature point corresponding to the target feature point located on 3' is B. Then optical flow calculation is performed according to 3, 3' and B, B2 to obtain an intermediate feature point B3 of the third cyclic process.
The optical flow meter sub-unit, when executing the fourth round of the loop process, is configured to perform optical flow calculation based on the second input parameter and the corresponding sub-image group. The second input parameter of the fourth cyclic process is the output result of the third cyclic process and the feature point corresponding to the target feature point of the sub-image of the second target image in the sub-image group corresponding to the fourth cyclic process; for example: as shown in fig. 5i, during the fourth cycle, the first target image in the sub-image group is 4, the second target image is 4 ', and the feature point corresponding to the target feature point on 4' is B. Then optical flow calculations are performed based on 4, 4' and B, B3 to obtain the intermediate feature point B4 for the fourth round.
Since the fourth cyclic process is the last cyclic process of the optical flow calculation subunit, the intermediate feature point of the fourth cyclic process is the verification feature point C of the first target image.
The corner screening subunit is configured to output the feature points of the second target image corresponding to the target feature points and a mark of successful tracking when the difference value between the target feature points of the first target image and the check feature points of the first target image is smaller than a preset value; and when the difference value between the target characteristic point of the first target image and the verification characteristic point of the first target image is greater than or equal to a preset value, outputting a sign of tracking failure.
In one embodiment, for example: and if the target characteristic point of the first target image is A, the verification characteristic point of the first target image is C, and the difference value between A and C is smaller than a preset value, outputting a characteristic point B corresponding to the target characteristic point of the second target image and a mark of successful tracking.
In the embodiment of the present application, in the forward tracking process, for each loop process of the optical-flow calculation subunit, an output result (intermediate feature point) of the optical-flow calculation subunit is a sum of a target feature point of a sub-image of the first target image within the sub-image group corresponding to the optical-flow calculation subunit and an optical-flow vector, wherein the optical-flow calculation subunit, when determining the optical-flow vector, is configured to:
acquiring a third input parameter, and respectively obtaining target gray value data of the first target image and the second target image according to the third input parameter; the target gradation value data includes any one of: the first gray value data of the image of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image in a first preset window taking the third input parameter as the center, and the second gray value data of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image;
in the embodiment of the present application, in order to save the calculation resources, the target gray-scale value data is preferentially selected as the first gray-scale value data.
If the optical flow calculation subunit executes a first loop process, the third input parameter of the optical flow calculation subunit is a target feature point of a sub-image corresponding to the first target image and a coordinate of a corresponding position of the target feature point on the sub-image corresponding to the second target image; if the optical flow computing subunit executes a non-first loop process, the input parameters of the optical flow computing subunit are the target feature points of the subgraph corresponding to the first target image and the output result of the previous optical flow computing subunit;
for example: when the optical flow calculation module executes a first loop process, the third input parameter is coordinates of corresponding positions of the target feature points A and A of the sub-image 1 corresponding to the first target object on the sub-image corresponding to the second target image;
when the optical flow calculation module executes the second loop process, the third input parameters are the target feature point a of sub-graph 2 corresponding to the first target image and the output result a1 of the first loop process.
Carrying out scharr filtering operation on the target gray value data to obtain gray value gradients of a subgraph corresponding to the first target image and a subgraph corresponding to the second target image; for example: obtaining gray value gradients of 1 and 1';
constructing a covariance matrix according to the gray value gradient; carrying out weighting operation on a subgraph corresponding to the first target image, a subgraph corresponding to the second target image and the third input parameter to obtain a weighted pixel value; obtaining a variance matrix according to the covariance matrix and the weighted pixel values; and obtaining an optical flow vector according to the covariance matrix and the differentiation matrix.
In the embodiment of the present application, in order to perform real-time control on the optical flow calculation module, the FPGA is further provided with a control unit 205, which is in communication connection with the memory unit and the tracking unit, respectively, and is configured to acquire the feature points of the first target image from the memory unit to the tracking unit, and control the processing sequence of each sub-image group by the tracking unit.
In the embodiment of the present application, as shown in fig. 6, a left and right frame forward tracking sub-unit 303, a left and right frame backward tracking sub-unit 304, a front and back frame forward tracking sub-unit 305, and a front and back frame backward tracking sub-unit 306 are further provided in the control unit, where:
a left and right frame forward tracking sub-unit 303 configured to control the tracking unit to perform left and right frame forward tracking on the sub-image group; for example: and obtaining the feature point of the first frame of the right eye according to the feature point of the first frame of the left eye.
A left and right frame back-tracking sub-unit 304 configured to control the tracking unit to perform left and right frame back-tracking on the sub-group of pictures with the tracking result of the left and right frame front-tracking sub-unit as an input after the left and right frame front-tracking sub-unit controls the tracking unit to perform left and right frame front-tracking on the sub-group of pictures;
a front-rear frame forward tracking sub-unit 305 configured to control the tracking unit to perform front-rear frame forward tracking on the sub-image group; for example: and obtaining the feature points of the left eye second frame according to the feature points of the left eye second frame.
And a front and back frame backward tracking sub-unit 306 configured to control the tracking unit to perform front and back frame backward tracking on the sub-image group using the tracking result of the front and back frame forward tracking sub-unit as an input after the front and back frame forward tracking sub-unit controls the tracking unit to perform front and back frame forward tracking on the sub-image group.
In the embodiment of the present application, the first target image and the second target image may be stored in a memory unit, so as shown in fig. 1c, a Double Data memory (DDR) 206 connected to the CPU is further provided in the embodiment of the present application, and is configured to store the first target image and the second target image and send the first target image and the second target image to the CPU.
In this embodiment of the application, in order to control the tracking unit to directly perform backward tracking after forward tracking, after the tracking unit obtains a feature point, corresponding to a target feature point, of a tracking target in a second target image, the second memory subunit is further configured to mark the feature point, corresponding to the target feature point, of the second image, and forward n sub-images of the marked second target image to the first memory subunit. Therefore, the result output in the forward direction can be used as the input of the reverse-direction Brother tracking, and the step of manual input is saved.
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (9)
1. An optical flow tracking apparatus, the apparatus comprising:
the system comprises a Central Processing Unit (CPU) and a Field Programmable Gate Array (FPGA), wherein the FPGA comprises a memory unit and a tracking unit, and the following steps are as follows:
the CPU is configured to acquire a first target image and a second target image, and respectively perform scaling processing with different scaling ratios on the first target image and the second target image to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image; each sub-image corresponding to the first target image comprises a pre-marked target feature point of the same tracking target, each sub-image of the first target image and each sub-image of the second target image have a one-to-one correspondence relationship, and the mutually corresponding sub-images form a sub-image group; n is a positive integer greater than 1;
the memory unit is configured to acquire and store the sub-images in each sub-image group;
the tracking unit is configured to sequentially process the sub-images in each sub-image group stored in the memory unit by adopting an optical flow tracking method based on the target feature points of the tracking target, so as to obtain feature points, corresponding to the target feature points, of the tracking target in the second target image.
2. The apparatus of claim 1, wherein the central processing unit CPU is configured to: acquiring the first target image and the second target image, and respectively carrying out scaling processing with different scaling ratios on the first target image and the second target image to obtain n sub-images corresponding to the first target image and n sub-images corresponding to the second target image;
the CPU is further configured to fill the boundaries of the n sub-images of the first target image and the n sub-images of the second target image according to a preset expansion ratio.
3. The apparatus of claim 1, wherein the memory cell comprises a first memory sub-cell, a second memory sub-cell, and wherein:
the second memory subunit is configured to acquire n preprocessed sub-images of the first target image and n preprocessed sub-images of the second target image, and transfer the n preprocessed sub-images of the first target image to the first memory subunit;
the first memory subunit is configured to acquire and store n sub-images of the first target image from the second memory subunit.
4. The apparatus of claim 1, wherein the tracking unit comprises a corner point filtering sub-unit and an optical flow calculation sub-unit that performs n times in a loop, wherein:
the optical flow calculation subunit is configured to perform forward tracking in sequence based on a first input parameter and a corresponding sub-image group to obtain a feature point, corresponding to a target feature point, of the second target image, where for a first-time loop process, the first input parameter is a target feature point of a sub-image of a first target image in the corresponding sub-image group, and for a non-first-time loop process, the first input parameter is an output result of a previous loop process and a target feature point of a sub-image of the first target image in the sub-image group corresponding to the optical flow calculation subunit in the current loop;
the optical flow calculating subunit is further configured to perform reverse tracking in sequence based on a second input parameter and a corresponding sub-image group to obtain a verification feature point of the first target image, where for a first cycle process, the second input parameter is a feature point, corresponding to a target feature point, of a sub-image of a second target image in each corresponding sub-image group, and for a cycle process not in the first cycle process, the second input parameter is an output result of the previous cycle process and a feature point, corresponding to a target feature point, of a sub-image of a second target image in the sub-image group corresponding to the optical flow calculating subunit during the current cycle;
the corner screening subunit is configured to output feature points, corresponding to the target feature points, of the second target image and a mark indicating successful tracking when a difference value between the target feature points of the first target image and the verification feature points of the first target image is smaller than a preset value; and when the difference value between the target characteristic point of the first target image and the verification characteristic point of the first target image is greater than or equal to the preset value, outputting a sign of tracking failure.
5. The apparatus according to claim 4, wherein, for each loop process, the output result of the optical flow calculation subunit is a sum of optical flow vectors and target feature points of sub-images of the first target image within the sub-image group corresponding to the optical flow calculation subunit, wherein the optical flow calculation subunit, when determining the optical flow vectors, is configured to:
acquiring a third input parameter, and respectively obtaining target gray value data of the first target image and the second target image according to the third input parameter; the target gray-scale value data includes any one of: the first gray value data of the image of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image in a first preset window taking the third input parameter as the center, and the second gray value data of the subgraph corresponding to the first target image and the subgraph corresponding to the second target image; if the optical flow calculation subunit is in a first loop process, the third input parameter of the optical flow calculation subunit is a target feature point of a sub-graph corresponding to the first target image and a coordinate of a corresponding position of the target feature point on a sub-graph corresponding to the second target image; if the optical flow calculation subunit is in a non-first cyclic process, the input parameters of the optical flow calculation subunit are the target feature points of the subgraph corresponding to the first target image and the output result of the last cyclic process;
performing scharr filtering operation on the target gray value data to obtain gray value gradients of a subgraph corresponding to the first target image and a subgraph corresponding to the second target image;
constructing a covariance matrix according to the gray value gradient;
performing weighting operation on the subgraph corresponding to the first target image, the subgraph corresponding to the second target image and the third input parameter to obtain a weighted pixel value;
obtaining a variance matrix according to the covariance matrix and the weighted pixel values;
and obtaining the optical flow vector according to the covariance matrix and the differentiation matrix.
6. The apparatus of claim 1, wherein the FPGA further comprises a control unit, wherein:
the control unit is in communication connection with the memory unit and the tracking unit respectively, and is configured to acquire the feature points of the first target image from the memory unit to the tracking unit and control the processing sequence of each sub-image group by the tracking unit.
7. The apparatus according to any one of claims 4-6, wherein the control unit comprises: left and right frame forward tracking subunit, left and right frame backward tracking subunit, front and back frame forward tracking subunit, front and back frame backward tracking subunit, wherein:
the left and right frame forward tracking subunit is configured to control the tracking unit to perform left and right frame forward tracking on the sub image group;
the left and right frame backward tracking subunit is configured to control the tracking unit to perform left and right frame backward tracking on the sub image group with the tracking result of the left and right frame forward tracking subunit as input after the left and right frame forward tracking subunit controls the tracking unit to perform left and right frame forward tracking on the sub image group;
the front-back frame forward tracking subunit is configured to control the tracking unit to perform front-back frame forward tracking on the sub image group;
the front-back frame backward tracking sub-unit is configured to control the tracking unit to perform front-back frame backward tracking on the sub-image group by taking a tracking result of the front-back frame forward tracking sub-unit as an input after the front-back frame forward tracking sub-unit controls the tracking unit to perform front-back frame forward tracking on the sub-image group.
8. The apparatus of claim 1, further comprising a double data memory storage unit, wherein:
the double data memory storage unit is configured to store the first target image and the second target image and send the first target image and the second target image to the CPU.
9. The apparatus according to claim 1, wherein after the tracking unit obtains the feature point of the tracking target in the second target image corresponding to the target feature point, the second memory subunit is further configured to mark the feature point of the second image corresponding to the target feature point, and forward n sub-images of the marked second target image to the first memory subunit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110194087.0A CN112991386A (en) | 2021-02-20 | 2021-02-20 | Optical flow tracking device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110194087.0A CN112991386A (en) | 2021-02-20 | 2021-02-20 | Optical flow tracking device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112991386A true CN112991386A (en) | 2021-06-18 |
Family
ID=76394047
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110194087.0A Withdrawn CN112991386A (en) | 2021-02-20 | 2021-02-20 | Optical flow tracking device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112991386A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103426184A (en) * | 2013-08-01 | 2013-12-04 | 华为技术有限公司 | Optical flow tracking method and device |
| CN105261042A (en) * | 2015-10-19 | 2016-01-20 | 华为技术有限公司 | Optical flow estimation method and apparatus |
| JP2016173795A (en) * | 2015-03-18 | 2016-09-29 | 株式会社リコー | Image processing apparatus, image processing method, and program |
| CN107103268A (en) * | 2016-02-23 | 2017-08-29 | 中国移动通信集团浙江有限公司 | A kind of method for tracking target and device |
| CN109978801A (en) * | 2019-03-25 | 2019-07-05 | 联想(北京)有限公司 | A kind of image processing method and image processing apparatus |
| CN110120063A (en) * | 2019-04-23 | 2019-08-13 | 深圳市道通智能航空技术有限公司 | A kind of target following processing method based on multiprocessor |
| CN110660086A (en) * | 2019-06-17 | 2020-01-07 | 珠海全志科技股份有限公司 | Motion control method and system based on optical flow algorithm |
| CN110751672A (en) * | 2018-07-24 | 2020-02-04 | 深圳地平线机器人科技有限公司 | Method and apparatus for implementing multi-scale optical flow pixel transform using dilution convolution |
| CN111383247A (en) * | 2018-12-29 | 2020-07-07 | 北京易讯理想科技有限公司 | Method for enhancing image tracking stability of pyramid LK optical flow algorithm |
| CN111563934A (en) * | 2020-06-10 | 2020-08-21 | 浙江欣奕华智能科技有限公司 | Monocular vision odometer scale determination method and device |
-
2021
- 2021-02-20 CN CN202110194087.0A patent/CN112991386A/en not_active Withdrawn
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103426184A (en) * | 2013-08-01 | 2013-12-04 | 华为技术有限公司 | Optical flow tracking method and device |
| JP2016173795A (en) * | 2015-03-18 | 2016-09-29 | 株式会社リコー | Image processing apparatus, image processing method, and program |
| CN105261042A (en) * | 2015-10-19 | 2016-01-20 | 华为技术有限公司 | Optical flow estimation method and apparatus |
| CN107103268A (en) * | 2016-02-23 | 2017-08-29 | 中国移动通信集团浙江有限公司 | A kind of method for tracking target and device |
| CN110751672A (en) * | 2018-07-24 | 2020-02-04 | 深圳地平线机器人科技有限公司 | Method and apparatus for implementing multi-scale optical flow pixel transform using dilution convolution |
| CN111383247A (en) * | 2018-12-29 | 2020-07-07 | 北京易讯理想科技有限公司 | Method for enhancing image tracking stability of pyramid LK optical flow algorithm |
| CN109978801A (en) * | 2019-03-25 | 2019-07-05 | 联想(北京)有限公司 | A kind of image processing method and image processing apparatus |
| CN110120063A (en) * | 2019-04-23 | 2019-08-13 | 深圳市道通智能航空技术有限公司 | A kind of target following processing method based on multiprocessor |
| CN110660086A (en) * | 2019-06-17 | 2020-01-07 | 珠海全志科技股份有限公司 | Motion control method and system based on optical flow algorithm |
| CN111563934A (en) * | 2020-06-10 | 2020-08-21 | 浙江欣奕华智能科技有限公司 | Monocular vision odometer scale determination method and device |
Non-Patent Citations (3)
| Title |
|---|
| AURELIEN PLYER等: "Massively Parallel Lucas Kanade Optical Flow for Real-Time Video Processing Applications", 《JOURNAL OF REAL-TIME IMAGE PROCESSING》, vol. 11, 24 April 2014 (2014-04-24), pages 713 - 730, XP035942295, DOI: 10.1007/s11554-014-0423-0 * |
| MINI猿要成长QAQ: "总结：光流--LK光流--基于金字塔分层的LK光流--中值流", pages 1 - 12, Retrieved from the Internet <URL:https://blog.csdn.net/sgfmby1994/article/details/68489944> * |
| 冉宝敬: "复杂自然场景下机器人视觉伺服研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》, no. 2018, 15 February 2018 (2018-02-15), pages 138 - 1870 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10970600B2 (en) | Method and apparatus for training neural network model used for image processing, and storage medium | |
| Meister et al. | Unflow: Unsupervised learning of optical flow with a bidirectional census loss | |
| CN110852383B (en) | Target detection method and device based on attention mechanism deep learning network | |
| Li et al. | Semi-direct monocular visual and visual-inertial SLAM with loop closure detection | |
| CN113139989B (en) | Particle image velocity measurement method and device based on deep learning | |
| CN114663496B (en) | Monocular vision odometer method based on Kalman pose estimation network | |
| CN112639878A (en) | Unsupervised depth prediction neural network | |
| CN111079507B (en) | Behavior recognition method and device, computer device and readable storage medium | |
| US10909764B2 (en) | Providing augmented reality target images in a web browser | |
| CN112149694A (en) | Image processing method, system, storage medium and terminal based on convolutional neural network pooling module | |
| CN112802202A (en) | Image processing method, image processing device, electronic equipment and computer storage medium | |
| CN112541460A (en) | Vehicle re-identification method and system | |
| Mehl et al. | M-fuse: Multi-frame fusion for scene flow estimation | |
| CN112233149A (en) | Scene flow determination method and device, storage medium and electronic device | |
| CN115661767A (en) | Image front vehicle target identification method based on convolutional neural network | |
| Wofk et al. | Monocular Visual-Inertial Depth Estimation | |
| Gehrig et al. | Dense continuous-time optical flow from events and frames | |
| CN108520532B (en) | Method and device for identifying motion direction of object in video | |
| WO2022120996A1 (en) | Visual position recognition method and apparatus, and computer device and readable storage medium | |
| CN112991386A (en) | Optical flow tracking device | |
| CN113570509A (en) | Data processing method and computer device | |
| KR102186764B1 (en) | Apparatus and method for estimating optical flow and disparity via cycle consistency | |
| CN112348854A (en) | Visual inertial mileage detection method based on deep learning | |
| CN111950599A (en) | Dense visual odometer method for fusing edge information in dynamic environment | |
| CN112464989A (en) | Closed loop detection method based on target detection network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210618 |