CN110147809A - Image processing method and device, storage medium and vision facilities - Google Patents

Abstract

The application provides a kind of image processing method and device, storage medium and vision facilities.Described image processing method, comprising: obtain the fisrt feature point set of the first image；Obtain the second feature point set of the second image；With at least two characteristic points for a matching unit, carries out the fisrt feature point set and second feature point concentrates the matching of characteristic point, obtain match parameter；According to the match parameter, the deformation of surface of the second image is determined.

Description

Image processing method and device, storage medium and image equipment

The application claims the application number as: 201910175084.5 application as priority of the Chinese application filed 2019 on 03/08.

Technical Field

The present application relates to the field of image technologies, and in particular, to an image processing method and apparatus, a storage medium, and an image device.

Background

In the process of image-based target tracking, due to the movement or flexible deformation of the target, the appearance of the target in the images acquired at different times changes, even is partially blocked, and the like. In the related art, various technologies are proposed to perform an image-based target tracking method, but most of the methods rely on establishing a local appearance matching relationship, which causes a problem of low accuracy, or a problem of high probability of target tracking failure due to poor texture quality of an image.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, a storage medium and an image device.

An image processing method comprising:

acquiring a first feature point set of a first image;

acquiring a second feature point set of a second image;

matching the feature points in the first feature point set and the second feature point set by taking at least two feature points as a matching unit to obtain matching parameters;

and determining the curved surface deformation of the second image according to the matching parameters.

An image processing apparatus comprising:

the first acquisition module is used for acquiring a first feature point set of a first image;

the second acquisition module is used for acquiring a second feature point set of a second image;

the matching module is used for matching the feature points in the first feature point set and the second feature point set by taking at least two feature points as a matching unit to obtain matching parameters;

and the first determining module is used for determining the curved surface deformation of the second image according to the matching parameters.

An image device, comprising:

a memory for storing at least computer-executable instructions;

and the processor is connected with the memory and used for realizing the image processing method provided by any technical scheme by executing the computer executable instruction.

According to the technical scheme provided by the embodiment of the application, in the image processing process applicable to target tracking, a first feature point set in a first image including target imaging to be tracked is obtained firstly, and then a second feature point set is extracted from a subsequently obtained second image. When the feature points are matched, single-point matching is not adopted any more, but matching units at least comprising two feature points are adopted to match feature points in a first feature point set and a second feature point set, so that the spatial distribution and other mutual constraint relations between a plurality of feature points in the matching units on a target can be utilized, the matching accuracy between the feature points is improved, the problem that the curved surface deformation error of a second image caused by the large single-point matching error is large, the target tracking accuracy is low and the target tracking loss phenomenon is large is solved. In a word, the technical scheme provided by the implementation of the application can improve the accuracy of target tracking and the success rate of tracking.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

fig. 1 is a schematic flowchart of a first image processing method according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a second image processing method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a process of iteratively determining a surface deformation of a second image based on a candidate surface deformation and a matching parameter according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

FIG. 5 is a diagram of several second images to be processed according to an embodiment of the present disclosure;

fig. 6 to 9 are schematic diagrams illustrating comparison of tracking effects of the method and related methods provided in the embodiments of the present application on several types of indications.

Detailed Description

The present application is described in further detail below with reference to the attached figures.

As shown in fig. 1, the present embodiment provides an image processing method, including:

step S110: acquiring a first feature point set of a first image;

step S120: acquiring a second feature point set of a second image;

step S130: matching the feature points in the first feature point set and the second feature point set by taking at least two feature points as a matching unit to obtain matching parameters;

step S140: and determining the curved surface deformation of the second image according to the matching parameters.

The image processing method provided by the embodiment can be applied to various image devices with image processing functions, for example, the image devices include but are not limited to: desktop computers, notebook computers, mobile phones, servers or server groups, etc.

In this embodiment, the first image may be an image including a target to be tracked. The second image is a newly acquired image that may contain the target.

In this embodiment, a first feature point set of the first image is obtained through various image processing, and the first feature point set includes one or more feature points. In step S120, a feature point extraction method for obtaining a first feature point set may be adopted to extract feature points of the second image from the second image, so as to form a second feature point set including one or more feature points.

For example, the first image and the second image are processed by a feature point extraction algorithm (SURF algorithm, SIFT algorithm, HOG algorithm, or the like) or an interest point detection algorithm, and feature points constituting the first feature point set and the second feature point set are determined. It is worth noting that: there are various ways to detect the feature points of the first feature point set and the second feature point set, and the specific implementation is not limited to any of the above.

The feature points in the first feature point set can be used for characterizing imaging characteristics of the target to be tracked in the image or visualization characteristics of the outer surface.

After the first feature point set and the second feature point set are obtained, matching of feature points in the first feature point set and the second feature point set is performed, and the matching parameters are obtained. The matching parameters are set to be matched with each other,

for the sake of distinction, in the present embodiment, a feature point in the first feature point set is referred to as a first feature point; the feature points in the second feature point set are referred to as second feature points. After the first feature point set and the second feature point set are obtained, matching of the first feature point and the second feature point is performed.

In this embodiment, when matching feature points, only single-point matching of feature points (also referred to as unary matching or unary projection) is not performed, but matching is performed in one matching unit including at least two feature points. For example, one matching unit may be: a matching pair consisting of two feature points. For example, one such matching unit may also be: and the matching group consists of 3 or more than 3 characteristic points.

For example, if the matching unit is a matching pair composed of two feature points, when matching the first feature point set and the second feature point set, the two feature points in the first feature point set are used as one matching unit, the two feature points in the second feature point set are used as the other matching unit, and the matching between the two matching units is performed to obtain the matching parameter representing whether and/or how much the matching is performed.

After obtaining the matching parameters in step S130, the surface deformation of the second image can be determined according to the matching parameters. The surface deformation may be a surface deformation of the second image relative to the first image, or a surface deformation of the second image relative to the original reference image. For example, the first image has a first surface deformation relative to the original reference image, and the surface deformation of the second image relative to the original reference image can be indirectly obtained based on the first image by performing steps S110 to S140. In the process of solving the curved surface deformation of the plurality of images, the original reference image can be an image without the curved surface deformation by default.

And determining whether the second image is the image of the target after deformation based on the curved surface deformation. Therefore, the target can be tracked by solving the deformation of the curved surface. In this embodiment, the matching unit including at least two feature points is used to perform the feature point matching, so that, compared with the single-point matching, the problem of poor target tracking accuracy caused by not considering the spatial structure characteristics of the target is reduced, and meanwhile, when the image texture quality is poor, the success rate of target tracking can be improved based on the mutual constraint relationship of the feature points in the matching unit, so that the method has the characteristics of high target tracking accuracy and high tracking success rate.

In some embodiments, the step S130 may include:

and matching the characteristic points in the first characteristic point set and the second characteristic point set by taking a characteristic point pair formed by two characteristic points as a matching unit to obtain a matching parameter indicating the matching condition between the characteristic point pairs.

The matching parameters are various parameters indicating the matching condition in the present embodiment. The matching condition may be whether and/or how well the match is. Alternatively, the matching parameter may be a matching probability between 0 and 1. The greater the probability of match, the greater the degree of match of two matching units from different sets of matching feature points. Therefore, the step S101 may include: and determining the matching probability of the single point matching of the first feature point in the first feature point set and the second feature point in the second feature point set.

Of course, in some embodiments, the matching parameter may also be an indication parameter indicating whether two states are matched.

In some embodiments, as shown in fig. 2, the method further comprises:

step S101: acquiring a first matching parameter of single-point soft matching in the first characteristic point set and the second characteristic point set;

the step S130 may include: and taking two feature points as a matching unit, and obtaining a second matching parameter of the matching unit based on the first matching parameter.

In the present embodiment, one matching unit is one matching pair. On one hand, the mutual relation restriction of two feature points in the matching pair is utilized, so that errors caused by unary projection are reduced, and the accuracy is improved; on the other hand, one matching unit and one matching pair can reduce the phenomena of large calculation amount and the like caused by excessive constraint relative to the matching unit formed by 3 or more than 3 characteristic points, thereby improving the efficiency.

In the present embodiment, matching in which two feature points are one matching unit is soft matching, not hard matching. Here, soft matching generates soft matching parameters, and hard matching generates hard matching parameters. The hard matching parameter is a parameter for indicating whether to match, and is generally indicated by a binarized value by using the hard matching parameter. And the soft matching parameter is a parameter capable of indicating the degree of matching, for example, a value between 0 and 1 is used to indicate the degree of matching.

In some embodiments, the step S101 may include: and generating a matching matrix based on the matching probability.

In this embodiment, the single point matching may also be referred to as a unary projection. A unary projection is a matching of a single feature point in a first set of feature points with a single feature point in a second set of feature points. And taking the matching probability of the single-point matching as an element of the matching matrix.

For example, if the first feature point set has N first feature points and the second feature point set has M second feature points, the N first feature points and the M second feature points form a matching matrix including N × M matching probabilities.

In other embodiments, the generated matching probability is not necessarily a matching matrix, but may be other forms of matching parameters, such as a matching vector, etc.

In some embodiments, the obtaining a second matching parameter of the matching unit based on the first matching parameter with two feature points as one matching unit includes:

and matching the feature points in the first feature point set and the second feature point set based on the initial candidate surface deformation of the second image by taking at least two feature points as a matching unit, and selecting the second matching parameter from the first matching parameters, wherein the initial candidate surface deformation of the second image is the surface deformation of the first image.

Since there are a plurality of feature points in both the first feature point set and the second feature point set, for example, the first feature point set includes M1 first feature points, and the second feature point set includes M2 second feature points. The candidate surface deformation determines which of the M1 first feature points matches with which of the second feature points. For example, if the serial number of a group of point pairs matching the first feature point is (i, j) and the serial number of the second feature point is (a, b), the specific values of (i, j) and (a, b) are known due to the existence of the candidate surface deformation.

In the embodiment, when the target tracking is performed, a plurality of pictures are continuously taken or directly collected to form a video. The first and second images may be from consecutive frames of a continuously captured photograph or video. Therefore, the second image is changed from the curved surface of the first image to the initial candidate curved surface deformation, and compared with the iteration started from any candidate curved surface deformation, the calculation amount can be greatly reduced, and the iteration efficiency is improved.

The determination of the surface deformation of the second image in step S140 may be an iterative process. At this time, the matching parameters in step S130 are initial matching parameters, and the initial surface deformation is also introduced in the process of confirming the surface deformation,

because the first image and the second image are from two adjacent frames of a video or two adjacent collected photos, the collection time interval is short, and the deformation of the target surface in the second image cannot generate step change based on the continuity of the target deformation.

In some embodiments, the first matching parameter in step S130 may also be determined based on the initial surface deformation of the second image, i.e. based on the surface deformation of the first image.

The method further comprises the following steps:

generating a candidate matching set containing a plurality of potential matches based on feature point matching with two feature points as a matching unit;

filtering out potential matches in the candidate match set that do not satisfy a constraint;

the step S130 may include:

second match parameters of potential matches in the filtered candidate match set are determined from the first match parameters.

For example, the first and second feature points of the first image each include N feature points, and the potential matches between feature points include N x N. These N x N matches constitute the set of candidate matches.

Some potential matches may not actually occur or have an extremely low probability of occurring, which may cause a computationally intensive problem if each potential match needs to be traversed.

In this embodiment, in order to reduce the problem of the amount of calculation, in this embodiment, potential matches that do not satisfy the constraint are filtered out based on the constraint, and after the potential matches that do not satisfy the constraint are filtered out, the number of potential matches included in the candidate match set is reduced.

In this way, only the second matching parameters of potential matches in the filtered candidate set need be considered in determining the second matching parameters.

Filtering out potential matches in the candidate match set that do not satisfy a constraint, including:

determining the deformation of a first feature point in the candidate matching set based on the surface deformation of the previous frame; determining whether the potential match satisfies a geometric constraint condition based on the deformation of the first feature point and the distance between the second feature points;

determining a photometric descriptor of a first feature point and a photometric descriptor of the second feature point, determining whether the potential match satisfies an appearance constraint;

filtering out the potential matches in the candidate match set that do not satisfy the geometric constraint and/or appearance constraint.

In some embodiments, the constraints that filter the potential matches include, but are not limited to, the geometric constraints and/or appearance constraints.

For example, the following formula may be employed to determine whether the corresponding potential match satisfies the constraint:

wherein,the candidate matching set after filtering is obtained; epsilon_gA tolerance threshold for determining whether the geometric constraint is satisfied; e is the same as_aTo determine whether a tolerance threshold for the appearance geometry is met. Psi_t-1For the surface deformation of the previous frame, here psi_t-1Can be a curved surface deformation of the first image;is the deformation of the first feature point.A photometric descriptor being a first feature point; f. of_aIs a photometric descriptor of the second feature point.

In some embodiments, determining whether the potential surface deformation satisfies the constraint includes, but is not limited to: appearance constraints and/or geometric constraints.

For example, assuming that the pre-filter candidate match set includes S potential matches, and then constraint-based filtering may filter out S1 potential matches, the post-filter candidate match set includes only S-S1 potential matches.

The curved surface deformation of the previous frame is changed into the initial curved surface deformation of the image (i.e. the second image) of the current frame.

In this embodiment, as shown in fig. 3, the step S140 may specifically include:

step A1: determining the matching error of a matching unit in the first characteristic point set and the second characteristic point set according to the alternative curved surface deformation and the matching parameters;

step A2: stopping iteration if the matching error meets a convergence condition or reaches the maximum iteration times, and outputting the alternative curved surface deformation obtained when the iteration is stopped as the final curved surface deformation;

step A3: if the matching error does not meet the convergence condition or the maximum iteration number, updating the alternative curved surface deformation of the second image according to the matching error;

step A4: and updating the matching parameters according to the alternative surface deformation, and returning to the step A1.

In this embodiment, the step S140 determines that the surface deformation is an iterative process.

In step a1, a matching error is first calculated based on the current candidate surface deformation and the matching parameters. The match error is determined, for example, by means of a table look-up or calculated by means of a predefined formula.

The matching error satisfying the convergence condition may include at least one of:

the current matching error is the minimum matching error in multiple iterations;

the current match error is less than the error threshold.

If the current matching error meets the convergence condition, which indicates that the matching of the feature points in the current different feature point sets is optimal, the deformation corresponding to the current matching can be considered as the optimal curved surface deformation, and the iteration can be stopped. In some cases, the matching error may not have a minimum value, and the errors of the deformations of the plurality of curved surfaces are the same or have small differences, and the matching error may still not satisfy the convergence condition after multiple iterations. At this time, the image device may obtain the iteration number, and if the iteration number has reached the maximum iteration number, the current candidate curved surface deformation is directly used as the curved surface deformation output of the second image.

If the matching error cannot satisfy the convergence condition and the maximum iteration number has been reached, in order to reduce the dead loop caused by repeated iteration, in the present embodiment, if the maximum iteration number is reached, the iteration is stopped even if the matching error does not satisfy the convergence condition.

And after the iteration is stopped, the alternative curved surface deformation corresponding to the iteration is used as the curved surface deformation output of the second image after the final optimization.

If the convergence condition is not met and the maximum iteration number is not reached, continuing iteration; if the iteration needs to be continued, the step A3 is performed, the candidate surface deformation is updated, and after the candidate surface deformation is updated, the matching parameters are recalculated correspondingly, so that the matching parameters are recalculated in the step a 4. Returning to step a1, recalculating the matching error according to the updated matching parameters and the candidate surface deformation. And repeating the steps to obtain the surface deformation of the second image accurately.

The following description will take a matching unit as an example.

The formula epsilon (C, psi) ═ Σ can be used in step a1_i，j∑_a，bd(ψ，i，j，a，b)C_i，aC_j，bAnd calculating a matching error.

Calculating a matching error epsilon (C, psi), wherein C is a matching matrix and corresponds to the first matching parameter; psi is the candidate surface deformation. i, j are two feature points from the first set of feature pointsA, b are two feature points (p) from the second set of feature points_a，p_b) Number of (2), C_i，aIs a characteristic pointMatching parameter of (C)_j，bIs a characteristic pointThe matching parameters of (1). Said C is_i，aAnd C_j，bCorresponding to the aforementioned second matching parameter. d (psi, i, j, a, b) is a matching pairAnd a matching pair (p)_a，p_b) The consistency of (c).

The candidate surface deformation may be updated in step a3 using the following function;

ψ^*deformation of the updated alternative curved surface; d_geo(ψ, i, j, a, b) is the geometric consistency of the matching pair (i, j) and the matching pair (a, b); e.g. of the type_ind(i，a)Is composed ofUnary projection error coding. And lambda is a penalty coefficient. s.t. represents a constraint. 0_m×nRepresenting an m × n all-zero matrix, 1_nRepresenting n column vectors of 1 components,. gtoreq.and. ltoreq.being greater and less than, respectively, at the element level,. gtoreq._i，jIndicating pointsAnd pointThe geodesic distance between them. The constraint on the matching relationship C ensures that each point can only participate in matching once at most. While the constraint on the morph psi is an inextensible constraint in order to prevent the euclidean distance between adjacent points from exceeding the limit. E_meshIs a set of edges formed by feature points.

In a4, the matching parameters may be updated according to the following formula:

c^*for the updated matching parameter, λ isPenalty factor, λ c^Te (ψ) is a penalty term; e (ψ) is a unary projection error; c is the matching parameter before updating, c^TIs the transposition of c; k (psi) is a matching pairAnd a matching pair (p)_a，p_b) The corresponding affinity matrix.

Bc≤1_m+nIs a one-to-one matching constraint. B can be obtained by:

b can be decomposed into two parts, B ═ B1; b2]Where B1 ∈ {0, 1}^m×mn，B2∈{0，1}^n×mn. B1 and B2 are defined as follows:

ind (,) represents a bijective function that maps a match between two nodes to an integer sequence number. B1_i，kLine i, the kth element of B1; b2_i，kThe jth line kth element of B2.

Through the iteration from the step A1 to the step A4, the curved surface deformation of the second image can be found, so that whether the second image is the image of the tracking target or not can be determined conveniently according to the curved surface deformation.

In some embodiments, the step a1 may include:

determining a matching probability of a first matching unit and a second matching unit, wherein the first matching unit comprises a plurality of first feature points, and the first feature points are feature points of the first feature point set; the second matching unit comprises a plurality of second feature points, and the second feature points are feature points of the second feature point set;

determining consistency of the first matching unit and the second matching unit;

based on the consistency, a match error is determined.

The match probability may be C in the foregoing formula_i，aAnd C_j，b。

The consistency may include: geometric consistency and/or appearance consistency. Appearance consistency can be used to measure the similarity or compatibility in appearance between feature points in two matching units; the geometric consistency can be used to measure the similarity or compatibility in geometric space between feature points in two matching units.

In some embodiments, the determining consistency of the first matching unit in the first feature point set and the second matching unit in the second feature point set includes:

determining the appearance consistency of the first matching unit and the second matching unit;

determining a geometric correspondence of the first matching unit and the second matching unit.

In this embodiment, the solution of the matching error is performed by combining the appearance consistency and the geometric consistency, so that the precise matching between two matching units is realized from two dimensions of the appearance and the geometric space, the accuracy of the surface deformation of the second image can be further improved, and the accuracy of the target tracking is improved.

In some embodiments, the determining the consistency of appearance of the first matching unit and the second matching unit comprises:

determining the appearance consistency from a first photometric descriptor and a second photometric descriptor, wherein the first photometric descriptor is a photometric descriptor of the first feature point; the second luminosity descriptor is a luminosity descriptor of the second feature point.

For example, the determination of the appearance consistency may utilize, but is not limited to, the following formula:

d_app(i, j, a, b) is the appearance uniformity;and f_aAre respectively a characteristic pointAnd p_aThe luminosity descriptor of (a); in the same way, the method for preparing the composite material,and f_bAre respectively a characteristic pointAnd p_bThe photometric descriptor of (1).

In some embodiments, the determining the geometric correspondence of the first matching unit and the second matching unit comprises:

determining the deformation of the first characteristic point based on the alternative curved surface deformation;

determining the geometric consistency based on the deformation of the first feature point.

For example, the geometric consistency may be determined using, but not limited to, the following formula:

d_geo(ψ, i, j, a, b) is the geometric uniformity.Is specially designed forSign pointDeformation under the curved surface deformation psi;characteristic pointDeformation under the curved surface deformation psi. τ is the mapping function:for combining each three-dimensionalMapping of points in a grid to 2DA point of the image. The external parameters of the camera are used to translate points in world coordinates into camera coordinates, and the internal parameters of the camera are used to further translate the camera coordinates into image coordinates.

In some embodiments, the determining, according to the candidate surface deformation and the matching parameter, a matching error of a matching unit in the first feature point set and the second feature point set includes:

determining a unary projection error between a single said second feature point and a single said first feature point;

and determining the matching error according to the unary projection error.

For example, the solving formula of the matching error can be written as ∈ (C, ψ) ═ C^TK(ψ)c+λc^Te (ψ); wherein e (ψ) is the unary projection error.

In some embodiments, the method further comprises:

determining a penalty coefficient according to the unary projection error and the compatibility between the second characteristic point and the first characteristic point;

calculating a penalty term based on the unary projection error and the penalty coefficient;

and determining the matching error according to the penalty item.

In the formula, epsilon (C, psi) ═ C^TK(ψ)c+λc^TAnd λ in e (ψ) is the penalty coefficient. Lambada c^Te (ψ) is the penalty term.

In some embodiments, the penalty factor may be predetermined or dynamically determined according to the current matching situation.

For example, the penalty factor may be dynamically determined using the following formula:

|K_i，j(ψ) | represents K_i，jAbsolute value of (ψ, | D^tI refers to the candidate match set D^tThe size of (2). K_i，j(psi) isAndconsistency of matching; n is the number of the characteristic points; sigma_ie_i(ψ) is an accumulated unary projection error of feature points in the first feature point set.

In some embodiments, if the matching error does not satisfy the convergence condition or does not satisfy the maximum iteration number, updating the candidate surface deformation of the second image according to the matching error, including:

and solving a relaxation problem of alternative curved surface deformation optimization based on the matching error so as to optimize the alternative curved surface deformation.

The relaxation problem solution may be: and (3) loosening discrete constraints or one-to-one mapping constraints in the graph matching process, wherein the loosened problem is called a loosened graph matching problem.

There are various ways to solve the relaxation problem of candidate surface deformation optimization based on the matching error, and an optional way is provided below, for example, an optimization solution that is minimized with respect to the matching relation c is performed by a method based on a Frank-Wolfe algorithm, and the optimization solution of candidate surface deformation is realized by the optimization solution of c.

Set,% ψ₀: given deformation; % Ω: a solution space of possible c; performing an optimization solution process for c may include:

1: initialization: computing matrix K (psi)₀) And vector e (psi)₀)

2: initialization: initializing the matching relation c to random

3: if c does not converge, executing the following steps:

4：g＝2K(ψ₀)c+e(ψ₀)

5：y＝arg min_yg^Ty，s.t.y∈Ω

6：β＝argmin_βε_λ(c+β(y-c))，s.t.0≤β≤1

7：c←c+β(y-c)

8: returning to the step 3;

9: and outputting c.

In some embodiments, the solving of the relaxation problem for candidate surface optimization based on the matching error to optimize the candidate surface deformation includes:

and solving the relaxation problem through linear programming based on the camera parameters for acquiring the second image and the distance between the projection point of the second characteristic point and the first characteristic point obtained based on the alternative curved surface deformation so as to optimize the alternative curved surface deformation.

For example, given a matching relationship C, (i.e., the correspondence matrix C), the optimization of candidate surfaces can be simplified to solve for the optimal deformation according to the following formula:

relaxing the first term in the above equation according to:

therefore, the optimization of the deformation of the candidate surface is relaxed as a linear fitting problem:

w_i，a＝C_i，a(∑_iC_i，a+∑_aC_i，a) + λ is the weight of each sample. The relaxation problem of the deformation of the candidate surface can be further restated as a well-conditioned linear system with respect to the coordinates of the mesh vertices:

where M is a coefficient matrix, A is a regularizer matrix, and r is a scalar coefficient that defines the degree of specification for the solution. M is the coordinate of the camera parameter, the characteristic point, the matching matrix c and the likeObtained through a series of calculations. x is the mesh vertex coordinates to be solved, e.g., x is the 3D coordinates of the mesh vertices to be solved; x is the number of^*Is the solved optimal mesh vertex coordinates.

In some embodiments, the step S120 may include:

and obtaining two-dimensional feature points for constructing the second feature point set according to the two-dimensional feature response of the pixels in the second image.

The second image is composed of a plurality of pixels having pixel values. For example, for an RGB image, the pixel value of one pixel includes an R value, a G value, and a B value. These values will produce corresponding two-dimensional responses in different image feature extraction algorithms. Therefore, in this embodiment, the two-dimensional feature points of the second feature point set can be constructed through the two-dimensional feature response of the pixels in the second image. The two-dimensional feature points here are feature points of a plane located within the image coordinate system. The two-dimensional feature point may be one of the feature points in the second feature point set.

Further, the obtaining two-dimensional feature points for constructing the second feature point set according to the two-dimensional feature response of the pixels in the second image includes:

determining a local extreme value of a predetermined image area in the second image based on a predetermined feature extraction algorithm;

and determining two-dimensional feature points for constructing the second feature point set according to the local extreme values.

For example, the feature extraction algorithm may employ, but is not limited to, a Scale-invariant feature transform (SIFT) algorithm, a Speeded Up Robust Features (SURF) algorithm, a Histogram of Oriented Gradients (HOG) algorithm, and the like. For example, determining local extrema of the predetermined image region in the second image using the SIFT algorithm may include: and obtaining the pixel point with the maximum response in the preset image area as a two-dimensional feature point by utilizing an SIFT algorithm. The local extremum here is a local maximum.

For another example, if the response of the pixel that can characterize the target to be tracked is the minimum after the feature extraction algorithm is processed, the pixel corresponding to the local minimum value may be used as the two-dimensional feature point.

As shown in fig. 4, the present embodiment provides an image processing apparatus including:

a first obtaining module 110, configured to obtain a first feature point set of a first image;

a second obtaining module 120, configured to obtain a second feature point set of a second image;

a matching module 130, configured to perform matching on feature points in the first feature point set and the second feature point set by using at least two feature points as a matching unit, so as to obtain matching parameters;

and the first determining module 140 is configured to determine the surface deformation of the second image according to the matching parameter.

In some embodiments, the first obtaining module 110, the second obtaining module 120, the matching module 130, and the first determining module 140 may be program modules; after being executed by the processor, the program module can realize the acquisition of the first characteristic point set and the second characteristic point set and the determination of the matching parameters and the curved surface deformation.

In other embodiments, the first obtaining module 110, the second obtaining module 120, the matching module 130, and the first determining module 140 may be a hardware and software module, which may include various programmable arrays; the programmable array includes but is not limited to a field programmable array, a complex programmable array, and the like.

In still other embodiments, the first obtaining module 110, the second obtaining module 120, the matching module 130, and the first determining module 140 may be pure hardware modules; including but not limited to application specific integrated circuits.

In other embodiments, the matching module 130 is specifically configured to perform matching on feature points in the first feature point set and the second feature point set by using a feature point pair formed by two feature points as a matching unit, and obtain a matching parameter indicating a matching condition between the feature point pairs.

In some embodiments, the apparatus further comprises:

a third obtaining module, configured to obtain a first matching parameter of single-point soft matching in the first feature point set and the second feature point set;

the matching module 130 is specifically configured to obtain a second matching parameter of a matching unit based on the first matching parameter by using two feature points as the matching unit.

In some embodiments, the third obtaining module is specifically configured to determine a matching probability of a single point matching between a first feature point in the first feature point set and a second feature point in the second feature point set.

In some embodiments, the third obtaining module is specifically configured to generate a matching matrix based on the matching probability.

In some embodiments, the matching module 130 is further specifically configured to perform matching between feature points in the first feature point set and feature points in the second feature point set based on the initial candidate surface deformation of the second image with at least two feature points as one matching unit, and select the second matching parameter from the first matching parameters, where the initial candidate surface deformation of the second image is the surface deformation of the first image.

In some embodiments, the first determining module 140 includes:

the first submodule is used for determining the matching error of a matching unit in the first characteristic point set and the second characteristic point set according to the deformation of the candidate curved surface and the matching parameter;

the second sub-module is used for stopping iteration if the matching error meets a convergence condition or reaches the maximum iteration times, and the alternative curved surface deformation obtained when the iteration is stopped is used as the final curved surface deformation output;

the third sub-module is used for updating the alternative curved surface deformation of the second image according to the matching error if the matching error does not meet the convergence condition or the maximum iteration times;

and the fourth sub-module is used for updating the matching parameters according to the alternative curved surface deformation.

In some embodiments, the first sub-module is specifically configured to determine a matching probability of a first matching unit and a second matching unit, where the first matching unit includes a plurality of first feature points, and the first feature points are feature points of the first feature point set; the second matching unit comprises a plurality of second feature points, and the second feature points are feature points of the second feature point set; determining consistency of the first matching unit and the second matching unit; based on the consistency, a match error is determined.

In some embodiments, the first submodule is further configured to determine an appearance consistency of the first matching unit and the second matching unit; determining a geometric correspondence of the first matching unit and the second matching unit.

In some embodiments, the first submodule is specifically configured to determine the appearance consistency according to a first luminosity descriptor and a second luminosity descriptor, where the first luminosity descriptor is a luminosity descriptor of the first feature point; the second luminosity descriptor is a luminosity descriptor of the second feature point.

In some embodiments, the first sub-module is specifically configured to determine a deformation of the first feature point based on the candidate surface deformation; determining the geometric consistency based on the deformation of the first feature point.

In some embodiments, the first sub-module is specifically configured to determine a unary projection error between a single second feature point and a single first feature point; and determining the matching error according to the unary projection error.

In some embodiments, the apparatus further comprises:

the second determining module is used for determining a penalty coefficient according to the unary projection error and the compatibility between the second characteristic point and the first characteristic point;

the calculation module is used for calculating a penalty item based on the unary projection error and the penalty coefficient;

the first sub-module is specifically configured to determine the matching error according to the penalty term.

In some embodiments, the third sub-module is specifically configured to perform a relaxation problem solution of candidate surface deformation optimization based on the matching error, so as to optimize the candidate surface deformation.

In some embodiments, the third sub-module is specifically configured to perform the relaxation problem solving by linear programming based on the camera parameters acquired from the second image and the distance between the projection point of the second feature point and the first feature point obtained based on the candidate surface deformation, so as to optimize the candidate surface deformation.

In some embodiments, the second obtaining module 120 is specifically configured to obtain the two-dimensional feature points for constructing the second feature point set according to the two-dimensional feature response of the pixels in the second image.

In some embodiments, the second obtaining module 120 is specifically configured to determine a local extreme value of a predetermined image region in the second image based on a predetermined feature extraction algorithm; and determining two-dimensional feature points for constructing the second feature point set according to the local extreme values.

In some embodiments, the apparatus further comprises:

the generating module is used for generating a candidate matching set containing a plurality of potential matches based on feature point matching taking two feature points as a matching unit;

a filtering module for filtering out potential matches in the candidate match set that do not satisfy a constraint;

the matching module is specifically configured to determine, from the first matching parameters, second matching parameters of potential matches in the filtered candidate matching set.

In some embodiments, the filtering module determines a deformation of a first feature point in the filtered candidate matching set based on a surface deformation of a previous frame; determining whether the potential match satisfies a geometric constraint condition based on the deformation of the first feature point and the distance between the second feature points;

determining a photometric descriptor of a first feature point and a photometric descriptor of the second feature point, determining whether the potential match satisfies an appearance constraint;

filtering out the potential matches in the candidate match set that do not satisfy the geometric constraint and/or appearance constraint.

Several specific examples are provided below in connection with any of the embodiments described above:

example 1:

the present example provides an image processing method usable for target tracking, comprising:

acquiring a first feature point set of a reference image, wherein the reference image can be a first frame image or a previous frame image and can correspond to the first image;

acquiring a second characteristic point set of a currently input image of the current frame, wherein the image of the current frame is the second image;

taking the curved surface deformation finally obtained by optimizing the previous frame image as the initial alternative curved surface deformation of the current frame image, and calculating the point pair matching of the first characteristic point set and the second characteristic point set;

based on point pair matching, obtaining a projection error of the point pair matching, and reconstructing alternative curved surface deformation; the projection error here may be a matching error between the aforementioned two matching units;

determining the projection error based on the reconstructed candidate surface deformation;

judging whether the optimal matching is found or the maximum iteration number is reached based on the projection error;

and if the optimal matching is obtained or the maximum iteration times are reached, outputting the alternative curved surface deformation when the iteration is stopped as the final curved surface deformation of the image of the frame, and if the optimal matching is not obtained or the iteration times are not reached, continuously calculating the point pair matching of the first characteristic point set and the second characteristic point set.

Example 2:

restoring the shape of the object having the non-rigid surface in the second image may comprise three steps, in particular as follows:

(1) the feature points correspond to: establishing a feature point matching relationship using local texture information calculated from a feature point descriptor algorithm;

(2) outlier rejection: measuring the geometric compatibility of the deformable model to eliminate incorrect matching relation;

(3) shape reconstruction, here shape reconstruction is equivalent to obtaining a surface deformation: the non-rigid shape of the target surface is estimated based on the known templates and the established feature point matching relationships.

Feature point correspondence refers to extracting feature points from a given image and then associating the feature points with the feature points in a nearest neighbor manner by a suitable distance metric. In detecting feature points, feature point detectors and descriptors (e.g., SIFT and SURF) are designed to be highly robust to changes in scale and rotation.

In this example, rather than utilizing the traditional unary projection errors between feature point sets, feature point correspondences and shape reconstructions are determined that are modeled from pairwise projection errors between graph structures. The paired projection is to match the feature points in different images by using two feature points as a matching unit.

An effective graph matching algorithm is developed under soft matching relaxation, an accurate and effective tracking effect is provided for a deformable surface target, and excellent experimental results clearly prove the advantages of work.

Unlike the conventional methods of processing the feature matching relationships separately, removing outliers, and shape reconstruction in the related art, these processes are integrated into a unified graph-based framework and an optimization problem is proposed that iteratively solves the solution matching relationships and solves the distortion. The strong matching constraint in the conventional graph matching problem is relaxed to a loose matching constraint in view of computational efficiency. This loose-match constraint enables more matching details to be retained, resulting in more accurate shapes, and also greatly improves computational efficiency through novel matching algorithms developed under soft-match constraints.

In this example, the tracking algorithm for the graph-based deformable surface target is advantageous in at least the following respects: introducing graph model and graph matching into deformable surface tracking through soft matching relaxation and well-designed candidate matched filtering strategies; and designing a unified optimization framework, and exploring all information of local appearance, spatial relation and a deformation model to obtain accurate shape reconstruction.

One, concrete algorithm

Representing known template shapes with triangulated meshesThis grid passes through N_VDotConsisting of N_eEdge set E composed of edges_meshAnd (4) connecting. Pressing points described in the camera reference frame (initial template) into a vectorIn (1). The known template is associated with the unknown morphed shape S by an unknown 3D continuously differentiable morph ψ, i.e. ψ willOne point in S is mapped into S. Similarly, N may be used_VA point v with unknown 3D coordinates_iTo represent the shape S and to press these points into a vectorThis vector is needed to be solved in the algorithm. It is assumed that the camera has been calibrated, with known intrinsic and extrinsic parameters. That is, there is a known mapping function τ:the points in each 3D mesh are mapped to points of the 2D image.

And P ═ P₁，…，p_nAre feature point sets extracted from the first image and the second image, respectively. For simplicity, for each feature point(and p_jE P) and also represent its homogeneous coordinates in the 2D image using the same notation. Since the 3D surface of the first image is known, for each feature pointCan calculate its 3D site

The unary matching relation of the characteristic points in the Pr and P point sets is represented by a matrixRepresenting, each element C in the matrix_i，j∈[0，1]To representAnd p_jThe matching probability of (2). The matrix C is the matching probability obtained from the soft matching relationship between the feature points. Soft-matching relationships enable more corresponding details to be maintained, thereby improving the accuracy of the recovered 3D shape, with another benefit of soft-matching in that subsequent quadratic programming problems are more easily solved by discarding discrete constraints, enabling reduced computational complexity and improved computational efficiency.

Minimizing the cost function ε (C, ψ) by solving C and ψ simultaneously can result in the optimal shape S being reconstructed:

wherein, 0_m×nRepresenting an m × n all-zero matrix, 1_nRepresenting n column vectors of 1 components,. gtoreq.and. ltoreq.being greater and less than, respectively, at the element level,. gtoreq._i，jIndicating pointsAnd pointThe geodesic distance between them. The constraint on the matching relationship C ensures that each point can only participate in matching once at most. While the constraint on the morph psi is an inextensible constraint in order to prevent the euclidean distance between adjacent points from exceeding the limit.

In the related art, the cost function ε (C, ψ) is often defined as all matching relationships under the morph ψThe accumulated error of (2). In an example, a graph-based metric is proposed that combines projection errors between graphical structures into:

where d (psi, i, j, a, b) is a measure below the morph psiAnd edge (p)_a，p_b) Consistency between them. Defining d as an appearance consistency function d_appAnd a geometric consistency function d_geoEach function is:

d(ψ，i，j，a，b)＝(1-α)d_app(i，j，a，b)+αd_geo(ψ，i，j，a，b)(3)

whereinAnd f_aAre respectively a characteristic pointAnd p_aAnd α e [0, 1 ]]The balance between local features and the graph structure used to reconstruct the shape is controlled.

For simplicity, equation (2) can be expressed in a point-to-point compatibility manner:

ε(C，ψ)＝c^TK(ψ)c (4)

whereinIs in the form of a vector of the C matrix,is the corresponding affinity matrix:

K_{ind(i，a)，ind(j，b)}(ψ)＝d(ψ，i，j，a，b)-κ (5)

wherein (i, a) represents a point in the first imageWith a point p in the second image_aA candidate match, ind (-) is a bijective function that maps point-match relationships to an integer index.

κ is chosen to be large enough to ensure that K (ψ) is non-positive, with the aim of avoiding trivial solutions solved because no matching relationship is activated.

In order to filter outlier matches with large projection errors under the warp ψ, the matching points are penalized by a projection error term, the projection error increases with increasing matching points;

ε(C，ψ)＝c^TK(ψ)c+λc^Te(ψ)， (6)

where lambda > 0 adaptively controls the degree to which outliers are rejected,the unary projection error for each point match is encoded as:

second, optimization solution

For a new frame (i.e. the image of the current frame to be processed), c and ψ are first predicted from the solution of the previous frame, and then the other term is optimized by alternately fixing one of the terms. This optimization process iterates until convergence or a maximum number of iterations of the algorithm is reached.

The optimal solution for the matching relationship may be as follows:

given the alternative morph ψ, the problem (1) is reduced to solve the optimal matching relationship as follows:

wherein Bc is less than or equal to 1_m+nIs a one-to-one matching constraint.

The solution of the updated matching parameters in equation (8) can be considered as a relaxation graph matching problem by removing the discrete constraints and adding penalty terms. Although some power-iterative algorithms for solving the conventional graph matching problem can be easily extended to solve the soft matching relationship c, these extended algorithms are also difficult to apply to the problem (8) due to the existence of penalty terms. In this section, a method based on the Frank-Wolfe algorithm is proposed to minimize the problem with respect to the matching relation c (8), which is described in algorithm 1.

The optimal solution for the deformation model may be as follows:

given a matching relationship C, (i.e., the correspondence matrix C), the solution of equation (1) can be simplified to solve for the optimal deformation according to the following equation:

the first term of equation (9) is relaxed according to:

equation (9) is therefore relaxed as a linear fitting problem:

wherein

Is the weight of each sample.

The solution of equation (12) can be further restated as a well-conditioned linear system with respect to the mesh vertex coordinates:

where M is a coefficient matrix, A is a regularization matrix, and r is a scalar coefficient that defines the degree of specification for the solution. x is the mesh vertex coordinates to be solved, e.g., x is the 3D coordinates of the mesh vertices to be solved; x is the number of^*Is the solved optimal mesh vertex coordinates.

Example 3:

the present example provides an image processing method based on example 1 and/or example 2, including: graph construction, candidate match filtering and adaptive outlier rejection.

The structure of the drawing:

an undirected graph with n nodes can be represented asWherein Respectively representing a point set and an edge set. Initial region of target surface of interest in given reference pictureA model map is created for the surface as follows. One node here can be considered as one feature point in the image.

And (3) node generation: feature points are typically extracted from the image to represent local regions, which are then modeled as vertices of a graph. Many feature-based methods obtain feature points as local minima/maxima of the cross-scale DoG image, i.e. SIFT features. However, the number of feature points obtained by this method cannot be controlled, so how many feature points are obtained depends on the operator and the content in the video frame. Furthermore, they are often sensitive to environmental changes, such as illumination changes and motion blur, thus compromising tracking accuracy.

In this example, a more robust and flexible method is used to extract feature points and generate vertices;

firstly, the following components are mixedEvenly divided into N grids, and SIFT response of each pixel in each grid is calculated.

Then, the feature point having the largest response per mesh is selected, and the selected feature points are regarded as the graph vertices.

Finally, the SIFT descriptors of these feature points are recorded as vertex attributes.

There are several alternative paradigms for edge generation, including but not limited to: full connectivity graphs, epsilon adjacency graphs, and k-nearest neighbor graphs. Fully-connected graphs have high computational complexity and are therefore not suitable for large-size graphs.

In this example, the Delaunay triangulation is preferentially used for edge generation, so as to construct a stable graph structure with invariance to scaling, translation, and rotation.

For each new frame t, the candidate map is constructed in the same wayThe match between feature points is then determined by graph matching.

And (3) potential matching filtering:

there are N vertices on each of the model map (corresponding to the first image) and the candidate map (corresponding to the second image), so there is a total of N between the vertices on the two maps²Is matched with the candidate. The size K (ψ) of the affinity matrix is thus N⁴Then large, which not only results in significant computational consumption on storage.

To improve computational efficiency, it is proposed to reduce the size of K (ψ) by filtering candidate matches under a reasonable continuity assumption. In particular, unreliable matches that result in large jumps in matches between consecutive frames are eliminated from the candidate match set. For a new frame t, for each vertexCandidate matches are constructed by geometric constraints and photometric constraints:

wherein epsilon_gAnd e_aRespectively, geometric and appearance variation tolerance thresholds. In addition, fromRemoving redundant matches and retaining at most n_cThe match with the largest appearance similarity. The final candidate match set is composed of the combination of the match sets of all vertices:

constructed D^tFollowed by deleting thoseThe compression is effected by the rows and columns in time, so that the size of the affinity matrix K (ψ) is compressed to at mostEmpirically set the parameters to ε in the experiment_g＝20，∈_a＝0.6，n_c＝5。

Adaptive noise removal:

the adaptive noise removing method integrates feature point matching, outlier rejection and shape reconstruction into a unified optimization framework of a formula (6) through a penalty term lambdac^Te (ψ) to drive rejection of outliers, where λ > 0 controls the degree to which outliers are rejected. It is often difficult to select a suitable lambda to reject outliers. Too small a does not bring about the de-noising effect, while too large a may reject many correct matches as outliers.

In this example, an outlier rejection method is also proposed that adaptively adjusts λ according to the affinity matrix K (ψ) and the projection error e (ψ):

wherein, | K_i，j(ψ) | represents K_i，jAbsolute value of (ψ, | D^tI refers to the candidate match set D^tThe size of (2). Is thus self-adaptiveThe objective should be to choose a suitable λ to avoid that any of the equations (6) will dominate the whole optimization process.

In summary, the present example first provides a new graph-based approach to solving the deformable surface reconstruction and tracking problem. The process of processing feature matching relationships, removing outliers, and shape reconstruction is integrated into a unified graph-based framework (feature correspondences are solved simultaneously by optimizing an objective function defined by pairwise projection errors between graph structures rather than unary projection errors between matching points, outlier removal, and shape recovery), and the optimization problem of solving matching relationships and solving deformations iteratively is proposed.

Second, the strong matching constraint in the traditional graph matching problem is relaxed to a loose matching constraint in view of computational efficiency. Such loose-matching constraints can retain more matching details, resulting in more accurate shapes, and computational efficiency is also greatly improved by novel matching algorithms developed under soft-matching constraints.

Again, given the candidate match filtering strategy, the present example graph-based approach is able to process thousands of points in a few seconds, which is much faster than traditional graph-based algorithms.

This example presents a novel graph-based deformable surface target tracking method aimed at improving tracking performance and efficiency. The proposed method solves the feature correspondence and shape recovery problems through pairwise projection errors between graph structures and improves computational efficiency using soft-matching relaxation. The method used on the standard dataset with the closed surface and the dataset with the rich, weak or repeated texture of different surfaces is widely compared with the existing advanced algorithm, and the experimental result shows that the method provided by the example has accurate and stable tracking performance on various types of curved surfaces, can realize stable tracking result on the curved surfaces with different types of texture, and is superior to the latest advanced algorithm in the aspects of tracking precision and computing efficiency.

To fully evaluate the comparison of the method provided in this example with the reference algorithm, a new data set is created for deformable surface tracking, called deformable surface tracking (DesurT). This data set was collected using a Kinect camera to evaluate tracking performance under various deformations and different lighting conditions. It contains 11 video streams and 3,361 frames showing variations of several different types of surfaces, including seven printed images of different content (campus, brick, cloth, cobblestone, scenery, stone, and sunset, respectively), two newspapers and two cushions.

As shown in fig. 5, these surfaces are roughly classified into three categories: (1) surfaces with good texture, including campus, cobblestone, landscape, newspaper 1, newspaper 2 and cushion 1; (2) repeating textured surfaces, including tiles, cloth and mats 2; (3) weakly textured surfaces, including stones and sunset.

To evaluate the reconstruction accuracy, a real mesh is constructed using the Kinect point cloud, and the average distance from the reconstructed mesh to the vertices in the real mesh is calculated. Thus, all videos have real mesh vertices manually labeled in each frame (130 vertices are used in print pictures and newspapers, and seat cushions are labeled with 121 vertices), except for depth information for each frame.

In order to test the robustness of the method provided by this example to occlusions, the tracking results of the method provided by this example on a common dataset (tracking surfaces with occlusions) comprising a video stream of two deformable surface objects with good and bad texture, respectively, for a total of 394 frames, and artificial and realistic occlusions in the dataset were also reported.

And (3) comparing and analyzing results:

the following shows the results of the method provided by this example compared to several most advanced baseline algorithms, including DIR, LM, and LLS:

the LM performs feature correspondence using SIFT matching, then performs an iterative outlier rejection step, and then reconstructs the shape by solving a linear system that is transformed from a degenerate linear system using an extended Laplace form.

LLS only focuses on the shape reconstruction step and takes the keypoint correspondences as input. In this experiment, keypoints derived from LM were used as inputs to the LLS after outlier rejection.

The DIR is a pixel-based method that uses dense template alignment for shape reconstruction, which depends largely on the initial estimate of the shape, which is initialized to the solution of the previous frame in this experiment, for the method provided by this example, predetermined values such as α -0.7 are fixed, and two sets of experimental results are published, N-1000 and N-2000, respectively.

TABLE 1 average tracking error (mm)

TABLE 2 average calculation time(s)

As shown in Table 1, the method provided by this example is robust to different types of surfaces with rich, weak or repetitive textures, and is significantly superior to all baseline algorithms even when relatively few feature points (N1000) are extracted from each surface to build a matching relationship. For occlusion surfaces (TSO datasets), the DIR achieves the best tracking results with the help of a well-designed occlusion detection strategy. Interestingly, without any specified occlusion surface process, the approach provided by this example achieved comparable results to DIR on TSO datasets and was generally superior to LM and LLS. When N is raised to 2000, the tracking accuracy of the method provided by this example is significantly improved on both datasets.

Given the calculation time (Table 2), DIRs are the most time consuming on both datasets. The method provided by this example defeats other algorithms on both datasets when N1000. When increasing the number of feature points to 2000, the approach provided by the present example is still most efficient on TSO datasets, but slower than LM on the proposed DeSurT dataset.

Fig. 6-9 illustrate several representative samples of various types of surface tracking provided by the comparison algorithm. For well-structured surfaces (fig. 6), all algorithms can provide reasonable tracking results, but the method provided by the present example can better handle the details. As shown in fig. 7 and 8, all comparative baseline algorithms were affected by weak textures and repeating textured surfaces, but the method provided by the present example is able to provide accurate tracking results from frame to frame. Furthermore, the method and DIR provided by this example are robust to occlusions (FIG. 9), whereas LM and LLS may not be able to track objects when there is some degree of occlusion.

To summarize, the present example proposes a graph-based deformable surface target tracking method, aiming at improving tracking performance and efficiency. The proposed method solves the feature correspondence and shape recovery problems through pairwise projection errors between graph structures and improves computational efficiency using soft-matching relaxation. Experimental results show that the algorithm has accurate and robust tracking performance on various types of curved surfaces and is superior to the latest most advanced algorithm.

The present embodiment also provides an image apparatus including:

a memory;

and a processor connected with the memory and used for realizing the image processing method provided by one or more of the foregoing embodiments through computer-executable instructions, for example, as one or more of the image processing methods shown in fig. 1 to 3.

The present embodiments also provide a computer-readable storage medium having stored thereon computer-executable instructions; the computer-executable instructions, when executed by a processor, enable the image processing methods provided by one or more of the foregoing embodiments, for example, one or more of the methods shown in fig. 1-3. The computer readable storage medium may be a non-transitory storage medium, which may also be referred to as a non-transitory storage medium, and typical non-transitory storage media include, but are not limited to, flash memory.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software programs of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including associated data structures) of the present application may be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.

In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Program instructions which invoke the methods of the present application may be stored on a fixed or removable recording medium and/or transmitted via a data stream on a broadcast or other signal-bearing medium and/or stored within a working memory of a computer device operating in accordance with the program instructions. An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform a method and/or a solution according to the aforementioned embodiments of the present application.

It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.

Claims (28)

1. An image processing method comprising:

acquiring a first feature point set of a first image;

acquiring a second feature point set of a second image;

matching the feature points in the first feature point set and the second feature point set by taking at least two feature points as a matching unit to obtain matching parameters;

and determining the curved surface deformation of the second image according to the matching parameters.

2. The method according to claim 1, wherein the matching of feature points in the first feature point set and the second feature point set is performed with at least two feature points as one matching unit to obtain matching parameters, and the method includes:

and matching the characteristic points in the first characteristic point set and the second characteristic point set by taking a characteristic point pair formed by two characteristic points as a matching unit to obtain a matching parameter indicating the matching condition between the characteristic point pairs.

3. The method of claim 2, further comprising:

acquiring a first matching parameter of single-point soft matching in the first characteristic point set and the second characteristic point set;

the matching of the feature points in the first feature point set and the second feature point set is performed by taking the two feature points as a matching unit to obtain matching parameters, and the matching parameters include:

and taking two feature points as a matching unit, and obtaining a second matching parameter of the matching unit based on the first matching parameter.

4. The method of claim 3, wherein obtaining the first matching parameters for the single-point soft matching in the first feature point set and the second feature point set comprises:

and determining the matching probability of the single point matching of the first feature point in the first feature point set and the second feature point in the second feature point set.

5. The method according to claim 4, wherein the obtaining of the first matching parameters of the single-point soft matching in the first feature point set and the second feature point set further comprises:

and generating a matching matrix based on the matching probability.

6. The method according to claim 3, wherein the obtaining a second matching parameter of the matching unit based on the first matching parameter with two feature points as one matching unit comprises:

and matching the feature points in the first feature point set and the second feature point set based on the initial candidate surface deformation of the second image by taking at least two feature points as a matching unit, and selecting the second matching parameter from the first matching parameters, wherein the initial candidate surface deformation of the second image is the surface deformation of the first image.

7. The method of claim 1, wherein determining the surface deformation of the second image according to the matching parameters comprises:

step A1: determining the matching error of a matching unit in the first characteristic point set and the second characteristic point set according to the deformation of the candidate curved surface and the matching parameters;

step A2: stopping iteration if the matching error meets a convergence condition or reaches the maximum iteration times, and outputting the alternative curved surface deformation obtained when the iteration is stopped as the final curved surface deformation;

step A3: if the matching error does not meet the convergence condition or the maximum iteration number, updating the alternative curved surface deformation of the second image according to the matching error;

step A4: and updating the matching parameters according to the alternative surface deformation, and returning to the step A1.

8. The method according to claim 7, wherein the determining the matching error of the matching unit in the first feature point set and the second feature point set according to the candidate surface deformation and the matching parameter comprises:

determining a matching probability of a first matching unit and a second matching unit, wherein the first matching unit comprises a plurality of first feature points, and the first feature points are feature points of the first feature point set; the second matching unit comprises a plurality of second feature points, and the second feature points are feature points of the second feature point set;

determining consistency of the first matching unit and the second matching unit;

based on the consistency, a match error is determined.

9. The method of claim 8, wherein determining the correspondence between the first matching unit in the first set of feature points and the second matching unit in the second set of feature points comprises:

determining the appearance consistency of the first matching unit and the second matching unit;

determining a geometric correspondence of the first matching unit and the second matching unit.

10. The method of claim 9, wherein determining the consistency of appearance of the first matching unit and the second matching unit comprises:

determining the appearance consistency from a first photometric descriptor and a second photometric descriptor, wherein the first photometric descriptor is a photometric descriptor of the first feature point; the second luminosity descriptor is a luminosity descriptor of the second feature point.

11. The method of claim 9, wherein determining the geometric correspondence of the first matching unit and the second matching unit comprises:

determining the deformation of the first characteristic point based on the alternative curved surface deformation;

determining the geometric consistency based on the deformation of the first feature point.

12. The method according to any one of claims 7 to 11, wherein the determining the matching error of the matching unit in the first feature point set and the second feature point set according to the candidate surface deformation and the matching parameter comprises:

determining a unary projection error between a single said second feature point and a single said first feature point;

and determining the matching error according to the unary projection error.

13. The method of claim 12, further comprising:

determining a penalty coefficient according to the unary projection error and the compatibility between the second characteristic point and the first characteristic point;

calculating a penalty term based on the unary projection error and the penalty coefficient;

and determining the matching error according to the penalty item.

14. The method of claim 7, wherein if the matching error does not satisfy a convergence condition or does not satisfy a maximum number of iterations, updating the candidate surface deformation of the second image according to the matching error, comprising:

and solving a relaxation problem of alternative curved surface deformation optimization based on the matching error so as to optimize the alternative curved surface deformation.

15. The method of claim 14, wherein solving a relaxation problem for candidate surface optimization based on the matching error to optimize the candidate surface deformation comprises:

and solving the relaxation problem through linear programming based on the camera parameters for acquiring the second image and the distance between the projection point of the second characteristic point and the first characteristic point obtained based on the alternative curved surface deformation so as to optimize the alternative curved surface deformation.

16. The method of claim 3, further comprising:

generating a candidate matching set containing a plurality of potential matches based on feature point matching with two feature points as a matching unit;

filtering out potential matches in the candidate match set that do not satisfy a constraint;

the obtaining a second matching parameter of the matching unit based on the first matching parameter by taking the two feature points as a matching unit includes:

second match parameters of potential matches in the filtered candidate match set are determined from the first match parameters.

17. The method of claim 16,

filtering out potential matches in the candidate match set that do not satisfy a constraint, including:

determining the deformation of a first feature point in the candidate matching set based on the surface deformation of the previous frame; determining whether the potential match satisfies a geometric constraint condition based on the deformation of the first feature point and the distance between the second feature points;

determining a photometric descriptor of a first feature point and a photometric descriptor of the second feature point, determining whether the potential match satisfies an appearance constraint;

filtering out the potential matches in the candidate match set that do not satisfy the geometric constraint and/or appearance constraint.

18. The method of claim 1 or 2, wherein said obtaining a second set of feature points for a second image comprises:

and obtaining two-dimensional feature points for constructing the second feature point set according to the two-dimensional feature response of the pixels in the second image.

19. The method of claim 18, wherein obtaining two-dimensional feature points from a two-dimensional feature response of pixels in the second image that construct the second set of feature points comprises:

determining a local extreme value of a predetermined image area in the second image based on a predetermined feature extraction algorithm;

and determining two-dimensional feature points for constructing the second feature point set according to the local extreme values.

20. An image processing apparatus comprising:

the first acquisition module is used for acquiring a first feature point set of a first image;

the second acquisition module is used for acquiring a second feature point set of a second image;

the matching module is used for matching the feature points in the first feature point set and the second feature point set by taking at least two feature points as a matching unit to obtain matching parameters;

and the first determining module is used for determining the curved surface deformation of the second image according to the matching parameters.

21. The apparatus according to claim 20, wherein the matching module is specifically configured to perform matching on feature points in the first feature point set and the second feature point set by taking a feature point pair formed by two feature points as a matching unit, and obtain a matching parameter indicating a matching condition between the feature point pairs.

22. The apparatus of claim 21, further comprising:

a third obtaining module, configured to obtain a first matching parameter of single-point soft matching in the first feature point set and the second feature point set;

the matching module is specifically configured to obtain a second matching parameter of the matching unit based on the first matching parameter by using two feature points as a matching unit.

23. The apparatus of claim 20, wherein the first determining module comprises:

the first submodule is used for determining the matching error of a matching unit in the first characteristic point set and the second characteristic point set according to the deformation of the candidate curved surface and the matching parameter;

the second sub-module is used for stopping iteration if the matching error meets a convergence condition or reaches the maximum iteration times, and the alternative curved surface deformation obtained when the iteration is stopped is used as the final curved surface deformation output;

the third sub-module is used for updating the alternative curved surface deformation of the second image according to the matching error if the matching error does not meet the convergence condition or the maximum iteration times;

and the fourth sub-module is used for updating the matching parameters according to the alternative curved surface deformation.

24. The apparatus according to claim 23, wherein the first sub-module is specifically configured to determine a matching probability of a first matching unit and a second matching unit, wherein the first matching unit includes a plurality of first feature points, and the first feature points are feature points of the first feature point set; the second matching unit comprises a plurality of second feature points, and the second feature points are feature points of the second feature point set; determining consistency of the first matching unit and the second matching unit; based on the consistency, a match error is determined.

25. The apparatus according to claim 23, wherein the third sub-module is specifically configured to perform a relaxation problem solution for candidate surface deformation optimization based on the matching error, so as to optimize the candidate surface deformation.

26. The apparatus of claim 22, further comprising:

the generating module is used for generating a candidate matching set containing a plurality of potential matches based on feature point matching taking two feature points as a matching unit;

a filtering module for filtering out potential matches in the candidate match set that do not satisfy a constraint;

the matching module is specifically configured to determine, from the first matching parameters, second matching parameters of potential matches in the filtered candidate matching set.

27. A computer-readable storage medium having stored thereon computer-executable instructions; the computer-executable instructions, when executed by a processor, are capable of performing the method as provided by any one of claims 1 to 19.

28. An image device, comprising:

a memory for storing at least computer-executable instructions;

a processor coupled to the memory for enabling implementation of the method provided in any one of claims 1 to 19 by executing the computer-executable instructions.