CN106846279A - A kind of adapting to image method for repairing and mending and its system based on interpolation by continued-fractions technology - Google Patents

Abstract

The present invention relates to a kind of adapting to image method for repairing and mending and its system based on interpolation by continued-fractions technology, the defect that repairing effect is poor, efficiency is low is solved compared with prior art.The present invention is comprised the following steps：Initialisation image signature analysis；The repairing of cut image breaking point is carried out using unitary interpolation by continued-fractions technology；Repairing for cut image is carried out using binary interpolation by continued-fractions technology, using initial repairing figure as A as frame, the position of each breaking point is determined by mask artwork, the Pixel Information for obtaining each breaking point is updated using binary interpolation by continued-fractions by the Pixel Information around breaking point, final repairing figure is obtained as B.The present invention improves the quality and efficiency of image mending, can change any type of cut, it is adaptable to all of image procossing.

Description

Adaptive image patching method and system based on continuous interpolation technology

Technical Field

The invention relates to the technical field of image processing, in particular to a self-adaptive image repairing method and a self-adaptive image repairing system based on a continuous fractional interpolation technology.

Background

The image restoration is to repair the damaged part of the image in a certain way to form a complete image. Image restoration has a wide range of applications in the industries of image processing, film industry and the like, and scratch removal in photos and old films, character and obstruction removal in images and the like are all related to image restoration. Therefore, image restoration has great application prospect, and is a research hotspot of computer vision and computer graphics at present.

At the present stage, many researchers have proposed different image inpainting methods with certain success, but these methods lack versatility, inpainting is not complete, and the boundary processed at the same time is fuzzy. Among them, the method based on partial differential equation and the method based on texture synthesis are the most common repairing methods at present. The image restoration method based on partial differential equation is characterized in that information around a restoration area is iterated to the area to be restored along the direction of an isophote of a pixel point in an image until the image is completely filled. The method based on partial differential equation has certain repairing effect, but lacks stability, and cannot obtain good effect especially when processing texture maps with large background. The texture synthesis method is another repair method, and a Markov random field model is adopted in texture synthesis for the first time. Practice proves that the method based on texture synthesis cannot solve the problem of boundary ambiguity in processing, and therefore cannot be practically applied.

For example, as shown in fig. 3, fig. 3 is an image to be patched.

1. The repairing is performed by using the methods of documents [1] and [2] ([1] Xing Huo, joining Tan, and MinHu. an automatic video scratch removal based on third type connectivity, Multimedia Tools and Applications, vol.71, No.2, pp.451-467,2014.[2] Xing Huo, joining Tan.A novel non-linear method of automatic video scratch removal, Proceedings-4th International reference Home, pp.39-45,2012.) the repairing method is performed by using a continuous repairing method (i.e. the latest method for performing image repairing by using a continuous method, specifically, the algorithm is described in [1] and [2], wherein the repairing method is only an improvement of the scratch processing method of documents [1] and [2], as shown in the figure 5).

2. After the image is processed by using the binary continuous method of the document [3], the results are shown in fig. 6, ([3] xingguo, Jieqing tan. binary temporal interpolant in image inpainting, Journal of information and Computational Science, vol.2, No.3, pp.487-492,2005. the document [3] is the latest method for image inpainting by using binary continuous method at present), which can better inpaint the scratch compared with the method of the binary continuous method, but the effect of the whole image inpainting is not good and the image is blurred.

Therefore, aiming at the limitations of various repair technologies at present, how to design an efficient and simple repair method under the existing hardware condition has become a technical problem which needs to be solved urgently at present.

Disclosure of Invention

The invention aims to solve the defects of poor repairing effect and low efficiency in the prior art, and provides a self-adaptive image repairing method based on a continuous interpolation technology and a system thereof to solve the problems.

In order to achieve the purpose, the technical scheme of the invention is as follows:

a self-adaptive image patching method based on a continuous fraction interpolation technology comprises the following steps:

initializing image characteristic analysis, analyzing an input scratch image, and judging whether the scratch image is a gray image or a color image; if the color image is a gray image, the color image is executed along R, G, B three color channels;

repairing damaged points of a scratch image by using a unitary continuous fractional interpolation technology, acquiring the position of each point to be repaired in the corresponding scratch image through an input mask image, detecting the points to be repaired line by taking an information matrix of the scratch image as a target, adaptively selecting surrounding known pixel points as sampling points for each point to be repaired, and reconstructing pixel information of each point to be repaired by combining the sampling points with unitary continuous fractional rational interpolation to obtain an initial repaired image A;

and (3) repairing the scratch image by using a binary continuous fractional interpolation technology, determining the position of each damaged point by using the initial repaired image A as an information image and a mask image, and updating pixel information of each damaged point by using pixel information around the damaged point by using binary continuous fractional interpolation to obtain the final repaired image B.

The repairing of the scratch image breakage point by using the one-element continuous fraction interpolation technology comprises the following steps:

determining a damage point of a scratch image, inputting a mask image of the scratch image, overlapping the mask image and the scratch image, and acquiring a to-be-repaired point of the scratch image;

adaptively selecting interpolation sampling points, reading a damage information matrix of the scratch image, adaptively selecting 4 known pixel points nearest to a point (x, y) to be repaired, and forming the 4 known pixel points into the interpolation sampling points;

the calculation of the pixel value of the damaged point, which is calculated by combining 4 known pixel points with an elementary run-length interpolation function, comprises the following steps:

the unitary run-length interpolation format is defined as:

wherein, b_i＝φ[x₀,x₁,…,x_i；y](i-0, …, m) is the function f (x, y) at point x₀,x₁,…,x_iM is the length of the input image, and satisfies the following condition:

φ[x_i；y]＝f(x_i；y),i＝0,1,2,…,m,

calculating the pixel value R of the point to be repaired by adopting the information of 4 known pixel points around the point (x, y) to be repaired and combining an elementary continuous interpolation function₁(x,y)；

Wherein,

b₀＝φ[x₀；y]＝f(x₀；y)，

in the above formula, f (x)₀；y),f(x₁；y),f(x₂；y),f(x₃(ii) a y) are respectively 4 known pixel points (x)₀,y),(x₁,y),(x₂,y),(x₃Y) pixel value, R₁(x, y) is the calculated pixel value of the point (x, y) to be repaired;

and detecting each point to be repaired in the scratch image from left to right and from top to bottom in sequence, and performing a step of adaptively selecting an interpolation sampling point and a step of calculating a pixel value of the damaged point to obtain an initial repaired image A.

The repainting of the scratch image by utilizing the binary continuous fractional interpolation technology comprises the following steps:

determining a binary interpolation sampling point, reading an initial repaired image A, taking the initial repaired image A as an information matrix, overlapping a mask image and the initial repaired image A to obtain the position of a point to be repaired of the initial repaired image A, and forming an interpolation pixel point by 16 pieces of known pixel information around the point (x, y) to be repaired;

searching the point (x, y) to be repaired to find 16 adjacent pixel points around the point, and sequentially selecting the following points according to the coordinate position of the point to be repaired:

taking the 16 pixel points as binary interpolation sampling points;

the updating of the pixel value of the damaged point, which is to calculate the pixel value of the damaged point by combining a binary interpolation sampling point and a binary run-length interpolation function, and update and replace the pixel value of the point in the original patch image A, comprises the following steps:

the binary run-length interpolation format is defined as:

wherein i is 0,1, …, m, n are the length and width of the input image respectively;

wherein,is a Newton-Thiele type mixed difference quotient;

constructed binary vector rational functionSatisfies the following conditions:

adopting 16 pixel points around the point (x, y) to be repaired as binary interpolation sampling points, namelyThe pixel value R of the point to be repaired is obtained by combining the calculation of the rational function of the binary vector₂(x,y)；

R₂(x,y)＝B₀(y)+2B₁(y)+2B₂(y)-2B₃(y)，

Wherein x is₀′＝x-2,x₁′＝x-1,x₂′＝x+1,x₃′＝x+2，y₀′＝y+2,y₁′＝y+1,y₂′＝y-1,y₃′＝y-2，

Wherein,

φ_NT[x₀′,…,x_i′；y₀′,…,y_j′]i-0, 1,2,3, j-0, 1,2,3, a Newton-Thiele type mixed difference quotient, x₀′＝x-2,x₁′＝x-1,x₂′＝x+1,x₃′＝x+2，y₀′＝y+2,y₁′＝y+1,y₂′＝y-1,y₃′＝y-2，

Middle phi of the above formula_NT[x_i′；y_j′]Satisfies the following conditions: phi is a_NT[x_i′；y_j′]＝f(x_i′,y_j') wherein f (x)_i′,y_j') is a corresponding known pixel point (x)_i′,y_j') pixel values;

the pixel value R₂(x, y) in place of the pixel value R of the point in the initial inpainting image A₁(x, y) the pixel value of the point (x, y) to be repaired after updating isR₂(x,y)；

And sequentially carrying out a determination step of binary interpolation sampling points and an updating step of pixel values of the damaged points on each point to be repaired of the initial repaired image A from left to right and from top to bottom to obtain a final repaired image B.

The self-adaptive selection of the interpolation sampling points comprises the following steps:

carrying out line-by-line detection by taking a damage information matrix of the scratch image as a target;

if the point (x, y) to be repaired is found, sequentially searching adjacent undamaged pixel points around the damaged point from near to far on two sides of the same row of the point (x, y) to be repaired, and taking the searched undamaged pixel points as a sampling point;

and when the number of the searched undamaged pixels reaches 4, stopping searching, and obtaining 4 effective sampling points on two sides of the same row of the point (x, y) to be repaired.

A system of an adaptive image inpainting method based on a continuous fraction interpolation technology comprises the following steps: the image inpainting device comprises an initial image input module for determining the type of an input image, a unary continuous fraction interpolation inpainting module for obtaining an initial inpainting image A and a binary continuous fraction interpolation refinishing module for obtaining a final inpainting image B;

the output end of the initialized image input module is connected with the input end of the unary continuous fraction interpolation repairing module, and the output end of the unary continuous fraction interpolation repairing module is connected with the input end of the binary continuous fraction interpolation repairing module.

Advantageous effects

Compared with the prior art, the self-adaptive image repairing method and the system thereof based on the continuous interpolation technology improve the quality and efficiency of image repairing, can modify scratches of any type, and are suitable for all image processing.

Through the matching use of the mask map, the position of the repair can be rapidly determined, so that the time for repairing can be saved, and meanwhile, the part of the image block which is interested can be flexibly repaired according to the requirements of a user. The method has the advantages of good image repairing effect, abundant texture details, greatly improved modification efficiency and strong practical applicability.

Drawings

FIG. 1 is a sequence diagram of the method of the present invention;

FIG. 2 is a schematic diagram of the system architecture of the present invention;

FIG. 3 is a prior art image to be repaired;

FIG. 4 is a mask diagram corresponding to FIG. 3;

FIG. 5 is a graph showing the effect of the method of reference [1] [2] after repairing FIG. 3;

FIG. 6 is a graph showing the effect of the repair of FIG. 3 using the method of reference [3 ];

FIG. 7 is a graph showing the effect of the repairing method of the present invention.

Detailed Description

So that the manner in which the above recited features of the present invention can be understood and readily understood, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings, wherein:

as shown in fig. 1, according to the adaptive image inpainting method based on the continuous interpolation technique, an initial inpainting image can be obtained through the one-element continuous interpolation, and the initial inpainting image is inpainted again through the two-element continuous interpolation, so that a final inpainting effect is obtained. The invention combines the two methods without bringing huge computation, because the simple unitary continuous interpolation repair is much smaller than the simple binary continuous interpolation repair, the unitary continuous interpolation repair method is used as the main repair technology, the binary continuous interpolation repair method is used as the secondary repair technology, and the pixel value of the damaged point is only updated, thereby reducing the overall computation.

Which comprises the following steps:

first, image feature analysis is initialized. And analyzing the input scratch image to judge whether the scratch image is a gray scale image or a color image. If the color image is a gray image, the color image is executed along R, G, B three color channels respectively, and the second step is carried out; if the image is a gray image, the second step is directly performed.

And secondly, repairing the damaged point of the scratch image by using a one-element continuous fractional interpolation technology.

The position of each point to be repaired in the corresponding scratch image is obtained through an input mask image, then the points to be repaired are detected line by taking an information matrix of the scratch image as a target, surrounding known pixel points are selected to be taken as sampling points in a self-adaptive mode aiming at each point to be repaired, and the pixel information of each point to be repaired is reconstructed by combining the sampling points and unitary continuous division type rational interpolation, so that an initial repaired image A is obtained.

As shown in fig. 3 and 4, the mask pattern is a scratch pattern of the image to be repaired, and scratches in the image to be repaired can be accurately positioned by using the mask pattern. Meanwhile, the matching repair of the mask image breaks through the traditional repair only aiming at the damaged image, and can also be suitable for repairing the interested position of a certain part of the image, not necessarily not a scratch image, such as the removal of a blocking object, the reconstruction of the detail of the image part and the like. In addition, the mask image can also shield certain areas on the image, so that the repairing process is not carried out, the repairing efficiency is improved, and meanwhile, the mask image can be better repaired for image repairing of some special shapes. The selected image, graphic or object is used to mask the image to be processed (in whole or in part) to control the area or process of image processing. The particular image or object used for overlay is referred to as a mask or template.

The method comprises the following specific steps:

(1) and determining the damage point of the scratch image. And inputting a mask image of the scratch image, overlapping the mask image and the scratch image, and acquiring a point to be repaired of the scratch image. The mask map is usually a binary image, the size of the image is the size of the scratch map, the pixel value is 0 for the part of the area which is not interested, and the pixel value is black, and the pixel value of the position where the interested or shielded or scratched is located is 1, and the position is white.

(2) And adaptively selecting an interpolation sampling point.

And reading a damage information matrix of the scratch image, adaptively selecting 4 known pixel points (non-damaged effective information points) nearest to the point (x, y) to be repaired, and forming interpolation sampling points by the 4 known pixel points. Generally, the more the number of interpolation sampling points, the more the advantage of the run-to-run equation can be reflected, but the interpolation using more than 4 pixel points causes the low efficiency of the algorithm, and the interpolation effect is similar to the interpolation of 4 pixel points, in addition, the interpolation of 2 or 3 pixel points does not reflect the advantage of the run-to-run equation interpolation, and the interpolation effect is not good, so that the interpolation processing using 4 pixel points is most suitable.

A. Carrying out line-by-line detection by taking a damage information matrix of the scratch image as a target;

B. if the point (x, y) to be repaired is found, sequentially searching adjacent undamaged pixel points around the damaged point from near to far on two sides of the same row of the point (x, y) to be repaired, and taking the searched undamaged pixel points as a sampling point;

C. and when the number of the searched undamaged pixels reaches 4, stopping searching, and obtaining 4 effective sampling points on two sides of the same row of the point (x, y) to be repaired.

The self-adaptive method is used for selecting a unitary continuous fraction interpolation sampling point and has more flexibility compared with the traditional interpolation window, because each row or each column can have a plurality of adjacent points to be repaired unknown, if the traditional interpolation window is adopted, the pixel value calculation of a damaged point is incorrect or the deviation is too large, the self-adaptive mode is to adopt a left and right nearest neighbor searching mode in the same row to search for available sampling points according to the position of the actual point to be repaired, and add the sampling points as interpolation points, once the number of the searched sampling points reaches the requirement, the searching is stopped, and the sampling points are used for interpolating to calculate the pixel value of the repairing point.

The invention combines the self-adaptive method and the continuous separation type theory to be applied to the image processing, and is the first creation in the field of image processing. The self-adaptive method and the continuous division theory are combined and used for image repairing, and the method is mainly embodied in that the self-adaptive method is used for selecting a unitary continuous division interpolation sampling point. The selection of the conventional unitary continuous fraction interpolation sampling point is random, and usually, several known pixel points near the damaged point are selected as sampling points (such as the method in the document [1] [2]) or several known pixel points are sequentially selected as sampling points along the forward direction of the coordinate where the damaged point is located. Due to this randomness, the patching algorithm is not stable, and even the pixel value of the damaged point obtained by each interpolation is different. When the self-adaptive method is used for selecting a sampling point, the sampling point is selected in a bilateral symmetry mode from near to far by taking the damaged point as a center, like pointer processing, a left pixel point and a right pixel point which are closest to the damaged point are searched firstly (the sequence of the left pixel point and the right pixel point is not different), whether the sampling point is a known pixel point is judged, if both the left pixel point and the right pixel point are known pixel points, the two pixel points are taken as the sampling point, if one of the two pixel points is a known pixel point, the pixel point is added as the sampling point, if both the left pixel point and the right pixel point are not known pixel points, the pointer is moved to the next pixel point which is next to the damaged point (the second-nearest is the position which is separated by one pixel point from the middle of the damaged point), whether the pixel point is a known pixel point is judged by adopting the. And by analogy, in the same operation, when the number of the searched sampling points reaches the requirement, stopping searching, and using the sampling points for interpolation to calculate the pixel value of the patch point.

(3) And calculating the pixel value of the breakage point. The pixel value of the damaged point is calculated by combining 4 known pixel points with an elementary run-length interpolation function, and the method comprises the following steps:

A. the unitary run-length interpolation format is defined as:

wherein, b_i＝φ[x₀,x₁,…,x_i；y](i-0, …, m) is the function f (x, y) at point x₀,x₁,…,x_iM is the length of the input image, and satisfies the following condition:

φ[x_i；y]＝f(x_i；y),i＝0,1,2,…,m,

B. calculating the pixel value R of the point to be repaired by adopting the information of 4 known pixel points around the point (x, y) to be repaired and combining an elementary continuous interpolation function₁(x,y)；

Wherein,

b₀＝φ[x₀；y]＝f(x₀；y)，

in the above formula, f (x)₀；y),f(x₁；y),f(x₂；y),f(x₃(ii) a y) are respectively 4 known pixel points (x)₀,y),(x₁,y),(x₂,y),(x₃Y) pixel value, R₁(x, y) is the calculated pixel value of the point (x, y) to be repaired;

(4) detecting each point to be repaired in the scratch image from left to right and from top to bottom, and performing a step of adaptively selecting an interpolation sampling point and a step of calculating a pixel value of the damaged point.

And thirdly, repairing the scratch image by using a binary continuous fractional interpolation technology. The position of each damaged point is determined from the mask map using the initial patch image a as an information image. Here, the mask map is reused once again in order to confirm the position of the damaged point, and since it is not already seen which damaged points are in the initial patch image a, it is necessary to determine the damaged point position information from the mask map. And updating the pixel information around the damaged point by adopting binary continuous fraction interpolation to obtain the pixel information of each damaged point, thereby obtaining the final repaired image B.

The binary continuous interpolation technology is used for repairing the scratch image, and the selection of the used interpolation function and the interpolation window is the same as that of the existing interpolation method, but the difference is that the objects acted by the interpolation function and the interpolation window are different. The prior art action objects are original breakage graphs, while the invention action object is an initial repair graph obtained in the last step. The initial repair map is used to improve the efficiency of the repair. Since the second step has already adopted the unitary continued fraction formula to repair, some damaged points may have already obtained the correct pixel values, if adopt the binary continued fraction formula again to repair the original damaged map, must carry on the repeated operation, this is unnecessary, will reduce the efficiency of the whole algorithm at the same time. Therefore, in this step, when the scratch image is repaired again by using the binary continuous interpolation technique, only the points to be repaired are regarded as damaged points, but other damaged points in the original damaged image are not regarded as damaged points, but these points are regarded as known pixel points, the pixel values of these points are taken as the pixel values of the corresponding points in the original repaired image, and then the pixel values of the points to be repaired are calculated by using the existing interpolation function and the existing interpolation window. The processing in this step is to consider the relevance among the image pixels, and the similarity between the pixels closer to the point to be repaired is larger, so that all other pixels around the point to be repaired are treated as known pixels to obtain a pixel value closer to the original pixel value. It can also be found from the processing of this step that since other pixel points except for the point to be repaired are all taken as known pixel points, it becomes difficult to determine the next point to be repaired, and at this time, the position of each damaged point needs to be determined by the mask map used in cooperation with the mask map.

Which comprises the following steps:

(1) and determining binary interpolation sampling points. Reading an initial repairing image A and taking the initial repairing image A as an information matrix, overlapping a mask image and the initial repairing image A to obtain a position of a point to be repaired of the initial repairing image A, and forming an interpolation pixel point by 16 pieces of known pixel information around the point (x, y) to be repaired;

searching the repairing point (x, y) to find 16 adjacent pixel points around the repairing point, and sequentially selecting the following points according to the coordinate position of the point to be repaired:

and taking the 16 pixel points as binary interpolation sampling points.

Here, the damage information is updated based on the initial patch image a, so that the 3 × 3 interpolation window (i.e., 9 pixels) is not used, but the 4 × 4 interpolation window (i.e., 16 pixels) is used for processing, and since the 3 × 3 interpolation window and the 4 × 4 interpolation window have almost the same effect here, and meanwhile, the image block obtained by dividing the image into 3 × 3 has more image blocks compared with the image block obtained by dividing the image into 4 × 4, and the more image blocks, the more the algorithm is executed, the more 16 pixels are preferably used as sampling points from the viewpoint of efficiency. For more than 16 pixels, the continuous division operation speed is low, the efficiency of the whole algorithm is low, and the algorithm is not suitable for use.

(2) And updating the pixel value of the breakage point. The pixel value of the damaged point is calculated by combining the binary interpolation sampling point with a binary run-length interpolation function, and the pixel value of the point in the original patch image A is updated and replaced, and the method comprises the following steps:

A. the binary run-length interpolation format is defined as:

wherein i is 0,1, …, m, n are the length and width of the input image respectively;

wherein,is a Newton-Thiele type mixed difference quotient;

constructed binary vector rational functionSatisfies the following conditions:

B. adopting 16 pixel points around the point (x, y) to be repaired as binary interpolation sampling points, namelyThe pixel value R of the point to be repaired is obtained by combining the calculation of the rational function of the binary vector₂(x,y)；

R₂(x,y)＝B₀(y)+2B₁(y)+2B₂(y)-2B₃(y)，

Wherein x is₀′＝x-2,x₁′＝x-1,x₂′＝x+1,x₃′＝x+2，y₀′＝y+2,y₁′＝y+1,y₂′＝y-1,y₃′＝y-2，

Wherein,

φ_NT[x₀′,…,x_i′；y₀′,…,y_j′]i-0, 1,2,3, j-0, 1,2,3, a Newton-Thiele type mixed differential quotient,

x₀′＝x-2,x₁′＝x-1,x₂′＝x+1,x₃′＝x+2，y₀′＝y+2,y₁′＝y+1,y₂′＝y-1,y₃′＝y-2，

middle phi of the above formula_NT[x_i′；y_j′]Satisfies the following conditions: phi is a_NT[x_i′；y_j′]＝f(x_i′,y_j') wherein f (x)_i′,y_j') is a corresponding known pixel point (x)_i′,y_j') pixel values;

C. the pixel value R₂(x, y) in place of the pixel value R of the point in the initial inpainting image A₁(x, y), the pixel value of the point (x, y) to be repaired after updating is R₂(x,y)。

(3) And (3) sequentially carrying out a determination step of binary interpolation sampling points and an updating step of pixel values of the damaged points on each point to be repaired of the initial repaired image A from left to right and from top to bottom to obtain a final repaired image B.

As shown in fig. 2, there is also provided a system of an adaptive image inpainting method based on a continuous interpolation technique, which includes: the image inpainting device comprises an initial image input module used for determining the type of an input image, a unary continuous fraction interpolation inpainting module used for obtaining an initial inpainting image A, and a binary continuous fraction interpolation refinishing module used for obtaining a final inpainting image B.

The output end of the initialized image input module is connected with the input end of the unary continuous fraction interpolation repairing module, and the output end of the unary continuous fraction interpolation repairing module is connected with the input end of the binary continuous fraction interpolation repairing module.

Fig. 3 shows a picture to be repaired, and fig. 4 shows a mask used in the repair. After processing by using a one-component continuous repairing method (see the detailed algorithm in document [1] [2]), as shown in fig. 5, the damaged part of the picture is partially modified, but there are still some damaged points that are not repaired. After the binary continuous repairing method (the specific algorithm is described in detail in document [3]) is used for processing, as shown in fig. 6, the overall visual effect and quality of the picture are improved. As shown in FIG. 7, after the repairing is carried out by adopting the method of the invention, the detail recovery of the marked part is better, the whole visual effect is better, and the method has optimization and promotion to a greater extent than the method of the document [1] [2] [3 ].

To demonstrate the higher efficiency of the present invention compared to the other three methods, a comparison of the run times of the four methods is shown in table 1.

TABLE 1 comparison of the run times of the method of the invention with the method of the document [1] [2] [3]

In order to show that the effect of the invention is better and is better improved compared with the other three methods, a theoretical parameter for evaluating the image quality, namely, the peak signal-to-noise ratio is used as an evaluation index, and the comparison of the peak signal-to-noise ratios of the images repaired by the four methods is shown in table 2.

TABLE 2 comparison of Peak SNR Using the method of the invention and the document [1] [2] [3] method

From the objective point of view, the comparison can find thatWhere m × n is the size of the matrix, max is 255, f (i, j) is the original image,the peak signal-to-noise ratio PSNR value is calculated by using the formula for the repaired image. The larger the peak signal-to-noise ratio is, the closer the repaired image is to the original image, i.e. the better the visual effect of the repaired image is, the higher the resolution is.

As shown in table 2, the peak snr of the picture to be patched using the above method is obviously higher than the peak snr of the picture after patched using the prior art method, and the image resolution and quality are higher.

As shown in table 1, as a result of using the running time of the above-mentioned method to patch the picture to be patched, running the method of the prior art and the method of the present invention on a computer in the same running environment, and recording the running time of the algorithm, it can be found that the efficiency of the present invention is higher than that of the binary run-length patching method (i.e., the method of document [3]), and the efficiency is higher than that of the binary run-length which processes simultaneously in the two-dimensional coordinate direction because the unitary run-length is the process in the one-dimensional coordinate direction. Therefore, compared with the prior art, the method has better operation efficiency and higher quality of the repaired picture.

The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are merely illustrative of the principles of the invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims (5)

1. A self-adaptive image inpainting method based on a continuous fraction interpolation technology is characterized by comprising the following steps:

11) initializing image characteristic analysis, analyzing an input scratch image, and judging whether the scratch image is a gray image or a color image; if the color image is a gray image, the color image is executed along R, G, B three color channels;

12) repairing damaged points of a scratch image by using a unitary continuous fractional interpolation technology, acquiring the position of each point to be repaired in the corresponding scratch image through an input mask image, detecting the points to be repaired line by taking an information matrix of the scratch image as a target, adaptively selecting surrounding known pixel points as sampling points for each point to be repaired, and reconstructing pixel information of each point to be repaired by combining the sampling points with unitary continuous fractional rational interpolation to obtain an initial repaired image A;

13) and (3) repairing the scratch image by using a binary continuous fractional interpolation technology, determining the position of each damaged point by using the initial repaired image A as an information image and a mask image, and updating pixel information of each damaged point by using pixel information around the damaged point by using binary continuous fractional interpolation to obtain the final repaired image B.

2. The adaptive image inpainting method based on the continuous and fractional interpolation technology as claimed in claim 1, wherein the repairing of the scratch image breakage point by using the unary continuous and fractional interpolation technology comprises the following steps:

21) determining a damage point of a scratch image, inputting a mask image of the scratch image, overlapping the mask image and the scratch image, and acquiring a to-be-repaired point of the scratch image;

22) adaptively selecting interpolation sampling points, reading a damage information matrix of the scratch image, adaptively selecting 4 known pixel points nearest to a point (x, y) to be repaired, and forming the 4 known pixel points into the interpolation sampling points;

23) the calculation of the pixel value of the damaged point, which is calculated by combining 4 known pixel points with an elementary run-length interpolation function, comprises the following steps:

231) the unitary run-length interpolation format is defined as:

$T_m\left(x\right)=b_0+\frac{x-x_0}{b_1}+\frac{x-x_1}{b_2}+\mathrm{...}+\frac{x-x_{m-1}}{b_m},$

wherein, b_i＝φ[x₀,x₁,…,x_i；y](i-0, …, m) is the function f (x, y) at point x₀,x₁,…,x_iM is the length of the input image, and satisfies the following condition:

φ[x_i；y]＝f(x_i；y),i＝0,1,2,…,m,

$\mathrm{\φ}\[x_p,x_q;y\]=\frac{x_q-x_p}{\mathrm{\φ}\[x_q;y\]-\mathrm{\φ}\[x_p;y\]},$

$\mathrm{\φ}\[x_i,\mathrm{...},x_j,x_k,x_l;y\]=\frac{x_l-x_k}{\mathrm{\φ}\[x_i,\mathrm{...},x_j,x_l;y\]-\mathrm{\φ}\[x_i,\mathrm{...},x_j,x_k;y\]};$

232) calculating the pixel value R of the point to be repaired by adopting the information of 4 known pixel points around the point (x, y) to be repaired and combining an elementary continuous interpolation function₁(x,y)；

$R_1(x,y)=T_3\left(x\right)=b_0+\frac{x-x_0}{b_1}+\frac{x-x_1}{b_2}+\frac{x-x_2}{b_3},$

Wherein,

b₀＝φ[x₀；y]＝f(x₀；y)，

$b_1=\mathrm{\φ}\[x_0,x_1;y\]=\frac{x_1-x_0}{\mathrm{\φ}\[x_1;y\]-\mathrm{\φ}\[x_0;y\]}=\frac{x_1-x_0}{f(x_1;y)-f(x_0;y)},$

$b_{12}=\mathrm{\φ}\[x_0,x_2;y\]=\frac{x_2-x_0}{\mathrm{\φ}\[x_2;y\]-\mathrm{\φ}\[x_0;y\]}=\frac{x_2-x_0}{f(x_2;y)-f(x_0;y)},$

$b_{13}=\mathrm{\φ}\[x_0,x_3;y\]=\frac{x_3-x_0}{\mathrm{\φ}\[x_3;y\]-\mathrm{\φ}\[x_0;y\]}=\frac{x_3-x_0}{f(x_3;y)-f(x_0;y)},$

$b_2=\mathrm{\φ}\[x_0,x_1,x_2;y\]=\frac{x_2-x_1}{\mathrm{\φ}\[x_0,x_2;y\]-\mathrm{\φ}\[x_0,x_1;y\]}=\frac{x_2-x_1}{b_{12}-b_1},$

$b_{22}=\mathrm{\φ}\[x_0,x_1,x_3;y\]=\frac{x_3-x_1}{\mathrm{\φ}\[x_0,x_3;y\]-\mathrm{\φ}\[x_0,x_1;y\]}=\frac{x_3-x_1}{b_{13}-b_1},$

$b_3=\mathrm{\φ}\[x_0,x_1,x_2,x_3;y\]=\frac{x_3-x_2}{\mathrm{\φ}\[x_0,x_1,x_3;y\]-\mathrm{\φ}\[x_0,x_1,x_2;y\]}=\frac{x_3-x_2}{b_{22}-b_2};$

in the above formula, f (x)₀；y),f(x₁；y),f(x₂；y),f(x₃(ii) a y) are respectively 4 known pixel points (x)₀,y),(x₁,y),(x₂,y),(x₃Y) pixel value, R₁(x, y) is the calculated pixel value of the point (x, y) to be repaired;

24) and detecting each point to be repaired in the scratch image from left to right and from top to bottom in sequence, and performing a step of adaptively selecting an interpolation sampling point and a step of calculating a pixel value of the damaged point to obtain an initial repaired image A.

3. The adaptive image inpainting method based on the continuous interpolation technique as claimed in claim 1, wherein the repairing of the scratch image by the binary continuous interpolation technique comprises the following steps:

31) determining a binary interpolation sampling point, reading an initial repaired image A, taking the initial repaired image A as an information matrix, overlapping a mask image and the initial repaired image A to obtain the position of a point to be repaired of the initial repaired image A, and forming an interpolation pixel point by 16 pieces of known pixel information around the point (x, y) to be repaired;

searching the point (x, y) to be repaired to find 16 adjacent pixel points around the point, and sequentially selecting the following points according to the coordinate position of the point to be repaired:

(x-2,y+2)(x-2,y+1)(x-2,y-1)(x-2,y-2)

(x-1,y+2)(x-1,y+1)(x-1,y-1)(x-1,y-2)

(x+1,y+2)(x+1,y+1)(x+1,y-1)(x+1,y-2)，

(x+2,y+2)(x+2,y+1)(x+2,y-1)(x+2,y-2)

taking the 16 pixel points as binary interpolation sampling points;

32) the updating of the pixel value of the damaged point, which is to calculate the pixel value of the damaged point by combining a binary interpolation sampling point and a binary run-length interpolation function, and update and replace the pixel value of the point in the original patch image A, comprises the following steps:

321) the binary run-length interpolation format is defined as:

$R_{m,n}^{NT}(x,y)=A_0\left(y\right)+(x-x_0)A_1\left(y\right)+\mathrm{...}+(x-x_0)\mathrm{...}(x-x_{m-1})A_m\left(y\right),$

wherein i is 0,1, …, m, n are the length and width of the input image respectively;

wherein,is a Newton-Thiele type mixed difference quotient;

${\mathrm{\φ}}_{NT}\[x_i;y_j\]=f(x_i,y_j),\∀(x_i,y_j)\∈{\mathrm{\Π}}_{x,y}^{m,n},$

${\mathrm{\φ}}_{NT}\[x_i,x_j;y_k\]=\frac{{\mathrm{\φ}}_{NT}\[x_j;y_k\]-{\mathrm{\φ}}_{NT}\[x_i;y_k\]}{x_j-x_i},$

${\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q,x_i,x_j;y_k\]=\frac{{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q,x_j;y_k\]-{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q,x_i;y_k\]}{x_j-x_i},$

${\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q;y_k,y_l\]=\frac{y_l-y_k}{{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q;y_l\]-{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q;y_k\]},$

${\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q,y_r,\mathrm{...},y_s,y_k,y_l\]=\frac{y_l-y_k}{{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q;y_r,\mathrm{...},y_s,y_l\]-{\mathrm{\φ}}_{NT}\[x_p,\mathrm{...},x_q;y_r,\mathrm{...},y_s,y_k\]};$

constructed binary vector rational functionSatisfies the following conditions:

$R_{m,n}^{NT}(x_i,y_j)=f(x_i,y_j),i=0,\mathrm{...},m;j=0,\mathrm{...},n;$

322) adopting 16 pixel points around the point (x, y) to be repaired as binary interpolation sampling points, namely

The pixel value R of the point to be repaired is obtained by combining the calculation of the rational function of the binary vector₂(x,y)；

R₂(x,y)＝B₀(y)+2B₁(y)+2B₂(y)-2B₃(y)，

Wherein, x'₀＝x-2,x′₁＝x-1,x′₂＝x+1,x′₃＝x+2，y′₀＝y+2,y′₁＝y+1,y′₂＝y-1,y′₃＝y-2，

Wherein,

φ_NT[x′₀,…,x′_i；y′₀,…,y′_j]i-0, 1,2,3, j-0, 1,2,3, a Newton-Thiele type mixed differential quotient,

x′₀＝x-2,x′₁＝x-1,x′₂＝x+1,x′₃＝x+2，y′₀＝y+2,y′₁＝y+1,y′₂＝y-1,y′₃＝y-2，

${\mathrm{\φ}}_{NT}\[x_i^{\′},x_j^{\′};y_k^{\′}\]=\frac{{\mathrm{\φ}}_{NT}\[x_j^{\′};y_k^{\′}\]-{\mathrm{\φ}}_{NT}\[x_i^{\′};y_k^{\′}\]}{x_j^{\′}-x_i^{\′}},$

${\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},x_i^{\′},x_j^{\′};y_k^{\′}\]=\frac{{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},x_j^{\′};y_k^{\′}\]-{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},x_i^{\′};y_k^{\′}\]}{x_j^{\′}-x_i^{\′}},$

${\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′};y_k^{\′},y_l^{\′}\]=\frac{y_l^{\′}-y_k^{\′}}{{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′};y_l^{\′}\]-{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′};y_k^{\′}\]},$

${\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},y_r^{\′},\mathrm{...},y_s^{\′},y_k^{\′},y_l^{\′}\]=\frac{y_l^{\′}-y_k^{\′}}{{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},y_r^{\′},\mathrm{...},y_s^{\′},y_l^{\′}\]-{\mathrm{\φ}}_{NT}\[x_p^{\′},\mathrm{...},x_q^{\′},y_r^{\′},\mathrm{...},y_s^{\′},y_k^{\′}\]},$

middle phi of the above formula_NT[x′_i；y′_j]Satisfies the following conditions: phi is a_NT[x′_i；y′_j]＝f(x′_i,y′_j) Wherein f (x'_i,y′_j) Is the corresponding known pixel point (x'_i,y′_j) A pixel value of (a);

323) the pixel value R₂(x, y) in place of the pixel value R of the point in the initial inpainting image A₁(x, y), the pixel value of the point (x, y) to be repaired after updating is R₂(x,y)；

33) And sequentially carrying out a determination step of binary interpolation sampling points and an updating step of pixel values of the damaged points on each point to be repaired of the initial repaired image A from left to right and from top to bottom to obtain a final repaired image B.

4. The adaptive image inpainting method based on the continuous interpolation technique as claimed in claim 2, wherein the adaptively selecting the interpolation sample points comprises the following steps:

41) carrying out line-by-line detection by taking a damage information matrix of the scratch image as a target;

42) if the point (x, y) to be repaired is found, sequentially searching adjacent undamaged pixel points around the damaged point from near to far on two sides of the same row of the point (x, y) to be repaired, and taking the searched undamaged pixel points as a sampling point;

43) and when the number of the searched undamaged pixels reaches 4, stopping searching, and obtaining 4 effective sampling points on two sides of the same row of the point (x, y) to be repaired.

5. The system of the adaptive image inpainting method based on the continuous interpolation technology as claimed in claim 1, comprising: the image inpainting device comprises an initial image input module for determining the type of an input image, a unary continuous fraction interpolation inpainting module for obtaining an initial inpainting image A and a binary continuous fraction interpolation refinishing module for obtaining a final inpainting image B;

the output end of the initialized image input module is connected with the input end of the unary continuous fraction interpolation repairing module, and the output end of the unary continuous fraction interpolation repairing module is connected with the input end of the binary continuous fraction interpolation repairing module.