WO2021093637A1

WO2021093637A1 - Focusing method and apparatus, electronic device, and computer readable storage medium

Info

Publication number: WO2021093637A1
Application number: PCT/CN2020/126139
Authority: WO
Inventors: 贾玉虎
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-11-12
Filing date: 2020-11-03
Publication date: 2021-05-20
Also published as: CN112866542A; CN112866542B

Abstract

A focusing method and apparatus, an electronic device and a computer-readable storage medium, the focusing method comprising: obtaining a target subject detection area in which a target subject is located in a preview image; when the target subject moves, determining a target subject prediction area according to the target subject detection area and movement data of the target subject, and obtaining a detection image corresponding to the target subject prediction region; using an image sensor to obtain phase differences of the detection image, the phase differences comprising a phase difference in a first direction and a phase difference in a second direction, and the first direction forming a preset included angle with the second direction; and according to the phase difference in the first direction and the phase difference in the second direction, controlling a lens to continuously focus on the moving target subject.

Description

Focus tracking method and device, electronic equipment, and computer readable storage medium

Cross-references to related applications

This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on November 12, 2019, the application number is 201911101390.0, and the application name is "focus tracking method and device, electronic equipment, computer-readable storage medium", and its entire content Incorporated in this application by reference.

Technical field

This application relates to the field of imaging, in particular to a method and device for tracking focus, electronic equipment, and computer-readable storage media.

Background technique

With the development of electronic device technology, more and more users take images through electronic devices. For moving objects, in order to ensure that the captured images are clear, it is usually necessary to focus on the camera module of the electronic device, that is, by continuously adjusting the distance between the lens and the image sensor, so that the subject is always on the focal plane. Focus tracking refers to the process of maintaining focus on the subject in the subsequent shooting process after the target camera focuses on the subject. Traditional focusing methods include phase detection auto focus (English: phase detection auto focus; referred to as PDAF).

Summary of the invention

According to various embodiments disclosed in the present application, a focus tracking method, device, electronic device, and computer-readable storage medium are provided.

A tracking focus method is applied to an electronic device. The electronic device includes an image sensor and a lens. The image sensor includes a plurality of pixel point groups arranged in an array, and each of the pixel point groups includes M arranged in an array. *N pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2; the method includes:

Acquiring the target subject detection area where the target subject in the preview image is located;

When the target subject moves, determine a target subject prediction area according to the target subject detection area and the movement data of the target subject, and obtain a detection image corresponding to the target subject prediction area;

The image sensor is used to obtain the phase difference value of the detection image, the phase difference value includes a phase difference value in a first direction and a phase difference value in a second direction; Set an angle; and

The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.

A tracking focus device is applied to an electronic device. The electronic device includes an image sensor and a lens. The image sensor includes a plurality of pixel point groups arranged in an array, and each of the pixel point groups includes M* arranged in an array. N pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2; including:

Recognition module, used to obtain the target subject detection area where the target subject in the preview image is located

A prediction module, configured to determine a target subject prediction area according to the target subject detection area and movement data of the target subject when the target subject moves, and obtain a detection image corresponding to the target subject prediction area;

The acquiring module is configured to acquire the phase difference value of the detection image by using the image sensor, the phase difference value including a phase difference value in a first direction and a phase difference value in a second direction; the first direction and the phase difference value The second direction is a preset angle; and

The focus tracking module is configured to control the lens to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.

An electronic device, including a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the Steps of tracking focus method.

One or more computer-readable storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors implement the steps of the focus tracking method when executed.

The details of one or more embodiments of the present application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the description, drawings and claims.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Figure 1 is a schematic diagram of the principle of phase detection autofocus;

FIG. 2 is a schematic diagram of phase detection pixel points arranged in pairs in the pixel points included in the image sensor; FIG.

FIG. 3 is a schematic diagram of a part of the structure of an image sensor in one or more embodiments;

4 is a schematic diagram of the structure of pixels in one or more embodiments;

FIG. 5 is a schematic structural diagram of an electronic device in one or more embodiments;

Fig. 6 is a schematic diagram of a filter set on a pixel point group in one or more embodiments;

FIG. 7 is a flowchart of a focus tracking method in one or more embodiments;

FIG. 8 is a flowchart of steps in one or more embodiments: the phase difference value in the first direction and the phase difference value in the second direction control the lens to continuously focus on the moving target subject;

FIG. 9 is a step in one or more embodiments: a flow chart of obtaining the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction;

10 is a step in one or more embodiments: a flow chart of obtaining a target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction;

FIG. 11 is a step in one or more embodiments: a flowchart of determining a target phase difference value according to the magnitude relationship between the first confidence level and the second confidence level;

Figure 12 is a step in one or more embodiments: a flow chart of obtaining the phase difference value of the detection image;

FIG. 13 shows the steps in one or more embodiments: acquiring the phase difference value in the first direction according to the phase relationship corresponding to the first segmented image and the second segmented image and corresponding to the third segmented image and the fourth segmented image A flow chart for obtaining the phase difference value in the second direction according to the phase relationship;

FIG. 14 is a structural block diagram of a focusing device in one or more embodiments;

Figure 15 is a block diagram of an electronic device in one or more embodiments.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

It can be understood that the terms "first", "second", etc. used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first direction may be referred to as the second direction, and similarly, the second direction may be referred to as the first direction. Both the first direction and the second direction are directions, but they are not the same direction.

When shooting images, in order to ensure that the image of the moving object is captured clearly, it is usually necessary to follow the focus of the electronic device. The focus is to maintain the focus of the subject in the subsequent shooting process after the target camera focuses on the subject. the process of. For example, in the preview process of the image taken by the electronic device, after the subject is focused, the subject is still focused in the subsequent acquired preview image, and the subject in the acquired preview image is still clearly imaged. The so-called "focusing" refers to the process of adjusting the distance between the lens of the electronic device and the image sensor to make the image sensor clear. Phase detection auto focus (English: phase detection auto focus; referred to as PDAF) is a relatively common auto focus technology.

Below, the embodiments of the present application will briefly describe the principle of PDAF technology.

Figure 1 is a schematic diagram of the principle of phase detection auto focus (PDAF). As shown in FIG. 1, M1 is the position of the image sensor when the electronic device is in the in-focus state, where the in-focus state refers to the state of successful focusing. When the image sensor is at the M1 position, the imaging light g reflected by the object W toward the lens Lens in different directions converges on the image sensor, that is, the imaging light g reflected by the object W toward the lens Lens in different directions is in the image The image is imaged at the same position on the sensor. At this time, the image of the image sensor is clear.

M2 and M3 are the possible positions of the image sensor when the electronic device is not in focus. As shown in Figure 1, when the image sensor is at the M2 position or the M3 position, the object W is reflected in different directions of the lens Lens. The imaging light g will be imaged at different positions. Please refer to Figure 1. When the image sensor is at the M2 position, the imaging light g reflected by the object W in different directions to the lens Lens is imaged at the position A and the position B respectively. When the image sensor is at the M3 position, the object W is reflected toward the lens. The imaging light g in different directions of the lens Lens is imaged at the position C and the position D respectively. At this time, the image of the image sensor is not clear.

In PDAF technology, the difference in position of the image formed by the imaging light entering the lens from different directions in the image sensor can be obtained. For example, as shown in Figure 1, the difference between position A and position B can be obtained, or, Obtain the difference between position C and position D; after obtaining the difference in position of the image formed by the imaging light entering the lens from different directions in the image sensor, the difference and the difference between the lens and the image sensor in the camera The geometric relationship is used to obtain the defocus distance. The so-called defocus distance refers to the distance between the current position of the image sensor and the position where the image sensor should be in the in-focus state; the electronic device can focus according to the obtained defocus distance.

It can be seen that when focusing, the calculated PD value is 0. On the contrary, the larger the calculated value, the farther the position of the clutch focal point is, and the smaller the value, the closer the clutch focal point. When using PDAF to focus, by calculating the PD value, and then obtaining the corresponding relationship between the PD value and the defocusing distance according to the calibration, the defocusing distance can be obtained, and then controlling the lens to move to the focal point according to the defocusing distance to achieve focusing.

In the related art, some phase detection pixel points may be provided in pairs in the pixel points included in the image sensor. As shown in FIG. 2, a phase detection pixel point pair (hereinafter referred to as a pixel point pair) A may be provided in the image sensor. Pixel point pair B and pixel point pair C. Among them, in each pixel point pair, one phase detection pixel is subjected to left shielding (English: Left Shield), and the other phase detection pixel is subjected to right shielding (English: Right Shield).

For the phase detection pixel point that has been blocked on the left, only the right beam of the imaging beam directed to the phase detection pixel point can be in the photosensitive part of the phase detection pixel point (that is, the part that is not blocked). ). For the phase detection pixel that has been occluded on the right, only the left beam of the imaging beam directed at the phase detection pixel can be in the photosensitive part of the phase detection pixel (that is, not The occluded part) is imaged. In this way, the imaging beam can be divided into left and right parts, and the phase difference can be obtained by comparing the images formed by the left and right parts of the imaging beam.

However, because the phase detection pixels set in the image sensor are usually sparse, therefore, only the horizontal phase difference can be obtained through the phase detection pixels, and the horizontal texture scene cannot be calculated. The calculated PD value will cause confusion. The correct result is obtained. For example, the shooting scene is a horizontal line, and the left and right images will be obtained according to the PD characteristics, but the PD value cannot be calculated.

In order to solve the situation that the phase detection autofocus cannot calculate the PD value to achieve focus for some horizontal texture scenes, an imaging component is provided in the embodiment of the application, which can be used to detect the phase difference value in the first direction and the second The phase difference value of the direction, for a horizontal texture scene, the phase difference value of the second direction can be used to achieve focusing.

In one of the embodiments, the present application provides an imaging assembly. The imaging component includes an image sensor. The image sensor may be a metal oxide semiconductor device (English: Complementary Metal Oxide Semiconductor; abbreviation: CMOS) image sensor, a charge-coupled device (English: Charge-coupled Device; abbreviation: CCD), a quantum thin film sensor, or an organic sensor.

Fig. 3 is a schematic diagram of a part of the image sensor in one of the embodiments. The image sensor 300 includes a plurality of pixel point groups Z arranged in an array, and each pixel point group Z includes a plurality of pixel points D arranged in an array, and each pixel point D corresponds to a photosensitive unit. The multiple pixels include M*N pixels, where both M and N are natural numbers greater than or equal to 2. Each pixel point D includes a plurality of sub-pixel points d arranged in an array. That is, each photosensitive unit can be composed of a plurality of photosensitive elements arranged in an array. The photosensitive element is an element that can convert light signals into electrical signals. In one of the embodiments, the photosensitive element may be a photodiode. In this embodiment, each pixel point group Z includes 4 pixel points D arranged in a 2*2 array, and each pixel point may include 4 sub-pixel points d arranged in a 2*2 array. Each pixel point D includes 2*2 photodiodes, and the 2*2 photodiodes are arranged correspondingly to the 4 sub-pixel points d arranged in a 2*2 array. Each photodiode is used to receive optical signals and perform photoelectric conversion, thereby converting the optical signals into electrical signals for output. The 4 sub-pixels d included in each pixel D are set corresponding to the same color filter, so each pixel D corresponds to a color channel, such as the red R channel, or the green channel G, or the blue channel B .

As shown in Figure 4, taking each pixel point including sub-pixel point 1, sub-pixel point 2, sub-pixel point 3, and sub-pixel point 4 as an example, sub-pixel point 1 and sub-pixel point 2 can be synthesized, and sub-pixel point 3 and sub-pixel point The pixel points 4 are synthesized to form a pair of PD pixels in the up and down direction, and the horizontal edge is detected to obtain the phase difference value in the second direction, that is, the vertical direction PD value (phase difference value); sub-pixel point 1 and sub-pixel point 3 are synthesized, The sub-pixel point 2 and the sub-pixel point 4 are combined to form a pair of PD pixels in the left and right directions, and vertical edges can be detected to obtain the phase difference value in the first direction, that is, the PD value (phase difference value) in the horizontal direction.

Fig. 5 is a schematic structural diagram of an electronic device in one of the embodiments. As shown in FIG. 5, the electronic device includes a micro lens 50, a filter 52, and an imaging component 54. The micro lens 50, the filter 52 and the imaging component 54 are sequentially located on the incident light path, that is, the micro lens 50 is disposed on the filter 52, and the filter 52 is disposed on the imaging component 54.

The filter 52 can include three types of red, green, and blue, and can only transmit light of corresponding wavelengths of red, green, and blue, respectively. One filter 52 is arranged on one pixel.

The imaging component 54 includes the image sensor in FIG. 3.

The lens 50 is used to receive incident light and transmit the incident light to the filter 52. After the filter 52 smoothes the incident light, the smoothed light is incident on the imaging component 54 on a pixel basis.

The photosensitive unit in the image sensor converts the light incident from the filter 52 into a charge signal through the photoelectric effect, and generates a pixel signal consistent with the charge signal. The charge signal is consistent with the received light intensity.

FIG. 6 is a schematic diagram of a filter set on the pixel point group in one of the embodiments. The pixel group Z includes 4 pixels D arranged in an array arrangement of two rows and two columns, wherein the color channel of the pixels in the first row and the first column is green, that is, the first row and the first row The filter set on the pixels in one column is a green filter; the color channel of the pixels in the first row and second column is red, that is, the filter set on the pixels in the first row and second column The filter is a red filter; the color channel of the pixel in the second row and the first column is blue, that is, the filter set on the pixel in the second row and the first column is a blue filter; The color channel of the pixel points in the second row and second column is green, that is, the filter set on the pixel points in the second row and second column is a green filter.

Fig. 7 is a flowchart of a focus tracking method in one of the embodiments. As shown in FIG. 7, the focus tracking method includes steps 702 to 708.

Step 702: Obtain the target subject detection area where the target subject in the preview image is located.

The preview image refers to the image obtained after the camera is focused. The subject refers to various objects, such as people, flowers, cats, dogs, cows, blue sky, white clouds, backgrounds, etc. The target subject refers to the subject in need, which can be selected according to needs. The target body detection area may be an area outlined according to the outline of the target body, or may be a rectangular frame or a round frame surrounding the target body. It should be noted that the shape of the target body area is not limited, and the target body area includes most of the target body.

Specifically, a camera device of an electronic device may be used to focus and obtain a preview image, and subject detection of the preview image may be performed to obtain a target subject detection area including the target subject. Focusing refers to the process of adjusting the focal length to make the image of the photographed object clear. Among them, the focal length refers to the distance from the optical center of the lens in the camera to the focal point of light gathering. Salient object detection refers to automatically processing regions of interest when facing a scene and selectively ignoring regions that are not of interest. In this embodiment, the region of interest is referred to as the target subject detection region. In one of the embodiments, the subject detection model is obtained by pre-collecting a large amount of training data, and inputting the training data into the subject detection model including the initial network weight for training. The subject detection model can be trained to recognize and detect various subjects, such as people, flowers, cats, dogs, backgrounds, etc.

Step 704: When the target subject moves, determine the target subject prediction area according to the target subject detection area and the movement data of the target subject, and obtain a detection image corresponding to the target subject prediction area.

The movement data of the target subject is data such as the speed, direction of movement, and trajectory of the target subject. The movement data of the target subject can be obtained by using a trained neural network model. The target subject prediction area is the area where the next time-series target subject is predicted. The detected image refers to an image including the target subject collected by the imaging device using the target subject prediction area as the focus area.

Specifically, when the target subject is a movable subject, the movement of the target subject can be detected, and the focus can be automatically tracked. According to the motion data of the target subject and the current target subject detection area, the target subject prediction area corresponding to the next time sequence is predicted. For example, you can input the first image and the second image in the trained neural network model. The first image and the second image include the same target subject. The trained neural network model can be based on the first image. The image and the second image include different movement data of the target subject, predicting the movement data of the next time sequence target subject and the target subject prediction area; it can also be inputting a first image including a moving target, and the first image includes: The current time sequence corresponds to the target subject detection area and the motion data of the target subject, and the corresponding network model can output a second image, which carries the target subject prediction area and the motion data of the target subject corresponding to the next time sequence. Focusing on the target subject prediction area, the electronic device obtains the detection image according to the pixel information of the pixel points included in each pixel point group in the image sensor. The sub-pixel included in the image sensor is a photosensitive element that can convert light signals into electrical signals. The intensity of the light signal received by the sub-pixel can be obtained according to the electrical signal output by the sub-pixel, and the intensity of the light signal received by the sub-pixel can be obtained according to the sub-pixel reception. The intensity of the obtained light signal can obtain the pixel information of the sub-pixel point.

Step 706: Obtain a phase difference value of the detection image, where the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle.

Specifically, the phase difference value of the detection image is acquired, and the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle

Possible values are 30°, 40°, 45°, 60° and other angles.

It may also be 90°, that is, when the phase difference value in the first direction refers to the phase difference value in the horizontal direction, the phase difference value in the second direction refers to the phase difference value in the vertical direction.

Step 708: Control the lens to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.

Specifically, according to the phase difference value in the first direction and the phase difference value in the second direction, the target phase difference value that is in a mapping relationship with the defocus distance value can be obtained, and then due to the difference between the target phase difference value and the target defocus distance value The corresponding relationship can be obtained through calibration, and the phase difference value in the first direction and the phase difference value in the second direction and the target phase difference value can be obtained to obtain the target defocus distance. The control lens continuously focuses on the moving target subject according to the target defocus distance value. Focus tracking refers to the process of maintaining the focus on the target subject in the subsequent shooting process after the lens focuses on the target subject, and the target subject in the acquired detection image remains clear imaging.

The focus tracking method provided in this embodiment obtains the target subject detection area in the preview image where the target subject is located. When the target subject moves, the target subject prediction area is determined according to the target subject detection area and the movement data of the target subject, and the detection image corresponding to the target subject prediction area is acquired. The phase difference value of the detection image is acquired, and the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle. The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction. The solution provided by the present application can effectively use the phase difference value to perform focus tracking for scenes with horizontal textures or vertical textures, and improve the accuracy and stability of focus tracking.

In one of the embodiments, the step of acquiring the detection image corresponding to the target subject prediction area includes: controlling the movement of the lens so that the focus is on the center of the target subject prediction area and collecting the detection image corresponding to the target subject prediction area.

Specifically, the lens is controlled to move so that the focus is on the center of the target subject prediction area and the detection image corresponding to the target subject prediction area is collected, and the detection image includes the target subject. For example, the process is as follows: when the target subject prediction area is rectangular, the focus is on the center of the rectangle; when the target subject prediction area is circular, the focus is on the dot; when the target subject prediction area is an irregular pattern , Then focus on the center of gravity of the target subject prediction area.

In one of the embodiments, determining the target subject prediction area according to the target subject detection area and the motion data of the target subject includes: inputting the first image into the prediction network model, the first image carrying the target subject detection area and the motion data of the target subject Information. Obtain a second image output by the prediction network model, and the second image is marked with a target subject prediction area.

Specifically, the predictive network model refers to a network model that has been trained. A first image including a moving target is input. The first image includes: the current time sequence corresponding to the target subject detection area and the target subject's motion data. The network model can output a second image, the second image is marked with the target subject prediction area corresponding to the next time sequence, and the motion data of the target subject can be obtained from the second image. In one of the embodiments, the predictive network model is a network model established based on a recurrent neural network algorithm. Recurrent neural network has memory, parameter sharing and Turing completeness, so it has certain advantages when learning the nonlinear characteristics of the sequence. Recurrent neural networks have applications in natural language processing (NLP) such as speech recognition, language modeling, machine translation and other fields, and are also used in various time series predictions.

In one of the embodiments, as shown in FIG. 8, the phase difference value in the first direction and the phase difference value in the second direction control the lens to continuously focus on the moving target subject, including

steps

802 and 804.

Step 802: Obtain the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction.

Specifically, the target phase difference value is determined according to the magnitude relationship between the phase difference value in the first direction and the phase difference value in the second direction or the carried confidence information. There is a mapping relationship between the target phase difference value and the target defocus distance, and the target phase difference value is input into a function used to characterize the mapping relationship, and the target defocus distance can be obtained.

Step 804: Control the lens movement of the electronic device according to the target defocus distance to continuously focus on the moving target subject.

Specifically, the target defocus distance refers to the distance between the current position of the image sensor and the position where the image sensor should be in the focus state; the electronic device can control the lens to move to the focus position according to the obtained target defocus distance Chase the focus. In one of the embodiments, the above-mentioned focus tracking method may further include: generating a depth value according to the target defocus distance value. The target defocus distance value can calculate the image distance in the in-focus state, and the object distance can be obtained according to the image distance and the focal length, and the object distance is the depth value.

In one of the embodiments, as shown in FIG. 9, the step of obtaining the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction includes: step 902 and step 904.

Step 902: Obtain a target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction.

Specifically, the target phase difference value can be determined according to the magnitude relationship between the phase difference value in the first direction and the phase difference value in the second direction or the confidence information, and the target defocus distance can be obtained according to the target phase difference value. For example, when both the phase difference value in the first direction and the phase difference value in the second direction exist, the confidence level of the phase difference value in the first direction and the confidence level of the phase difference value in the second direction can be obtained from Select a phase difference value as the target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction, and then obtain the mapping relationship between the phase difference value and the defocus distance value according to the determined target phase difference value The corresponding target defocus distance value.

Step 904: Obtain the target defocus distance according to the target phase difference value.

Specifically, there is a mapping relationship between the target phase difference value and the target defocus distance, and the target phase difference value is input into a function used to characterize the mapping relationship, and the target defocus distance can be obtained.

In one of the embodiments, obtaining the target defocus distance according to the target phase difference value includes: calculating the target defocus distance according to the calibrated defocus function and the target phase difference value, and the calibrated defocus function is used to characterize the target phase difference The relationship between the value and the target defocus distance.

Specifically, the corresponding relationship between the target defocus distance value and the target phase difference value is as follows: defocus=PD*slope(DCC), where the defocus coefficient (Defocus Conversion Coefficient, DCC) can be obtained by calibration, and PD is the target phase Difference. According to the calibrated defocus function and the target phase difference value, the target defocus distance can be calculated. The calibration process of the corresponding relationship between the target phase difference value and the defocus distance value includes: dividing the effective focus stroke of the camera module into N (N≥3) equal parts, namely (near focus DAC-far focus DAC)/N, In this way, the focus range of the motor is covered; focus is performed at each focus DAC (DAC can be 0 to 1023) position, and the phase difference of the current focus DAC position is recorded; after completing the motor focus stroke, take a group of N focus DACs Compare the PD values of, generate N similar ratios K, and fit the two-dimensional data composed of DAC and PD to obtain a straight line with slope K.

In one of the embodiments, as shown in FIG. 10, the step of obtaining the target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction includes: step 1002 to step 1006.

Step 1002: Obtain a first confidence degree corresponding to the phase difference value in the first direction.

Specifically, the phase difference value in the first direction carries information about the first confidence level, and the first confidence level is used to characterize the accuracy of the phase difference value in the first direction.

Step 1004: Obtain a second confidence level corresponding to the phase difference value in the second direction.

Specifically, the phase difference value in the second direction carries information about the second confidence level, and the second confidence level is used to characterize the accuracy of the phase difference value in the second direction.

Step 1006: Determine the target phase difference value according to the magnitude relationship between the first confidence level and the second confidence level.

Specifically, the target phase difference value is determined according to the magnitude relationship between the first confidence level and the second confidence level. For example, when the confidence level of the phase difference value in the first direction is greater than the confidence level of the phase difference value in the second direction, the accuracy of identifying the phase difference value in the first direction is higher than the accuracy of the phase difference value in the second direction. Degree, the phase difference value in the first direction can be selected as the target phase difference value; when the confidence level of the phase difference value in the first direction is less than the confidence level of the phase difference value in the second direction, the phase difference value in the first direction is identified The accuracy is lower than the accuracy of the phase difference in the second direction, the phase difference in the second direction can be selected as the target phase difference; when the confidence in the phase difference in the first direction is equal to that of the phase difference in the second direction For confidence, the accuracy of identifying the phase difference value in the first direction is equal to the accuracy of the phase difference value in the second direction. The phase difference value can be selected from the phase difference value in the first direction and the phase difference value in the second direction. The larger value is used as the target phase difference value.

In one of the embodiments, as shown in FIG. 11, the step of determining the target phase difference value according to the magnitude relationship between the first confidence level and the second confidence level includes: step 1102 to step 1106.

Step 1102: When the first degree of confidence is greater than the second degree of confidence, use the phase difference value in the first direction corresponding to the first degree of confidence as the target phase difference value.

Specifically, when the confidence level of the phase difference value in the first direction is greater than the confidence level of the phase difference value in the second direction, the phase difference value in the first direction is selected, and the corresponding defocus distance is obtained according to the phase difference value in the first direction. Value, and confirm that the moving direction is horizontal.

Step 1104: When the second degree of confidence is greater than the first degree of confidence, use the phase difference value in the second direction corresponding to the second degree of confidence as the target phase difference value.

Specifically, when the confidence level of the phase difference value in the first direction is less than the confidence level of the phase difference value in the second direction, the phase difference value in the second direction is selected, and the corresponding defocus distance is obtained according to the phase difference value in the second direction. Value, and determine that the moving direction is vertical.

Step 1106: When the first degree of confidence is equal to the second degree of confidence, both the phase difference in the first direction and the phase difference in the second direction are used as the target phase difference value.

Specifically, when the confidence level of the phase difference value in the first direction is equal to the confidence level of the phase difference value in the second direction, the defocus distance value in the horizontal direction may be determined according to the phase difference value in the first direction, and the defocus distance value in the horizontal direction may be determined according to the second direction. The phase difference value of the direction determines the defocus distance value in the vertical direction. It first moves according to the defocus distance value in the horizontal direction, and then moves according to the defocus distance value in the vertical direction, or first moves according to the defocus distance value in the vertical direction. The focus distance value moves, and then moves according to the defocus distance value in the horizontal direction. It should be noted that for scenes with horizontal textures, because the PD pixel pair in the horizontal direction cannot obtain the phase difference value in the first direction, the PD pixel pair in the vertical direction can be compared to calculate the second direction in the vertical direction. According to the phase difference value in the second direction, the defocus distance value is calculated according to the phase difference value in the second direction, and then the lens movement is controlled according to the defocus distance value in the vertical direction to achieve focus; for scenes with vertical texture, because the vertical direction The PD pixel pair cannot get the phase difference value in the second direction. You can compare the PD pixel pair in the horizontal direction to calculate the phase difference value in the first direction in the horizontal direction, and calculate the defocus distance value based on the phase difference value in the first direction. , And then control the movement of the lens according to the defocus distance value in the horizontal direction to achieve focus.

In one of the embodiments, as shown in FIG. 12, the step of obtaining the phase difference value of the detection image includes: step 1202 and step 1204.

Step 1202: According to the first direction, the detected image is divided into a first segmented image and a second segmented image. The phase difference value in the first direction is obtained according to the phase relationship corresponding to the first segmented image and the second segmented image.

Specifically, the electronic device may perform segmentation processing on the target image along the line direction (the x-axis direction in the image coordinate system). In the process of segmenting the target image along the line direction, each segment of the target image can be segmented. A dividing line is perpendicular to the direction of the row. The first segmented image and the second segmented image obtained after segmenting the target image in the direction of the row can be referred to as the left image and the right image, respectively. Obtain the phase difference value in the first direction according to the phase difference of the "mutually matched pixels" in the left and right images.

Step 1204: According to the second direction, the detected image is divided into a third segmented image and a fourth segmented image. Acquire the phase difference value in the second direction according to the phase relationship corresponding to the third segmented image and the fourth segmented image.

Specifically, the electronic device may perform segmentation processing on the target image along the direction of the column (the y-axis direction in the image coordinate system). In the process of segmenting the target image along the direction of the column, each segment of the target image is segmented. A dividing line is perpendicular to the direction of the column. The first segmented image and the second segmented image obtained after segmenting the target image in the direction of the column can be referred to as the upper image and the lower image, respectively.

In one of the embodiments, the first direction is the row direction, and the second direction is the column direction. The step of dividing the detected image into a first segmented image and a second segmented image according to the first direction includes: performing segmentation processing on the detected image according to the first direction to obtain multiple image regions, each of which includes a detected image A row of pixels in. Acquire multiple first segmented image areas and multiple second segmented image areas from multiple image areas. The first segmented image area includes even-numbered rows of pixels in the detected image, and the second segmented image area includes the detected image. Pixels in odd rows. A plurality of first segmented image areas are used to stitch together a first segmented image, and a plurality of second segmented image areas are used to form a second segmented image.

Specifically, the first direction is the row direction, and the detection image is segmented according to the first direction to obtain multiple image regions, and each image region includes a row of pixels in the detection image. Obtain multiple first segmented image areas and multiple second segmented image areas from multiple image areas. The first segmented image area refers to the pixels of the even-numbered rows in the detection image, and the second segmented image area refers to It is to detect odd-numbered rows of pixels in the image. The multiple first segmented image areas are sequentially stitched according to their positions in the detected image to obtain the first segmented image, and multiple second segmented image areas are sequentially stitched according to their positions in the detected image to obtain the second segmented image. image.

The step of dividing the detected image into a third segmented image and a fourth segmented image according to the second direction includes: performing segmentation processing on the detected image according to the second direction to obtain a plurality of image regions, and each image region includes a detected image A column of pixels in. Acquire multiple third segmented image areas and multiple fourth segmented image areas from multiple image areas, the third segmented image area includes even-numbered columns of pixels in the detected image, and the fourth segmented image area includes the detected image Pixels in odd columns. A plurality of third segmented image regions are used to stitch together a third segmented image, and a plurality of fourth segmented image regions are used to form a fourth segmented image.

Specifically, the second direction is the column direction, and the detection image is segmented according to the column direction to obtain multiple image regions, and each image region includes a column of pixels in the detection image. Obtain multiple third segmented image areas and multiple fourth segmented image areas from multiple image areas. The third segmented image area refers to the pixels in the even-numbered column of the detection image, and the fourth segmented image area refers to It is to detect odd-numbered columns of pixels in the image. The multiple third segmented image areas are sequentially stitched according to their positions in the detected image to obtain the third segmented image, and the multiple fourth segmented image areas are sequentially stitched according to their positions in the detected image to obtain the fourth segmented image. image.

In one of the embodiments, as shown in FIG. 13, the step obtains the phase difference value in the first direction according to the phase relationship corresponding to the first segmented image and the second segmented image and according to the third segmented image and the fourth segmented image. Obtaining the phase difference value in the second direction by the phase relationship corresponding to the image includes: step 1302 and step 1304.

Step 1302: Determine the phase difference value of the pixels that match each other according to the position difference of the pixels that match each other in the first segmented image and the second segmented image. The phase difference value in the first direction is determined according to the phase difference value of the pixels that match each other.

Specifically, when the first segmented image includes even-numbered rows of pixels, the second segmented image includes odd-numbered rows, and the pixel a in the first segmented image and the pixel b in the second segmented image are mutually exclusive. Matching, the phase difference value in the first direction can be determined according to the phase difference of the pixel a and the pixel b that are matched with each other.

Step 1304: Determine the phase difference value of the pixels that match each other according to the position difference of the pixels that match each other in the third segmented image and the fourth segmented image. The phase difference value in the second direction is determined according to the phase difference value of the pixels that match each other.

Specifically, when the first segmented image includes even-numbered columns of pixels, the second segmented image includes odd-numbered columns, and the pixel a in the first segmented image and the pixel b in the second segmented image match each other, Then, the phase difference value in the second direction can be determined according to the phase difference of the pixel a and the pixel b that are matched with each other.

"Matched pixels" means that the pixel matrix composed of the pixel itself and the surrounding pixels are similar to each other. For example, the pixel a and the surrounding pixels in the first segmented image form a pixel matrix with 3 rows and 3 columns, and the pixel value of the pixel matrix is:

The pixel b and the surrounding pixels in the second segmented image also form a pixel matrix with 3 rows and 3 columns, and the pixel value of the pixel matrix is:

It can be seen from the above that the two matrices are similar, and it can be considered that the pixel a and the pixel b match each other. There are many ways to judge whether the pixel matrix is similar. Usually, the pixel value of each corresponding pixel in the two pixel matrices can be calculated, and then the absolute value of the difference obtained is added, and the result of the addition is used to determine Whether the pixel matrix is similar, that is, if the result of the addition is less than a preset threshold, the pixel matrix is considered to be similar; otherwise, the pixel matrix is considered to be dissimilar.

For example, for the above two pixel matrices with 3 rows and 3 columns, the difference of 1 and 2, the difference of 15 and 15, the difference of 70 and 70, ..., and then the absolute difference Values are added, and the result of the addition is 3. If the result of the addition of 3 is less than the preset threshold, it is considered that the two pixel matrices with 3 rows and 3 columns are similar.

Another way to judge whether the pixel matrix is similar is to use the Sobel convolution kernel calculation method or the high Laplacian calculation method to extract the edge characteristics, and judge whether the pixel matrix is similar by the edge characteristics.

In the embodiments of the present application, "the positional difference of pixels that match each other" refers to the difference between the positions of the pixels in the first segmented image and the positions of the pixels in the second segmented image among the matched pixels . As in the above example, the position difference between the pixel a and the pixel b that match each other refers to the difference between the position of the pixel a in the first segmented image and the position of the pixel b in the second segmented image.

The pixels that match each other correspond to the different images formed by the imaging light entering the lens from different directions in the image sensor. For example, the pixel a in the first segmented image and the pixel b in the second segmented image match each other, where the pixel a may correspond to the image formed at position A in FIG. 1, and the pixel b may correspond to FIG. In the image formed at position B.

Since the matched pixels correspond to the different images in the image sensor formed by the imaging light entering the lens from different directions, the phase difference of the matched pixels can be determined according to the position difference of the matched pixels. .

The brightness value of the pixel points in the above pixel point group obtains the target image. After the target image is divided into two segmented images, through pixel matching, the phase difference value of the matching pixels can be quickly determined, and it also contains a wealth of phases. The difference value can improve the accuracy of the phase difference value, and improve the accuracy and stability of focusing.

It should be understood that although the various steps in the flowcharts of FIGS. 7-13 are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figure 7-13 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. These sub-steps or stages The execution order of is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

The embodiment of the present application provides a focus tracking device, which is applied to electronic equipment. As shown in FIG. 14, the focus tracking device includes: an identification module 1402, a prediction module 1404, an acquisition module 1406, and a focus tracking module 1408.

The recognition module 1402 is used to obtain the target subject detection area where the target subject in the preview image is located.

Specifically, the recognition module 1402 is configured to use an electronic device to focus and obtain a preview image, and perform subject detection on the preview image to obtain a target subject detection area including the target subject. Focusing refers to the process of adjusting the focal length to make the image of the photographed object clear. The focal length refers to the distance from the optical center of the lens in the camera to the focal point of the light. Salient object detection refers to automatically processing regions of interest when facing a scene and selectively ignoring regions that are not of interest. In this embodiment, the region of interest is referred to as the target subject detection region. In one of the embodiments, the subject detection model is obtained by pre-collecting a large amount of training data, and inputting the training data into the subject detection model including the initial network weight for training. The subject detection model can be trained to recognize and detect various subjects, such as people, flowers, cats, dogs, backgrounds, etc.

The prediction module 1404 is configured to determine the target subject prediction area according to the target subject detection area and the movement data of the target subject when the target subject moves, and obtain a detection image corresponding to the target subject prediction area.

The movement data of the target subject is data such as the speed, direction of movement, and trajectory of the target subject. The movement data of the target subject can be obtained by using a trained neural network model. The target subject prediction area is the area where the next time-series target subject is predicted. The detected image refers to an image including the target subject collected by the electronic device using the target subject prediction area as the focus area.

Specifically, the prediction module 1404 is configured to detect the movement of the target subject when the target subject is a movable subject, and automatically perform focus tracking. According to the motion data of the target subject and the current target subject detection area, the target subject prediction area corresponding to the next time sequence is predicted. For example, you can input the first image and the second image into the trained neural network model. The first image and the second image include the same target subject. The trained neural network model can be based on the first image. The image and the second image include different movement data of the target subject, which can predict the movement data of the next time series target subject and the target subject prediction area; it can also be inputting a first image including a moving target, the first image Including: the current time sequence corresponding to the target subject detection area and the motion data of the target subject, the corresponding network model can output a second image, and the second image carries the target subject prediction area and the motion data of the target subject corresponding to the next time sequence. Focusing on the target subject prediction area, the electronic device obtains the detection image according to the pixel information of the pixel points included in each pixel point group in the image sensor. The sub-pixel included in the image sensor is a photosensitive element that can convert light signals into electrical signals. The intensity of the light signal received by the sub-pixel can be obtained according to the electrical signal output by the sub-pixel. The intensity of the obtained light signal can obtain the pixel information of the sub-pixel point.

The obtaining module 1406 is used to obtain the phase difference value of the detection image, the phase difference value including the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle.

Specifically, the acquiring module 1406 is configured to acquire the phase difference value of the detection image, and the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle

Possible values are 30°, 40°, 45°, 60° and other angles.

The focus tracking module 1408 is configured to control the lens to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.

Specifically, the focus tracking module 1408 is configured to obtain the target phase difference value that is in a mapping relationship with the defocus distance value according to the phase difference value in the first direction and the phase difference value in the second direction. The correspondence between the focal distance values can be obtained by calibration, and the target defocus distance can be obtained by obtaining the phase difference value in the first direction and the phase difference value in the second direction and the target phase difference value. The control lens continuously focuses on the moving target subject according to the target defocus distance value. Focus tracking refers to the process of maintaining the focus on the target subject in the subsequent shooting process after the lens focuses on the target subject, and the target subject in the acquired detection image remains clear imaging.

The focus tracking device provided in this embodiment acquires the target subject detection area in the preview image where the target subject is located. When the target subject moves, the target subject prediction area is determined according to the target subject detection area and the movement data of the target subject, and the detection image corresponding to the target subject prediction area is acquired. The phase difference value of the detection image is acquired, and the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle. The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction. The solution provided by the present application can effectively use the phase difference value to perform focus tracking for scenes with horizontal textures or vertical textures, and improve the accuracy and stability of focus tracking.

In one of the embodiments, the prediction module is used to control the movement of the lens so that the focus is at the center of the predicted region of the target subject, and the detection image corresponding to the predicted region of the target subject is collected based on the focus.

In one of the embodiments, the prediction module is used to input the first image into the prediction network model, the first image carries the information of the target subject detection area and the motion data of the target subject, and the second image output by the prediction network model is obtained. The image carries the information of the predicted area of the target subject.

In one of the embodiments, the acquiring module is used to acquire the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction; according to the target defocus distance, the lens movement of the electronic device is controlled to continue to the moving target subject Focus.

In one of the embodiments, the obtaining module is configured to obtain the target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction; and obtain the target defocus distance according to the target phase difference value.

In one of the embodiments, the acquisition module is used to calculate the target defocus distance according to the calibrated defocus function and the target phase difference value, and the calibrated defocus function is used to characterize the relationship between the target phase difference value and the target defocus distance.

In one of the embodiments, the acquiring module is configured to acquire a first confidence level corresponding to a phase difference value in a first direction; acquire a second confidence level corresponding to a phase difference value in a second direction; according to the first confidence level and the second confidence level The degree of relationship determines the target phase difference value.

In one of the embodiments, the acquiring module is configured to use the phase difference value in the first direction corresponding to the first confidence degree as the target phase difference value when the first confidence degree is greater than the second confidence degree; When the degree of confidence is one, the phase difference in the second direction corresponding to the second degree of confidence is taken as the target phase difference value; when the first degree of confidence is equal to the second degree of confidence, the phase difference in the first direction and the phase difference in the second direction are both As the target phase difference value.

In one of the embodiments, the acquisition module is used to divide the detected image into the first segmented image and the second segmented image according to the first direction. The phase difference value in the first direction is obtained according to the phase relationship corresponding to the first segmented image and the second segmented image; the detected image is segmented into the third segmented image and the fourth segmented image according to the second direction. Acquire the phase difference value in the second direction according to the phase relationship corresponding to the third segmented image and the fourth segmented image.

In one of the embodiments, the acquisition module is used to perform segmentation processing on the detection image according to the first direction to obtain multiple image areas, each image area includes a row of pixels in the detection image, and multiple image areas are obtained from the multiple image areas. The first segmented image area and a plurality of second segmented image areas, the first segmented image area includes pixels in even-numbered lines in the detection image, and the second segmented image area includes pixels in odd-numbered lines in the detection image. The segmented image areas are spliced into the first segmented image, and multiple second segmented image areas are used to form the second segmented image; the detection image is segmented according to the second direction to obtain multiple image areas, each image The area includes a column of pixels in the detection image; multiple third segmented image areas and multiple fourth segmented image areas are obtained from the multiple image areas, and the third segmented image area includes even-numbered columns of pixels in the detected image. The four-segmented image area includes pixels of odd columns in the detection image; a plurality of third-segmented image areas are used to stitch together a third segmented image, and a plurality of fourth-segmented image areas are used to form a fourth segmented image.

In one of the embodiments, the acquisition module is configured to determine the phase difference value of the pixels that match each other according to the position difference of the pixels that match each other in the first segmented image and the second segmented image. The phase difference value in the first direction is determined according to the phase difference values of the mutually matched pixels; the phase difference value of the mutually matched pixels is determined according to the position difference of the mutually matched pixels in the third segmented image and the fourth segmented image. The phase difference value in the second direction is determined according to the phase difference value of the pixels that match each other.

The division of the various modules in the focus tracking device described above is only for illustration. In other embodiments, the focus tracking device may be divided into different modules as needed to complete all or part of the functions of the focus tracking device.

For the specific definition of the focus tracking device, please refer to the above definition of the focus tracking method, which will not be repeated here. Each module in the above-mentioned tracking device can be implemented in whole or in part by software, hardware, and a combination thereof. The foregoing modules may be embedded in the form of hardware or independent of the processor in the electronic device, or may be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.

An electronic device that includes a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the various implementations described above. The focus tracking method in the example.

The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to execute the tracking in each of the foregoing embodiments焦方法。 Focus method.

A computer program product containing instructions, when it runs on a computer, causes the computer to execute the focus tracking method in each of the above-mentioned embodiments.

In one of the embodiments of the present application, an electronic device is provided. The electronic device may be an electronic device, and the electronic device may be an electronic device with a digital image capturing function. For example, the electronic device may be a smart phone or a tablet. Computer, camera or video camera, etc. The internal structure diagram can be as shown in Figure 15. The electronic device includes a processor and a memory connected through a system bus. Among them, the processor of the electronic device is used to provide calculation and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system and computer readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. When the computer-readable instruction is executed by the processor, the focus tracking method provided in the embodiment of the present application is realized.

In addition, although not shown in FIG. 15, the electronic device may also include a lens and an image sensor, where the lens may be composed of a set of lenses, and the image sensor may be a metal oxide semiconductor element (English: Complementary Metal Oxide Semiconductor. Abbreviation) : CMOS) image sensor, charge-coupled device (English: Charge-coupled Device. Abbreviation: CCD), quantum thin film sensor or organic sensor, etc. The image sensor may be connected to the processor through a bus, and the processor may implement a focus tracking method provided by the embodiment of the present application through a signal output from the image sensor to the processor.

Those skilled in the art can understand that the structure shown in FIG. 15 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

In an embodiment of the present application, an electronic device is provided, the electronic device includes a memory and one or more processors. The memory stores computer-readable instructions. When one or more processors execute the computer-readable instructions, Implement the following steps:

Obtain the target subject detection area where the target subject in the preview image is located. When the target subject moves, the target subject prediction area is determined according to the target subject detection area and the movement data of the target subject, and the detection image corresponding to the target subject prediction area is acquired. The phase difference value of the detection image is acquired, and the phase difference value includes the phase difference value in the first direction and the phase difference value in the second direction. The first direction and the second direction form a preset angle. The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.

The implementation principles and technical effects of the computer-readable storage medium provided in this embodiment are similar to those of the foregoing method embodiments, and will not be repeated here.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer readable. In the storage medium, when the computer-readable instructions are executed, they may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

A tracking focus method is applied to an electronic device. The electronic device includes an image sensor and a lens. The image sensor includes a plurality of pixel point groups arranged in an array, and each of the pixel point groups includes M* arranged in an array. N pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2; the method includes:

Acquiring the target subject detection area where the target subject in the preview image is located;

When the target subject moves, determine a target subject prediction area according to the target subject detection area and the movement data of the target subject, and obtain a detection image corresponding to the target subject prediction area;

The image sensor is used to obtain the phase difference value of the detection image, the phase difference value includes a phase difference value in a first direction and a phase difference value in a second direction; Set an angle; and

The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.
The method according to claim 1, wherein said and acquiring a detection image corresponding to said target subject prediction area comprises:

Controlling the movement of the lens so that the focus is on the center of the target subject prediction area and collecting a detection image corresponding to the target subject prediction area.
The method according to claim 1, wherein the determining the target subject prediction area according to the target subject detection area and the motion data of the target subject comprises:

Inputting a first image into a prediction network model, the first image carrying information about the target subject detection area and the motion data of the target subject;

Acquire a second image output by the prediction network model, where the second image is marked with the target subject prediction area.
The method according to claim 3, wherein the predictive network model is a network model established based on a recurrent neural network algorithm.
The method according to claim 1, wherein the phase difference value in the first direction and the phase difference value in the second direction controlling the lens to continuously focus on the moving target subject comprises:

Acquiring the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction;

The lens movement of the electronic device is controlled according to the target defocus distance to continuously focus the moving target subject.
The method according to claim 5, wherein the obtaining the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction comprises:

Obtaining a target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction;

Obtain the target defocus distance according to the target phase difference value.
The method according to claim 6, wherein the obtaining the target defocus distance according to the target phase difference value comprises:

The target defocus distance is calculated according to the calibrated defocus function and the target phase difference value, and the calibrated defocus function is used to characterize the relationship between the target phase difference value and the target defocus distance.
The method according to claim 5, wherein the obtaining the target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction comprises:

Acquiring a first confidence level corresponding to the phase difference value in the first direction;

Acquiring a second confidence level corresponding to the phase difference value in the second direction;

The target phase difference value is determined according to the magnitude relationship between the first confidence level and the second confidence level.
The method according to claim 8, wherein the determining the target phase difference value according to the magnitude relationship between the first confidence level and the second confidence level comprises:

When the first degree of confidence is greater than the second degree of confidence, use the phase difference value in the first direction corresponding to the first degree of confidence as the target phase difference value;

When the second confidence level is greater than the first confidence level, use the phase difference value in the second direction corresponding to the second confidence level as the target phase difference value;

When the first degree of confidence is equal to the second degree of confidence, both the first direction phase difference and the second direction phase difference are used as the target phase difference value.
The method according to claim 1, wherein said obtaining the phase difference value of the detection image by the image sensor comprises:

The detection image is divided into a first segmented image and a second segmented image according to the first direction; the first segmented image is obtained according to the phase relationship corresponding to the second segmented image Phase difference in one direction;

The detection image is divided into a third segmented image and a fourth segmented image according to the second direction; the first segmented image is acquired according to the phase relationship corresponding to the third segmented image and the fourth segmented image The phase difference between the two directions.
The method according to claim 10, wherein the first direction is a row direction, and the second direction is a column direction; and the detection image is divided into a first segmented image according to the first direction And the second segmentation image, including:

The detection image is segmented according to the first direction to obtain a plurality of image regions, each of the image regions includes a row of pixels in the detection image; a plurality of first images are obtained from the plurality of image regions A segmented image area and a plurality of second segmented image areas, the first segmented image area includes pixels of even-numbered lines in the detection image, and the second segmented image area includes odd-numbered lines in the detection image的pixels; using the plurality of first segmented image areas to stitch together into the first segmented image, using the multiple second segmented image areas to form the second segmented image;

The segmenting the detection image into a third segmented image and a fourth segmented image according to the second direction includes:

The detection image is segmented according to the second direction to obtain a plurality of image regions, each of the image regions includes a column of pixels in the detection image; and a plurality of first images are obtained from the plurality of image regions A three-segmented image area and a plurality of fourth-segmented image areas, the third segmented image area includes pixels in even-numbered columns in the detection image, and the fourth segmented image area includes odd-numbered columns in the detection image的pixels; using the plurality of third segmented image areas to form the third segmented image, and using the plurality of fourth segmented image areas to form the fourth segmented image.
The method according to claim 10, wherein the phase difference value in the first direction and the phase difference value in the first direction according to the phase relationship corresponding to the first divided image and the second divided image are obtained and the phase difference value according to the first divided image Obtaining the phase difference value in the second direction by the phase relationship corresponding to the three segmented image and the fourth segmented image includes:

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the first segmented image and the second segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels Phase difference in one direction;

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the third segmented image and the fourth segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels The phase difference between the two directions.
A tracking focus device, wherein, applied to an electronic device, the electronic device includes an image sensor and a lens, the image sensor includes a plurality of pixel point groups arranged in an array, and each of the pixel point groups includes an array arranged M*N pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2, including:

Recognition module, used to obtain the target subject detection area where the target subject in the preview image is located

A prediction module, configured to determine a target subject prediction area according to the target subject detection area and movement data of the target subject when the target subject moves, and obtain a detection image corresponding to the target subject prediction area;

The acquiring module is configured to acquire the phase difference value of the detection image by using the image sensor, the phase difference value including a phase difference value in a first direction and a phase difference value in a second direction; the first direction and the phase difference value The second direction is a preset angle; and

The focus tracking module is configured to control the lens to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.
An electronic device, including a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the one or more Each processor performs the following steps:

The electronic device includes an image sensor and a lens, the image sensor includes a plurality of pixel point groups arranged in an array, and each pixel point group includes M*N pixel points arranged in an array; each pixel point corresponds to one The photosensitive unit, where M and N are both natural numbers greater than or equal to 2;

Acquiring the target subject detection area where the target subject in the preview image is located;

When the target subject moves, determine a target subject prediction area according to the target subject detection area and the movement data of the target subject, and obtain a detection image corresponding to the target subject prediction area;

The image sensor is used to obtain the phase difference value of the detection image, the phase difference value includes a phase difference value in a first direction and a phase difference value in a second direction; Set an angle; and

The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.
The electronic device according to claim 14, wherein the processor further executes the following steps when executing the computer-readable instructions:

Controlling the movement of the lens so that the focus is on the center of the target subject prediction area and collecting a detection image corresponding to the target subject prediction area.
The electronic device according to claim 14, wherein the processor further executes the following steps when executing the computer-readable instructions:

Inputting a first image into a prediction network model, the first image carrying information about the target subject detection area and the motion data of the target subject;

Acquire a second image output by the prediction network model, where the second image is marked with the target subject prediction area.
The electronic device according to claim 16, wherein the predictive network model is a network model established based on a recurrent neural network algorithm.
The electronic device according to claim 16, wherein the processor further executes the following steps when executing the computer-readable instructions:

Acquiring the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction;

The lens movement of the electronic device is controlled according to the target defocus distance to continuously focus the moving target subject.
The electronic device according to claim 18, wherein the processor further executes the following steps when executing the computer-readable instructions:

Obtaining a target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction;

Obtain the target defocus distance according to the target phase difference value.
The electronic device according to claim 19, wherein the processor further executes the following steps when executing the computer-readable instructions:

The target defocus distance is calculated according to the calibrated defocus function and the target phase difference value, and the calibrated defocus function is used to characterize the relationship between the target phase difference value and the target defocus distance.
The electronic device according to claim 18, wherein the processor further executes the following steps when executing the computer-readable instructions:

Acquiring a first confidence level corresponding to the phase difference value in the first direction;

Acquiring a second confidence level corresponding to the phase difference value in the second direction;

The target phase difference value is determined according to the magnitude relationship between the first confidence level and the second confidence level.
The electronic device according to claim 21, wherein the processor further executes the following steps when executing the computer-readable instructions:

When the first degree of confidence is greater than the second degree of confidence, use the phase difference value in the first direction corresponding to the first degree of confidence as the target phase difference value;

When the second confidence level is greater than the first confidence level, use the phase difference value in the second direction corresponding to the second confidence level as the target phase difference value;

When the first degree of confidence is equal to the second degree of confidence, both the first direction phase difference and the second direction phase difference are used as the target phase difference value.
The electronic device according to claim 14, wherein the processor further executes the following steps when executing the computer-readable instructions:

The detection image is divided into a first segmented image and a second segmented image according to the first direction; the first segmented image is obtained according to the phase relationship corresponding to the second segmented image Phase difference in one direction;

The detection image is divided into a third segmented image and a fourth segmented image according to the second direction; the first segmented image is acquired according to the phase relationship corresponding to the third segmented image and the fourth segmented image The phase difference between the two directions.
The electronic device according to claim 14, wherein the first direction is a row direction and the second direction is a column direction, and the processor further executes the following steps when executing the computer-readable instruction:

The detection image is segmented according to the first direction to obtain a plurality of image regions, each of the image regions includes a row of pixels in the detection image; a plurality of first images are obtained from the plurality of image regions A segmented image area and a plurality of second segmented image areas, the first segmented image area includes pixels of even-numbered lines in the detection image, and the second segmented image area includes odd-numbered lines in the detection image的pixels; using the plurality of first segmented image areas to stitch together into the first segmented image, using the multiple second segmented image areas to form the second segmented image;

The detection image is segmented according to the second direction to obtain a plurality of image regions, each of the image regions includes a column of pixels in the detection image; and a plurality of first images are obtained from the plurality of image regions A three-segmented image area and a plurality of fourth-segmented image areas, the third segmented image area includes pixels in even-numbered columns in the detection image, and the fourth segmented image area includes odd-numbered columns in the detection image的pixels; using the plurality of third segmented image areas to form the third segmented image, and using the plurality of fourth segmented image areas to form the fourth segmented image.
The electronic device according to claim 23, wherein the processor further executes the following steps when executing the computer-readable instructions:

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the first segmented image and the second segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels Phase difference in one direction;

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the third segmented image and the fourth segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels The phase difference between the two directions.
One or more computer-readable storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors perform the following steps:

Acquiring the target subject detection area where the target subject in the preview image is located;

When the target subject moves, determine a target subject prediction area according to the target subject detection area and the movement data of the target subject, and obtain a detection image corresponding to the target subject prediction area;

The image sensor is used to obtain the phase difference value of the detection image, the phase difference value includes a phase difference value in a first direction and a phase difference value in a second direction; Set an angle; the image sensor includes a plurality of pixel point groups arranged in an array, each of the pixel point groups includes M*N pixel points arranged in an array; each pixel point corresponds to a photosensitive unit, where M And N are both natural numbers greater than or equal to 2; and

The lens is controlled to continuously focus on the moving target subject according to the phase difference value in the first direction and the phase difference value in the second direction.
The storage medium according to claim 26, wherein the processor further executes the following steps when executing the computer-readable instructions:

Controlling the movement of the lens so that the focus is on the center of the target subject prediction area and collecting a detection image corresponding to the target subject prediction area.
The storage medium according to claim 26, wherein the processor further executes the following steps when executing the computer-readable instructions:

Inputting a first image into a prediction network model, the first image carrying information about the target subject detection area and the motion data of the target subject;

Acquire a second image output by the prediction network model, where the second image is marked with the target subject prediction area.
The storage medium according to claim 28, wherein the predictive network model is a network model established based on a recurrent neural network algorithm.
The storage medium according to claim 26, wherein the processor further executes the following steps when executing the computer-readable instructions:

Acquiring the target defocus distance according to the phase difference value in the first direction and the phase difference value in the second direction;

The lens movement of the electronic device is controlled according to the target defocus distance to continuously focus the moving target subject.
The storage medium according to claim 30, wherein the processor further executes the following steps when executing the computer-readable instructions:

Obtaining a target phase difference value according to the phase difference value in the first direction and the phase difference value in the second direction;

Obtain the target defocus distance according to the target phase difference value.
The storage medium according to claim 31, wherein the processor further executes the following steps when executing the computer-readable instructions:

The target defocus distance is calculated according to the calibrated defocus function and the target phase difference value, and the calibrated defocus function is used to characterize the relationship between the target phase difference value and the target defocus distance.
The storage medium according to claim 30, wherein the processor further executes the following steps when executing the computer-readable instructions:

Acquiring a first confidence level corresponding to the phase difference value in the first direction;

Acquiring a second confidence level corresponding to the phase difference value in the second direction;

The target phase difference value is determined according to the magnitude relationship between the first confidence level and the second confidence level.
The storage medium according to claim 33, wherein the processor further executes the following steps when executing the computer-readable instructions:

When the first degree of confidence is greater than the second degree of confidence, use the phase difference value in the first direction corresponding to the first degree of confidence as the target phase difference value;

When the second confidence level is greater than the first confidence level, use the phase difference value in the second direction corresponding to the second confidence level as the target phase difference value;

When the first degree of confidence is equal to the second degree of confidence, both the first direction phase difference and the second direction phase difference are used as the target phase difference value.
The storage medium according to claim 26, wherein the processor further executes the following steps when executing the computer-readable instructions:

The detection image is divided into a first segmented image and a second segmented image according to the first direction; the first segmented image is obtained according to the phase relationship corresponding to the second segmented image Phase difference in one direction;

The detection image is divided into a third segmented image and a fourth segmented image according to the second direction; the first segmented image is acquired according to the phase relationship corresponding to the third segmented image and the fourth segmented image The phase difference between the two directions.
The storage medium according to claim 26, wherein the first direction is a row direction and the second direction is a column direction, and the processor further executes the following steps when executing the computer-readable instruction:

The detection image is segmented according to the first direction to obtain a plurality of image regions, each of the image regions includes a row of pixels in the detection image; a plurality of first images are obtained from the plurality of image regions A segmented image area and a plurality of second segmented image areas, the first segmented image area includes pixels of even-numbered lines in the detection image, and the second segmented image area includes odd-numbered lines in the detection image的pixels; using the plurality of first segmented image areas to stitch together into the first segmented image, using the multiple second segmented image areas to form the second segmented image;

The detection image is segmented according to the second direction to obtain a plurality of image regions, each of the image regions includes a column of pixels in the detection image; and a plurality of first images are obtained from the plurality of image regions A three-segmented image area and a plurality of fourth-segmented image areas, the third segmented image area includes pixels in even-numbered columns in the detection image, and the fourth segmented image area includes odd-numbered columns in the detection image的pixels; using the plurality of third segmented image areas to form the third segmented image, and using the plurality of fourth segmented image areas to form the fourth segmented image.
The storage medium according to claim 35, wherein the processor further executes the following steps when executing the computer-readable instructions:

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the first segmented image and the second segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels Phase difference in one direction;

Determine the phase difference value of the mutually matched pixels according to the position difference of the matched pixels in the third segmented image and the fourth segmented image; determine the phase difference value of the mutually matched pixels according to the phase difference value of the mutually matched pixels The phase difference between the two directions.