WO2020000262A1

WO2020000262A1 - Light source estimating method, image processing method and related products

Info

Publication number: WO2020000262A1
Application number: PCT/CN2018/093144
Authority: WO
Inventors: 林威丞
Original assignee: 华为技术有限公司
Priority date: 2018-06-27
Filing date: 2018-06-27
Publication date: 2020-01-02
Also published as: CN112313946A

Abstract

A light source estimating method, an image processing method and related products. The light source estimating method maps a color brightness information group of a sub-block of image segmentation into a color brightness three-dimensional space to obtain a plurality of color brightness sample points in the color brightness three-dimensional space, performs hierarchical cluster processing on the plurality of color brightness sample points, and calculates the blending degree of the color brightness sample point cluster of each layer to calculate the blending degree of the entire image, and estimates whether a plurality of light sources of different color temperatures exists in the photographing environment of the image based on the blending degree. Since the light source estimating method can estimate whether a plurality of light sources of different color temperatures exists in the photographing environment of the image to some extent, this provides a basis for performing corresponding image correction based on the case that a plurality of light sources of different color temperatures is comprised, so that it is possible to perform targeted image correction in the case that a plurality of light sources of different color temperatures is comprised, thereby facilitating improving the targeted characteristic and rationality of image correction.

Description

Light source estimation method, image processing method and related products

Technical field

The present application relates to the field of image processing, and in particular, to a light source estimation method, an image processing method, and related products.

Background technique

Daylight is a natural light source. Except for the natural light source such as daylight, artificial light sources can be generated by different lamps, and the color temperature of daylight and light sources of different lamps are different. Color temperature is a physical quantity that evaluates the color of a light source. It is defined as heating a black body (absolute black radiator, which is similar to a closed carbon block that does not reflect incident light) to a certain temperature, and it will emit light. Black body temperature changes. When the color of the light emitted by a light source is the same as the color of the black body, the temperature of the black body is the color temperature of the light source.

The color temperature of the light source can be classified into high color temperature, medium color temperature, and low color temperature from high to low. The color of high color temperature light sources is light blue, and the color of low color temperature light sources is light yellow. When shooting with a digital camera, the unprocessed original image suffers from overall color cast (blue, yellow, or green). The color of the deviation is affected by the color (color temperature) of the light source in the scene. To eliminate this overall color cast, it is generally achieved by automatic white balance (AWB) correction technology. AWB correction technology can calculate the color temperature of the light source in the shooting environment. This corrects the color cast of the image, and strives to make the white objects in the original scene appear white in the image.

When there are multiple light sources in the shooting environment and the color temperature of each light source is different, such as high, medium, and low color temperatures at the same time, the traditional AWB correction technology cannot determine whether there is a mixed color temperature in the environment, so usually only one light source color temperature can be estimated as the color deviation of the corrected image in accordance with. For example, if the color temperature of the light source calculated by the AWB correction technology is closer to the high color temperature in the environment, the low color temperature area in the corrected image will be yellowish, and if the calculated light source color temperature is closer to the low color temperature, the high color temperature area in the image There is a bluish problem. Because the human eye is not sensitive to the difference in the color of the light source in the mixed color temperature light source scene (the brain will think of the light source as close to white), in the mixed color temperature scene, the light blue and yellow light source colors in the image may trigger the user Feel the color is wrong. Therefore, how to estimate the number of light sources with different color temperatures in the environment has become an urgent need.

Summary of the invention

The embodiments of the present application provide a light source estimation method, an image processing method, and related products.

In a first aspect, an embodiment of the present application provides a method for estimating a light source, which may include:

Divide the image into m sub-blocks (m sub-blocks may be all the same size, partially the same, or different from each other, and the shape of the sub-blocks may be square, rectangular, or other shapes), where m is an integer greater than 1. Acquiring m color luminance information groups of the m sub-blocks, each color luminance information group corresponding to one sub-block (that is, one-to-one correspondence between the m sub-blocks and m color luminance information groups), the color luminance The information group includes brightness information and color information. The image may be an original image or another image captured by a camera.

Map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where each color brightness sample corresponds to one color brightness information group ( That is, there is a one-to-one correspondence between m color brightness sample points and m color brightness information groups in a color brightness three-dimensional space. The color brightness three-dimensional space includes two color dimensions and one brightness dimension.

When the m color brightness sample points are divided into k layers along the brightness dimension (it can be understood that when k is equal to 1, it means that m color brightness sample points are divided into the same layer, then the action of layering is also (Not actually performed), and in the case where each of the k layers is divided into P color luminance sample clusters, a first blending degree of the number of clusters P corresponding to each of the k layers is calculated. Each of the k layers corresponds to a brightness interval (continuous brightness interval), where P is an integer greater than 1, and k is a positive integer.

A second blending degree of the number of clusters P corresponding to the image is determined based on the first blending degree of the number of clusters P corresponding to each of the k layers. It can be understood that when k is equal to 1, it means that m color brightness samples are divided into the same layer, then only the first blending degree of the number of clusters P corresponding to the only layer is obtained. At this time, the The first blending degree of the number of clusters P corresponding to this layer is the second blending degree of the number of clusters P corresponding to the image.

Compare the second blending degree of the corresponding number of clusters P of the image with the blending degree threshold. In a case where the second blending degree of the corresponding grouping number P of the image is greater than the blending degree threshold, it is estimated that there are at least P light sources with different color temperatures in the shooting environment corresponding to the image.

The blending degree between the two color luminance sample clusters can indicate the closeness of the connection between the two color luminance sample clusters. Among them, the greater the degree of blending between the two clusters, the higher the degree of closeness between the color brightness samples of the two clusters; the smaller the degree of blending between the two clusters, the color brightness samples of the two clusters The tighter the connection is.

When k is greater than 1, the number of color brightness samples included in different layers of the k layers may be all the same, partly the same, or different from each other.

When k is greater than 1, the heights of the brightness intervals corresponding to different layers in the k layers may be all the same, partially the same, or different from each other.

In addition, in some possible implementation manners, when the second blending degree of the corresponding clustering number P of the image is less than or equal to the blending degree threshold, it may be estimated that there are differences in the shooting environments corresponding to the images. The number of color temperature light sources is not P (in this case, there may be a monochrome temperature light source or a mixed color temperature light source in the shooting environment corresponding to the image).

It can be seen that the embodiment of the present application provides an effective light source estimation method. This light source estimation method obtains the color brightness by mapping the color brightness information group of the image segmented sub-blocks to the color brightness three-dimensional space. Multiple color brightness sample points in three-dimensional space, and perform multiple layer clustering processing on multiple color brightness sample points, and then calculate the blending degree of the entire image by calculating the blending degree of the color brightness sample point clustering of each layer, and based on this It is estimated that there are multiple light sources with different color temperatures in the shooting environment corresponding to the image, which lays a foundation for image correction based on the situation that includes multiple light sources with different color temperatures.

In some possible implementation manners, determining the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers may include: The first blending degree corresponding to the number of clusters P of each layer is summed or weighted to obtain a second blending degree corresponding to the number of clusters P of the image.

Wherein, when the first blending degree P corresponding to the number of clusters P in each of the k layers is weighted and summed to obtain the second blending degree corresponding to the number of clusters P in the image, then the number of clusters corresponding to each layer The weighted summation weight of the first blending degree of P may be determined based on the height of the brightness interval of each layer. For example, the higher the weighted summation weight of a layer with a relatively higher brightness interval, the greater the correspondingly smaller height of the brightness interval. Layer, the corresponding weighted summation weight can be smaller. Or the weighted summation weight of the first blending degree corresponding to the number of clusters P of each layer can be determined based on the number of color brightness samples of each layer. For example, the layer with a relatively large number of color brightness samples has a corresponding weighted summation weight The larger the layer with a relatively small number of color luminance samples, the smaller the corresponding weighted summation weight can be. Of course, the weighted summation weight of each layer can also be determined based on other parameters.

In some possible implementation manners, when the P is greater than 2, calculating the first blending degree of the number of clusters P corresponding to the i-th layer in the k-layer may include: calculating each of the P-groups of the i-th layer A third degree of fusion between the two clusters; summing or weighting the third degree of fusion between each two clusters in the P clusters to obtain the number of clusters P corresponding to the i-th layer First degree of blending. The i-th layer is any one of the k layers.

In some possible implementation manners, when the P is equal to 2; calculating the first blending degree of the number of clusters P corresponding to the i-th layer in the k layers may include: calculating between two clusters of the i-th layer A third degree of integration; wherein the first degree of integration of the number of clusters P corresponding to the i-th layer is equal to the third degree of integration between the two clusters. The i-th layer is any one of the k layers.

In some possible implementation manners, calculating the third blending degree between the cluster gi and the cluster gj may include inserting a continuous array of measurement cells between the center point of the cluster gi and the center point of the cluster gj, the continuous array The number of measurement cells is T. The statistics include the number of measurement cells Q of the color brightness samples in the clusters gi and gj. A third blending degree between the cluster gi and the cluster gj is determined as Q / T, where the T and the Q are integers, the T is greater than 0 and the Q is greater than or equal to 0. The cluster gi and the cluster gj are any two clusters among the P clusters in the i-th layer.

In some possible implementations, the length of a single measurement cell may be Dist_D65_D50. The width of a single measurement cell can be Dist_D65_D50 / 32. Of course, the length and width of the measurement cell are also designed to other values, and the specific values can be set based on the needs of the scene.

In some possible implementation manners, the length direction of the continuously arranged measurement cells may be perpendicular to a line connecting the center point of the cluster gi and the center point of the cluster gj. Of course, the length direction of the continuously arranged measurement cells may not be perpendicular to the line connecting the center point of the cluster gi and the center point of the cluster gj (the angle between the length direction and the line connecting the center points) The range may be, for example, 60 ° to 90 °).

In some possible implementation manners, the two color dimensions included in the three-dimensional color luminance space are a first color dimension and a second color dimension, wherein a first color dimension coordinate of a center point of any cluster is equal to the any cluster The average value of the first color dimension coordinates of all color brightness samples in the cluster, and the second color dimension coordinate of the center point of the any cluster is equal to the average of the second color dimension coordinates of all color brightness samples in the any cluster value. Specifically, for example, the first color dimension coordinate of the center point of the cluster gi is equal to the average value of the first color dimension coordinates of all color brightness samples in the cluster gi, and the second color dimension coordinate of the center point of the cluster gi is equal to the cluster. The average value of the second color dimension coordinates of all color brightness samples in gi.

In some possible implementations, since P is an integer greater than 1, the calculation, determination, comparison, and estimation may be performed for P = 2,... X, respectively, where X is an integer greater than 2. When it is estimated that there are P = 2, ... Y light sources with different color temperatures in the shooting environment, it can be determined that P = Y light sources with different color temperatures exist in the shooting environment. The Y is an integer greater than 2 and less than or equal to the X.

For example, set P = P1 and P = P2 respectively;

Then, calculate a first blending degree of the number of clusters P1 corresponding to each layer in the k layers; and determine a number of clusters P1 corresponding to the image based on the first blending degree of the number of clusters P1 corresponding to each layer in the k layers The second blending degree of the image; comparing the second blending degree of the corresponding clustering number P1 of the image with the blending degree threshold of the clustering number P1. In addition, a first blending degree of the number of clusters P2 corresponding to each of the k layers is calculated; and a number of clusters P2 corresponding to the image is determined based on the first blending degree of the number of clusters P2 corresponding to each of the k layers. The second blending degree of the image; comparing the second blending degree of the corresponding number of clusters P2 of the image with the blending degree threshold of the number of clusters P2.

The second degree of fusion of the corresponding number of clusters P1 in the image is greater than the threshold of the degree of fusion of the number of clusters P1, and the second degree of fusion of the corresponding number of clusters P2 of the image is less than or equal to the threshold of the degree of fusion of the number of clusters P2 In this case, it can be estimated that there are P1 light sources with different color temperatures in the shooting environment of the image.

In addition, the second degree of fusion of the corresponding number of clusters P1 in the image is greater than the threshold of the degree of fusion of the number of clusters P1 and the second degree of fusion of the corresponding number of clusters P2 of the image is greater than the threshold of the degree of fusion of the number of clusters P2 In the case, when the P2 is larger than the P1 (for example, P2 = P1 + 1), it can be estimated that there are P2 light sources with different color temperatures in the shooting environment of the image.

Or in another optional implementation manner i, the second blending degree of the corresponding number of clusters P1 of the image is greater than the threshold of the blending degree of the number of clusters P1, and the second corresponding number of clusters of the image P2 When the blending degree is greater than the blending degree threshold of the number of clusters P2, when the second blending degree of the corresponding clustering number P2 of the image and the blending degree threshold of the clustering number P2 are different (this difference can be expressed as (the The second degree of fusion corresponding to the number of clusters P2 of the image-the degree of fusion threshold of the number of clusters P2) / the threshold of the degree of fusion) is greater than the second degree of fusion of the corresponding number of clusters P1 and the threshold of the degree of fusion of the number of clusters P1 It is estimated that the shooting environment corresponding to the image includes P2 light sources with different color temperatures.

In a second aspect, an embodiment of the present application provides another method for estimating a light source, which may include:

Step S1: Divide the image into m sub-blocks, where m is an integer greater than 1.

Step S2: Obtain m color brightness information groups of the m sub-blocks, where each color brightness information group corresponds to one sub-block, and the color brightness information group includes brightness information and color information.

Step S3: Map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, each color brightness sample point corresponding to one color brightness information group The color brightness three-dimensional space includes two color dimensions and one brightness dimension.

Step S4: divide the m color brightness samples into k layers along the brightness dimension;

Step S5: Assign a value that has not been selected from the set of candidate numbers of light sources with different color temperatures to P.

Step S6: Divide each of the k layers into P color brightness sample point clusters, and calculate a first blending degree of the number of clusters P corresponding to each of the k layers. Each of the k layers corresponds to a brightness interval, where P is an integer greater than 1, and k is a positive integer.

Step S7: Determine a second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers.

Step S8: Compare the second blending degree of the corresponding clustering number P of the image with the blending degree threshold. In a case where the second blending degree of the corresponding grouping number P of the image is greater than the blending degree threshold, it is estimated that there are at least P light sources with different color temperatures in the shooting environment corresponding to the image. Return to step S5.

Among them, the candidate quantity set includes, for example, X-1 candidate quantities such as 2, ... X. Since each time a quantity that has not been selected in the candidate quantity set is assigned to P, then the process is performed X-1 times. After execution (that is, steps S5 to S8 are executed cyclically X-1 times), each candidate quantity of the candidate quantity set will be assigned to P in turn, that is, P = 2, ... X, which is equivalent to respectively The steps of calculation, determination, comparison and estimation are performed for P = 2, ... X.

In a third aspect, an embodiment of the present application further provides an image processing method, including:

The light source estimation method according to any one of the first aspect or the second aspect is performed. The image is corrected according to P light sources with different color temperatures, wherein the correction includes at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction.

In a fourth aspect, an embodiment of the present application further provides a light source estimation device. The light source estimation device may include a segmentation unit, an acquisition unit, a mapping unit, a calculation unit, a determination unit, a comparison unit, and an estimation unit.

The segmentation unit is configured to segment an image into m sub-blocks, where m is an integer greater than 1.

The obtaining unit is configured to obtain m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group includes brightness information and color information.

A mapping unit, configured to map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where each color brightness sample is associated with a color The brightness information group corresponds, and the three-dimensional space of color brightness includes two color dimensions and one brightness dimension.

A calculation unit, configured to calculate all the m color luminance sample points into k layers along the luminance dimension, and each of the k layers is divided into P color luminance sample groups. The first blending degree of the number of clusters P corresponding to each of the k layers is described. Each of the k layers corresponds to a brightness interval (continuous brightness interval). Wherein, P is an integer greater than 1, and k is a positive integer.

The determining unit is configured to determine a second blending degree of the number of clusters P corresponding to the image based on a first blending degree of the number of clusters P corresponding to each of the k layers.

The comparison unit is configured to compare the second blending degree of the corresponding grouping number P of the image with the blending degree threshold.

Wherein, an estimation unit is configured to, when the second blending degree corresponding to the corresponding number of clusters P of the image is greater than the blending degree threshold, estimate that at least P different color temperatures exist in the shooting environment corresponding to the image. light source.

In addition, the estimation unit may be further configured to estimate light sources with different color temperatures in the shooting environment corresponding to the image when the second blending degree of the corresponding clustering number P of the image is less than the blending degree threshold. The number is not P.

In some possible implementation manners, where k is greater than 1, in determining an aspect of the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers, The determining unit is specifically configured to perform summing or weighted summing processing on a first blending degree of the number of clusters P corresponding to each of the k layers to obtain a second blend of the corresponding number of clusters P of the image degree.

In some possible implementation manners, the P is greater than 2; in calculating a first blending degree of the number of clusters P corresponding to the i-th layer in the k-layers, the calculation unit is specifically configured to calculate the first a third blending degree between every two clusters in the P clusters of the i-layer; summing or weighted summing the third blending degree between every two clusters in the P clusters to obtain the first The first blending degree of the grouping number P corresponding to the i-layer, and the i-th layer is any one of the k-layers.

In some possible implementation manners, in terms of calculating a third blending degree between the cluster gi and the cluster gj, the calculation unit is specifically configured to:

A continuous array of measurement cells is inserted between the center point of the cluster gi and the center point of the cluster gj, and the number of the continuous array measurement cells is T. Wherein, the cluster gi and the cluster gj are any two of the P clusters in the i-th layer.

The statistics include the number of measurement cells Q of the color brightness sample points in the cluster gi and the cluster gj; it is determined that the third blending degree between the cluster gi and the cluster gj is Q / T, where T and all Said Q is an integer, said T is greater than 0 and said Q is greater than or equal to 0.

In some possible implementation manners, a length direction of the continuously arranged measurement cells is perpendicular to a line connecting a center point of the cluster gi and a center point of the cluster gj.

In some possible implementation manners, the two color dimensions included in the three-dimensional color luminance space are a first color dimension and a second color dimension, wherein a first color dimension coordinate of a center point of any cluster is equal to the any cluster The average value of the first color dimension coordinates of all color brightness samples in the cluster, and the second color dimension coordinate of the center point of the any cluster is equal to the average of the second color dimension coordinates of all color brightness samples in the any cluster value.

In some possible implementation manners, the calculation unit, the determination unit, the comparison unit, and the estimation unit may perform the calculation, the determination, the comparison, and In the estimation, X is an integer greater than two.

The estimation unit is further configured to determine, when it is estimated that P = 2, ... Y light sources with different color temperatures exist in the shooting environment, determine that P = Y light sources with different color temperatures exist in the shooting environment. , Wherein Y is an integer greater than 2 and less than or equal to X.

In a fifth aspect, an embodiment of the present application further provides a light source estimation device. The light source estimation device may include a segmentation circuit, an acquisition circuit, a mapping circuit, a calculation circuit, a determination circuit, a comparison circuit, and an estimation circuit.

A segmentation circuit is used to segment an image into m sub-blocks, where m is an integer greater than 1.

The obtaining circuit is configured to obtain m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group includes brightness information and color information.

A mapping circuit, configured to map the m color luminance information groups to a color luminance three-dimensional space to obtain m color luminance samples located in the color luminance three-dimensional space, where each color luminance sample is associated with a color The brightness information group corresponds, and the three-dimensional space of color brightness includes two color dimensions and one brightness dimension.

A calculation circuit, configured to calculate all the m color luminance sample points divided into k layers along the luminance dimension, and each of the k layers is divided into P color luminance sample groups. The first blending degree of the number of clusters P corresponding to each of the k layers is described. Each of the k layers corresponds to a brightness interval (continuous brightness interval), where P is an integer greater than 1, and k is a positive integer.

A determining circuit is configured to determine a second blending degree of the number of clusters P corresponding to the image based on a first blending degree of the number of clusters P corresponding to each of the k layers.

The comparison circuit is configured to compare the second blending degree and the blending degree threshold of the corresponding grouping number P of the image.

An estimation circuit is configured to estimate that at least P light sources with different color temperatures exist in the shooting environment corresponding to the image when the second blending degree of the corresponding clustering number P of the image is greater than the blending degree threshold.

In some possible implementation manners, where k is greater than 1, in determining an aspect of the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers, The determining circuit is specifically configured to perform summing or weighted summing processing on a first blending degree of the number of clusters P corresponding to each of the k layers to obtain a second blend of the corresponding number of clusters P of the image degree.

In some possible implementation manners, the P is greater than 2; in calculating a first blending degree of the number of clusters P corresponding to the i-th layer in the k-layers, the calculation circuit is specifically configured to calculate the first a third blending degree between every two clusters in the P clusters of the i-layer; summing or weighted summing the third blending degree between every two clusters in the P clusters to obtain the first The first blending degree of the grouping number P corresponding to the i-layer, and the i-th layer is any one of the k-layers.

In some possible implementation manners, when the P is equal to 2; in terms of calculating a first blending degree of the number of clusters P corresponding to the ith layer in the k layer, the calculation circuit may be specifically configured to: The third degree of integration between the two subgroups of the i-th layer is described, wherein the first degree of integration of the number of subgroups P corresponding to the i-th layer is equal to the third degree of integration between the two subgroups. The i-th layer is any one of the k layers.

In some possible implementation manners, in terms of calculating a third degree of fusion between the cluster gi and the cluster gj, the calculation circuit is specifically configured to:

A continuous array of measurement cells is inserted between the center point of the cluster gi and the center point of the cluster gj, and the number of the continuously arrayed measurement cells is T; where the cluster gi and the cluster gj are the i-th Any two of the P groupings of the layer;

According to a sixth aspect, an embodiment of the present application further provides an image processing apparatus, including:

Mutually coupled light source estimation device and correction device.

The light source estimation device is any one of the light source estimation devices provided in the fifth aspect or the fourth aspect.

The correction device is configured to correct the image according to P light sources with different color temperatures, and the correction includes at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction.

The correction device may be, for example, an image signal processor (ISP). The ISP may include at least one of the following correction circuits: an automatic white balance correction circuit, a color correction circuit, and a saturation correction circuit. , Or contrast correction circuit.

In a seventh aspect, an embodiment of the present application further provides a light source estimation device. The light source estimation device includes a processor and a memory coupled to each other. The memory stores a computer program. The processor is configured to call the memory. A computer program stored in the computer to execute any one of the light source estimation methods provided in the first aspect or the second aspect.

According to an eighth aspect, an embodiment of the present application further provides an image processing apparatus. The light source estimation apparatus includes a processor and a memory coupled to each other. The memory stores a computer program. The processor is configured to call the memory. A stored computer program to execute any one of the image processing methods provided by the third aspect.

In a ninth aspect, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, wherein the computer program is executed by related hardware to complete the first aspect or the second aspect Any of the light source estimation methods provided.

According to a tenth aspect, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by related hardware to complete any one of the images provided by the third aspect. Approach.

According to an eleventh aspect, an embodiment of the present application further provides a computer program product, and when the computer program product runs on a computer, the computer is caused to execute any one of the light source estimation methods provided in the first aspect or the second aspect.

In a twelfth aspect, an embodiment of the present application further provides a computer program product, and when the computer program product runs on a computer, the computer is caused to execute any one of the image processing methods provided in the third aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic diagram of a device system architecture according to an example of the present application.

FIG. 1B is a schematic structural diagram of an image processing component provided as an example in an embodiment of the present application.

FIG. 1C is a color temperature level division method of a standard light source provided by way of example in the embodiment of the present application.

FIG. 2 is a schematic diagram of mapping a color information group to a two-dimensional color coordinate plane according to an example of the present application.

3A and 3B are schematic diagrams of sub-speed division of several images provided by way of example in the embodiment of the present application.

FIG. 4 is a schematic diagram of the distribution of color brightness samples in a mixed color temperature light source scene provided by way of example in the embodiment of the present application.

FIG. 5 is a schematic diagram of the distribution of color brightness samples in a monochrome temperature light source scene provided by way of example in the embodiment of the present application.

FIG. 6 is a schematic flowchart of a light source estimation method according to an embodiment of the present application.

FIG. 7 is a schematic diagram of color brightness sample points layered along the brightness dimension according to an embodiment of the present application.

FIG. 8 is a schematic diagram of clustering of color brightness samples according to an embodiment of the present application.

FIG. 9 is a schematic diagram of inserting a continuously arranged measurement cell between the center points of two color brightness sample point clusters according to an embodiment of the present application.

FIG. 10 is a schematic flowchart of an image processing method according to an embodiment of the present application.

FIG. 11A is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

FIG. 11B is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application.

FIG. 12 is a schematic structural diagram of a light source estimation device according to an embodiment of the present application.

FIG. 13 is a schematic structural diagram of another light source estimation device according to an embodiment of the present application.

FIG. 14 is a schematic diagram of another light source estimation device according to an embodiment of the present application.

FIG. 15 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

FIG. 16 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application.

detailed description

The terms "including" and "having" in the specification and claims of the present application and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally includes Other steps or units inherent to these processes, methods, products or equipment. In addition, the terms "first", "second", "third", and "fourth" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order .

The following first introduces some system architectures of this application. Referring to FIG. 1A, FIG. 1A is a schematic diagram of a system architecture provided by the present application. The system 100 includes a camera 110, an image processing unit 120, and a memory 130. The camera 110 is configured to capture an image (original image). The image processing unit 120 is configured to perform some related processing on the image obtained by the camera 110 (for example, perform automatic white balance correction, color correction, saturation correction, and / or contrast correction on the image obtained by the camera 110). The memory 130 is used to store some program code or data related to image processing.

Among them, the image processing component 120 may be composed of one or more processors; or, the image processing component 120 may include both one or more processors or some hardware circuits; or the image processing component 120 may not include processing Controller, but includes some hardware circuits.

Referring to FIG. 1B, the image processing section 120 includes, for example, an image signal processor 121, a light source estimation device 122, and the like. The light source estimation device 122 can be used to estimate the existence of a light source in the image shooting environment (for example, it can be estimated whether a monochrome temperature light source or a mixed color temperature light source exists in the image shooting environment). Among them, the estimation result of the existence of the light source in the image shooting environment output by the light source estimation device 122 can be performed by the image signal processor 121 for related image processing (for example, automatic white balance correction, color correction, saturation of the image). (Such as degree correction and / or contrast correction).

Among them, in FIG. 1A, the camera 110, the image signal processor 121, the light source estimation device 122, and the like are physically set independently as an example. However, in actual applications, some components may be physically integrated as a whole. For example, in some actual product designs, the light source estimation device 122 may be integrated into the image signal processor 121, and the light source estimation device 122 may also be integrated into the camera 110. When the light source estimation device is integrated into the image signal processor, in this case, the image signal processor may have the function of the light source estimation device described in the above example, and the image signal processor integrated with the light source estimation device It can still be called an image signal processor. Similarly, when the light source estimation device is integrated into the camera, in this case, the camera can also have the function of the light source estimation device described in the above example, and the camera integrated with the light source estimation device can still be called Camera, and so on in other cases.

The method mentioned in this application may be specifically implemented based on the architecture exemplified above. The following is a brief introduction to the standard light source. The standard light source generally refers to a light source whose color temperature is a standard color temperature. The color temperature level of a standard light source can be divided into three levels: high color temperature, medium color temperature, and low color temperature. High color temperature, medium color temperature, and low color temperature can be divided into several sub-levels. Referring to FIG. 1C, FIG. 1C illustrates a color temperature level division method of a standard light source by way of example. In the example shown in FIG. 1C, the color temperature is divided into 10 sub-levels. , CWF, TL84 and U30 are classified as medium color temperature, and A and H are classified as low color temperature. Among them, the color of high color temperature light sources is light blue, and the color of low color temperature light sources is light yellow.

The following first introduces an AWB correction method. First, the image is cut into n × m subblocks, and all pixels of each subblock are added to obtain the color average value (R, G, B) of this subblock. Based on this color average value, this can be obtained. The color information group (R / G, B / G) of the sub-block, or (R, G, B) can be converted to (Y, Cb, Cr) or (Y, U, V) color space, then the color information Groups can also be expressed as (Cb, Cr) or (U, V). Then, the color information group of each sub-block is mapped on a two-dimensional color coordinate plane (for example, as shown in FIG. 2), and then a color sample point located in the two-dimensional color coordinate plane is formed, and each color sample point corresponds to one color information. group.

It can be understood that block (block) and subblock (subblock) is a relative concept. Dividing a block (image block) can get the subblock of this block. If you continue to divide the subblock, you can get this The sub-block of the sub-block, that is, the block is composed of the sub-blocks, and the sub-blocks are obtained by segmenting the blocks. Of course, both blocks and sub-blocks can be called blocks (image blocks).

In the example shown in FIG. 2, small dots represent color sample points of a color information group of a sub-block, and nine large dots represent calibration points of nine standard light sources. For example, the color samples within the range enclosed by the dotted line can be regarded as close enough to the nine light sources. These color samples can be regarded as valid color samples. The effective color samples can be used as the basis for subsequent calculations. Outside the range enclosed by the dotted line The color samples can be regarded as invalid color samples, and the invalid color samples can not be used as the basis for subsequent calculations. Of course, all color samples can also be regarded as valid color samples.

The main goal of the above AWB correction method is to calculate the color temperature of the light source. Therefore, all color samples within the circled range of the dashed line can be added and the average Avg (R / G, B / G) can be calculated; according to the calibration point of the standard light source (Such as the calibration points of the nine standard light sources in Figure 2) and the relative positional relationship with Avg (R / G, B / G) to estimate the color temperature of the only light source existing in the image shooting environment; based on the average Avg (R / G, B / G) and the estimated color temperature of the only light source in the image shooting environment is converted to obtain the gain values of the three RGB channels, namely (R-gain, G-gain, B-gain); Gain, G-gain, B-gain) is multiplied by (R, G, B) of each pixel in the image to correct the color temperature deviation of the sole light source, which results in AWB correction.

The above AWB correction method can estimate the color temperature of the only light source in the image shooting environment (for example, the color temperature of this unique light source can be specifically output 5000K), and when there are multiple light sources in the shooting environment and the color temperature of each light source is different, such as at the same time There are light sources with high, medium, and low color temperatures. The above AWB correction method cannot effectively determine whether there are multiple light sources with different color temperatures in the image shooting environment, so the color temperature of the only light source can usually be estimated as the basis for correcting the color cast of the image. Among them, the only light source can be called a single light source. In contrast, multiple different color temperatures can be called mixed color temperatures. Multiple different color temperature light sources can be called mixed color temperature light sources.

Therefore, how to estimate whether there are multiple light sources with different color temperatures in the image shooting environment has become of great research and application value. Based on this, the embodiments of the present application further provide some light source estimation methods. These light source estimation methods strive to estimate whether there are multiple light sources with different color temperatures in the image shooting environment.

The inventors of the present application have found through extensive research that when there are multiple light sources with different color temperatures in the image shooting environment, the physical characteristics of light sources projected by different color temperatures are usually different from the physical characteristics of a single light source projection. Therefore, the physical characteristics of the light source projection can be analyzed to determine whether there are multiple light sources with different color temperatures in the image shooting environment, and this idea is helpful to overcome the misjudgement of the color temperature of the light source caused by the color of the object itself.

Specifically, some solutions in the embodiments of the present application use color information and brightness information of an image to estimate whether there are two or more light sources with different color temperatures in an image shooting environment. The color and brightness information of an image can be obtained, for example, by dividing the image into m sub-blocks, where m is an integer greater than 1. Acquire m color luminance information groups of the m sub-blocks. Each color luminance information group corresponds to one sub-block (that is, a one-to-one correspondence between the m sub-blocks and m color luminance information groups), and the color luminance information group includes luminance information and color information. The image may be an original image obtained by a camera or other images. The color information and brightness information of the image include m color brightness information groups of m sub-blocks of the image.

For example, m may be equal to 2, 3, 4, 8, 12, 16, 32, 64, 128, 256, or other values. The sizes of the m sub-blocks may be all the same, partially the same, or different from each other. The shape of the sub-blocks can be square, rectangular, or other shapes. For example, referring to FIG. 3A and FIG. 3B, in the example shown in FIG. 3A, the image is divided into m sub-blocks of the same size, and in the example shown in FIG. 3B, the image is divided into m sub-blocks of different sizes. Of course, the sub-block segmentation method of the image is not limited to the method illustrated in FIGS. 3A and 3B.

The color information may be expressed in the following form: (R / G, B / G) or (Cb, Cr) or (U, V), and the brightness information may be expressed as BV (bright value). Therefore, the color luminance information group of the sub-block can be expressed as (R / G, B / G, BV) or (Cb, Cr, BV) or (U, V, BV), for example. In order to analyze the color and brightness information groups of each sub-block of the image, and then analyze the correlation between the color and brightness information groups of each sub-block. Further, the m color brightness information groups may be mapped to a color brightness three-dimensional space to obtain m color brightness sample points located in the color brightness three-dimensional space. Each color brightness sample point corresponds to one color brightness information group (that is, one-to-one correspondence between m color brightness sample points and m color brightness information groups in a color brightness three-dimensional space), and the color brightness three-dimensional space Includes two color dimensions and one brightness dimension. Further, by performing clustering processing on m color brightness sample points located in the color brightness three-dimensional space, multiple color brightness sample point groups can be obtained. Alternatively, the m color brightness samples located in the three-dimensional color brightness space are first layered along the brightness dimension, and then the color brightness samples of each layer are grouped, and multiple color brightness samples can be obtained for each layer. Point group.

For the two types of scenes of monochrome temperature and mixed color temperature, the inventors of the present application analyzed the distribution characteristics of relevant color brightness sample points and found that in the mixed color temperature scene, there is a strong degree of blending between the color brightness sample groups; in the monochrome temperature scene, , There is a weaker blending degree between the color brightness sample clusters. The blending degree between the two color luminance sample clusters can indicate the closeness of the connection between the two color luminance sample clusters. Among them, the greater the degree of blending between the two clusters, the higher the degree of closeness between the color brightness samples of the two clusters; the smaller the degree of blending between the two clusters, the color brightness samples of the two clusters The tighter the connection is. Here are some examples through two experimental examples.

Referring to FIG. 4, the right image in FIG. 4 is a captured image. Daylight with a high color temperature in the shooting environment is illuminated through the window and A light with a low color temperature, that is, there is a mixed color temperature in the shooting environment. The left figure in FIG. 4 shows the distribution of the color brightness samples of each sub-block of the image in the three-dimensional space of color brightness. The horizontal plane of the three-dimensional space of color brightness has two color dimensions and one brightness dimension in the vertical direction. . The 9 points above the left are the marked points of the standard light source, and the points below the left are the color brightness samples of the subblocks on the right. If the color brightness samples are divided into two groups (Group1, Group2), Group1 belongs to a high color temperature, and Group2 belongs to a low color temperature. The analysis found that the distributions of these two groups have strong mutual blending characteristics. And this blending characteristic is mainly caused by light sources with high and low color temperatures being projected on the same object (such as the floor, etc.), and the two light sources will have a fusion effect in color and brightness.

Please refer to Figure 5. The right image of Figure 5 is the captured image. In the shooting environment, there is only a single high color temperature daylight light source, and no other artificial light source, that is, there is no mixed color temperature in the shooting environment. However, there is a pale yellow wood grain wall in the shooting environment. The color brightness samples of this wood texture wall are distributed in the low color temperature region. The left figure in Figure 5 shows the color brightness samples of each sub-block in the color brightness. For the distribution in three-dimensional space, the 9 points on the upper left are labeled points of the standard light source, and the points on the lower left are sample points of the color brightness of each sub-block on the right. If the color brightness samples are divided into two groups (Group1, Group2), Group1 belongs to a high color temperature, and Group2 belongs to a low color temperature. The analysis found that the degree of blending between the two groups is weak, mainly because the wood grain wall itself does not emit light, and the light yellow color is limited to the wall surface itself cannot project on other objects.

Through the above experiments, it was found that light sources of different color temperatures will be projected on the same object, and the light sources have a fusion effect. This characteristic is reflected in the distribution characteristics of color brightness samples, and specifically shows that there is a strong degree of fusion between color brightness sample groups. Some solutions in the embodiments of this application use this characteristic to estimate whether there is a mixed color temperature light source in the image shooting environment. .

Referring to FIG. 6, FIG. 6 is a schematic flowchart of a light source estimation method according to an embodiment of the present application. A light source estimation method may be implemented in the system architecture shown in FIG. 1A or FIG. 1B. For example, the light source estimation method may be mainly performed by the image processing unit 120, and specifically, for example, by the light source estimation device in the image processing unit 120. Mainly implemented from 121, the methods can specifically include:

601. Divide an image into m sub-blocks. The m is an integer greater than 1. The image may be an original image or another image captured by a camera.

602. Acquire m color luminance information groups of the m sub-blocks. Each color luminance information group corresponds to one sub-block (that is, a one-to-one correspondence between the m sub-blocks and m color luminance information groups), and the color luminance information group includes luminance information and color information.

603: Map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness sample points located in the color brightness three-dimensional space. Each color brightness sample point corresponds to a color brightness information group, that is, there is a one-to-one correspondence between m color brightness sample points and m color brightness information groups in a color brightness three-dimensional space. As mentioned above, the three-dimensional space of color brightness includes two color dimensions and one brightness dimension.

For example, assuming m = 128, it means that the image is cut into 128 sub-blocks, and then the color brightness information groups of 128 sub-blocks can be obtained, and a total of 128 color brightness information groups are mapped to the 128 color brightness information groups. In the three-dimensional space of brightness, a total of 128 color brightness samples can be obtained. Each color brightness sample corresponds to a color brightness information group, and each color brightness information group corresponds to a sub-block.

604. Calculate the case where the m color brightness sample points are divided into k layers along the brightness dimension, and each of the k layers is divided into P color brightness sample groupings. The first blending degree of the number of clusters P corresponding to each of the k layers. In some possible implementations, for example, to cluster color brightness samples, any one of the following clustering algorithms can be used: k-means algorithm, hierarchical clustering algorithm, and density-based clustering algorithms. Clustering (density based clustering, DBSCAN) algorithms, or the use of hierarchical structure iterative reduction and clustering (balanced reducing clustering) algorithms, etc. Of course, other clustering algorithms may be used for clustering the color and brightness samples, which is not specifically limited in this embodiment.

It can be understood that when k is equal to 1, it means that m color luminance samples are divided into the same layer, so the action of layering is not actually performed. Each of the k layers corresponds to a brightness interval. Wherein, P is an integer greater than 1. K is a positive integer. For example, P may be equal to 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 20, or 35 or other values. For example, k may be equal to 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 17, or other values.

When k is greater than 1, the number of color brightness samples included in different layers of the k layers may be all the same, partly the same, or different from each other. When k is greater than 1, the heights of the brightness intervals corresponding to different layers in the k layers may be all the same, partially the same, or different from each other.

For example, in some possible implementations, the color brightness sample points are divided into k layers along the brightness dimension, and each layer is distributed from dark to light, and the number of color brightness sample points of each layer are S1, S2, ..., Sk, respectively. Among them, the number of color brightness samples in different layers may be all the same, some of them may be the same or different from each other. For example, the number of color brightness samples of brighter layers may be more or less, and for example, the number of color brightness samples of each layer is equal.

605. Determine a second blending degree of the grouping number P corresponding to the image based on the first blending degree of the grouping number P corresponding to each of the k layers.

It can be understood that when k is equal to 1, it means that m color brightness samples are divided into the same layer, then only the first blending degree of the number of clusters P corresponding to the only layer is obtained. At this time, the The first blending degree of the unique clustering number P corresponding to this layer is directly used as the second blending degree of the clustering number P corresponding to the image.

606. Compare the second blending degree of the corresponding clustering number P of the image with the blending degree threshold. The blending degree threshold here is a blending degree threshold corresponding to the number of clusters P, that is, when the value of the number of clusters P is different, the blending threshold may also be different. Of course, the blending degree threshold may not correspond to the number of clusters P, that is, when the value of the number of clusters P is different, the blending threshold may not change. The blending threshold may be an empirical value or obtained based on experimental data, which is not limited in this application.

607. In the case where the second blending degree of the corresponding number of clusters P of the image is greater than the blending degree threshold, it is estimated that there are P light sources with different color temperatures in the shooting environment of the image, that is, a suitable number of light sources is obtained. P. In some possible implementation manners, when the second blending degree of the corresponding grouping number P of the image is less than or the blending degree threshold, it may be estimated that there are no P color temperatures in the shooting environment corresponding to the image Different light sources. Optionally, the present application may further combine other technical means to estimate whether there is only a single light source or multiple light sources with different color temperatures in the shooting environment corresponding to the image.

It can be seen that the embodiment of the present application provides a feasible light source estimation method. This light source estimation method obtains the color brightness by mapping the color brightness information group of the image segmented sub-blocks to the color brightness three-dimensional space. Multiple color brightness sample points in three-dimensional space, and perform multiple layer clustering processing on multiple color brightness sample points, and then calculate the blending degree of the entire image by calculating the blending degree of the color brightness sample point clustering of each layer, and based on this It is estimated that there are multiple light sources with different color temperatures in the shooting environment corresponding to the image. Then, this lays the foundation for image correction based on the situation that includes multiple light sources with different color temperatures. For example, it makes it possible to perform targeted image correction when multiple light sources with different color temperatures are included, which in turn helps to improve the rationality of image correction.

In some possible implementations, since P is an integer greater than 1, the calculation, determination, comparison, and estimation may be performed for P = 2,... X, respectively, where X is an integer greater than 2. When it is estimated that there are P = 2, ... Y light sources with different color temperatures in the shooting environment (that is, it is determined that the appropriate number of light sources is P = 2, ... Y), it is determined that P exists in the shooting environment. = Y light sources with different color temperatures, where Y is an integer greater than 2 and less than or equal to X. That is to say, the partial steps related to the estimation in the method corresponding to FIG. 6 are repeatedly performed for different values of P to determine whether each value is suitable as the number of light sources and the maximum number of light sources The value of P is determined as the final number of light sources. Therefore, each value of P is actually a hypothetical light source number, and this value is applied to the light source estimation method to determine whether the hypothetical value is appropriate. When there are multiple suitable values, for example, the maximum value can be used as the final number of light sources. It can be understood that this is because a larger number of light sources include a part of the number of light sources. For example, suppose P = 2 to 10 and execute the method for 2 to 10 respectively. When P = 2 to 5 are determined as the appropriate number of light sources, P = 5 can be selected as the final number of light sources. This is because the 5 light sources include 2, 3, and 4 light sources previously estimated, so it is reasonable to choose the maximum value. In contrast, when P = 6 to 10 is applied to the above method, it is determined through the method that there are no P = 6 to 10 light sources, and the value of P = 6 to 10 previously assumed is not appropriate, not There are such a number of light sources.

For example, set P = P1 and P = P2 respectively; then, calculate the first blending degree of the number of clusters P1 corresponding to each of the k layers; based on the number of clusters corresponding to each of the k layers The first blending degree of P1 determines the second blending degree of the number of clusters P1 corresponding to the image; comparing the second blending degree of the corresponding number of clusters P1 of the image with the blending degree threshold of the number of clusters P1. And calculate a first blending degree of the number of clusters P2 corresponding to each of the k layers; and determine a number of clusters P2 corresponding to the image based on the first blending degree of the number of clusters P2 corresponding to each of the k layers A second blending degree; comparing the second blending degree of the corresponding number of clusters P2 of the image with the blending degree threshold of the number of clusters P2.

Or in another optional embodiment, the second blending degree of the corresponding number of clusters P1 in the image is greater than the threshold of the blending degree of the number of clusters P1, and the second blending degree of the corresponding number of clusters P2 of the image is greater than In the case of the blending degree threshold of the number of clusters P2, when the second blending degree of the corresponding clustering number P2 of the image is different from the blending degree threshold of the clustering number P2 (this difference can be expressed as (the image's The second degree of fusion corresponding to the number of clusters P2-the threshold of the degree of fusion of the number of clusters P2) / the threshold of the degree of fusion) is greater than the difference between the second degree of fusion of the corresponding number of clusters P1 and the threshold of the degree of fusion of the number of clusters P1 It is estimated that the shooting environment corresponding to the image includes P2 light sources with different color temperatures. That is, the number of clusters corresponding to the second blending degree that exceeds the corresponding blending degree threshold more is determined as the number of light sources.

In some possible implementation manners, determining the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers may include: The first blending degree corresponding to the number of clusters P of each layer is summed or weighted to obtain a second blending degree corresponding to the number of clusters P of the image. Wherein, when the first blending degree P corresponding to the number of clusters P in each of the k layers is weighted and summed to obtain the second blending degree corresponding to the number of clusters P in the image, then the corresponding clusters in each layer The weighted summation weight of the first blending degree of the number P can be determined based on the height of the brightness interval of each layer. For example, the higher the weighted summation weight of a layer with a relatively higher brightness interval, the greater the height of the brightness interval. The relatively smaller the layer, the smaller the corresponding weighted summation weight can be. Alternatively, the weighted summation weight of the first blending degree corresponding to the number of clusters P of each layer may be determined based on the number of color brightness samples of each layer. For example, the layer with a relatively large number of color brightness samples has a corresponding weighted calculation. The larger the sum weight, the smaller the number of color brightness samples, and the smaller the corresponding weighted sum weight. Of course, the weighted summation weight of each layer can also be determined based on other parameters.

There are multiple ways to calculate the first blending degree of the number of clusters P corresponding to each of the k layers. For example, in some possible implementation manners, when the P is greater than 2; calculating the first blending degree of the number of clusters P corresponding to the i-th layer in the k layers may include: calculating the P-th clusters of the i-th layer The third degree of integration between each two subgroups. The third blending degree between each two clusters in the P clusters is summed or weighted to obtain a first blending degree of the number of clusters P corresponding to the i-th layer. The i-th layer is any one of the k layers. For another example, in some possible implementation manners, when the P is equal to 2; calculating the first blending degree of the number of clusters P corresponding to the i-th layer in the k-layer may include: calculating two of the i-th layer The third degree of integration between the clusters; wherein the first degree of integration of the number of clusters P corresponding to the i-th layer is equal to the third degree of integration between the two clusters. The i-th layer is any one of the k layers.

Among them, there can be multiple ways to calculate the blending degree between the two clusters. For example, in some possible implementations, calculating the third degree of blending between the cluster gi and the cluster gj may include inserting a continuous array of measurement cells between the center point of the cluster gi and the center point of the cluster gj, the continuous The number of arranged measurement cells is T. The statistics include the number of measurement cells Q of the color brightness samples in the clusters gi and gj. It is determined that the third blending degree between the cluster gi and the cluster gj is Q / T. The T and the Q are integers. The T is greater than 0 and the Q is greater than or equal to 0. The cluster gi and the cluster gj are any two clusters among the P clusters in the i-th layer.

Further, in order to further simplify the calculation, the third blending degree between two pairs of subgroups can also be calculated in the same color plane. For example, calculating the third degree of fusion between cluster gi and cluster gj includes: projecting cluster gi and cluster gj to the same color plane (that is, the brightness of the color brightness sample points in cluster gi and cluster gj take the same value). Continuously arranged measurement cells are inserted between the center point of the cluster gi and the center point of the cluster gj that are projected onto the same color plane. The number of the continuously arranged measurement cells is T. The statistics include the number of measurement cells Q of the color brightness samples in the clusters gi and gj. It is determined that the third blending degree between the cluster gi and the cluster gj is Q / T. among them. The T is greater than 0 and the Q is greater than or equal to 0, and T and Q are integers. The cluster gi and the cluster gj are any two clusters among the P clusters in the i-th layer.

For example, the distance between the color brightness sample point in the cluster gi and the center point of the group gi is less than or equal to the distance between this color brightness sample point and the center point of the group gj.

In some possible implementations, the length of a single measurement cell may be equal to, for example, Dist_D65_D50, Dist_D75_D65, or Dist_D55_D50 or other experience values. Among them, Dist_D75_D65 represents the distance between the calibration points of the standard light sources D75 and D65 in the color plane; Dist_D65_D50 represents the distance between the calibration points of the standard light source D65 and D50 in the color plane; Dist_D55_D50 represents the color plane, the standard The distance between the calibration points of the light sources D55 and D50; in other cases and so on. The width of a single measurement cell is equal to length × 1/32, length × 1/20, length × 1/16, length × 1/19, or other experience values, such as the width of a single measurement cell Dist_D65_D50 / 32, Dist_D75_D65 / 32 , And so on in other cases.

It can be understood that the third blending degree between two pairs of subgroups can be calculated and calculated in the three-dimensional space of color brightness, and the third blending degree between two pairs of subgroups can also be calculated in the same color plane. The two clusters are projected onto the same color plane to obtain all the samples of the two clusters located on the same color plane). When the third blending degree between two pairs of subgroups is calculated in the three-dimensional space of color and brightness, then a single measurement cell is a three-dimensional measurement cell having a length, a width, and a height, and in this case, a single measurement cell The height may be greater than or equal to the layer height of the layer where the two clusters are located. For the length and width, refer to the above examples. When the third blending degree between two pairs of subgroups is calculated in the two-dimensional color plane, then a single measurement cell is a measurement cell with a length and a width but not a high plane. In this case, the single measurement cell For the length and width, please refer to the above examples.

For example, see FIG. 9. In the example shown in FIG. 9, the center points of the two clusters that are projected onto the same color plane are C1 and C2 respectively. The points in the figure represent the color brightness samples of the two clusters. The box indicates the measurement cell. The number above the measurement cell indicates the number of color brightness samples that fall into the measurement cell. For example, when the number above the measurement cell is 1, it indicates the color that falls into the measurement cell in two subgroups. The number of brightness samples is 1, when the number above the measurement cell is 0, it means that the number of color brightness samples that fall into the measurement cell in the two clusters is 0, and when the number above the measurement cell is 3, it means two The number of color brightness samples that fall into the measurement cell in the cluster is 3, and so on in other cases. It can be understood that some or all of the measurement cells inserted into the measurement cells of C1 and C2 may fall into the color brightness samples, that is, some or all of the measurement cells inserted into the measurement cells of C1 and C2 include the color brightness samples. If the number of measured measurement cells including color brightness samples is Q and the total number of measurement cells inserted into C1 and C2 is T, then the third degree of integration between the two clusters can be calculated as Q / T (also expressed as (Q / T) × 100%). For example, assuming that the number of measurement cells including color brightness samples is 30, and the total number of measurement cells inserted into C1 and C2 is 40, then the third degree of integration between the two clusters can be calculated as 30 / 40 = 0.75 (75%), for example, assuming that the number of measurement cells including color brightness samples is 45, and the total number of measurement cells inserted into C1 and C2 is also 45, then two can be calculated The third degree of integration between the subgroups is 45/45 = 1 (100%), and so on in other cases.

In some possible implementation manners, the length direction of the continuously arranged measurement cells may be perpendicular to a line connecting a center point of the cluster gi and a center point of the cluster gj (for example, as shown in FIG. 9). Of course, the length direction of the continuously arranged measurement cells may not be perpendicular to the line connecting the center point of the cluster gi and the center point of the cluster gj. For example, the included angle range between the longitudinal direction and the line connecting the center point may be, for example, 60 ° to 90 °.

In some possible implementation manners, the two color dimensions included in the three-dimensional color luminance space are a first color dimension and a second color dimension, wherein a first color dimension coordinate of a center point of the cluster gi is equal to that in the cluster gi The average value of the first color dimension coordinates of all the color brightness sample points, and the second color dimension coordinate of the center point of the cluster gi is equal to the average value of the second color dimension coordinates of all the color brightness sample points in the cluster gi.

In the following, a more specific example of a method for calculating the degree of blending between clusters will be given with reference to the drawings.

First, the color brightness sample points in the color brightness three-dimensional space are divided into k layers along the brightness dimension. The number of color brightness samples for each layer is S1, S2 ... Sk. Please refer to FIG. 7. In the example shown in FIG. 7, the color and brightness samples are divided into 5 layers along the brightness dimension, that is, k = 5, that is, divided into 5 layers. The horizontal plane (color plane) in FIG. 7 is two color dimensions, and the vertical direction is one brightness dimension. The four planes perpendicular to the brightness dimension divide the color brightness three-dimensional space into five layers, so that multiple color brightness samples are distributed in these five layers. The layer division rules have been described in the previous embodiments, and will not be repeated here.

After the layering is completed, the color brightness samples in each layer are grouped. Referring to FIG. 8, in the example shown in FIG. 8, the color brightness samples of each layer are divided into two clusters (P = 2). The plane in FIG. 8 is a color plane formed by two color dimensions. FIG. 8 specifically shows a result of dividing a plurality of color luminance samples included in a layer into two clusters, and the center points of each cluster are respectively C1 and C2. The clustering algorithm can refer to the introduction of the previous embodiment.

In some possible implementations, for example, if the distance between the center points (C1 and C2) of two clusters is less than the minimum distance threshold (can be expressed as Cent_Dist_min), the two clusters may not be included in the calculation of the image blending degree. . Here, the distance between the center points of two clusters is greater than Cent_Dist_min and is included in the calculation as an example. For example, the distance between the calibration points of the standard light sources D65 and D50 on the color planes (R / G and B / G planes) is Dist_D65_D50. For example, the above minimum distance threshold Cent_Dist_min = Dist_D65_D50 × 70% can be set, and This is used as a criterion to select whether this layer is included in the calculation of image blending. Of course, in some cases, even if the distance between the center points (C1 and C2) of two clusters is less than the minimum distance threshold, the two clusters may be included in the calculation of the image fusion degree. In this case, it can be considered as the smallest. The distance threshold Cent_Dist_min = 0.

After grouping is complete, successively arranged measurement cells are inserted between C1 and C2. Referring to FIG. 9, an example is shown in which a plurality of measurement cells arranged consecutively are inserted between C1 and C2, and the length of a single measurement cell may be Dist_D65_D50. The width of a single measurement cell can be Dist_D65_D50 / 32. The length and width of a single measurement cell actually used may also be other suitable values, which are not limited in this application. Assuming that the number of consecutively arranged measurement cells is T (in the example shown in FIG. 9, T represents the total number of measurement cells inserted between C1 and C2), the statistics include the colors in the two clusters Number of measurement cells for brightness samples Q (In the example shown in Figure 9, Q represents the total number of measurement cells whose number above is greater than 0, and the number above each measurement cell indicates the color included in this measurement cell Number of brightness samples); furthermore, it can be determined that the blending degree between the two clusters is Q / T (or Q / T × 100%). A larger Q / T indicates a greater degree of blending between the two clusters. Since P = 2 is taken as an example in FIG. 9, the fusion degree of each layer is equal to the fusion degree between two subgroups of this layer.

In FIG. 9, the number of clusters P = 2 is taken as an example. Of course, if P is greater than 2, then the degree of blending between the two subgroups of each layer can be calculated according to the above example. Perform the summation or weighted summation process to obtain the blending degree of this layer. For example, a certain layer is divided into (g1, g2, g3, g4) four groups. Among them, it is assumed that L (g1, g2) represents the blending degree between g1 and g2, and so on. Then L (g1, g2), L (g1, g3), L (g1, g4), L (g2, g3), L (g2, g4), and L (g3, g4), etc. (Or weighted summation) to obtain the blending degree of this layer, and so on for other clusters. After calculating the blending degree of each layer, the blending degree of each layer is summed (or weighted summation) to obtain the blending degree of the entire image. Specifically, after multiplying the blending degree of each layer by the corresponding weighted value, the summation of the image's blending degree Total_Con ^P (P represents how many groups each layer is divided into), if the blending degree of the image exceeds the blending threshold Th ^P ( The blending threshold Th ^P corresponds to the number of clusters P), and it can be estimated that there are P light sources of different color temperatures in the image shooting environment.

In a scenario, it is assumed that there may be at most 4 light sources in the image shooting environment. Set P = 2, 3, 4 and set the blending threshold to Th ² , Th ³ and Th ^4. If the blending degree of the image is Total_Con ² , Total_Con ³ and Total_Con ⁴ are all smaller than the corresponding blending thresholds, it is determined that there is only a monochromatic temperature light source in the image shooting environment, that is, there is no mixed color temperature light source. If Total_Con ² , Total_Con ³ and Total_Con ⁴ are all larger than the corresponding blending threshold, then it can be estimated that there are four different color temperature light sources in the image shooting environment. For another example, if Total_Con ² and Total_Con ^{3 are} larger than the corresponding blending threshold and Total_Con ^{4 is} smaller than the corresponding blending threshold Th ⁴ , it can be estimated that there are three different color temperature light sources in the image shooting environment. For another example, if Total_Con ^{2 is} larger than the corresponding blending threshold Th ² , Total_Con ³ and Total_Con ^{4 are} smaller than the corresponding blending threshold, it can be estimated that there are two different color temperature light sources in the image shooting environment.

In another scenario, when the blending degree of the images obtained under multiple grouping conditions is greater than the corresponding blending degree threshold, the following example may also be used to estimate that there are several different color temperature light sources in the image shooting environment. For example, suppose that there may be up to 4 light sources in the image shooting environment. Set P = 2, 3, 4 and set the blending thresholds as Th ² , Th ³ and Th ^4. If the blending levels of the images are Total_Con ² , Total_Con ³ and Total_Con ⁴ are all smaller than the corresponding blending threshold, it is determined that only a light source with a single color temperature exists in the image shooting environment, that is, a light source with a mixed color temperature does not exist. For another example, if Total_Con ^{2 is} larger than the corresponding blending threshold Th ² , Total_Con ³ and Total_Con ^{4 are} smaller than the corresponding blending threshold, it can be estimated that there are two different color temperature light sources in the image shooting environment. If Total_Con ^{4 is} less than the corresponding blending degree threshold Th ⁴ , Total_Con ² and Total_Con ^{3 are} greater than the corresponding blending degree threshold, then the image blending degree that exceeds the corresponding blending degree threshold the most (that is, the difference between the image blending degree and the blending degree threshold is the largest ), The number of clusters is determined as the number of light sources with different color temperatures. For example, (Total_Con ² -Th ² ) / Th ² = 0.36, (Total_Con ³ -Th ³ ) / Th ³ = 0.25. From the perspective of the degree of transcendence (the difference), 0.36> 0.25, so it can be estimated There are two different color temperature light sources in the image shooting environment, and so on in other cases.

In the case where it is estimated that there are multiple light sources with different color temperatures in the image shooting environment, the image may be correlated and corrected according to P light sources with different color temperatures.

Referring to FIG. 10, FIG. 10 is a schematic flowchart of an image processing method according to an embodiment of the present application. An image processing method may be implemented in the system architecture shown in FIG. 1A or FIG. 1B. For example, the image processing method may be mainly performed by the image processing unit 120. Specifically, for example, steps related to light source estimation in the image processing method may be performed by The light source estimation device 121 in the image processing component 120 is mainly executed, and the steps related to the correction in the image processing method may be mainly performed by the image signal processor 122 in the image processing component 120. The method may specifically include:

1001. Perform a light source estimation method. The light source estimation method may be any one of the light source estimation methods provided in the foregoing embodiments.

1002. When it is estimated that there are P light sources with different color temperatures in the shooting environment corresponding to the image, the image is corrected according to the P light sources with different color temperatures. When it is estimated that a monochrome temperature light source exists in the shooting environment corresponding to the image, the image is corrected according to the monochrome temperature light source. The correction may include at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction.

For example, the correction may be performed by an image signal processor (ISP). The light source estimation method can be performed by a light source estimation device. A structural diagram of a specific image processing apparatus may be shown in FIG. 11A. For example, after an original image is input to the ISP, the color shift of the light source is corrected by the AWB correction circuit. For the image after AWB correction, white objects will appear as white as possible, but other colors may not be accurate. A color correction (CC) circuit corrects each color to the correct color through color correction. For the image after the color correction circuit CC, the saturation correction circuit can further use the color enhancement (CE) mechanism to specify a specific color in the image, and increase or decrease its saturation to complete the saturation correction, thereby improving the color style of the image. A contrast correction circuit (such as Gamma) is used to correct the contrast of the image brightness. The order of the circuits in the ISP in FIG. 11A can be adjusted and changed. For example, it can also be adjusted to the order shown in the example in FIG. 11A to obtain the structure in FIG. 11B. Of course, it can also be adjusted to other orders according to needs. This application does not go into details. After estimating the number of light sources, the light source estimation device transmits information with the number to the ISP, so that the ISP performs the correction using the information, and the execution order of different types of corrections can be adjusted and changed. For specific correction methods, refer to other existing literatures, which are not described in this application.

When the light source estimation device detects that there is a multi-color temperature light source in the image shooting environment, the captured image may appear light blue (caused by a high color temperature light source) and light yellow (caused by a low color temperature light source). After CC and CE color enhancement, the color of the light source will be more intense on the image. Because the brain recognizes that the light source is white. People are not sensitive to the color of the light source. For images taken under a multi-color temperature light source environment, the excessively strong color of the light source will be considered by the user to be the wrong color, especially a light blue with a high color temperature. In order to reduce the sense of color deviation in mixed color temperature scenes, the AWB is adjusted so that the correction of the light source is biased to a high color temperature to reduce the color cast of light blue, reduce the color intensity of CC and CE, reduce the color of the light source, and reduce the contrast of Gamma , To reduce the brightness difference between high and low color temperature light sources, so that photos taken under a multi-color temperature light source environment, is conducive to closer to the scene seen by the human eye.

Referring to FIG. 12, an embodiment of the present application further provides a light source estimation device 1200. The light source estimation device 1200 may include:

The segmentation unit 1210, the acquisition unit 1220, the mapping unit 1230, the calculation unit 1240, the determination unit 1250, the comparison unit 1260, and the estimation unit 1270. The segmentation unit 1210 is configured to segment an image into m sub-blocks, where m is an integer greater than 1. The obtaining unit 1220 is configured to obtain m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group includes brightness information and color information.

A mapping unit 1230 is configured to map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where each color brightness sample is associated with one The color brightness information group corresponds, and the color brightness three-dimensional space includes two color dimensions and one brightness dimension.

A calculation unit 1240 is configured to calculate when the m color luminance samples are divided into k layers along the luminance dimension, and each of the k layers is divided into P color luminance sample groups. The first blending degree of the number of clusters P corresponding to each of the k layers. Each of the k layers corresponds to a brightness interval (continuous brightness interval). Wherein, P is an integer greater than 1, and k is a positive integer. A determining unit 1250 is configured to determine a second blending degree of the grouping number P corresponding to the image based on a first blending degree of the grouping number P corresponding to each of the k layers. The comparing unit 1260 is configured to compare the second blending degree of the corresponding grouping number P of the image with the blending degree threshold. The estimation unit 1270 is configured to estimate that at least P different color temperatures exist in the shooting environment corresponding to the image when the second blending degree of the corresponding clustering number P of the image is greater than the blending degree threshold. Light source. In addition, the estimation unit 1270 may be further configured to, when the second blending degree of the corresponding clustering number P of the image is less than the blending degree threshold, estimate the different color temperatures in the shooting environment corresponding to the image. The number of light sources is not P.

In some possible implementation manners, where k is greater than 1, in determining an aspect of the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers, The determining unit 1250 is specifically configured to perform summing or weighted summing processing on a first blending degree P corresponding to the number of clusters P of each of the k layers to obtain a second blending degree corresponding to the number of clusters P of the image .

In some possible implementation manners, the P is greater than 2; in calculating a first blending degree of the number of clusters P corresponding to the i-th layer in the k-layers, the calculation unit 1240 is specifically configured to calculate the i-th A third blending degree between every two clusters in the P subgroups of the layer; summing or weighting the third blending degree between each two clusters in the P clusters to obtain the i-th The first blending degree of the grouping number P corresponding to the layer, and the i-th layer is any one of the k layers.

In some possible implementation manners, in terms of calculating a third blending degree between the cluster gi and the cluster gj, the calculation unit 1240 is specifically configured to: insert between the center point of the cluster gi and the center point of the cluster gj The number of consecutively arranged measurement cells is T. The cluster gi and the cluster gj are any two clusters among the P clusters in the i-th layer. In addition, the calculation unit 1240 may count the number of measurement cells Q including the color brightness sample points in the cluster gi and the cluster gj, and determine that the third blending degree between the cluster gi and the cluster gj is Q / T, wherein the T and the Q are integers, the T is greater than 0 and the Q is greater than or equal to 0. In some possible implementation manners, a length direction of the continuously arranged measurement cells is perpendicular to a line connecting a center point of the cluster gi and a center point of the cluster gj. In some possible implementation manners, the two color dimensions included in the three-dimensional color luminance space are a first color dimension and a second color dimension, and a first color dimension coordinate of a center point of any cluster is equal to all The average value of the first color dimension coordinates of the color brightness sample points, and the second color dimension coordinate of the center point of the any group is equal to the average value of the second color dimension coordinates of all the color brightness sample points in the any group.

In some possible implementation manners, the calculation unit, the determination unit, the comparison unit, and the estimation unit may perform the calculation, the determination, the comparison, and In the estimation, X is an integer greater than two. The estimation unit is further configured to determine, when it is estimated that P = 2, ... Y light sources with different color temperatures exist in the shooting environment, determine that P = Y light sources with different color temperatures exist in the shooting environment. , Wherein Y is an integer greater than 2 and less than or equal to X.

Among them, all units in FIG. 12 may be specifically implemented by software codes (specifically, may be implemented by software codes executed by a processor); or some units in FIG. 12 may be specifically implemented by software codes, and another part of the units may be implemented by hardware circuits. ; Or all units in FIG. 12 may be specifically implemented by hardware circuits. In the example shown in FIG. 13, all units in FIG. 12 are implemented by hardware circuits as an example.

Referring to FIG. 13, an embodiment of the present application further provides a light source estimation device 1300. The light source estimation device 1300 may be implemented by a hardware circuit, and may include a segmentation circuit 1310, an acquisition circuit 1320, a mapping circuit 1330, and a calculation. A circuit 1340, a determination circuit 1350, a comparison circuit 1360, and an estimation circuit 1370. Among them, any circuit may include multiple transistors, logic gates, or basic circuit logic units.

A segmentation circuit 1310 is configured to segment an image into m sub-blocks, where m is an integer greater than 1. The obtaining circuit 1320 is configured to obtain m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group includes brightness information and color information. A mapping circuit 1330 is configured to map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, and each color brightness sample is associated with one color brightness. Corresponding to the information group, the three-dimensional space of color brightness includes two color dimensions and one brightness dimension.

A calculation circuit 1340 is configured to calculate when the m color luminance samples are divided into k layers along the luminance dimension, and each of the k layers is divided into P color luminance sample groups. The first blending degree of the number of clusters P corresponding to each of the k layers. Each of the k layers corresponds to a brightness interval (continuous brightness interval). Wherein, P is an integer greater than 1, and k is a positive integer. A determining circuit 1350 is configured to determine a second blending degree of the number of clusters P corresponding to the image based on a first blending degree of the number of clusters P corresponding to each of the k layers. The comparison circuit 1360 is configured to compare the second blending degree and the blending degree threshold of the corresponding grouping number P of the image. Wherein, an estimation circuit 1370 is configured to estimate that at least P different color temperatures exist in the shooting environment corresponding to the image when the second blending degree of the corresponding clustering number P of the image is greater than the blending degree threshold. Light source.

In addition, the estimation circuit 1370 may be further configured to estimate a difference in a shooting environment corresponding to the image in a case where the second blending degree of the corresponding clustering number P of the image is smaller than the blending degree threshold. The number of light sources of color temperature is not P.

In some possible implementation manners, where k is greater than 1, in determining an aspect of the second blending degree of the number of clusters P corresponding to the image based on the first blending degree of the number of clusters P corresponding to each of the k layers, The determining circuit 1350 is specifically configured to perform summing or weighted summing processing on the first blending degree P of the number of clusters P corresponding to each of the k layers to obtain the second blending degree of the corresponding number of clusters P of the image .

In some possible implementation manners, the P is greater than 2; in calculating a first blending degree of the number of clusters P corresponding to the i-th layer in the k-layers, the calculation circuit 1340 is specifically configured to calculate the i-th A third blending degree between every two clusters in the P subgroups of the layer; summing or weighting the third blending degree between each two clusters in the P clusters to obtain the i-th The first blending degree of the grouping number P corresponding to the layer, and the i-th layer is any one of the k layers.

In some possible implementation manners, in terms of calculating a third degree of fusion between cluster gi and cluster gj, the calculation circuit 1340 is specifically configured to: insert a continuous between a center point of cluster gi and a center point of cluster gj The number of the arranged measurement cells is T. The cluster gi and the cluster gj are any two clusters among the P clusters in the i-th layer. In addition, the calculation circuit 1340 is configured to count the number of measurement cells Q including color brightness samples in the cluster gi and the cluster gj, and determine that the third degree of integration between the cluster gi and the cluster gj is Q / T, wherein the T and the Q are integers, the T is greater than 0 and the Q is greater than or equal to 0. In some possible implementation manners, a length direction of the continuously arranged measurement cells is perpendicular to a line connecting a center point of the cluster gi and a center point of the cluster gj.

In some possible implementation manners, the two color dimensions included in the three-dimensional color luminance space are a first color dimension and a second color dimension, and a first color dimension coordinate of a center point of any cluster is equal to all The average value of the first color dimension coordinates of the color brightness sample points, and the second color dimension coordinate of the center point of the any group is equal to the average value of the second color dimension coordinates of all the color brightness sample points in the any group.

In some possible implementation manners, the calculation circuit, the determination circuit, the comparison circuit, and the estimation circuit may perform the calculation, the determination, the comparison, and In the estimation, X is an integer greater than two. The estimation circuit is further configured to: when it is estimated that there are P = 2, ... Y light sources with different color temperatures in the shooting environment, determine that P = Y light sources with different color temperatures exist in the shooting environment , Wherein Y is an integer greater than 2 and less than or equal to X.

In the example shown in FIG. 14, some or all of the units in FIG. 12 are implemented by software codes executed by a processor as an example. Referring to FIG. 14, an embodiment of the present application further provides a light source estimation device 1400. The light source estimation device 1400 includes a processor 1410 and a memory 1420 that are coupled to each other. A computer program is stored in the memory 1410. The processor 1410 is configured to call a computer program stored in the memory 1420 to execute any one of the light source estimation methods provided in the embodiments of the present invention. For details, refer to the previous embodiments.

The processor 1410 may include a central processing unit (CPU) or other processors, such as a digital signal processor (DSP), a microprocessor, a microcontroller, or a neural network calculator. In some specific applications, the components of the light source estimation device are coupled together, for example, through a bus system. The bus system may include a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are marked as the bus system 1430 in the figure. The light source estimation method disclosed in the foregoing embodiment of the present application may be applied to the processor 1410, or implemented by the processor 1410. The processor 1410 may be an integrated circuit chip, and has processing capabilities of image signals.

In some implementation processes, each step of the above-mentioned light source estimation method may be completed by an integrated logic circuit of hardware in the processor 1410 or an instruction in the form of software. That is, the processor 1410 may include, in addition to a computing unit executing software instructions, other hardware accelerators, such as an application-specific integrated circuit, an off-the-shelf programmable gate array or other programmable logic device, a discrete gate or transistor logic device, and a discrete Hardware components. The processor 1410 may implement or execute various light source estimation methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The steps of the light source estimation method disclosed in the embodiments of the present application can be directly implemented as hardware, software, or a combination of hardware and software modules. The software module may be located in a random storage, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, and the like, which are mature storage media in the art. The storage medium is located in the memory 1420. For example, the processor 1410 can read information in the memory 1420 and complete the steps of the foregoing method in combination with its hardware.

For example, the processor 1410 may be used, for example, to divide an image into m sub-blocks, where m is an integer greater than 1, and obtain m color luminance information groups of the m sub-blocks, each color luminance information group and one sub-block Correspondingly, the color brightness information group includes brightness information and color information; the m color brightness information groups are mapped to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where Each color brightness sample corresponds to a color brightness information group, and the color brightness three-dimensional space includes two color dimensions and a brightness dimension; the m color brightness samples are divided into k layers along the brightness dimension And when each of the k layers is divided into P color luminance sample clusters, the first blending degree of the number of clusters P corresponding to each of the k layers is calculated; Each layer corresponds to a brightness interval, the P is an integer greater than 1, and the k is a positive integer; the number of clusters corresponding to the image is determined based on the first blending degree of the number of clusters P corresponding to each layer in the k layers The second blending degree of P; A second blending degree and a blending degree threshold value corresponding to the corresponding clustering number P of the image; and in a case where the second blending degree of the corresponding clustering number P of the image is greater than the blending degree threshold, the image is estimated There are P light sources with different color temperatures in the shooting environment.

Referring to FIG. 15, an embodiment of the present application further provides an image processing apparatus 1500. The image processing apparatus 1500 includes a processor 1510 and a memory 1520 coupled to each other. A computer program is stored in the memory 1520. The processor 1510 is configured to call a computer program stored in the memory 1520 to execute any image processing method provided by an embodiment of the present invention. In addition to light source estimation, this method also performs the previously mentioned correction operations.

Referring to FIG. 16, an embodiment of the present application further provides an image processing device 1600. The image processing device 1600 includes a light source estimation device 1610 and a correction device 1620 coupled to each other. The light source estimation device 1610 may be, for example, the light

source estimation device

1200 or 1300 or 1400. The correction device 1620 is configured to correct the image according to P light sources with different color temperatures. The correction includes at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction. The correction device may be, for example, an image signal processor (ISP). The ISP may include at least one of the following correction circuits: an automatic white balance correction circuit 1621, a color correction circuit 1622, a saturation correction circuit 1623, or a contrast correction. Circuit 1624.

For example, the automatic white balance correction circuit 1621 may be used to perform automatic white balance correction on an image according to P light sources with different color temperatures. The color correction circuit 1622 can be used to perform color correction on an image according to P light sources with different color temperatures. The saturation correction circuit 1623 can be used to perform saturation correction on an image according to P light sources with different color temperatures. The contrast correction circuit 1624 can be used to perform contrast correction on an image according to P light sources with different color temperatures.

An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, wherein the computer program is executed by related hardware to complete the execution of any one of the light sources provided by the embodiments of the present invention. Estimate method. In addition, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by related hardware to complete execution of any image provided by the embodiment of the present invention. Approach.

An embodiment of the present application further provides a computer program product, wherein when the computer program product runs on a computer, the computer is caused to execute any one of the light source estimation methods provided by the embodiments of the present invention. In addition, an embodiment of the present application further provides a computer program product, and when the computer program product runs on a computer, the computer is caused to execute any one of the image processing methods provided by the embodiments of the present invention.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions in other embodiments.

It should be noted that, for the foregoing method embodiments, for simplicity of description, they are all described as a series of action combinations, but those skilled in the art should know that this application is not limited by the described action order. Because according to this application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for this application.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated. To another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical or other forms.

In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

When the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-accessible storage medium. Based on this understanding, the technical solution of the present application is essentially a part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, where the computer software product is stored in a The computer-readable storage medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, and specifically a processor in the computer device) to perform all of the foregoing methods of the embodiments of the present application. Or some steps. Among them, the foregoing storage medium may include: various programs that can store programs such as a U disk, a mobile hard disk, a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM). The medium of the code.

As mentioned above, the above embodiments are only used to describe the technical solution of the present application and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still apply the foregoing The technical solutions described in the embodiments are modified, or some technical features are equivalently replaced; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

A light source estimation method, comprising:

Segment the image into m sub-blocks, where m is an integer greater than 1;

Acquiring m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group including brightness information and color information;

Mapping the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where each color brightness sample point corresponds to one color brightness information group, The three-dimensional space of color and brightness includes two color dimensions and one brightness dimension;

In the case where the m color brightness sample points are divided into k layers along the brightness dimension, and each of the k layers is divided into P color brightness sample groupings, calculate the A first blending degree of the number of clusters P corresponding to each layer; each of the k layers corresponds to a brightness interval, where P is an integer greater than 1, and k is a positive integer;

Determining a second blending degree of the number of clusters P corresponding to the image based on a first blending degree of the number of clusters P corresponding to each of the k layers;

Comparing the second degree of fusion with a threshold of degree of fusion;

When the second degree of fusion is greater than the threshold of the degree of fusion, it is estimated that there are at least P light sources with different color temperatures in the shooting environment of the image.
The method according to claim 1, wherein the k is greater than 1, and determining the number of clusters P corresponding to the image based on a first blending degree of the number of clusters P corresponding to each of the k layers The second blending degree includes: summing or weighted summing the first blending degree P corresponding to the number of clusters P in each of the k layers to obtain a second blending degree corresponding to the number of clusters P of the image.
The method according to claim 1 or 2, wherein the P is greater than 2; calculating the first blending degree of the number of clusters P corresponding to the ith layer in the k layers comprises:

Calculating a third blending degree between every two clusters in the P clusters of the i-th layer;

Summing or weighted summing the third degree of integration between each two of the P groups to obtain the first degree of integration of the number of groups P corresponding to the i-th layer, where the i-th layer is Any one of the k layers.
The method according to claim 3, wherein calculating the third blending degree between the cluster gi and the cluster gj comprises:

Insert continuously arranged measurement cells between the center point of the cluster gi and the center point of the cluster gj, and the number of the continuously arranged measurement cells is T;

The statistics include the number of measurement cells Q of the color brightness samples in the clusters gi and gj;

Determining a third degree of integration between the cluster gi and the cluster gj as Q / T;

Wherein, the T and the Q are integers, the T is greater than 0 and the Q is greater than or equal to 0, and the cluster gi and the cluster gj are any two of the P clusters of the i-th layer Subgroups.
The method according to claim 4, wherein:

The length direction of the continuously arranged measurement cells is perpendicular to the line connecting the center point of the cluster gi and the center point of the cluster gj.
The method according to claim 4 or 5, wherein the two color dimensions are a first color dimension and a second color dimension, wherein the first color dimension coordinate of the center point of any cluster is equal to the any one The average value of the first color dimension coordinates of all color brightness samples in a cluster, and the second color dimension coordinate of the center point of any one cluster is equal to the second color dimension coordinates of all color brightness samples in any one cluster. average value.
The method according to any one of claims 1 to 6, characterized in that the calculation, the determination, the comparison, and the estimation are performed for P = 2, ... X, respectively, and X is an integer greater than 2 ;

When it is estimated that P = 2, ... Y light sources with different color temperatures exist in the shooting environment, it is determined that P = Y light sources with different color temperatures exist in the shooting environment, where Y is 2 or more and An integer less than or equal to said X.
An image processing method, comprising:

Performing the light source estimation method according to any one of claims 1 to 7;

The image is corrected according to P light sources with different color temperatures, wherein the correction includes at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction.
A light source estimation device, comprising:

A segmentation unit, configured to segment an image into m sub-blocks, where m is an integer greater than 1;

An obtaining unit, configured to obtain m color brightness information groups of the m sub-blocks, each color brightness information group corresponding to one sub-block, and the color brightness information group includes brightness information and color information;

A mapping unit, configured to map the m color brightness information groups to a color brightness three-dimensional space to obtain m color brightness samples located in the color brightness three-dimensional space, where each color brightness sample is associated with a color The brightness information group corresponds, and the three-dimensional space of color brightness includes two color dimensions and one brightness dimension;

A calculation unit, configured to calculate all the m color luminance sample points into k layers along the luminance dimension, and each of the k layers is divided into P color luminance sample groups. The first blending degree of the number of clusters P corresponding to each of the k layers is described; wherein each of the k layers corresponds to a brightness interval, where P is an integer greater than 1 and k is a positive integer ;

A determining unit, configured to determine a second blending degree of the grouping number P corresponding to the image based on the first blending degree of the grouping number P corresponding to each of the k layers;

A comparison unit, configured to compare the second degree of fusion with a threshold of degree of fusion;

An estimation unit is configured to estimate that, in a case where the second blending degree is greater than the blending degree threshold, at least P light sources with different color temperatures exist in a shooting environment of the image.
The device according to claim 9, wherein the k is greater than 1, and the first number of the grouping number P corresponding to the image is determined based on a first blending degree of the number of groupings P corresponding to each of the k layers. In the aspect of two blending degrees, the determining unit is specifically configured to: perform summing or weighted summing processing on the first blending degree of the number of clusters P corresponding to each of the k layers to obtain corresponding clusters of the image Count the second degree of blending.
The device according to claim 9 or 10, wherein the P is greater than 2; in calculating a first blending degree of the number of clusters P corresponding to the i-th layer in the k layers, the calculation unit is specifically For: calculating the third degree of fusion between every two subgroups in the P subgroups of the i-th layer; and summing or weighting the third degree of integration between every two subgroups in the P subgroups And processing to obtain a first blending degree of the clustering number P corresponding to the i-th layer, where the i-th layer is any one of the k-layers.
The device according to claim 11, wherein in the aspect of calculating a third degree of fusion between the cluster gi and the cluster gj, the calculation unit is specifically configured to:

Insert continuously arranged measurement cells between the center point of the cluster gi and the center point of the cluster gj, and the number of the continuously arranged measurement cells is T;

The statistics include the number of measurement cells Q of the color brightness sample points in the cluster gi and the cluster gj; determining that the third degree of fusion between the cluster gi and the cluster gj is Q / T;

Wherein, the T and the Q are integers, the T is greater than 0 and the Q is greater than or equal to 0; the cluster gi and the cluster gj are any two of the P clusters of the i-th layer Subgroups.
The device according to claim 12, wherein:

The length direction of the continuously arranged measurement cells is perpendicular to the line connecting the center point of the cluster gi and the center point of the cluster gj.
The device according to claim 12 or 13, wherein the two color dimensions are a first color dimension and a second color dimension, wherein a first color dimension coordinate of a center point of any cluster is equal to the any one The average value of the first color dimension coordinates of all color brightness samples in a cluster, and the second color dimension coordinate of the center point of any one cluster is equal to the second color dimension coordinates of all color brightness samples in any one cluster. average value.
The device according to any one of claims 9 to 14, wherein the calculation unit, the determination unit, the comparison unit, and the estimation unit respectively perform the calculation for P = 2, ... X , The determination, the comparison, and the estimation, X is an integer greater than 2;

The estimation unit is further configured to determine, when it is estimated that P = 2, ... Y light sources with different color temperatures exist in the shooting environment, determine that P = Y light sources with different color temperatures exist in the shooting environment. , Wherein Y is an integer greater than 2 and less than or equal to X.
An image processing device, comprising:

Mutually coupled light source estimation device and correction device;

The light source estimation device is the light source estimation device according to any one of claims 9 to 15;

The correction device is configured to correct the image according to P light sources with different color temperatures, and the correction includes at least one of the following corrections: automatic white balance correction, color correction, saturation correction, or contrast correction.
A light source estimation device, characterized in that the light source estimation device includes a processor and a memory coupled with each other, and the memory stores a computer program; the processor is configured to call the computer program stored in the memory, The light source estimation method according to any one of claims 1 to 7 is performed.
An image processing device, wherein the light source estimation device includes a processor and a memory coupled to each other, and the memory stores a computer program; the processor is configured to call the computer program stored in the memory to The image processing method according to claim 8 is executed.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, wherein the computer program is executed by related hardware to complete the light source estimation according to any one of claims 1 to 7. method.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is executed by related hardware to complete the image processing method of claim 8.