JP2018148281A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow an optimal white balance gain (WBGain) to be set by using a classification category obtained by machine learning.SOLUTION: A teaching data storage unit 641 has stored teaching data including plural pieces of image data and category numbers of light sources. A machine learning unit 642 performs machine learning of criteria for determining categories of the light sources, and stores the learned criteria (classifier) in a related parameter storage unit 643. A light source identifying unit 644 performs category classification of an initial estimation vector with the classifier stored in the related parameter storage unit 643, and outputs categories of one or more light sources at a scene photographing a frame and data such as distance between the categories of the light sources and the initial estimation vector on a feature space to a white balance correction unit 65.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing apparatus.

Conventionally, in a digital camera, digital video camera, camera-equipped communication terminal (mobile phone, smartphone, tablet, etc.) classified as an imaging device, color reproduction correction processing is performed on a captured image.
That is, in general, image data obtained by an image sensor such as a CMOS image sensor is used for automatic exposure correction and automatic whitening using a semiconductor such as a system-on-chip (SoC) including an image signal processor (ISP). Image processing such as balance, autofocus, and shading correction is performed.

Here, in a system-on-chip including an image signal processor, various types of image processing such as color adjustment processing for improving color reproducibility is performed on image data. For this reason, the correction of the image data received by the imaging device, which is the preceding stage, greatly affects the color reproducibility, and plays an important role in the image processing apparatus and the image processing method.

Further, the light received by the image sensor is determined by various external factors. For example, in a scene shot outside a building, it is affected by various external factors such as weather, time, and a dark state such as a building shadow. Further, for example, in a scene photographed in a building, it is affected by various external factors such as the type and brightness of a lighting fixture. For this reason, from the image data obtained by an image sensor such as a CCD sensor or a CMOS image sensor, the type of external factor (for example, the type of ambient light) that is affected in the scene being photographed is identified and identified. It is necessary to perform color correction corresponding to each external factor (for example, whether the ambient light is sunlight or fluorescent light).

In this regard, a technique for specifying the type of ambient light in a scene being photographed using machine learning or the like has been proposed (see, for example, Patent Document 1 and Patent Document 2).

JP 2012-134625 A JP 2014-515587 A

However, in the conventional techniques including Patent Document 1 and Patent Document 2, specific means using machine learning are not clearly described, and it has been pointed out that the contents are technically difficult to implement.
On the other hand, conventionally, a method for specifying the type of ambient light based on “predetermined color” is known. Assuming that a plurality of ambient lights exist as the light source of the photographed scene, the “predetermined color” exists as a color approximate to the ambient light by the light emitted from each ambient light. It will be.
In addition, for example, it is difficult to always maintain a constant color of light irradiated on an object depending on every situation such as the state of the object surface, material, gloss, reflection, and the like. For this reason, the captured image data is not necessarily composed only of a specific color of ambient light, and it is difficult to specify the type of ambient light from a predetermined color.
Furthermore, technology that makes it possible to change the combination and method of image processing according to the input image data is also present. However, the input image data is infinite, and it can be easily imagined that it is difficult to use realistically.

  The present invention has been made in view of such a situation, and an object thereof is to set a more optimal color correction coefficient (WB Gain: White Balance Gain) using a classification category obtained by machine learning.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention includes:
First acquisition means for acquiring a predetermined light source estimation vector on the feature space by acquiring a predetermined parameter from the predetermined image data and mapping a value based on the parameter onto the predetermined feature space;
Classifying means for classifying the light source estimation vector acquired by the first acquisition means into a category on the feature space of the light source estimation vector by a classifier previously machine-learned;
Second acquisition means for acquiring an optimum WBGain from data of a predetermined frame of the image data based on the light source estimation vector and the category classified by the classification means;
Is provided.
Thereby, the optimal WBGain can be set using the classification category obtained by machine learning.

  According to the present invention, it is possible to set a more optimal WBGain using a classification category obtained by machine learning.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a functional block diagram illustrating details of a functional configuration of a white balance processing unit in the image processing apparatus of FIG. 1. 3 is a flowchart illustrating a flow of white balance processing executed by a white balance processing unit of the image processing apparatus in FIG. 2. It is a figure which shows the distribution image of the initial estimation vector obtained by mapping from the some teaching data acquired from the predetermined light source. It is a figure which shows the output image of the classifier obtained by machine learning. It is a figure which shows the initial estimation vector mapped from the data of the flame | frame. It is an image figure for demonstrating the optimal WBGain of the light source of each category. It is an image figure with a special category. It is an image figure for demonstrating the optimal WBGain corresponding to a special category. It is an image figure for demonstrating the case where an initial estimated vector exists in the overlapping distribution range of B light source and C light source.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention.
The image processing apparatus 1 includes a sensor Bayer data acquisition unit 2, an image pipe 3, and an output image data output unit 4.

  The sensor Bayer data acquisition unit 2 sequentially acquires each frame of data constituting a moving image captured by an imaging device (not shown) in the form of sensor Bayer data and outputs it to the image pipe 3.

  The image pipe 3 includes a shading correction processing unit 5, a white balance processing unit 6, a color interpolation processing unit 7, a color reproduction gain processing unit 8, and a gamma processing so as to perform various image processing on the data of each frame. Unit 9 and a color space conversion processing unit 10.

  The shading correction processing unit 5 performs a shading correction process on the frame data (sensor Bayer data) supplied from the sensor Bayer data acquisition unit 2. The data of the frame subjected to the shading correction process is supplied to the white balance processing unit 6.

  The white balance processing unit 6 performs white balance processing on the frame data supplied from the shading correction processing unit 5. The data of the frame subjected to the white balance process is supplied to the color interpolation processing unit 7.

  The color interpolation processing unit 7 performs color interpolation processing on the frame data supplied from the white balance processing unit 6. The data of the frame subjected to the color interpolation process is supplied to the color reproduction gain processing unit 8.

  The color reproduction gain processing unit 8 performs color reproduction gain processing on the frame data supplied from the color interpolation processing unit 7. The frame data subjected to the color reproduction gain processing is supplied to the gamma processing unit 9.

  The gamma processing unit 9 performs gamma processing on the frame data supplied from the color reproduction gain processing unit 8. The data of the frame subjected to the gamma processing is supplied to the color space conversion processing unit 10.

  The color space conversion processing unit 10 performs color space conversion processing on the frame data supplied from the gamma processing unit 9. The frame data subjected to the color space conversion process is supplied to the output image data output unit 4.

  The output image data output unit 4 outputs each frame data sequentially output from the image pipe 3 to an unillustrated circuit, which will be described later, as output image data.

  FIG. 2 is a functional block diagram mainly showing details of the functional configuration of the white balance processing unit 6 in the image processing apparatus 1 of FIG.

In the white balance processing unit 6, an RGB calculation unit 61, an initial estimation unit 62, a feature space mapping unit 63, a light source determination unit 64, a white balance correction unit 65, and a correction data output unit 66 function.
Details of the functions and the like of the RGB calculation unit 61 to the correction data output unit 66 will be described together with the description of the processing of the image processing apparatus 1 with reference to the flowcharts in FIG.

In the light source determination unit 64, a teaching data storage unit 641, a machine learning unit 642, a related parameter storage unit 643, and a light source identification unit 644 function.
The teaching data storage unit 641 stores teaching data including a plurality of image data and a category number of a light source of a scene in which each of the images is captured.
Based on the teaching data stored in the teaching data storage unit 641, the machine learning unit 642 performs machine learning on the basis for determining the category of the light source from the feature vector extracted from each image data, and learns the determination criterion (classification) Is stored in the related parameter storage unit 643. As a machine learning method, for example, SVM (support vector machine) can be employed.
The light source identification unit 644 uses the classifier stored in the related parameter storage unit 643 as the initial estimated vector mapped to the feature space by the feature space mapping unit 63 from the frame data described later and supplied to the light source determination unit 64. A white balance correction unit that classifies one or more light sources of a scene in which the frame is captured and data such as a distance in the feature space between the one or more light source categories and the initial estimated vector. Output to 65.

Next, the white balance processing executed by the white balance processing unit 6 of the image processing apparatus 1 will be described with reference to FIG.
Here, “white balance processing” refers to a series of processing in which the light source of the scene in which each frame is photographed is determined and white balance correction is performed on the data of each frame.

  FIG. 3 is a flowchart illustrating white balance processing executed by the white balance processing unit 6 of the image processing apparatus 1.

  The white balance processing is executed each time the pixel group data (hereinafter referred to as “frame data”) of the frame subjected to the shading correction processing is supplied from the shading correction processing unit 5 to the white balance processing unit 6. . The statistic data is the same data as the frame data.

  In step S <b> 11, the RGB calculation unit 61 calculates (R value, G value, B value) in the frame data as three channels, and supplies them to the initial estimation unit 62.

  In step S12, the initial estimation unit 62 extracts the maximum value for each of (R value, G value, B value) calculated in step S11. The maximum values (R value, G value, B value) are supplied to the feature space mapping unit 63 as initial estimated values (Rmax value, Gmax value, Bmax value) of the light source color.

In step S13, the feature space mapping unit 63 maps the initial estimated value (Rmax value, Gmax value, Bmax value) of the light source color extracted in step S12 to the feature space, and uses the obtained feature vector as the light source determination unit. 64. In this embodiment, the feature space is described as a two-dimensional space of (R / G, B / G).
The feature vector (R / G, B / G) in the feature space is a light source color value normalized by G, and this reciprocal is used as the optimum white balance gain of a predetermined light source. Is abbreviated as “optimal WBGain” below. The feature vector (Rmax / Gmax, Bmax / Gmax) obtained by mapping the initial estimated value (Rmax, Gmax, Bmax) extracted in step S12 to the feature space is hereinafter referred to as “initial estimated vector”.

In step S14, the light source determination unit 64 determines the type of light source into which the initial estimated vector mapped in the feature space in step S13 is classified.
That is, in step S14, the light source determination unit 64 determines one or more light source categories into which the initial estimated vector is classified based on the classifier. This classifier is stored in the related parameter storage unit 643 together with various related parameters in which related parameters learned in advance are stored.

  In step S15, in step S15, the white balance correction unit 65, based on the light source category determined by the light source determination unit 64 in step S14, prepares a LUT (not shown) prepared in advance for white balance correction of each light source. Lookup Table) is used to perform optimal white balance correction on the frame data. Note that the white balance correction when it is determined to be close to a plurality of light source categories will be described later with reference to FIG.

  In step S <b> 16, the correction data output unit 66 supplies the white balance corrected image data derived in step S <b> 15 to the color interpolation processing unit 7 that is the subsequent stage in the image processing apparatus 1.

  FIG. 4 is a diagram showing a distribution image of initial estimated vectors obtained by mapping onto a feature space from a plurality of teaching data acquired from a predetermined light source.

In FIG. 4, symbols A to E indicate category numbers into which initial estimation vectors obtained from a plurality of teaching data are classified.
That is, teaching data captured under a light source of an arbitrarily selected category is classified into each category and clustered.
In the example of FIG. 4, it is assumed that the light source classified into category A (hereinafter referred to as “A light source”) is sunlight. That is, the cluster classified as the A light source is obtained by classifying the teaching data imaged under sunlight into a cluster.
Here, light sources that are similarly classified into categories B to E are referred to as B light source to E light source, similarly to the A light source.
The output image of the classifier thus calculated will be described with reference to FIG. Note that the data of each frame of the scene captured in the present embodiment may include one or more of the light sources A to E.

FIG. 5 is a diagram illustrating an output image of the classifier obtained by machine learning of a plurality of teaching data in the feature space according to the present embodiment.
Note that symbols A to E indicate the category numbers into which the initial estimated vectors obtained from the plurality of teaching data are classified, as in FIG.

In the example of FIG. 5, the boundaries of the clusters representing the light sources A to E are defined by machine learning of the teaching data.
That is, by the above-described machine learning, the boundaries of the clusters representing the light sources A to E can be determined, and the classification of the clusters can be easily realized.
In this embodiment, the classifier has a function of determining the boundaries of the clusters representing the light sources of various light sources (A to E) by machine learning as described above. By using this classifier, the optimum WBGain can be easily calculated as will be described later.
Next, a method for acquiring the WBGain of the initial estimated light source calculated from the actually input predetermined frame data using the classifier calculated in this way will be described with reference to the following drawings.

  FIG. 6 is a diagram showing an initial estimated vector calculated from data of a predetermined frame in the feature space according to the present embodiment.

In the example of FIG. 6, the initial estimated vector H calculated from the data of the frame exists within the category range of the A light source.
A method of acquiring the newly calculated initial estimated vector H WBGain will be described with reference to FIG.

  That is, FIG. 7 is an image diagram for explaining the optimum WBGain of the light sources classified into the categories A to E in the feature space according to the present embodiment.

In the example of FIG. 7, an example in which the optimum WBGain is set for the initial estimated vector H classified as the A light source is shown.
Here, as described above, there is a possibility that the initial estimated vector distributed in the space divided into each category divided by the classifier is out of the optimum WBGain of each category due to various factors. For this reason, it is not desirable to perform white balance correction using the initial estimated vector H as it is.
Therefore, in the example of FIG. 7, the initial estimated vector H exists in the feature space classified as the A light source. Therefore, instead of the initial estimated vector, the optimum WBGain of the A light source set by the machine learning described above. Set.

  FIG. 8 is an image diagram in which a special category exists in the feature space according to the present embodiment.

In the example of FIG. 8, a case where a new special category having a certain distribution exists in the space of the C light source category is shown.
That is, when data of a plurality of frames are input, a new cluster may be created in a predetermined region in the feature space that is difficult to classify into any of the above categories A to E.
In such a case, a new cluster generated as a result of accumulating frame data can be referred to as a special category and identified as another light source. Therefore, the case where the initial estimation vector is classified into this special category will be described with reference to FIG.

  FIG. 9 is an image diagram for explaining the optimum WBGain corresponding to the special category in the feature space according to the present embodiment.

Referring to FIG. 9, the initial estimated vectors are classified into special categories newly set in FIG. When such a special category is set, a new classifier is calculated again for the special category by the machine learning described above. Then, the optimum WBGain corresponding to the special category is set separately.
That is, in the example of FIG. 9, as the initial estimated vector, the optimum WBGain corresponding to the special category separately set here is set as the optimum WBGain of the initial estimated vector.

  FIG. 10 is an image diagram for explaining a case where the initial estimated vector is within the overlapping distribution range of the B light source and the C light source in the feature space according to the present embodiment.

Referring to FIG. 10, for example, the initial estimated vector exists in the overlapping distribution range of the B light source and the C light source.
In such a case, first, the light source identification unit 644 calculates a similarity degree from the algorithm using the initial estimated vector and the optimum WBGain of each light source. Then, based on the similarity, it is selected whether the initial estimated vector is classified as a B light source or a C light source.
In the example of FIG. 10, the light source identification unit 644 selects the initial estimated vector as the optimum WBGain of the C light source as the similarity calculation result.

  Although one embodiment of the image processing apparatus of the present invention has been described above, the present invention is not limited to the above-described embodiment. In addition, the effects described in the present embodiment are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the present embodiment.

For example, in the above-described embodiment, it has been described that the SVM is used as the machine learning method. However, the machine learning method is not particularly limited thereto.
Furthermore, for example, the machine learning has been described on the assumption that it is completed before the white balance process is executed, but the present invention is not particularly limited thereto. That is, when performing white balance processing, it is possible to perform machine learning in parallel.

For example, in the above-described embodiment, it is described that the initial estimation unit 62 extracts the maximum value (Rmax value, Gmax value, Bmax value) of each channel of (R, G, B), but is not particularly limited thereto.
For example, (R value, G value, B value) may be converted into (Y value, U value, V value), and the maximum value may be extracted.
Further, for example, the average value (Rave value, Gave value, Bave value) of each channel may be extracted without extracting (Rmax value, Gmax value, Bmax value).

  For example, in the above-described embodiment, the feature space mapping unit 63 is described as mapping the (R / G, B / G) two-dimensional space as the feature space. However, the present invention is not particularly limited thereto. For example, more feature quantities may be added and mapped as a multidimensional feature space.

For example, in the example of FIG. 10, the light source identification unit 644 has been described as selecting the initial estimated vector classification using the concept of similarity, but how to define the concept of similarity, Further, how to select the classification of the initial estimated vector can be freely determined.
For example, the optimum WBGain of the initial estimated vector may be selected using the average value of the optimum WBGain of the B light source and the C light source weighted by the calculated similarity.

  The hardware configuration shown in FIG. 1 is merely an example for achieving the object of the present invention, and is not particularly limited.

The block diagram shown in FIG. 2 is merely an example, and is not particularly limited. That is, it is sufficient if the white balance processing unit 6 has a function capable of executing the above-described series of processing as a whole, and what functional block is used to realize this function is particularly illustrated in the example of FIG. It is not limited.
Furthermore, the location of the functional block is not limited to that shown in FIG.
One functional block may be constituted by hardware alone or in combination with software.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

In summary, the image processing apparatus to which the present invention is applied only needs to have the following configuration, and can take various embodiments.
That is, the image processing apparatus to which the present invention is applied (for example, the image processing apparatus 1 in FIG. 1)
Acquisition means for acquiring a predetermined light source estimation vector on the feature space by acquiring a predetermined parameter from the predetermined image data and mapping a value based on the parameter onto the predetermined feature space (for example, FIG. 2) An initial estimation unit 62 and a feature space mapping unit 63),
Classifying means (for example, a light source identification unit 644 in FIG. 2) for classifying the light source estimation vector acquired by the first acquisition means into a category in the feature space of the light source estimation vector by a classifier previously machine-learned. ,
Based on the light source estimation vector and the category classified by the classification unit, a correction unit (for example, a white balance correction unit in FIG. 2) that obtains an optimum WBGain from data of a predetermined frame of the image data and performs WB correction 65)
It is enough to have

As a result, even if multiple light sources exist as ambient light in the photographed scene, the feature vector acquired from the frame data is categorized using the classifier obtained by machine learning in advance. By specifying the light source, the optimum WBGain can be acquired more easily.
Therefore, it is possible to perform the optimum white balance correction more easily as compared with the conventional technique that performs (direct) white balance correction based on the light source color estimated from the frame data.

DESCRIPTION OF SYMBOLS 1 ... Image processing device 2 ... Sensor Bayer data acquisition part 3 ... Image pipe 4 ... Output image data output part 5 ... Shading correction process part 6 ... White balance process part 7. Color interpolation processing unit 8 ... Color reproduction gain processing unit 9 ... Gamma processing unit 10 ... Color space conversion processing unit 61 ... RGB calculation unit 62 ... Initial estimation unit 63 ... Feature space Mapping unit 64 ... Light source determination unit 65 ... White balance correction unit 66 ... Correction data output unit 641 ... Teaching data storage unit 642 ... Machine learning unit 643 ... Related parameter storage unit 644 ..Light source identification part

Claims (4)

First acquisition means for acquiring a predetermined light source estimation vector on the feature space by acquiring a predetermined parameter from the predetermined image data and mapping a value based on the parameter onto the predetermined feature space;
Classifying means for classifying the light source estimation vector acquired by the first acquisition means into a category on the feature space of the light source estimation vector by a classifier previously machine-learned;
Second acquisition means for acquiring an optimum WBGain from data of a predetermined frame of the image data based on the light source estimation vector and the category classified by the classification means;
An image processing apparatus comprising: The second acquisition unit acquires an optimum vector set in advance for each category classified by the classification unit as an optimum WBGain.
The image processing apparatus according to claim 1. The classification means estimates and identifies an optimum category by calculating a similarity when a plurality of category spaces overlap with respect to the classified category.
The image processing apparatus according to claim 1. The light source estimation vector acquired by the first acquisition unit is compared with the category classified by the classification unit, and when the classification is far from the category classified by the classification unit, a newly generated constant distribution is set as a special category. Set,
Based on the special category, an optimum WBGain different from the optimum WBGain obtained by the second obtaining unit is separately obtained.
The image processing apparatus according to claim 1.

Images