JP2013115571A - Information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a novel and improved information processing apparatus which is capable of controlling white balance more accurately.SOLUTION: An information processing apparatus is provided which comprises a pickup image capturing section which captures a pickup image, a block dividing section which divides the pickup image into a plurality of blocks, a statistic value calculation section which calculates a block statistic value by taking statistics of pixel values for each block, a light source estimation section which estimates a light source illuminating a subject of the pickup image based on the block statistic value and generates light source estimation information on an estimation result, a reliability calculation section which calculates reliability of the light source estimation information based on eccentricity of the block statistic value, and a white balance control section which control white balance of the pickup image based on the light source estimation information and the reliability of the light source estimation information.

Description

  The present invention relates to an information processing apparatus.

  An imaging apparatus such as a digital still camera performs white balance adjustment so that the color of the subject can be accurately reproduced regardless of the light source color. For example, the imaging apparatus divides the captured image into a plurality of blocks, determines the type of the light source based on the RGB statistical value of each block, and uses each correction coefficient based on the type of the light source to each pixel of the captured image. Correct the value. Thereby, the imaging apparatus performs white balance adjustment.

Japanese Patent No. 4081213 Japanese Patent No. 4754227 JP 2008-219414 A

  However, the conventional image pickup apparatus may erroneously determine the type of light source. This possibility is particularly high when the color distribution of the captured image is uniform (for example, when the ground or sky is drawn on the entire captured image). If the imaging device misjudged the type of light source, the actual color information (luminance, hue, saturation, etc.) of the subject does not match the color information in the captured image, that is, the subject color is regarded as the light source As a result, a color failure, which is a phenomenon in which colors are biased in the complementary color direction, has occurred.

  As techniques for preventing the occurrence of color feria, techniques disclosed in Patent Documents 1 to 3 are known. The techniques disclosed in Patent Documents 1 and 2 divide a captured image into a plurality of blocks, and calculate the reliability of the light source for each block for each type of light source. This reliability is low when the color distribution in the block varies (various colors are included in the block), and is high when the color distribution in the block is uniform. As a result, the techniques disclosed in Patent Documents 1 and 2 can prevent light source misjudgment due to color mixture in the block. For example, the techniques disclosed in Patent Documents 1 and 2 prevent erroneous determination of the color of a block as yellow, which is the color of a light bulb, when a green pixel and a red pixel are included in the block. be able to.

  However, even with these techniques, the light source cannot be accurately specified when the color distribution of the captured image is uniform. Furthermore, in these techniques, when the color distribution of the captured image is uniform, the color distribution in the block is uniform, so the reliability is also high for light sources other than the light source at the imaging location. There was also a problem. Therefore, these techniques cannot solve the above problem.

  On the other hand, the technique disclosed in Patent Document 3 divides an image into a plurality of regions (for example, a human face region and other regions), and performs light source estimation and white balance adjustment for each region. However, Patent Document 3 does not disclose any processing for a case where the color distribution of the captured image is uniform. Therefore, even the technique disclosed in Patent Document 3 cannot solve the above problem.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved information processing apparatus capable of adjusting white balance more accurately. There is.

  In order to solve the above-described problem, according to an aspect of the present invention, a captured image acquisition unit that acquires a captured image, a block division unit that divides the captured image into a plurality of blocks, and a pixel value for each block is statistics. Thus, a statistical value calculation unit that calculates a block statistical value, a light source estimation unit that estimates a light source that illuminates the subject of the captured image based on the block statistical value, and generates light source estimation information related to the estimation result, and a block statistical value A reliability calculation unit that calculates the reliability of the light source estimation information based on the bias, a white balance adjustment unit that adjusts the white balance of the captured image based on the light source estimation information and the reliability of the light source estimation information, and , An information processing apparatus is provided.

  According to this aspect, the information processing apparatus calculates the reliability of the light source estimation information based on the bias of the block statistical value, and based on the light source estimation information and the reliability of the light source estimation information, Adjust the balance. Therefore, since the information processing apparatus can reduce the reliability when the color distribution of the captured image is uniform, the white balance can be adjusted more accurately.

  Here, the information management unit that stores the reliability correspondence information in which the light source estimation information and the reliability of the light source estimation information are associated is stored in the storage unit, and the white balance adjustment unit stores the reliability correspondence information stored in the storage unit Among them, the white balance of the captured image may be adjusted based on the reliability correspondence information with the highest reliability of the light source estimation information.

  According to this aspect, the information processing apparatus is based on the reliability correspondence information with higher reliability when the reliability in the current frame is low, such as when the color distribution of the captured image is uniform. The white balance can be adjusted. Thereby, the information processing apparatus can adjust the white balance more accurately. In particular, when the reliability decreases due to a change in the imaging environment, the information processing apparatus can cause the white balance to slowly follow the change in the imaging environment.

  In addition, when the number of reliability correspondence information stored in the storage unit exceeds a predetermined number of frames, the information management unit obtains the oldest reliability correspondence information stored in the storage unit from the storage unit. It may be deleted.

  According to this aspect, since the information processing apparatus can delete the reliability correspondence information that is most likely not to match the imaging environment of the current frame, the white balance can be adjusted more accurately.

  Further, the information management unit may delete the reliability correspondence information stored in the storage unit when the luminance of the captured image exceeds a predetermined range.

  According to this aspect, the information processing apparatus can delete the reliability correspondence information that is very likely not to match the imaging environment of the current frame. Therefore, the information processing apparatus can adjust the white balance more accurately.

  The reliability calculation unit may calculate the reliability based on the bias of the block statistical value and at least one of the subject brightness of the captured image and the color temperature of the light source.

  According to this aspect, since the information processing apparatus calculates the reliability based on the bias of the block statistical value and at least one of the subject brightness of the captured image and the color temperature of the light source, the reliability is calculated more accurately. can do.

  In addition, the reliability calculation unit classifies the block into one of a plurality of color groups based on the block statistical value, and blocks the variance value or standard deviation of the number of blocks classified into each color group. You may calculate as a bias of a statistical value.

  According to this aspect, the information processing apparatus classifies the blocks into one of a plurality of color groups based on the block statistics, and blocks the variance value of the number of blocks classified into each color group. Calculated as statistical value bias. Therefore, the information processing apparatus can calculate the reliability more accurately.

  In addition, the white balance adjustment unit calculates a first white balance correction coefficient based on the light source estimation information and the reliability of the light source estimation information, and filters the first white balance correction coefficient to obtain the first white balance correction coefficient. A second white balance correction coefficient that follows the white balance correction coefficient of 1 by following a predetermined convergence time is calculated, and the white balance of the captured image is adjusted based on the second white balance correction coefficient. , It may be set shorter.

  According to this aspect, since the information processing apparatus sets the convergence time of the second white balance correction coefficient to be short (for example, 1 to 2 seconds), when the reliability increases due to a change in the imaging environment, the white balance Can quickly follow changes in the imaging environment.

  As described above, according to the present invention, when the color distribution of the captured image is uniform, the reliability can be lowered, so that the white balance can be adjusted more accurately.

It is a block diagram which shows the structure of the imaging device which concerns on embodiment of this invention. It is a block diagram which shows the structure of the image signal processing part concerning the embodiment. It is explanatory drawing which shows a mode that a captured image is divided | segmented into a some block. It is explanatory drawing which shows an example of a color group. It is explanatory drawing which shows an example of a color group. It is explanatory drawing which shows the number of the blocks which belong to each color group. It is explanatory drawing which shows the specific example of the number of the blocks which belong to each color group. It is a graph which shows the correspondence of the dispersion value of the number of blocks which belong to each color group, and reliability. It is a graph which shows the correspondence of the to-be-photographed object's brightness and reliability of a captured image. It is a graph which shows the correspondence of the color temperature of a light source, and reliability. It is explanatory drawing which shows the reliability information memorize | stored in a memory | storage part. It is explanatory drawing which shows a mode that reliability falls according to progress of time. It is explanatory drawing which shows a mode that reliability increases with progress of time. It is a flowchart which shows the procedure of the process by an imaging device.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

<1. Configuration of Imaging Device>
First, a schematic configuration of an imaging apparatus (information processing apparatus) 10 according to the first embodiment of the present invention will be described with reference to FIG. The imaging device 10 includes a lens unit 11, an imaging device 12, an AFE circuit 13, an image signal processing unit 14, an exposure control unit 16, a timing generator 17, a driver 18, an image display unit 19, and an image recording. Part 20.

  Note that the imaging apparatus 10 has hardware such as a CPU, ROM, RAM, display, auxiliary light source, and flash memory, and the CPU reads and executes a program stored in the ROM, whereby each of the above functions is performed. Processing by blocks is realized. That is, the ROM stores the above-described functional blocks, particularly the image signal processing unit 14, the exposure control unit 16, the timing generator 17, the driver 18, the image display unit 19, and the image recording unit 20. A program for realizing is stored. The ROM stores information such as various tables and graphs necessary for processing of the imaging apparatus 10.

  As another embodiment, an information processing apparatus (for example, a computer such as a personal computer or a server) including the image signal processing unit 14 can be considered. In this embodiment, the information processing apparatus acquires a captured image from an imaging apparatus or the like, and adjusts the white balance of the captured image by performing processing described later on the captured image. Further, the present embodiment is applicable to both the case where a still image is captured and the case where a moving image is captured. However, as will be described later, since the white balance correction coefficient follows the change in the imaging environment, the present embodiment is particularly preferably applied when capturing a moving image.

  The lens unit 11 is detachable from the imaging apparatus 10 and includes a plurality of lenses, a diaphragm mechanism, a shutter, and the like. The lens unit 11 captures visible light from the outside of the imaging device 10 and causes the captured light to enter the image sensor 12 via a lens or the like.

  The image sensor 12 includes a plurality of unit elements arranged in a matrix and an image sensor control unit (not shown). The unit element corresponds to a unit pixel of the captured image. Each unit element includes an R element that receives red light, a G element that receives green light, and a B element that receives blue light. The unit element outputs R information, G information, and B information (hereinafter collectively referred to as “RGB information”) relating to the reception intensity of each of red light, green light, and blue light to the image sensor control unit. The imaging element control unit generates a captured image having RGB information as pixel values based on the RGB information given from each unit element. Thereby, the image sensor 12 performs imaging.

  The image sensor control unit outputs the generated captured image to the AFE circuit 13. Note that an xy plane is set in the image sensor 12, and a coordinate value on the xy plane is given to each pixel of the captured image. The x-axis and y-axis directions are arbitrary. For example, the y-axis is an axis extending in the vertical direction of the imaging device 12, that is, the vertical direction of the imaging device 10, and the x-axis extends in a direction perpendicular to the y-axis. Is the axis.

  The AFE circuit 13 performs analog / digital conversion on the captured image supplied from the image sensor 12 and outputs the converted captured image to the image signal processing unit 14. As shown in FIG. 2, the image signal processing unit 14 includes a captured image acquisition unit 141, a preprocessing unit 142, a block division unit 143, a statistical value calculation unit 144, a light source estimation unit 145, and a reliability calculation unit. 146, an information management unit 147, a storage unit 148, a white balance adjustment unit 149, and a post-processing unit 150.

  The captured image acquisition unit 141 acquires a captured image from the AFE circuit 13 and outputs it to the preprocessing unit 142. The preprocessing unit 142 performs various types of preprocessing such as defective pixel correction, black level correction, shading correction, and camera shake correction on the captured image. Further, the preprocessing unit 142 assigns a frame number to the captured image that has been preprocessed. This frame number is information for uniquely identifying each captured image. The preprocessing unit 142 outputs the captured image to the block dividing unit 143 and the white balance adjusting unit 149.

  The block dividing unit 143 divides the captured image into a plurality of blocks. For example, the block dividing unit 143 divides the captured image into 16 × 12 blocks 100 as illustrated in FIG. 3. Each block 100 includes a plurality of unit pixels. Each unit pixel includes an R pixel 100R having R information, two G pixels 100G having G information, and a B pixel 100B having B information. In addition, the block division unit 143 gives xy coordinates to each block. For example, in the example shown in FIG. 2, each block is given an integer in the range of 0 to 15 as the x coordinate, and an integer in the range of 0 to 11 is given as the y coordinate. The block dividing unit 143 outputs block information regarding the generated block to the statistical value calculating unit 144.

  The statistical value calculation unit 144 performs the following processing based on the block information. That is, the statistical value calculation unit 144 integrates (statistics) R information, G information, and B information for each block, so that the R block statistical value, the G block statistical value, and the B block statistical value (hereinafter, these are calculated). (Also collectively referred to as RGB block statistics). The statistical value calculation unit 144 generates RGB block statistical value information in which the RGB block statistical value and the xy coordinates of the block are associated with each other, and outputs the RGB block statistical value information to the light source estimation unit 145.

  Then, the statistical value calculation unit 144 adds up the R block statistical value, the G block statistical value, and the B block statistical value for all the blocks, thereby collecting the R statistical value, the G statistical value, and the B statistical value (hereinafter, these are summarized). (Also referred to as RGB statistical values). Then, the statistical value calculation unit 144 calculates a luminance statistical value of the captured image based on the following formula (1), and calculates the average luminance by dividing the luminance statistical value by the number of pixels of the captured image.

  In Equation (1), Y is a luminance statistical value, R is an R statistical value, G is a G statistical value, and B is a B statistical value. The statistical value calculation unit 144 outputs average luminance information related to the average luminance to the information management unit 147 and the exposure control unit 16. On the other hand, the statistical value calculation unit 144 converts the RGB block statistical values of each block into color coordinates in the color space map. Here, the color space map is a map showing the correspondence between the color coordinates in the color space and the color group.

4 and 5 show examples of color space maps. The color space map shown in FIG. 4 shows the correspondence between color coordinates in the CbCr plane and color groups 0-13. For example, the color coordinates in the area 110 correspond to the color group 8. The color space map shown in FIG. 5 shows the correspondence between the color coordinates in the CIExy plane and the color groups 0, 1, 2,. For example, the color coordinates in region 120 correspond to color group 0. The color space used for the color space map is not limited to these examples, and may be any color space, for example, each color space such as RGB, L ^* a ^* b ^* , and YPbPr.

  Then, the statistical value calculation unit 144 classifies each block into one of color groups based on the color coordinates and color space map of each block. Then, as illustrated in FIG. 6, the statistical value calculation unit 144 generates block distribution information indicating a correspondence relationship between the type of color group and the number of blocks belonging to the color group, and outputs the block distribution information to the reliability calculation unit 146. .

  Here, an example of the block distribution information is shown in FIG. In the block distribution information example 1 shown in FIG. 7, the block distribution is biased toward a specific color group, and in the block distribution information example 4, the block distribution is the most varied. At the bottom of each example, the variance value of the number of blocks classified into each color group, that is, the variance value “VAR” of the block distribution information is shown. The block distribution information example 1 is block distribution information obtained when a plant is imaged in proximity, and the block distribution information example 2 is block distribution information obtained when an underground passage is imaged. Block distribution information example 3 is block distribution information obtained when a bird is imaged in close proximity, and block distribution information example 4 is block distribution information obtained when a plate on which many dishes are placed is photographed.

  As shown in FIG. 7, when the block distribution is biased toward a specific color group, the variance value becomes large, but when the block distribution varies, the variance value becomes small. When the color distribution of the captured image is uniform, the distribution of blocks is biased toward a specific color group, and thus the variance value becomes large. Therefore, as described later, as the variance value increases, the reliability of the light source estimation information decreases.

  The light source estimation unit 145 estimates a light source (light source color in a specific example) that illuminates the subject of the captured image based on the RGB block statistical value information. The method for estimating the light source is not particularly limited, and a known method is arbitrarily applied. The light source estimation unit 145 outputs estimation result information regarding the estimation result to the reliability calculation unit 146 and the information management unit 147. The estimation result information indicates, for example, the light source color in RGB coordinates (EstR, EstG, EstB).

  The reliability calculation unit 146 calculates a variance value of the number of blocks classified into each color group, that is, a variance value of block distribution information based on the following equation (2).

  The variance value takes the maximum value when all the blocks are classified into one color group. For example, when the total number of blocks is 16 × 16 = 256 and the total number of color groups is 14, the maximum value of the variance value is 4346.8.

  And the reliability calculation part 146 calculates the dispersion value corresponding | compatible reliability which is one of the reliability of light source estimation information based on the dispersion value corresponding | compatible reliability graph L1 shown in FIG. 8, and a dispersion value. Here, the dispersion value correspondence reliability graph L1 shows the correspondence between the dispersion value and the dispersion value correspondence reliability (Ra). As described above, the larger the variance value, the more uniform the color distribution of the captured image, making it difficult to estimate the light source. Therefore, the greater the variance value, the smaller the reliability corresponding to the variance value.

  Further, the reliability calculation unit 146 calculates a subject brightness correspondence reliability that is one of the reliability of the light source estimation information, based on the subject brightness correspondence reliability graph L2 illustrated in FIG. 9 and the setting value information. Here, the subject brightness correspondence reliability graph L2 indicates the correspondence between the subject brightness of the captured image and the subject brightness correspondence reliability (Rb). The greater the brightness of the subject, the brighter the light source that illuminates the subject, making it easier to identify the light source (for example, if the light source is bright, the light source is likely to be sunlight). Therefore, the greater the subject brightness, the greater the subject brightness correspondence reliability. The set value information is given by the exposure control unit 16 described later. Specifically, the reliability calculation unit 146 calculates the subject brightness based on the set value information and the following equation (2-1).

  In equation (2-1), BV is the subject brightness, TV is the shutter speed, AV is the aperture value, and SV is the gain of the AFE circuit 13. Then, the reliability calculation unit 146 calculates the subject brightness reliability based on the subject brightness and the subject brightness correspondence reliability graph L2.

  Further, the reliability calculation unit 146 identifies the estimated color temperature of the light source based on the light source estimation information. Then, the reliability calculation unit 146 calculates a color temperature correspondence reliability, which is one of the reliability of the light source estimation information, based on the color temperature correspondence reliability graph L3 illustrated in FIG. 10 and the color temperature of the light source. . Here, the color temperature correspondence reliability graph L3 shows the correspondence between the color temperature of the light source and the color temperature correspondence reliability (Rc). When the color temperature of the light source is about 5000K to 8000K, it is easy to identify the light source (the light source is likely to be sunlight). On the other hand, if the color temperature of the light source is too high or too low, it is difficult to specify the light source. Therefore, when the color temperature of the light source is about 5000K to 8000K, the reliability corresponding to the color temperature is increased.

  Then, the reliability calculation unit 146 calculates the reliability of the estimation result information by multiplying the dispersion value correspondence reliability, the subject brightness correspondence reliability, and the color temperature correspondence reliability. Then, the reliability calculation unit 146 generates reliability information regarding the reliability of the estimation result information and outputs the reliability information to the information management unit 147.

  Note that the reliability calculation unit 146 may calculate a standard deviation instead of the variance value. In this case, a standard deviation correspondence reliability graph is prepared instead of the variance value correspondence reliability graph L1, and the reliability calculation unit 146 performs standard deviation correspondence reliability based on the standard deviation and the standard deviation correspondence reliability graph. Calculate the degree. Then, the reliability calculation unit 146 calculates the reliability of the estimation result information by multiplying the reliability corresponding to the standard deviation, the reliability corresponding to the subject brightness, and the reliability corresponding to the color temperature.

  In addition, the reliability calculation unit 146 may use the dispersion value corresponding reliability as it is as the reliability of the estimation result information. In addition, the reliability calculation unit 146 may calculate the reliability of the estimation result information by multiplying the dispersion value correspondence reliability by the subject lightness correspondence reliability or the color temperature correspondence reliability.

  The information management unit 147 first determines whether or not the average luminance exceeds a predetermined range based on the average luminance information. As a result, when the information management unit 147 determines that the average luminance exceeds the predetermined range, the information management unit 147 determines that the imaging environment, specifically, the light source has changed significantly, and clears the storage unit 148.

  Then, the information management unit 147 causes the storage unit 148 to store reliability correspondence information in which the light source estimation information, the reliability information, and the frame number of the captured image are associated. When the number of reliability correspondence information stored in the storage unit 148 exceeds a predetermined number of frames, the information management unit 147 stores the oldest reliability correspondence information stored in the storage unit 148. Delete from 148. That is, the information management unit 147 manages the reliability correspondence information stored in the storage unit 148 by performing a FIFO process.

  The state of the FIFO processing by the information management unit 147 is shown in FIG. FIG. 11 shows the reliability correspondence information stored in the storage unit 148 as a bar graph. The horizontal axis indicates the frame number, the right end indicates the newest frame number, and the left end indicates the oldest frame number. The vertical axis shows the reliability. The information management unit 147 stores the reliability correspondence information R1 generated for the captured image of the current frame in the storage unit 148. When the number of reliability correspondence information stored in the storage unit 148 exceeds a predetermined number of frames, the information management unit 147 deletes the oldest reliability correspondence information R3.

  Further, the information management unit 147 outputs, to the white balance adjustment unit 149, the highest reliability among the reliability correspondence information stored in the storage unit 148. In the example illustrated in FIG. 11, the information management unit 147 outputs the reliability correspondence information R <b> 2 having the highest reliability to the white balance adjustment unit 149. As a result, even if the reliability of the current frame decreases from the reliability of the previous frame due to a change in the imaging environment (for example, panning or subject change), the information management unit 147 displays the reliability correspondence information with the maximum reliability. This can be provided to the white balance adjustment unit 149.

  FIG. 12 shows an example in which the reliability of the current frame decreases from the reliability of the previous frame. The reliability correspondence information in the storage unit 148 transitions from (a) to (i) in FIG. The horizontal and vertical axes in FIGS. 12A to 12I are the same as those in FIG. At the time shown in FIG. 12A, the information management unit 147 stores the reliability correspondence information R1a in the storage unit 148, and outputs the reliability correspondence information R2a having the highest reliability to the white balance adjustment unit 149.

  Similarly, at each time point shown in FIGS. 12B to 12I, the information management unit 147 stores the reliability correspondence information R1b to R1i in the storage unit 148, while having the highest reliability at each time point. The degree correspondence information R2b to R2i is output to the white balance adjustment unit 149. That is, the information management unit 147 outputs reliability correspondence information of a frame before the current frame to the white balance adjustment unit 149. Therefore, when the reliability decreases due to a change in the imaging environment, the white balance adjustment unit 149 can cause the white balance to slowly follow the change in the imaging environment.

  The white balance adjustment unit 149 calculates a first white balance correction coefficient based on the light source color indicated by the reliability correspondence information. The calculation method of the first white balance correction coefficient is not particularly limited, and a known method is arbitrarily applied. For example, the white balance adjustment unit 149 calculates the first white balance correction coefficient based on the following equation (3).

In Expression (3), Max (EstR, EstG, EstB) indicates the maximum value of EstR, EstG, EstB, and K _r , K _g , and K _b are first white balance correction coefficients, respectively. That is, the first white balance correction coefficient _Kr is a correction coefficient for the R information of the captured image, and the first white balance correction coefficient _Kg is a correction coefficient for the G information of the captured image. The balance correction coefficient _Kb is a correction coefficient for the B information of the captured image.

  Further, the white balance adjustment unit 149 calculates a second white balance correction coefficient by applying a low pass filter to the first white balance correction coefficient. Then, the white balance adjustment unit 149 adjusts the white balance of the captured image based on the second white balance correction coefficient.

  Therefore, the second white balance correction coefficient follows the time change of the first white balance correction coefficient with a delay of a predetermined convergence time. Here, the convergence time is a time from when a first white balance correction coefficient is calculated until the second white balance correction coefficient converges (matches) with the first white balance correction coefficient. For example, it is set as short as 1-2 seconds. Therefore, the second white balance correction coefficient quickly follows the first white balance correction coefficient.

  Here, as described above, when the reliability decreases due to a change in the imaging environment, the information management unit 147 outputs the reliability correspondence information of the frame before the current frame to the white balance adjustment unit 149. Accordingly, the first white balance correction coefficient slowly follows the change in the imaging environment, so the second white balance correction coefficient also slowly follows the change in the imaging environment.

  On the other hand, since the information management unit 147 outputs the reliability correspondence information having the highest reliability among the reliability correspondence information stored in the storage unit to the white balance adjustment unit 149, the reliability increases due to a change in the imaging environment. In this case, the reliability correspondence information of the current frame is output to the white balance adjustment unit 149. Therefore, since the first white balance correction coefficient quickly follows the change in the imaging environment, the second white balance correction coefficient also quickly follows the change in the imaging environment.

  As a result, the white balance adjustment unit 149 can ensure the robustness of the second white balance correction coefficient and suppress the occurrence of the dead zone time when the reliability increases due to a change in the imaging environment. Furthermore, the user can grasp the change of the captured image more naturally.

  Conventionally, since the focus was only on ensuring robustness, the convergence time was very long (for example, about 100 seconds). For this reason, the white balance of the captured image does not readily follow the change in the imaging environment, and the user may feel uncomfortable with the captured image. On the other hand, since the white balance adjustment unit 149 sets the convergence time to about 1 to 2 seconds, when the reliability increases due to the change in the imaging environment, the white balance of the captured image is quickly changed to the change in the imaging environment. Can be followed.

  FIG. 13 shows an example in which the reliability of the current frame increases from the reliability of the previous frame. The reliability correspondence information in the storage unit 148 transitions from FIG. 13A to FIG. 13C. 13A to 13C are the same as those in FIG. At the time shown in FIG. 13A, the information management unit 147 stores the reliability correspondence information R1j in the storage unit 148, while the reliability correspondence information R2j (= R1j) having the highest reliability is stored in the white balance adjustment unit 149. Output to.

  Similarly, at each time point illustrated in FIGS. 13B to 13C, the information management unit 147 stores the reliability correspondence information R1k to R1l in the storage unit 148, while the reliability having the highest reliability at each time point. The degree correspondence information R2k to R2l (= R1k to R1l) is output to the white balance adjustment unit 149. Therefore, the reliability correspondence information increases as time elapses, that is, as the imaging environment changes. In this case, the white balance adjustment unit 149 can quickly follow the change in the imaging environment with the second white balance correction coefficient, that is, the white balance of the captured image.

  The white balance adjustment unit 149 outputs the captured image whose white balance has been adjusted to the post-processing unit 150. The post-processing unit 150 performs various post-processing on the captured image, for example, demosaic processing, smoothing, edge enhancement processing, color reproduction correction, tone control processing, gamma correction, and YCbCr (luminance, hue, saturation) conversion. Do. The post-processing unit 196 displays the post-processing noise-removed image on the image display unit 19 shown in FIG.

  The exposure control unit 16 adjusts the drive timing of the image sensor 12 via the timing generator 17. Furthermore, the exposure control unit 16 adjusts the aperture, shutter speed, and shutter speed of the lens unit 11 so that the average brightness of the captured image matches the preset target brightness (that is, the exposure of the image sensor 12 is appropriate). And the gain of the AFE circuit 13 is set. Then, the exposure control unit 16 outputs setting value information regarding these setting values to the driver 18, the AFE circuit 13, and the reliability calculation unit 146. The driver 18 and the AFE circuit 13 are driven based on these set values. The set value information is given by, for example, an APEX value.

<2. Processing procedure by imaging device>
Next, a processing procedure by the imaging apparatus 10 will be described with reference to a flowchart shown in FIG. In step S <b> 10, the captured image acquisition unit 141 acquires a captured image from the AFE circuit 13 and outputs it to the preprocessing unit 142.

  In step S20, the preprocessing unit 142 performs various types of preprocessing such as defective pixel correction, black level correction, shading correction, and camera shake correction on the captured image. Further, the preprocessing unit 142 assigns a frame number to the captured image that has been preprocessed. The preprocessing unit 142 outputs the captured image to the block dividing unit 143 and the white balance adjusting unit 149.

  In step S30, the block dividing unit 143 divides the captured image into a plurality of blocks. In addition, the block division unit 143 gives xy coordinates to each block. The block dividing unit 143 outputs block information regarding the generated block to the statistical value calculating unit 144.

  In step S40, the statistical value calculation unit 144 performs the following processing based on the block information. That is, the statistical value calculation unit 144 calculates R, G, and B block statistical values by integrating (statistics) R information, G information, and B information for each block. The statistical value calculation unit 144 generates RGB block statistical value information in which the RGB block statistical value and the xy coordinates of the block are associated with each other, and outputs the RGB block statistical value information to the light source estimation unit 145.

  Next, the statistical value calculation unit 144 calculates an RGB statistical value by integrating the R block statistical value, the G block statistical value, and the B block statistical value for all blocks. Then, the statistical value calculation unit 144 calculates a luminance statistical value of the captured image based on the above-described formula (1), and calculates the average luminance by dividing the luminance statistical value by the number of pixels of the captured image.

  Next, the statistical value calculation unit 144 outputs average luminance information regarding the average luminance to the information management unit 147 and the exposure control unit 16. On the other hand, the statistical value calculation unit 144 converts the RGB block statistical values of each block into color coordinates in the color space map.

  Next, the statistical value calculation unit 144 classifies each block into one of the color groups based on the color coordinates of each block and the color space map. Then, as illustrated in FIG. 6, the statistical value calculation unit 144 generates block distribution information indicating a correspondence relationship between the type of color group and the number of blocks belonging to the color group, and outputs the block distribution information to the reliability calculation unit 146. .

  In step S50, the light source estimation unit 145 estimates the light source (light source color in the specific example) that illuminates the subject of the captured image based on the RGB block statistical value information. The light source estimation unit 145 outputs estimation result information regarding the estimation result to the reliability calculation unit 146 and the information management unit 147.

  On the other hand, in step S60, the reliability calculation unit 146 calculates the variance value of the number of blocks classified into each color group, that is, the variance value of the block distribution information based on the above-described equation (2).

  Next, the reliability calculation unit 146 calculates a dispersion value corresponding reliability, which is one of the reliability of the light source estimation information, based on the dispersion value corresponding reliability graph L1 illustrated in FIG. 8 and the dispersion value. Further, the reliability calculation unit 146 calculates a subject brightness correspondence reliability that is one of the reliability of the light source estimation information, based on the subject brightness correspondence reliability graph L2 illustrated in FIG. 9 and the setting value information. Further, the reliability calculation unit 146 identifies the estimated color temperature of the light source based on the light source estimation information. Next, the reliability calculation unit 146 calculates a color temperature correspondence reliability, which is one of the reliability of the light source estimation information, based on the color temperature correspondence reliability graph L3 illustrated in FIG. 10 and the color temperature of the light source. .

  Next, the reliability calculation unit 146 calculates the reliability of the estimation result information by multiplying the dispersion value correspondence reliability, the subject brightness correspondence reliability, and the color temperature correspondence reliability. Then, the reliability calculation unit 146 generates reliability information regarding the reliability of the estimation result information and outputs the reliability information to the information management unit 147.

  In step S70, the information management unit 147 determines whether the average luminance exceeds a predetermined range based on the average luminance information. If the information management unit 147 determines that the average luminance exceeds the predetermined range, the information management unit 147 proceeds to step S80. If the information management unit 147 determines that the average luminance is within the predetermined range, the information management unit 147 proceeds to step S90.

  In step S80, the information management unit 147 determines that the imaging environment, specifically, the light source has changed significantly, and clears the storage unit 148. In step S90, the information management unit 147 causes the storage unit 148 to store reliability correspondence information in which the light source estimation information, the reliability information, and the frame number of the captured image are associated. Next, the information management unit 147 determines whether or not the number of reliability correspondence information stored in the storage unit 148 has exceeded a predetermined number of frames. When the information management unit 147 determines that the number of reliability correspondence information exceeds the predetermined number of frames, the information management unit 147 deletes the oldest reliability correspondence information stored in the storage unit 148 from the storage unit 148. That is, the information management unit 147 manages the reliability correspondence information stored in the storage unit 148 by performing a FIFO process.

  In step S <b> 100, the information management unit 147 outputs the highest reliability among the reliability correspondence information stored in the storage unit 148 to the white balance adjustment unit 149. In step S110, the white balance adjustment unit 149 calculates a first white balance correction coefficient based on the light source color indicated by the reliability correspondence information. For example, the white balance adjustment unit 149 calculates the first white balance correction coefficient based on the above-described equation (3).

  In step S120, the white balance adjustment unit 149 calculates a second white balance correction coefficient by applying a low-pass filter to the first white balance correction coefficient. Next, the white balance adjustment unit 149 adjusts the white balance of the captured image based on the second white balance correction coefficient, and outputs the adjusted captured image to the post-processing unit 150.

  In step S130, the post-processing unit 150 performs various post-processing on the captured image, for example, demosaic processing, smoothing, edge enhancement processing, color reproduction correction, tone control processing, gamma correction, YCbCr (luminance, hue, saturation). Degree) conversion. The post-processing unit 196 displays the post-processing noise-removed image on the image display unit 19 shown in FIG. Thereafter, the imaging device 10 ends the process.

  As described above, in the present embodiment, the imaging apparatus 10 calculates the reliability of the light source estimation information based on the bias of the block statistical value, specifically, the variance value of the block distribution information, and the light source estimation information and the light source estimation The white balance of the captured image is adjusted based on the reliability of the information. Therefore, the imaging apparatus 10 can reduce the reliability when the color distribution of the captured image is uniform, and thus can adjust the white balance more accurately.

  More specifically, the imaging device 10 stores the reliability correspondence information in the storage unit 148, and based on the reliability correspondence information having the highest reliability among the reliability correspondence information stored in the storage unit 148, Adjust the white balance of the captured image. Therefore, when the reliability in the current frame is low as in the case where the color distribution of the captured image is uniform, the imaging device 10 performs white balance based on the reliability correspondence information with higher reliability. Can be adjusted. Thereby, the imaging device 10 can adjust the white balance more accurately. In particular, when the reliability decreases due to a change in the imaging environment, the imaging apparatus 10 can cause the white balance to slowly follow the change in the imaging environment.

  Further, when the number of reliability correspondence information stored in the storage unit 148 exceeds the predetermined number of frames, the imaging device 10 stores the oldest reliability correspondence information stored in the storage unit 148. Delete from the part 148. As a result, the imaging apparatus 10 can delete the reliability correspondence information that is most likely not to match the imaging environment of the current frame, so that the white balance can be adjusted more accurately.

  Furthermore, since the imaging device 10 deletes the reliability correspondence information stored in the storage unit 148 when the average luminance of the captured image exceeds a predetermined range, there is a possibility that the imaging device 10 does not match the imaging environment of the current frame. High reliability correspondence information can be deleted. Therefore, the imaging apparatus 10 can adjust the white balance more accurately.

  Furthermore, since the imaging apparatus 10 calculates the reliability based on the bias of the block statistical value, that is, the variance value of the block distribution information, the subject brightness of the captured image, and the color temperature of the light source, the reliability can be more accurately determined. Can be calculated.

  Further, the imaging apparatus 10 classifies the blocks into one of a plurality of color groups based on the RGB block statistical values, and a variance value of the number of blocks classified into each color group, that is, block distribution information Is calculated as the bias of the block statistical value. Therefore, the imaging apparatus 10 can calculate the reliability more accurately.

  Furthermore, since the imaging apparatus 10 sets the convergence time of the second white balance correction coefficient to be short (for example, 1 to 2 seconds), when the reliability increases due to a change in the imaging environment, the white balance is set to the imaging environment. The change can be followed quickly.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Image pick-up device 11 Lens unit 12 Image pick-up element 13 AFE circuit 14 Image signal processing part 141 Captured image acquisition part 142 Pre-processing part 143 Block division part 144 Statistical value calculation part 145 Light source estimation part 146 Reliability calculation part 147 Information management part 148 Memory | storage 149 White balance adjustment unit 150 Post-processing unit 16 Exposure control unit 17 Timing generator 18 Driver 19 Image display unit 20 Image recording unit

Claims (7)

A captured image acquisition unit for acquiring a captured image;
A block division unit for dividing the captured image into a plurality of blocks;
A statistical value calculation unit that calculates a block statistical value by statistically calculating a pixel value for each block;
A light source estimation unit that estimates a light source that illuminates a subject of the captured image based on the block statistics, and generates light source estimation information related to an estimation result;
A reliability calculation unit that calculates the reliability of the light source estimation information based on the bias of the block statistical value;
An information processing apparatus comprising: a white balance adjustment unit that adjusts a white balance of the captured image based on the light source estimation information and a reliability of the light source estimation information. An information management unit that stores in the storage unit reliability correspondence information in which the light source estimation information and the reliability of the light source estimation information are associated;
The white balance adjustment unit adjusts the white balance of the captured image based on the reliability correspondence information having the highest reliability of the light source estimation information among the reliability correspondence information stored in the storage unit. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized.   When the number of the reliability correspondence information stored in the storage unit exceeds a predetermined number of frames, the information management unit selects the oldest reliability correspondence information stored in the storage unit. The information processing apparatus according to claim 2, wherein the information processing apparatus is deleted from the storage unit.   The information processing unit according to claim 2 or 3, wherein the information management unit deletes the reliability correspondence information stored in the storage unit when the luminance of the captured image exceeds a predetermined range. apparatus.   The reliability calculation unit calculates the reliability based on a bias of the block statistic and at least one of a subject brightness of the captured image and a color temperature of the light source. Item 5. The information processing device according to any one of Items 1 to 4.   The reliability calculation unit classifies the block into one of a plurality of color groups based on the block statistical value, and calculates a variance value or standard deviation of the number of blocks classified into each color group. The information processing apparatus according to claim 1, wherein the information processing apparatus calculates the bias of the block statistical value. The white balance adjustment unit calculates a first white balance correction coefficient based on the light source estimation information and the reliability of the light source estimation information, and filters the first white balance correction coefficient. , Calculating a second white balance correction coefficient that follows the first white balance correction coefficient with a predetermined convergence time and adjusting the white balance of the captured image based on the second white balance correction coefficient And
The information processing apparatus according to claim 1, wherein the convergence time is set short.

Images