JP4860512B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus capable of adjusting white balance, and more particularly, to a technique for white balance adjustment suitable for underwater photography and a technique for restoring disappearance of colors peculiar to underwater.

  With the spread of digital cameras and small home video cameras in recent years, an increasing number of people bring their cameras underwater and enjoy underwater shooting during snorkeling and scuba diving.

  However, when photographing underwater, an image with an unnatural hue with strong bluishness can be obtained as it is. The reason for this is that the natural temperature falling into the water has a higher color temperature depending on the water depth.

  For example, even at a water depth of about 5 m, the natural light that has passed through the water reaches a color temperature of 10,000 K (Kelvin). Further, when the distance between the subject and the camera is long, the distance passing through the water is also long, so that the photographed image becomes more bluish.

  Such a captured image with a strong bluish color becomes a problem not only when shooting underwater, but also when performing snorkeling for shooting underwater from near the water surface. For example, when shooting a fish or a carp with a depth of about 3 m from the vicinity of the water surface, the sunlight passes through the water of 3 m and is reflected by the subject, and further passes through the water of 3 m and enters the camera. This is because it will pass through 6m of water.

  On the other hand, since humans have high adaptability to light and become accustomed to eyes, a gap in hue between a captured image and visual observation becomes large.

  Since digital cameras and video cameras that are currently on the market do not assume shooting under special light sources such as underwater, auto white balance does not function properly during underwater shooting. For this reason, a photographer attempts to obtain a normal white balance by shooting with an orange or magenta optical filter attached to the lens. However, the filter is not easy to exchange in water and cannot handle underwater light that changes every moment depending on the water depth.

Some high-end cameras can manually adjust the white balance (see Patent Document 1). If such a camera is used, normal white balance can be obtained although it takes time for adjustment. Alternatively, even when the white balance cannot be adjusted as a function on the camera side, it is possible to adjust the white balance using photo retouching software, RAW development software, video editing apparatus (software), or the like after shooting.
JP 2006-157316 A

  By adjusting the white balance, an image of an achromatic subject can be adjusted to an achromatic color, but normal color reproduction cannot be expected with this alone. If so, the spectrum of light falling into the water is very different from that of the ground, and the color rendering properties are extremely low. This is a phenomenon caused by a decrease in long-wavelength light and is understood by divers as “disappearance of color in water”. That is, even when a normal white balance is obtained, the color becomes light (saturation decreases), the red color becomes dark (lightness decreases), and the hue changes.

  For this reason, some photographers bring light sources such as video lights and strobes into the water, but there are many disadvantages to bringing light sources into the water.

  First of all, the size and weight of the photographic equipment may increase, which may limit or become inconvenient in water. In addition, a shadow appears strongly because it emits light with strong directionality. Furthermore, when shooting with a wide angle of view, a wide-angle lens is usually used. In such a case, the light from the light source is radiated over a wide range and the subject distance is set to be long. The distance to the time becomes longer, and the color change still occurs. More specifically, the difference in luminance between a place close to the light source and a place far away from the light source increases, making it difficult to obtain appropriate exposure over a wide range. In addition, when a strobe is used as a light source, it takes a long time to charge, so there is a problem that the photographing interval becomes long.

  For the above reasons, underwater photography with a light source is actually limited to macro photography with a short photography distance.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus capable of realizing a highly accurate color restoration process during underwater shooting with a simple technique.

  According to an aspect of the present invention, an underwater optical path length calculation unit that calculates an underwater optical path length of subject light, a color restoration information generation unit that generates color restoration information based on the calculated underwater optical path length, and the color restoration There is provided an image processing apparatus comprising: a pixel value correcting unit that corrects an input pixel value based on information and a white balance gain.

  Further, according to one aspect of the present invention, an underwater optical path length calculating unit that calculates an underwater optical path length of subject light, a color recovery matrix generating unit that generates a color recovery matrix based on the calculated underwater optical path length, There is provided an image processing apparatus comprising matrix multiplication means for correcting an input pixel value based on a color restoration matrix and a white balance gain.

  According to another aspect of the invention, a white balance setting unit that sets a white balance gain around a subject during underwater shooting, and a color restoration matrix generation that generates a color restoration matrix based on the set white balance gain. There is provided an image processing apparatus comprising: means; and matrix multiplication means for correcting an input pixel value based on the color restoration matrix and the white balance gain.

  According to the present invention, highly accurate color restoration processing can be performed by a simple method during underwater photography.

  An embodiment of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention. This image processing apparatus is built in, for example, a digital camera.

  1 includes an image sensor 1, an A / D converter 2, an OB (Optical Black) subtractor 3, a demosaic processing unit 4, a white balance gain multiplier 5, and an underwater optical path length calculation unit. 6, a color restoration matrix generation unit 7, a color restoration matrix multiplier 8, a land / land switch 9, a matrix multiplier 10, a gamma processing unit 11, and a video recording unit 12.

  An image signal captured by the image sensor 1 is converted into a digital pixel value by the A / D converter 2. The demosaic processing unit 4 performs a process of giving all color information to each pixel. The reason why the demosaic processing is necessary is that the pixel array of the general image sensor 1 is a Bayer array in which more G pixels are arranged than other color pixels. In the demosaic processing unit 4, color interpolation is performed. Processing is performed so that each pixel has RGB color information.

  The white balance gain multiplier 5 performs white balance adjustment on the pixel value after the demosaic process using the white balance gain set by a predetermined method. A method for setting the white balance gain will be described later.

  The underwater optical path length calculation unit 6 calculates an optical path length (hereinafter referred to as an underwater optical path length) until the light that has poured onto the water surface reaches the image sensor 1. This calculation method will also be described later.

  Based on the underwater optical path length calculated by the underwater optical path length calculation unit 6, the color recovery matrix generation unit 7 generates a color recovery matrix for recovering colors lost in water.

  The color restoration matrix multiplier 8 performs processing for multiplying the result obtained by the white balance gain multiplier 5 by the color restoration matrix.

  The land / land switch 9 switches between underwater and land. The matrix multiplier 10 performs a process of multiplying the multiplication result of the color restoration matrix multiplier 8 by the color space conversion matrix.

  The gamma processing unit 11 performs gamma correction on the multiplication result of the matrix multiplier 10. The pixel value after the gamma correction is recorded in the video recording unit 12 in a format such as JPEG.

  The underwater optical path length calculation unit 6, the color restoration matrix generation unit 7, the color restoration matrix multiplier 8, and the underwater switch 9 that are indicated by broken lines in FIG. 1 are the main parts of this embodiment. That is, this embodiment can be realized by adding the configuration of the broken line portion in FIG. 1 to the configuration of the conventional digital camera.

  FIG. 2 is a flowchart showing in detail a processing procedure from obtaining the matrix multiplication result from the pixel value after multiplication of the white balance gain, which is the main part of FIG. First, a white balance gain is set (step S1). Here, the white balance gain indicates the gains of R and B with respect to the RGB values of the image sensor 1 when the image sensor 1 handles RGB primary colors.

  There are several ways to set the white balance gain. Hereinafter, first to fifth methods for setting the white balance will be described. Note that white balance may be set by a method other than the following.

  In the first method, white balance is set by an image obtained by photographing an achromatic gray color sample in water. More specifically, a gray color sample is shot in advance near the target shooting location in water, and the white balance is adjusted by the software so that the image of the color sample becomes gray. The obtained white balance gain is also applied to other images. Thereby, a fairly accurate white balance can be obtained.

  However, it is very troublesome for the photographer to bring a gray color sample and take a picture in the water where the behavior is restricted. Therefore, for example, as shown in FIG. 3, a gray sticker 15 is attached to the diving fin 14, and the sticker 15 is also photographed when the original photographing is performed.

  Underwater, the buoyancy works and the body can move relatively freely, so the fin seal 15 can be photographed without difficulty, and the photographer does not have to carry a gray color sample.

  As a further method for saving the photographer, a barcode pattern, for example, is pasted on the side of the seal 15 as shown in FIG. 4, and the barcode 15a pattern is also photographed when the seal 15 is photographed. The white balance gain is automatically set by detecting the position of the barcode 15a from the video imaged inside the camera, and calculating the gain for balancing the seal 15 at the center in gray, before and after the setting. It is conceivable that the set white balance gain is applied to other images taken in (1). This eliminates the need for the user to adjust the white balance later, which saves time and improves usability. Of course, in addition to being incorporated in the camera, in the image processing software performed after shooting, similarly, it is possible to detect the position of the barcode 15a and calculate the white balance gain that balances the sticker 15 in gray. Can be raised.

  The seal 15 may include various color patterns as shown in FIG. 4 in addition to the gray single color as shown in FIG.

  The second method is to attach an ambient light sensor capable of measuring white balance to the camera. In this method, the ambient light sensor directly measures the underwater light around the camera and sets the white balance gain. For this reason, the white balance cannot be measured with the optical path length between the subject and the camera taken into account. Therefore, for example, by detecting the distance to the subject using a camera and adjusting the white balance based on the detected distance, the accuracy of white balance can be improved.

  The third method is so-called auto balance. This method is effective both when the image processing apparatus according to the present embodiment is applied to a camera and when it is applied to image processing software performed after photographing.

  If the invention previously filed by the present inventor (Japanese Patent Laid-Open No. 2007-13415) is used, the color of the light source that falls on the object can be identified fairly accurately. In fact, it has obtained good results in many underwater images and is one of the most promising methods.

  However, when both land and underwater images can be processed, it is desirable to switch between underwater and land before image processing in order to prevent erroneous determination. On land, blue light sources such as underwater are unlikely at all, and even if they exist, it may be desirable to reproduce blue in order to leave the atmosphere in place without balancing blue to white. This is because it is necessary to change the balance calculation method and the limit for limiting the detected white balance gain value according to human chromatic adaptation.

  When switching between land and water is provided in the camera, a changeover switch may be provided, or switching may be performed on a menu screen. When a camera provided with a changeover switch is housed in an underwater housing for underwater photography, the camera is provided with an underwater changeover switch 16 as shown in FIGS. It is desirable to provide a protrusion 17 for pressing the changeover switch. FIG. 5 shows a state in which the underwater housing is expanded, and the protrusion 17 protrudes from the inside of the housing. FIG. 6 shows a state in which the camera is housed in the underwater housing and closed. Pressed by the projection 17 of the housing, the underwater switch 16 of the camera is pressed.

  In the fourth method, the user sets the white balance. In this case, the user sets the white balance through a camera switch or menu screen, or through a software user interface. Many cameras allow you to set the white balance by specifying the type of light source (bulb, fluorescent light, clear, cloudy, shade, etc.) or by specifying the color temperature. Useless. Underwater, the chromaticity coordinates deviate significantly from blackbody radiation (and daylight), and the chromaticity coordinates are distributed over a wide range depending on the underwater path length. There is a high possibility that adjustments cannot be made.

  Therefore, it is convenient to allow the user to set the white balance by giving the underwater path length as a parameter. If the underwater optical path length is specified following the specification of the color temperature installed in many cameras and RAW development processing software, the white balance is set for the chromaticity coordinates of the transmitted light in water determined by the optical path length. Design becomes possible.

  In the fifth method, a depth meter is attached to a camera or an underwater housing that houses the camera, and wiring for transmitting the depth measured by the depth meter to the camera is provided. In the camera, the white balance is set with a value obtained by multiplying the water depth by, for example, 1 to 1.5 as an underwater optical path length. This is because, on a clear day near the equator where the sun is high and the sunlight falls at a right angle to the water surface, the water depth and the underwater light path length are almost the same, but in cloudy days and areas where the sun altitude is low, This is because the underwater optical path length tends to be larger than the actual water depth because the light falls obliquely. Around the northern Kanto region where sports divers can dive in winter, the actual water depth and underwater light path length may differ by a factor of 1.5. For this reason, it is necessary to multiply 1 to 1.5 times. The magnification may be set from the camera menu, or the latitude may be input, the season may be calculated from the clock inside the camera, and the result may be automatically set.

  This fifth method may not be practical because it cannot set an accurate white balance, and the mechanism is complicated and costly. However, it is possible to record the water depth data as ancillary information of the video, and there is an advantage that the water depth data can be utilized afterwards. For example, in the EXIF information that records the attached information of still images, the tag that records the water depth is not defined by the standard at the present time, but the maker note tag (MakerNote Tag) that the manufacturer records unique information If the data is recorded, it is very convenient because it is possible to accurately specify how many meters the depth of each image was taken after the fact. Therefore, in the future, it is possible to change the standard so that water depth data can be recorded as part of EXIF information.

  When the white balance gain is set by any one of the first to fifth methods described above, the land-and-land switch 9 next selects whether to shoot underwater or on land (FIG. 2). Step S2). As described above, the land / land switch 9 may be realized by a dedicated switch provided in the camera, for example, or may be realized by software selection on the menu screen of the camera.

  If underwater shooting is selected in step S2, the underwater optical path length calculation unit 6 calculates the underwater optical path length (step S3). Below, the processing content of the underwater optical path length calculation part 6 is explained in full detail.

  Here, it is assumed that the camera has an image sensor 1 of three colors of RGB. To show the color characteristics of the image sensor, it is converted from CIE1931XYZ, which is the standard color expression method of CIE (International Commission on Illumination), into three color components that are imaged by the camera (which are usually three colors of RGB) Let C be the color matrix for this purpose. In this case, C is a 3 × 3 matrix and is determined by the characteristics of the camera color filter, image sensor 1, low-pass filter, and the like.

  Note that CIE 1931XYZ is a color expression method that can represent all the existing colors using three values of XYZ instead of RGB.

  Here, if a value when a certain light source color is represented by CIE1931XYZ is (WP_X, WP_Y, WP_Z), C * (WP_X, WP_Y, WP_Z) on the right side of the following equation (1) is linear in the camera. RAW value (value after OB subtraction processing).

The white balance gains WBR and WBB are set so that the light source color is gray. The R component of C * (WP_X, WP_Y, WP_Z) is WBR, the G component is 1, and the B component is WBB. When weighted, it is adjusted to gray. This weighting is expressed as a 3 × 3 matrix on the right side of equation (1).

  The adjustment to the gray color also matches the linear RAW value when the CIE D65 is photographed, and the right side of the equation (1) matches the XYZ value of the C * D65 standard light source on the left side. become.

  By the way, D65 is one of the standard light sources defined by the CIE, and is an average value of light falling on the ground. D65 is currently most widely used as a white point, and sRGB and Adobe (registered trademark) RGB, which are standard color spaces of digital cameras, also define the white point as D65. Other light sources may be used as the standard light source. Ideally, it is desirable to use as a standard light source daylight calculated from a complete radiator having a predetermined color temperature or an average value of light falling on the ground.

  When R = G = B in sRGB or Adobe RGB, for example, when RGB = (118, 118, 118), this value is achromatic, and is displayed as D65 on a normally calibrated monitor.

  (WP_X, WP_Y, WP_Z) can be calculated by the above-described equation (1). However, the three-axis notation of XYZ includes the light intensity. The intensity of light is independent of the color, and when only the color is described, the CIE 1931 xy coordinate is expressed by the color on the plane (equal energy plane) where X + Y + Z = 1.

  Here, not the intensity of the light source but only its color (chromaticity coordinates) is a problem. Therefore, the chromaticity coordinates in which the light source color is expressed by two axes (xy) of the equal energy plane instead of the three axes of XYZ. wp_xy = (WP_X / (WP_X + WP_Y + WP_Z), WP_Y / (WP_X + WP_Y + WP_Z)) is calculated. This chromaticity coordinate wp_xy corresponds to the light source color.

  When the light source color is calculated by the above procedure, the underwater optical path length is calculated by the following procedure. FIG. 7 is a diagram showing CIE1931xy color coordinates of transmitted light in water. An inverted U-shaped curve cb1 in FIG. 7 is a color coordinate curve of monochromatic light, and an actual color exists inside the inverted U-shaped curve. A curve cb2 in FIG. 7 is a color coordinate curve of black body radiation.

  This curve cb2 plots the light emitted when the temperature of the object is raised, with the temperature on the left side of FIG. 7 being high and the temperature on the right side being low. In FIG. 7, it is a color coordinate when the place displayed as 5000K is 5000 degrees. Also, the daylight that falls on the ground is almost similar to this blackbody radiation, and the color coordinates of the light that falls on the ground at noon on a sunny day are about 5000K.

  A curve cb3 in FIG. 7 shows how the color coordinates of the underwater light change depending on the underwater optical path length when 5000 K of light is poured on the water surface. That is, the curve cb3 is a color coordinate curve for each underwater optical path length.

  The light source color wp_xy calculated by the above equation (1) is plotted in FIG. 7, and a perpendicular line is drawn from the plotted position toward the curve cb3, and the underwater optical path length can be obtained from the position where the curve cb3 intersects. .

  The inventor has found that the underwater optical path length can be calculated by the above-described method through various experiments.

  When the underwater optical path length is calculated in step S3 of FIG. 2 by the above method, the color restoration matrix generation unit 7 generates a color restoration matrix (step S4). Here, a table is prepared in advance so that a color restoration matrix corresponding to the underwater optical path length can be selected.

  FIG. 8 shows an example of the table. The table of FIG. 8 stores a table number, an underwater optical path length, and a 3 × 3 matrix in association with each other. In this table, for example, a reference chart is brought into water, photographing is performed at each water depth to obtain an image sample, and color matching is performed to generate a 3 × 3 matrix. Further, the underwater optical path length is calculated using the above-described equation (1) based on the white balance corresponding to each image, and the table of FIG. 8 is generated in advance based on the obtained result.

  In step S4 of FIG. 2, a 3 × 3 matrix corresponding to the underwater optical path length calculated in step S3 is extracted from the table of FIG. 8, and the extracted one is used as a color restoration matrix.

  In the table of FIG. 8, the underwater optical path length is registered only in increments of 100 cm, but a 3 × 3 matrix may be prepared in finer units, or calculated in step S3 of FIG. When the underwater optical path length is an intermediate value of the underwater optical path length registered in the table of FIG. 8, a new 3 × 3 matrix may be generated by performing interpolation processing such as linear interpolation.

  In the above description, an example in which the color restoration matrix is selected using the calculated underwater optical path length as a parameter has been described. However, the processing of the underwater optical path length calculation unit 6 and the color restoration matrix generation unit 7 is combined into one, The color restoration matrix may be generated without calculating. In this case, a table associating the white balance gain with the color restoration matrix is prepared, and the corresponding color restoration matrix is extracted using the white balance gain as a parameter.

  FIG. 9 is a diagram showing an example of this type of table. In the table of FIG. 9, white balance gains GainR and GainB and a 3 × 3 matrix are registered in association with each other. This table is created, for example, by bringing a reference chart into the water, taking images at each water depth to obtain image samples, and matching the 3 × 3 matrix resulting from color matching and the white balance gain at that time Is done.

  In order to obtain a final color restoration matrix using the table of FIG. 9, for example, the processing of FIG. 10 is performed. First, each element Matrix [j] (j = 0 to 9) of the 3 × 3 color restoration matrix is initialized to zero (step S11). Next, a variable i representing the type of matrix is initialized to zero (step S12). If i = 0, the white balance gain and matrix at the top level in FIG. 9 are selected, and if i = 3, the white balance gain and matrix at the bottom level are selected.

  Next, the difference dR, dB between the reciprocal of the white balance gains WBR, WBB and the reciprocal of the product of the 3 × 3 matrix value and GainR or GainB is calculated (step S13).

  Next, it is determined whether or not the difference dR or dB is zero (step S14). If it is not zero, the weight Fact [i] is calculated from dR and dB (step S15). On the other hand, if the determination in step S14 is NO, it indicates that WBR and WBB match the gain values recorded in the table, so that the weight Fact [i] is a very large fixed value (1.0E + 99). ) (Step S16).

  When the processing of steps S15 and S16 is completed, each element Matrix [j] = Matrix [j] + Fact [i] * Table [i]. Matrix [j] is calculated (step S17). Next, the total weight value FactSum = FactSum + Fact [i] is calculated (step S18).

  Next, it is determined whether or not processing has been performed up to the last stage of the table of FIG. 9 (step S19). If there is still a matrix to be processed, the variable i is incremented (step S20), and step The processes after S13 are repeated.

  When all the rows have been processed, a color restoration matrix having a value obtained by weighting and averaging each element of the 3 × 3 matrix with the weight Fact [i] is generated (step S21).

  When the color restoration matrix is generated using the table of FIG. 8 or 9 in the above-described processing procedure, the color restoration matrix multiplier 8 generates the multiplication result of the white balance gain multiplier 5 and the generated result. A process of multiplying by the color restoration matrix is performed (step S5 in FIG. 2).

  The processes in steps S3 to S5 in FIG. 2 are performed only when underwater shooting is performed. When the processing in step S5 is completed, matrix multiplication processing by the matrix multiplier 10 is performed in both cases of underwater shooting and land shooting (step S6). In this process, a pixel value is calculated by performing a multiplication process using a color space conversion matrix for converting RGB values captured by the image sensor 1 into a color space such as sRGB.

For example, the pixel value obtained in step S6 when performing underwater photography is expressed by the following equation (2).
Pixel value after matrix multiplication = color space conversion matrix × (color restoration matrix × white balance gain multiplier 5 multiplication result) (2)

On the other hand, the pixel value obtained in step S6 when performing land photography is expressed by the following equation (3).
Pixel value after matrix multiplication = color space conversion matrix × multiplication result of white balance gain multiplier 5 (3)

  The pixel value obtained by the process of step S6 in FIG. 2 is recorded in the video recording unit 12 after being gamma-corrected by the gamma processing unit 11, as shown in FIG.

  As described above, in the first embodiment, during underwater shooting, a color restoration matrix corresponding to the underwater optical path length or the white balance gain is generated, and the generated color restoration matrix, the color space conversion matrix, and the white balance gain are used to generate pixels. Since the value is calculated, a natural image with less bluish color can be obtained as on land.

  In this embodiment, if the water depth and the underwater optical path length are known, accurate color restoration processing can be performed with relatively simple processing. Therefore, it can be easily incorporated into a camera, and can also be easily realized with image processing software after shooting. Yes.

  Even if the depth of water is not known, accurate color restoration processing can be performed by acquiring the white balance gain using the result of photographing a color sample underwater.

  Thus, according to the present embodiment, the color restoration process during underwater photography can be automated, and the original color of the underwater subject can be restored without bothering the photographer.

(Second Embodiment)
In the first embodiment, in order to calculate the pixel value at the time of underwater shooting, the arithmetic processing according to the above-described equation (2) is performed. Since the combination rule is established for the matrix operation, the equation (2) is the same as the following equation (4).
Pixel value after matrix multiplication = (color space conversion matrix × color restoration matrix) × multiplication result of white balance gain multiplier 5 (4)

  In this equation (4), the multiplication result of the matrix obtained by the matrix synthesis process and the white balance gain multiplier 5 after the multiplication process of the color space conversion matrix and the color restoration matrix (hereinafter, matrix synthesis process) is performed in advance. Multiply by the pixel value of. In the second embodiment described below, the pixel value is calculated based on the above equation (4).

  FIG. 11 is a block diagram showing a schematic configuration of an image processing apparatus according to the second embodiment of the present invention. In FIG. 11, the same reference numerals are given to the components common to FIG. 1, and the differences will be mainly described below.

  The image processing apparatus of FIG. 11 includes a matrix synthesis unit 21 that is not shown in FIG. The matrix synthesis unit 21 multiplies the color restoration matrix generated by the color restoration matrix generation unit 7 and the color space conversion matrix.

  Further, the matrix multiplier 10 in FIG. 11 multiplies the multiplication result of the matrix synthesizing unit 21 and the pixel value multiplied by the white balance gain multiplier 5 to calculate a final pixel value.

  The underwater optical path length calculation unit 6, the color restoration matrix generation unit 7, the matrix synthesis unit 21, and the underwater switch 9 that are indicated by broken lines in FIG. 11 are the main parts of this embodiment.

  Also in the second embodiment, the calculation process itself is the same as in the first embodiment except that the calculation order is different. Therefore, as in the first embodiment, high-precision color during underwater shooting Restoration processing is possible.

(Third embodiment)
In the first and second embodiments, the white balance gain multiplier 5 and the matrix multiplier 10 are provided separately. However, these multipliers can be combined into one.

When the pixel value is represented by a third-order vector (for example, R value, G value, B value) and the matrix is 3 × 3, the white balance gain multiplier 5 calculates the matrix shown in the following equation (5). be equivalent to.

  Therefore, the white balance gain multiplier 5 and the matrix multiplier 10 can be combined into one and the calculation result can be expressed by one matrix.

In the second embodiment described above, the pixel value after matrix multiplication is calculated based on the equation (4). However, the equation (4) can also be rewritten as the following equation (6).
Pixel value after matrix multiplication = (color space conversion matrix × color restoration matrix) × (white balance matrix × pixel value after demosaic processing) (6)

Since the combination rule is established in the matrix operation, the equation (6) can be rewritten as the equation (7).
Pixel value after matrix multiplication = (color space conversion matrix × color restoration matrix × white balance matrix) × pixel value after demosaic processing (7)

  From the above background, the third embodiment described below is characterized in that the white balance gain multiplier 5 and the matrix multiplier 10 are combined into one and the calculation based on the above equation (7) is performed.

  FIG. 12 is a block diagram showing a schematic configuration of an image processing apparatus according to the third embodiment of the present invention. In FIG. 12, the same reference numerals are given to the components common to those in FIG. 1 and FIG. 11, and the differences will be mainly described below.

  The image processing apparatus in FIG. 12 includes a white balance matrix synthesis unit 22, a first matrix synthesis unit 23, and a second matrix synthesis unit 24 as configurations not shown in FIGS. Further, although the white balance gain multiplier 5 exists in FIGS. 1 and 11, it is omitted in FIG.

  The underwater optical path length calculation unit 6, the color restoration matrix generation unit 7, the white balance matrix generation unit 22, the first matrix synthesis unit 23, the underwater switch 9 and the second matrix synthesis unit 24 shown by broken lines in FIG. It is the main part of the form.

  The white balance matrix generation unit 22 generates a 3 × 3 matrix including white balance gain.

  The first matrix synthesis unit 23 performs multiplication of the color restoration matrix and the white balance matrix. The second matrix synthesis unit 24 multiplies the matrix that is the calculation result of the first matrix synthesis unit 23 and the color space conversion matrix. As a result, the calculation result in parentheses in the above equation (7) is output from the second matrix synthesis unit 24.

  Next, the matrix multiplier 10 multiplies the pixel value after the demosaic process by the matrix that is the output result of the second matrix synthesis unit 24, and the calculation of the equation (7) is completed.

  Also in the third embodiment, the calculation process itself is the same as in the first and second embodiments except that the calculation order of the matrix is different. Therefore, as in the first and second embodiments, High-accuracy color restoration processing during underwater photography is possible.

(Fourth embodiment)
In the first to third embodiments, the image processing apparatus that can be incorporated in the camera has been described. However, in the fourth embodiment described below, development processing of RAW data output from the image sensor 1 is performed. It is an image processing apparatus that can be incorporated into RAW development software to be performed.

  FIG. 13 is a block diagram showing a schematic configuration of an image processing apparatus according to the fourth embodiment of the present invention. The basic configuration of the image processing apparatus of FIG. 13 is the same as that of the image processing apparatus of FIG. 1, but the image sensor 1 and the A / D converter 2 are omitted because they are incorporated into the RAW development software in the form of software. . Instead, an image input unit 31 for capturing RAW data is provided.

  Instead of FIG. 13, the basic configuration may be the same as that of FIG. 11 or FIG. 12, and in that case, the configuration is shown in the block diagram of FIG. 14 or FIG.

  On the other hand, if the input data is not RAW data but data that has already undergone color space conversion, such as JPEG, the block configuration is as shown in FIG. The image processing apparatus of FIG. 16 has an inverse gamma processing unit 32 as a configuration not shown in FIG. 13, and the OB subtraction unit 3, demosaic processing unit 4 and matrix multiplier 10 of FIG. 13 are omitted.

  The inverse gamma processing unit 32 performs non-linear conversion between the gamma characteristic and the inverse characteristic of the image data recorded in, for example, JPEG format, and performs a process in which the light amount and the pixel value are in a proportional relationship. That is, the inverse gamma processing unit 32 performs a process reverse to that of the gamma processing unit 11.

In terms of an expression, the inverse gamma processing unit 32 performs, for example, the following expression (8), and the gamma processing unit 11 performs the expression (9).
Output = power (input / 255, 2.2) × 255 (8)
Output = power (input / 255, 1 / 2.2) × 255 (9)

  As described above, the present invention can be incorporated in software for performing RAW development and image processing, and can perform white balance adjustment with high accuracy simply using a PC after underwater photography.

  In the above-described first to third embodiments, examples of incorporation into the camera have been described. However, at least a part of the configuration illustrated in FIGS. 1, 11, and 12 may be configured by hardware or software. It may be configured. Further, the configuration other than that indicated by the broken line in FIG. 1 and the like is not an essential part of the present invention, and the configuration may be variously changed.

  In each of the above-described embodiments, the example using the primary color image sensor 1 has been described. However, the present invention can also be applied to the case where a complementary color image sensor is used.

  In the first to fourth embodiments described above, the example in which the matrix calculation is performed using the color restoration matrix multiplier 8 and the matrix multiplier 10 to perform the underwater color restoration processing has been described. Underwater color restoration processing may be performed using the conversion information. If the underwater optical path length is included as a parameter of the equation for obtaining such color conversion information, it is not necessary to separately perform an arithmetic process with the underwater optical path length, and the color restoration process can be simplified.

  In this case, a color restoration information generation unit that generates color restoration information based on the underwater optical path length is provided instead of the color restoration matrix generation unit 7, and the color restoration information and the white balance gain are provided instead of the matrix multiplier 10. Based on the above, a pixel value correction unit for correcting the input pixel value is provided.

  As an example of color conversion information, it is conceivable to use color mapping that associates colors with each other. In such color mapping, a color conversion map is provided for each underwater optical path length, or when the underwater optical path length is directly included in the color conversion formula, only one type of color conversion map is provided. The memory capacity can be reduced.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. The flowchart which shows in detail the process sequence from the multiplication of the white balance gain which is the principal part of FIG. 1 until the multiplication result of a matrix is obtained. The external view which shows an example of the fin which stuck the gray seal | sticker. The top view which shows an example of the seal | sticker which has arrange | positioned barcode around a gray pattern. The figure which shows the state which expand | deployed the underwater housing. The figure which shows the state which accommodated the camera in the underwater housing and closed. The figure which shows the CIE1931xy color coordinate of the transmitted light in water. The figure which shows an example of a table. The figure which shows an example of another table. 10 is a flowchart illustrating an example of a processing procedure for generating a color restoration matrix using the table of FIG. 9. The block diagram which shows schematic structure of the image processing apparatus by the 2nd Embodiment of this invention. The block diagram which shows schematic structure of the image processing apparatus which concerns on the 3rd Embodiment of this invention. The block diagram which shows schematic structure of the image processing apparatus by the 4th Embodiment of this invention. The block diagram which shows the modification of FIG. The block diagram which shows the modification of FIG. The block diagram which shows schematic structure of the image processing apparatus in the case of inputting the data after color space conversion.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image sensor 2 A / D converter 3 OB subtractor 4 Demosaic processing part 5 White balance gain multiplier 6 Underwater optical path length calculation part 7 Color restoration matrix production | generation part 8 Color restoration matrix multiplier 9 Aqua land switch 10 Matrix multiplier 11 Gamma processing unit 12 Video recording unit 21 Matrix synthesis unit 22 White balance matrix generation unit 23 First matrix synthesis unit 24 Second matrix synthesis unit 31 Image input unit 32 Inverse gamma processing unit

Claims (11)

An underwater optical path length calculating means for calculating an underwater optical path length of subject light;
Color restoration information generating means for generating color restoration information based on the calculated underwater optical path length;
Pixel value correcting means for correcting an input pixel value based on the color restoration information and the white balance gain;
A white balance setting means for setting a white balance gain around the subject when shooting underwater;
Light source color calculation means for calculating the light source color of the subject based on the set white balance gain,
The underwater optical path length calculating unit calculates an underwater optical path length of subject light based on the light source color calculated by the light source color calculating unit.   The image processing apparatus according to claim 1, wherein the color restoration information generation unit generates the color restoration information using color mapping that associates colors with each other. An underwater optical path length calculating means for calculating an underwater optical path length of subject light;
Color restoration matrix generating means for generating a color restoration matrix based on the calculated underwater optical path length;
Matrix multiplication means for correcting input pixel values based on the color restoration matrix and white balance gain;
A white balance setting means for setting a white balance gain around the subject when shooting underwater;
Light source color calculation means for calculating the light source color of the subject based on the set white balance gain,
The underwater optical path length calculating unit calculates an underwater optical path length of subject light based on the light source color calculated by the light source color calculating unit. The underwater optical path length calculation means is a correspondence between the underwater optical path length and the color coordinates when light from daylight calculated from an average value of a perfect radiator having a predetermined color temperature or light falling on the ground falls on the water surface. It refers to data indicating a relationship between an image according to any one of claims 1 to 3, characterized in that to calculate the water path length of the object light using a light source color that is calculated by the light source color calculation means Processing equipment. The light source color calculation means, based on the condition in which the light source colors when given the white balance gain is achromatic image according to any one of claims 1 to 4, characterized in that to calculate the light source color Processing equipment. The white balance setting means, according to claim 1, characterized in that setting the white balance gain so that the color of an image obtained by imaging by the imaging device an object of achromatic in water is the original achromatic 6. The image processing device according to any one of 5 to 5 . The white balance setting means any of claims 1 to 5, characterized in that setting the white balance gain based on the detected information white balance attached to the imaging apparatus in the measurable ambient light sensor An image processing apparatus according to claim 1. Equipped with a land-and-land selection means for selecting whether the imaging by the imaging device was performed underwater or on land,
The white balance setting unit performs image processing on a pixel value captured and input by the imaging device according to a selection result of the land / land selection unit, and the white balance gain corresponding to underwater shooting, or the image processing apparatus according to any one of claims 1 to 5, characterized in that setting the white balance gain corresponding to the land photography. Underwater optical path length information input means for inputting information on the underwater optical path length or water depth,
The white balance setting means, based on the information input, the image processing apparatus according to any one of claims 1 to 5, characterized in that setting the white balance gains. The input pixel value is RAW data;
The matrix multiplication unit calculates a pixel value corresponding to the RAW data by multiplying a pixel value corresponding to the RAW data, the color restoration matrix, the white balance gain, and a color space conversion matrix. An image processing apparatus according to any one of claims 1 to 9 . The input pixel value is obtained by gamma-converting image data having a predetermined image file format so as to be proportional to the input light amount,
The matrix multiplying unit calculates a pixel value corresponding to the image data by multiplying a pixel value corresponding to the image data, the color restoration matrix, the white balance gain, and a color space conversion matrix. An image processing apparatus according to any one of claims 1 to 9 .

Images