JP6704611B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to a photographing device and a photographing method for performing image processing on image data.

The image pickup device is equipped with an image pickup device such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Dark current noise having a fixed pattern of stripes or spots is superimposed on the captured image data generated by the image capturing device. In order to remove this dark current noise, a photographing device that performs dark correction processing of photographed image data is known. The dark correction process is a process of subtracting image data (dark image data) imaged with the image sensor shielded from the image data generated by the main exposure process. As a result, dark current noise is removed from the captured image data.

Patent Document 1 discloses a configuration in which dark current noise is removed from a pixel signal output from a focus detection pixel in an image sensor. Specifically, in Patent Document 1, the pixel signal (dark output) output from the shaded focus detection pixel is stored in advance. The dark current noise of the pixel signal is removed by subtracting the dark output from the pixel signal (pixel output) output from the exposed focus detection pixel.

Further, in Patent Document 2, an effective pixel area for generating captured image data and a shielded reference pixel area (optical black pixel area) are provided in an image sensor, and a reference signal output from the reference pixel area is provided. A configuration for removing dark current noise of captured image data by using it is disclosed.

JP-A-63-263881 JP, 2011-50016, A

When subtracting dark image data from captured image data to remove dark current noise, the reference signal level for captured image data and dark image data is set to prevent offset (change in signal level) from occurring in captured image data. Need to be matched. The reference signal level of each image data is calculated using the pixel signal output from the shaded pixel. However, in addition to dark current noise having a fixed pattern, pixel signals usually include random noise that changes with time and has different magnitudes between pixels. Due to this random noise, the reference signal level fluctuates, and there is a problem that offset occurs in the captured image data.

In the imaging device described in Patent Document 1, the pixel output and the dark output are output from the same focus detection pixel at different timings, and have different random noises. Therefore, in the imaging device described in Patent Document 1, the pixel output and the dark output have different reference signal levels, and there is a problem in that an offset occurs in the pixel output.

In addition, in the image capturing apparatus described in Patent Document 2, it is possible to suppress random noise included in the reference signal level of the reference pixel by taking the average value of the pixel signals of the plurality of reference pixels. However, since the number of reference pixels is smaller than the number of effective pixels, the reference signal level may vary even if the average value of the pixel signals of a small number of reference pixels is taken.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image capturing apparatus and an image capturing apparatus suitable for removing noise in captured image data while suppressing an offset generated in the captured image data. It is to provide a method.

According to one embodiment of the present invention, an effective pixel area including a plurality of effective pixels that outputs a signal corresponding to a light flux from a subject and a reference pixel area including a plurality of reference pixels shielded from the light flux are provided. The image pickup device having the image pickup device and the effective pixel generate light image data based on the signal output from the effective pixel area when the effective pixel is exposed to the light beam. In the light-shielded state, the dark image data is generated based on the signal output from the effective pixel area, and the reference signal level during light-shielding is based on the signal output from the area including the effective pixel area. And a captured image correction means for correcting captured image data using the light-shielding reference signal level.

With such a configuration, the reference signal level of dark image data is calculated by using effective pixels having a large number of pixels. Therefore, the fluctuation of the reference signal level due to random noise is suppressed, and the occurrence of offset in the captured image data is suppressed when the dark image data is subtracted from the captured image data.

Further, according to the embodiment of the present invention, the captured image correction unit corrects the captured image data by subtracting dark image data from the captured image data at a predetermined ratio, for example.

Further, according to the embodiment of the present invention, the captured image generation means calculates the exposure reference signal level based on the signal output from the reference pixel area when the effective pixel is in the exposure state, for example.

Further, according to an embodiment of the present invention, the image capturing apparatus further includes, for example, a reference signal correction unit that matches the exposure reference signal level and the light-shielding reference signal level.

Further, according to one embodiment of the present invention, the reference signal correction means, for example, when the effective pixel is in the light-shielded state, calculates the signal level difference between the light-shielded reference signal level and the signal level output from the reference pixel region. The signal level of the captured image data is calculated and corrected based on the signal level difference.

Further, according to an embodiment of the present invention, the dark image generating means, for example, by calculating any one of the average value, the median value, the mode value of the signal output from the area including the effective pixel area, Determines the reference signal level when light is blocked.

Further, according to one embodiment of the present invention, the reference signal correction unit uses the signal output from the effective pixel area to determine the reference for light shielding in the effective pixel area when the plurality of effective pixels are in the light shielding state. An approximate curve representing the distribution of signal levels is calculated, and the signal level difference is calculated using the approximate curve.

Further, according to one embodiment of the present invention, the reference signal correction unit uses the signals output from the effective pixel region and the reference pixel region, for example, when the plurality of effective pixels are in the light-shielding state. An approximate curve that represents the distribution of the signal levels inside the pixel area and the reference pixel area is calculated, and the signal level difference is calculated using the approximate curve.

Further, according to an embodiment of the present invention, the dark image generating means calculates the light-shielding reference signal level by using, for example, the approximate curve calculated by the reference signal correcting means.

Further, according to one embodiment of the present invention, the captured image generating means calculates, for example, an average value, a median value, or a mode value of the signals output from the reference pixel area to thereby obtain the exposure reference value. Determine the signal level.

Further, according to the embodiment of the present invention, the dark image generating means is used to calculate the light-shielding reference signal level in the effective pixel area based on, for example, the photographing condition when the effective pixel is in the exposure state. Change the signal output area.

Further, according to an embodiment of the present invention, the image capturing apparatus further includes, for example, dark image storage means for storing dark image data. In this case, the captured image correction means corrects the captured image data by using the dark image data stored in the dark image storage stage in advance.

According to an embodiment of the present invention, an image capturing method includes an effective pixel region including a plurality of effective pixels that output a signal corresponding to a light flux from a subject, and a reference including a plurality of reference pixels shielded from the light flux. A photographing method executed in a photographing device including an image sensor having a pixel region, wherein photographing is performed based on a signal output from the effective pixel region in an exposure state in which an effective pixel is exposed to a light flux. In the captured image generation step of generating image data, and when the effective pixel is not exposed to the light flux and is in the light-shielded state, dark image data is generated based on the signal output from the effective pixel area, and the effective pixel area is generated. Using the dark image data, the dark image generation means for calculating the reference signal level at the time of light shielding based on the signal output from the region including, and correcting the signal level of the dark image data based on the reference signal level at the time of light shielding. A captured image correction step of correcting captured image data.

According to the present embodiment, a photographing apparatus and a photographing method suitable for removing noise in photographed image data while suppressing an offset generated in photographed image data are provided.

FIG. 1 is a block diagram of an image capturing apparatus according to the first embodiment of the present invention. FIG. 2 is a flowchart of the shooting process according to the first embodiment of the present invention. FIG. 3 is a schematic view of an image pickup surface of the image pickup device according to the first embodiment of the present invention. FIG. 4 is a graph showing row representative values and approximate curves of pixel signals according to the first embodiment of the present invention. FIG. 5 is a graph showing row representative values and approximate curves of pixel signals according to the first embodiment of the present invention. FIG. 6 is a flowchart of the shooting process according to the second embodiment of the present invention.

(First embodiment)
Hereinafter, the image capturing apparatus according to the first embodiment of the present invention will be described with reference to the drawings. In the following, a digital single lens reflex camera will be described as an embodiment of the present invention. Note that the image capturing device is not limited to a digital single-lens reflex camera, and includes, for example, a mirrorless single-lens camera, a compact digital camera, a camcorder, a tablet terminal, a PHS (Personal Handy phone System), a smartphone, a feature phone, a portable game machine, and the like. May be replaced with another type of device having

FIG. 1 is a block diagram showing the configuration of the image capturing apparatus 1 of this embodiment. As shown in FIG. 1, an image processing apparatus is incorporated in the photographing apparatus 1, and a system controller 100, an operation unit 102, a diaphragm/shutter drive circuit 104, a photographing lens 106, a diaphragm 108, a shutter 110, an image sensor 112. An image pickup element drive circuit 114, an image processing engine 116, a memory 118, a card interface 120, an LCD (Liquid Crystal Display) control circuit 122, an LCD 124 and a ROM (Read Only Memory) 126.

The operation unit 102 includes various switches necessary for the user to operate the image capturing apparatus 1, such as a power switch, a release switch, and a shooting setting switch. When the power switch is pressed by the user, power is supplied from a battery (not shown) to various circuits of the image capturing apparatus 1 through power lines. After power is supplied, the system controller 100 accesses the ROM 126, reads a control program, loads the control program into a work area (not shown), and executes the loaded control program to control the entire imaging apparatus 1.

When the release switch is operated, the system controller 100 sets an aperture/aperture so that a proper exposure can be obtained based on a photometric value measured by a TTL (Through The Lens) exposure meter (not shown) built in the photographing apparatus 1. The diaphragm 108 and the shutter 110 are drive-controlled via the shutter drive circuit 104. More specifically, the drive control of the aperture 108 and the shutter 110 is performed based on an AE function designated by a shooting setting switch, such as a program AE (Automatic Exposure), a shutter speed priority AE, and an aperture priority AE. The system controller 100 also performs AF (Autofocus) control in addition to AE control. An active method, a phase difference detection method, a contrast detection method, or the like is applied to the AF control. Since the configuration and control of this type of AE and AF are well known, detailed description thereof is omitted here.

The light flux from the subject passes through the taking lens 106, the diaphragm 108, and the shutter 110, and is received by the image sensor 112. The image sensor 112 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. Hereinafter, the image sensor 112 will be described as a CMOS image sensor.

The image sensor drive circuit 114 subjects the pixel signal input from the image sensor 112 to predetermined signal processing of amplification processing, and outputs the signal to the image processing engine 116. The image processing engine 116 performs clamp correction processing, demosaicing processing, gradation conversion processing, edge enhancement processing, matrix calculation processing, Y/C separation processing, white balance adjustment on pixel signals input from the image sensor drive circuit 114. Photographed image data (luminance signal Y, color difference signals Cb, Cr) is generated by performing signal processing such as processing, and compressed in a predetermined format such as JPEG (Joint Photographic Experts Group). The memory 118 is used as a storage location of processed data when the image processing engine 116 executes signal processing.

The memory card 200 is removably inserted into the card slot of the card interface 120. The image processing engine 116 can communicate with the memory card 200 via the card interface 120. The image processing engine 116 stores the compressed photographed image data in the memory card 200 (or a built-in memory (not shown) provided in the photographing apparatus 1).

The image processing engine 116 also buffers the generated captured image data in a frame memory (not shown) on a frame-by-frame basis. The image processing engine 116 sweeps out the buffered signal from each frame memory at a predetermined timing, converts it into a video signal of a predetermined format, and outputs it to the LCD control circuit 122. The LCD control circuit 122 controls the modulation of the liquid crystal based on the video signal input from the image processing engine 116. As a result, the captured image of the subject is displayed on the display screen of the LCD 124. The user can visually recognize a real-time through image captured with proper brightness and focus based on the AE control and the AF control through the display screen of the LCD 124.

When the user performs a reproduction operation of the captured image, the image processing engine 116 reads the captured image data designated by the operation from the memory card 200 or the built-in memory and converts it into a video signal of a predetermined format, and the LCD control circuit 122. Output to. The LCD control circuit 122 modulates and controls the liquid crystal based on the video signal input from the image processing engine 116, so that the captured image of the subject is displayed on the display screen of the LCD 124.

Next, of the shooting processing by the image processing engine 116, the dark current noise removal processing of the shot image will be described in detail. FIG. 2 shows a flowchart of processing relating to removal of dark current noise in the shooting processing. This process is started by operating the release switch.

[Processing Step S101 (Main Exposure Processing)]
In the main processing step S101, main exposure processing is performed according to the shooting settings set by the user. The shooting settings include, for example, settings relating to exposure time (shutter speed of the shutter 110), aperture value of the aperture 108, ISO (International Organization for Standardization) sensitivity, and signal processing. The settings relating to the signal processing include, for example, the amplification factor of the amplification processing by the image sensor drive circuit 114, the parameters of the white balance adjustment processing by the image processing engine 116, and the like.

In the main exposure process S101, a process of generating captured image data according to the light flux from the subject and a process of correcting the reference signal level of the captured image data (clamp correction process) are performed. The pixel signal used to generate the captured image data and the pixel signal for calculating the reference signal level of the captured image data (hereinafter, referred to as “exposure reference signal level”) are different from each other in different areas in the image sensor 112. It is output from the pixel.

FIG. 3 is a schematic diagram of the image pickup surface 130 of the image pickup device 112 according to the present embodiment. On the image pickup surface 130, a plurality of pixels are arranged side by side in the vertical direction (vertical direction) of the image pickup element 112 in m rows and in the horizontal direction (horizontal direction) in n columns (m and n are positive). Integer). Further, the imaging surface 130 is provided with an effective pixel area 130A including effective pixels and a reference pixel area 130B including reference pixels. The effective pixel region 130A is provided on the lower right side of the imaging surface 130, and within the effective pixel region 130A, there are a plurality of effective pixels between the p+1th row to the mth row and the q+1th column to the nth column. Are arranged (p and q are positive integers). The reference pixel region 130B is provided in an L shape from the upper side to the left side of the effective pixel region 130A, as shown by hatching in FIG. 3, for example. The pixel signal output from the effective pixel (effective pixel signal) is used to generate captured image data based on the light flux from the subject. The pixel signal output from the reference pixel (reference pixel signal) is used to calculate the exposure reference signal level. The arrangement of the reference pixel area 130B is not limited to the arrangement shown in FIG. For example, the reference pixel area 130B may be provided only on the upper side of the imaging surface 130 (from the first row to the q-th row), and the effective pixel area 130A is formed along the four sides around the imaging surface 130. You may arrange so that it may surround.

The effective pixel is in an exposure state in which a light flux is received from the subject when the shutter 110 is open, and is in a light-shielding state in which the light flux from the subject is shielded by the shutter 110 when the shutter 110 is closed. The effective pixel outputs the charge accumulated in the exposure state as an effective pixel signal. Captured image data is generated based on the effective pixel signal. The captured image data includes a signal and a noise component according to the light flux from the subject. The noise component is, for example, offset noise uniformly generated in the effective pixel region 130A, dark current noise (fixed pattern noise) having a fixed pattern according to the image sensor 112, random noise having different levels between pixels, or the like. is there.

The reference pixel is an optical black pixel that is always shielded from the light flux from the subject regardless of the operation of the shutter 110. The average value of the reference pixel signals output from the plurality of reference pixels during the main exposure process S101 is stored in the storage area in the image capturing apparatus as the exposure reference signal level. The exposure reference signal level may be a mode value or a median value instead of the average value of the reference pixel signals.

When the exposure reference signal level is stored, clamp correction processing is performed on the captured image data using the exposure reference signal level.

[Processing Step S102 (Dark Exposure Processing)]
In this processing step S102, dark exposure processing is performed. The dark exposure processing is executed in the same shooting setting as the main exposure processing S101, except that the shutter is closed and the effective pixels are shielded from light, for example.

In the dark exposure processing, dark image data is generated based on the pixel signal output from the shaded effective pixel. Dark image data is a signal containing only noise components. Further, the reference signal level of dark image data (hereinafter, referred to as “reference signal level during light shielding”) is calculated based on the signals output from the shaded effective pixel and the reference pixel. Using this light-shielding reference signal level, reference signal level correction processing (clamp correction processing) is performed on dark image data.

The light-shielding reference signal level is approximately calculated using the pixel signal of each pixel. Specifically, first, a representative value (row representative value) of pixel signals is calculated for each row of effective pixels and reference pixels. The row representative value is a value in which the pixel signals included in the row are reflected, and is, for example, an average value of pixel signals output from pixels excluding defective pixels and pixels for phase difference detection. The row representative value may be the median value or the mode value of the pixel signals. FIG. 4 is a graph showing the row representative value of each row of the image sensor 112 and its approximate curve. The horizontal axis of FIG. 4 represents the row numbers 1 to m, and the vertical axis represents the row representative value of each row. In FIG. 4, the row representative value is shown by a solid line, and the approximate curve is shown by a broken line. As shown in FIG. 4, the row representative value has variations between rows due to random noise. In order to suppress the influence of this random noise, an approximate curve of row representative values is obtained by fitting. This approximate curve is, for example, a polynomial obtained by the method of least squares. The approximate curve of the row representative value is not limited to the polynomial. The approximated curve may be approximated using a function other than a polynomial, or may be obtained by a moving average.

Further, as shown in FIG. 4, in the row representative value, the signal level increases as the row number increases in the region where the row number is larger than the r-th row. This increase in signal level is called shading, and changes depending on the specifications of the image sensor 112, individual differences, and drive modes. The shading is superimposed on the pixel signal as an offset. Therefore, an area with a small offset due to shading is used for the calculation of the reference signal level during light shielding.

In the example illustrated in FIG. 4, the light-shielding reference signal level is calculated using the pixel signal output from the region (reference region) 130C of the first row to the r-th row of the image sensor 112. Specifically, of the approximate curves of the row representative values shown in FIG. 4, the average value V1 of the first row to the r-th row is stored in the memory 118 as the light-shielding reference signal level. The light-shielding reference signal level may be the median value or the mode value of the pixel signals output from the reference area 130C in the approximate curve of the row representative value.

When the light-shielded reference signal level is stored in the memory 118, the clamp correction process is performed on the dark image data using the light-shielded reference signal level.

Note that shading may occur not only in the vertical direction as shown in FIG. 4 but also in the horizontal direction. In this case, in order to prevent the offset due to shading from being superimposed on the row representative value of the pixel signal, the reference regions 130C shown in FIG. It should not be used to calculate the reference signal level.

Further, the shading in the vertical direction is not limited to the example shown in FIG. For example, depending on the shooting settings of the shooting device 1, shading may occur on both the upper side and the lower side of the pixel area. FIG. 5 is a graph showing the row representative value of each row and the approximate curve thereof when shading occurs on both the upper side and the lower side of the pixel region. The horizontal axis of FIG. 5 represents the row numbers 1 to m, and the vertical axis represents the row representative value of each row. In the example shown in FIG. 5, shading also occurs in the reference pixel area 130B (first row to p-th row). Therefore, in this case, in the effective pixel area 130A, the average value of the row representative values of the reference areas 230C of the sth to rth rows in which shading has not occurred is stored as the light-shielding reference signal level.

As described above, in the present embodiment, the light-shielding reference signal level is calculated using the pixel signals output from the effective pixel area 130A configured by a larger number of pixels than the reference pixel area 130B. By using a large number of pixel signals, it is possible to suppress fluctuation of the reference signal level during light shielding due to random noise. As a result, the clamp correction processing to the proper signal level can be performed on the dark image data.

[S103 (Offset correction process)]
In this processing step S103, processing for matching the light-shielding reference signal level with the exposure reference signal level is performed. For example, as shown in FIG. 5, when shading occurs in the reference pixel area 130B (first row to p-th row), the shading reference signal level of dark image data is the shading in the effective pixel area 130A. It is calculated by using the signal output from the reference area 230C that does not exist. On the other hand, the exposure reference signal level is calculated using the signals output from the reference pixel regions 130B in the first row to the p-th row in which shading is occurring. In this case, in the dark exposure process S102, the clamp correction process is performed on the captured image data based on the exposure reference signal level whose signal level is increased (offset) by shading. Therefore, in this processing step S103, the magnitude of the offset due to shading is detected, and the captured image data is corrected based on this offset magnitude. Thereby, the reference signal level of the captured image data and the reference signal level of the dark image data can be matched.

As the amount of offset due to shading, an approximate curve of row representative values at the time of dark exposure shown in FIGS. 4 and 5 is used. For example, when the shading shown in FIG. 5 occurs in the pixel signal, the light-shielding reference signal level is a region having a flat characteristic and a small signal level (from the s-th row) in the approximate curve of the row representative value. It is the average value V2 of the (rth row). The exposure reference signal level used for calculating the offset is obtained by the average value Vb of the signal levels corresponding to the reference pixel area 130B (first row to p-th row) in the approximate curve of the row representative value. The offset is obtained by the difference ΔV between the light-shielding reference signal level V2 and the exposure reference signal level Vb obtained from the approximate curve.

As described above, in this embodiment, in order to calculate the offset of the reference signal level, the approximate curve of the row representative value obtained from the pixel signal in the dark exposure processing is used. As a result, the influence of random noise on the offset calculation result can be suppressed. Further, the signal level of the captured image data is corrected based on the calculated offset. As a result, the reference signal level of the captured image data and the reference signal level of the dark image data are made uniform so as not to have an offset due to shading.

For example, consider a case where the exposure reference signal level is calculated using only the reference pixel signal output from the reference pixel in the dark exposure processing S102. Since the number of reference pixels is smaller than the number of effective pixels, when the average value of a plurality of reference signals is used as the exposure reference signal level, fluctuations in the signal level due to random noise may not be sufficiently suppressed. When the level of the reference signal fluctuates, the magnitude of the offset due to shading cannot be calculated correctly, so that the offset of the captured image data cannot be properly corrected. If the offset remains in the captured image data, the captured image may change in brightness or tint.

On the other hand, in the present embodiment, the approximate curve of the row representative value obtained in the dark exposure processing S102 is used for calculating the offset between the exposure reference signal level and the light-shielding reference signal level. Since the number of effective pixels is larger than the number of reference pixels, it is possible to suppress the fluctuation of the signal level due to random noise, as compared with the case where the signal level is calculated using only the reference pixel signal.

In this processing step S103, the offset generated in the entire captured image data by shading is corrected, but the shading itself is not corrected. The shading of the shape as shown in FIGS. 4 and 5 is superimposed on the captured image data whose offset is corrected. The shading superimposed on the captured image data is removed in dark image subtraction processing S104 described later.

In addition, when shading as shown in FIG. 4 occurs in the pixel signal, the reference pixel area 130B in the first row to the p-th row (that is, the exposure reference signal level calculated in the main exposure processing S101) No offset due to shading has occurred. Therefore, before the execution of this processing step S103, the offset due to shading between the light-shielding reference signal level and the exposure-time reference signal level can be regarded as zero. In addition, whether or not shading occurs and what shape of shading occurs depends on the driving mode of the image capturing apparatus 1j and the like. Therefore, when the drive mode is set such that shading does not occur in the reference signal level during exposure, execution of this offset correction processing S103 may be omitted.

[Processing Step S104 (Dark Image Data Subtraction Process)]
In this processing step S104, the captured image data whose offset has been corrected is corrected using the dark image data. Specifically, the dark current noise and shading included in the captured image data are removed by subtracting the dark image data from the captured image data.

Dark current noise is caused by unevenness of dark current of pixels in the image sensor 112. The magnitude (amplitude) of the dark current noise changes depending on the imaging conditions such as the integration time of the image sensor 112, temperature, or sensitivity. Further, the dark current noise includes a component in which the pattern (noise appearance distribution in the captured image) changes depending on the shooting conditions and a component in which the pattern does not change. Of these, the component of the dark current noise pattern that does not change is also called fixed pattern noise. The dark current noise pattern includes, for example, striped noise that extends in the vertical direction and the horizontal direction of the imaging surface 130, and spot-shaped noise that discretely occurs on the imaging surface 130. This dark current noise is included in both the captured image and the dark image. Therefore, by subtracting the dark image data from the captured image data, it is possible to remove the component (fixed pattern noise) in which the pattern does not change, from the dark current noise included in the captured image data.

Moreover, the captured image data and the dark image data include shading. This shading is also removed by subtracting the dark image data from the captured image data.

[Processing Step S105 (Image Processing)]
In this processing step S105, predetermined image processing such as gamma correction and white balance correction is performed on the captured image data in which dark current noise and shading have been corrected. The photographed image data that has undergone image processing is compressed in a predetermined format and stored in the memory card 200 or the built-in memory of the photographing apparatus 1.

As described above, in the present embodiment, the shading reference signal level of dark image data is calculated using the pixel signals output from the area (reference area) including the effective pixel area 130A. The effective pixel area 130A includes more pixels than the reference pixel area 130B. By calculating the light-shielding reference signal level using the pixel signals output from the large number of pixels, the influence of random noise on the light-shielding reference signal level can be suppressed. As a result, the clamp correction process using an appropriate signal level can be performed on the dark image data.

Further, in the present embodiment, the difference (offset) between the exposure reference signal level of the captured image data and the exposure reference signal level of the dark image data generated by shading is used by using the pixel signal output from the effective pixel area 130A. ) Is calculated. This makes it possible to properly correct the offset of the captured image data caused by shading.

In addition, in the present embodiment, the dark image data generated in the processing step S102 is subtracted from the captured image data whose offset is corrected in the processing step S103, but the present invention is not limited to this. For example, the offset correction processing may be performed on the dark image data after the dark image data is generated in the processing step S102. In this case, the offset of the captured image data is corrected by subtracting the dark image data from the captured image data.

Further, the captured image data or the dark image data may be subjected to a filter process for removing random noise. The captured image data and dark image data include noise components such as dark current noise, shading, and random noise. Of these, random noise usually has a higher frequency than dark current noise and shading. Therefore, by performing a filtering process such as a low-pass filtering process on the captured image data or the dark image data, it is possible to reduce only random noise included in these image data. For example, by performing a filtering process on the captured image data, it is possible to suppress the offset of the exposure reference signal level of the captured image data. Further, by performing filtering processing on at least one of the captured image data and the dark image data, when subtracting the dark image data from the captured image data, random noise included in the captured image data and the dark image data is generated. It is possible to suppress the increase in the amplitude of the random noise due to the addition.

(Second embodiment)
Next, an image pickup apparatus according to the second embodiment of the present invention will be described. In the second embodiment, unlike the first embodiment, the dark exposure processing is executed before the main exposure processing. In the following, the dark exposure processing and the dark image data subtraction processing in the second embodiment will be described, and for convenience of description, the same reference numerals are used for the same components as those in the first embodiment. I will. The dark exposure process may be executed before the main exposure process, and its execution timing is, for example, a stage before factory shipment of the image capturing apparatus 1 or a timing desired by the user.

The shading included in the captured image data or the dark image data changes depending on the capturing condition (hereinafter referred to as “fixed shading”), which does not change regardless of the capturing condition (temperature, sensitivity, exposure time, etc.). Including a component (hereinafter referred to as “variable shading”). Therefore, the exposure reference signal level of the captured image data includes both an offset caused by fixed shading and an offset caused by variable shading. The offset caused by the variable shading becomes larger as the integration time (exposure time) of the image sensor 112 is longer, the temperature is higher, and the sensitivity is set higher.

In the first embodiment, since the dark exposure process is executed under the same shooting conditions as the main exposure process, one dark image signal is used to perform both the offset caused by the fixed shading and the offset caused by the variable shading. Can be corrected. On the other hand, in the second embodiment, the shooting conditions of the dark exposure process are not necessarily the same as the shooting conditions of the main exposure process. Therefore, in the second embodiment, before the main exposure processing is performed, dark image data used to correct the offset caused by the fixed shading and dark image data used to correct the offset caused by the variable shading are generated. It

It is desirable that the shading included in the dark image data (hereinafter, referred to as “fixed dark image data”) used to correct the offset caused by the fixed shading includes only the fixed shading. Therefore, the dark exposure processing for generating the fixed dark image data is executed under the shooting conditions in which the variable shading is unlikely to occur, that is, the conditions in which the integration time of the image sensor 112 is short, the temperature is low, and the sensitivity is low. It The generated fixed dark image data is stored in the memory 118.

It is desirable that dark shading is largely included in dark image data (hereinafter, referred to as “varying dark image data”) used for correcting an offset caused by the variable shading. Specifically, the size of the variable shading of the variable dark image data is about the same as or larger than the fixed shading. Therefore, the dark exposure processing for generating the variable dark image data is executed under the photographing condition in which the variable shading is likely to occur, that is, the condition in which the integration time of the image sensor 112 is long, the temperature is high, and the sensitivity is set high. It The generated variable dark image data is stored in the memory 118. The memory 118 also stores shooting conditions in dark exposure processing for generating variable dark image data, that is, gain Gd in pixel signal amplification processing, exposure time ITd, and ambient temperature Td.

Note that the fixed dark image data and the variable dark image data include dark current noise, like the dark image data in the first embodiment. Therefore, the dark current noise included in the captured image data can be removed by using the fixed dark image data and the variable dark image data. In the following, of the shooting processing by the image processing engine 116 according to the second embodiment, the dark current noise removal processing of the shot image data will be described in detail. FIG. 6 shows a flowchart of processing relating to dark current noise removal processing. The process shown in FIG. 6 is a process executed by the user operating the release switch.

[Processing Step S201 (Main Exposure Processing)]
In the main processing step S201, main exposure processing is performed according to the shooting settings set by the user. The main exposure process is started by operating the release switch. The main exposure process of the second embodiment is the same as the main exposure process of the first embodiment. Specifically, the captured image data is generated based on the effective pixel signal. Further, the exposure reference signal level is calculated based on the reference pixel signal and stored in the memory 118. When the exposure reference signal level is stored, clamp correction processing is performed on the captured image data using the exposure reference signal level.

In the main exposure processing S201 according to the second embodiment, the shooting conditions at the time of the main exposure processing are stored in the memory 118. The stored shooting conditions are the gain Ge in the pixel signal amplification processing, the exposure time ITe, and the environmental temperature Te.

[Processing Step S202 (Offset Correction Processing by Fixed Shading)]
In this processing step S202, the correction processing of the offset included in the captured image data due to the fixed shading is performed. The magnitude of the offset caused by the fixed shading is calculated by using the fixed dark image data stored in the memory 118 in advance. Specifically, similar to the processing step S102 of the first embodiment, a row representative value of pixel signals is calculated for each row of effective pixels and reference pixels of fixed dark image data. Then, an approximate curve of the row representative value with the row number as a variable is obtained. Further, similar to the processing step S103 of the first embodiment, the difference (offset) between the exposure-time reference signal level and the light-shielding reference signal level due to the fixed shading is calculated by the approximation curve. The signal level of the captured image data is corrected based on this offset.

[Processing Step S203 (Offset Correction Processing by Variable Shading)]
In this processing step S203, the correction processing of the offset included in the captured image data due to the variable shading is performed. The magnitude of the offset caused by the variable shading is calculated using the variable dark image data stored in advance in the memory 118 and the shooting condition when the variable dark image data is acquired (at the time of dark exposure processing). Specifically, similar to the processing step S102 of the first embodiment, a row representative value of pixel signals is calculated for each row of effective pixels and reference pixels of the variable dark image data. Then, an approximate curve of the row representative value with the row number as a variable is obtained. Further, similar to the processing step S103 of the first embodiment, the difference (offset) between the exposure-time reference signal level and the light-shielding reference signal level is calculated by the approximation curve.

The offset calculated using the variable dark image data includes both the offset caused by the fixed shading and the offset caused by the variable shading. Although the offset caused by the variable shading changes according to the shooting conditions, the shooting conditions at the time of acquiring the variable dark image data (at the time of dark exposure) are the same as the shooting conditions at the time of acquiring the captured image data (at the main exposure). Not necessarily the same. Therefore, in this processing step, the offset adjustment processing is executed so that the offset calculated using the variable dark image data becomes the same as the offset caused by the variable shading included in the captured image data.

In the offset adjustment processing, first, the offset resulting from the fixed shading calculated in processing step S202 is subtracted from the offset calculated using the variable dark image data. As a result, only the offset caused by the variable shading is acquired.

Next, the adjustment coefficient k for adjusting the offset caused by the variable shading is calculated based on the shooting conditions. The formula for calculating the adjustment coefficient k is represented by the following formula (1).
(Equation 1)
k=α×(Ge/Gd)×(ITe/ITd)×(2 ^(Te-Td)/KT )
Here, the coefficient α is a constant that is set automatically or manually so that the offset is accurately adjusted.

As shown in the equation (1), the adjustment coefficient k is a term for the gain of the amplification processing of the fluctuation dark image data and the captured image data (Ge/Gd), a term for the integration time (ITe/ITd), and a temperature. (2 ^(Te−Td)/KT ) and the coefficient α. The magnitude of the offset caused by the variable shading included in the captured image data increases as the integration time increases, and increases as the temperature increases. Further, the offset caused by the variable shading is amplified according to the gain. Therefore, the adjustment coefficient k substantially represents the ratio of the magnitude of the offset caused by the variation shading included in the captured image data and the variation dark image data.

Next, the adjustment factor k is multiplied by the offset caused by the variable shading. The signal level of the captured image data is corrected based on the offset multiplied by the adjustment coefficient k.

Through the processing of processing steps S202 and S203, the reference signal levels of the captured image data, the fixed dark image data, and the variable dark image data are made uniform so as to have no offset due to shading.

[Processing Step S204 (Dark Image Data Subtraction Process)]
In this processing step S204, dark current noise and shading removal processing is performed on the captured image data. Specifically, the fixed dark image data and the variable dark image data are subtracted from the captured image data.

The captured image data and the fixed dark image data include fixed shading. Therefore, the fixed shading included in the captured image data is removed by subtracting the fixed dark image data from the captured image data.

Also, the captured image data and the variable dark image data include dark current noise and variable shading of a fixed pattern. The dark current noise and the magnitude of the variable shading change according to the shooting conditions. Specifically, both dark current noise and variable shading become larger as the exposure time is longer, the environmental temperature is higher, and the ISO sensitivity is set higher. Therefore, the variable dark image data is adjusted according to the shooting conditions before being subtracted from the shot image data.

The fixed dark image data and the adjustment coefficient k calculated in the processing step S203 are used for the adjustment of the variable dark image data. The variable dark image data includes variable shading and fixed shading. Therefore, by subtracting the fixed dark image data from the changed dark image data, the changed dark image data that does not include fixed shading can be obtained. Further, by multiplying the variable dark image data that does not include fixed shading by the adjustment coefficient k, the dark current noise included in the variable dark image data and the magnitude of the variable shading can be calculated as the dark current noise included in the captured image data. And variable shading size. By subtracting the adjusted varying dark image data from the captured image data, dark current noise and varying shading included in the captured image data are removed.

[Processing Step S205 (Image Processing)]
In this processing step S205, predetermined image processing such as gamma correction and white balance correction is performed on the captured image data from which dark current noise has been removed. The photographed image data that has undergone image processing is compressed in a predetermined format and stored in the memory card 200 or the built-in memory of the photographing apparatus 1.

As described above, in the second embodiment, since dark current noise is removed by using dark image data stored in the memory 118 in advance, it is not necessary to perform dark exposure processing every time main exposure processing is performed. As a result, dark current noise can be removed in a short time.

In addition, in the second embodiment, the shooting conditions for performing the dark exposure processing can be set arbitrarily. Therefore, the dark current noise included in the captured image data may be larger than the dark current noise included in the dark image data depending on the shooting conditions of the dark exposure process and the main exposure process. In this case, the dark current noise of the captured image data can be removed by making the adjustment coefficient k calculated in the dark image data correction processing S205 larger than 1.

Further, when the light-shielding reference signal level is calculated using the reference pixel signal as in the conventional technique, there is a problem that the reference signal level fluctuates due to random noise. In addition, when the light-shielding reference signal level is calculated using the reference pixel signal and the adjustment coefficient k is larger than 1, the fluctuation of the light-shielding reference signal level due to this random noise also increases. However, in the present embodiment, the light-shielding reference signal level is calculated using the approximate curve obtained based on the effective pixel signal. Since fluctuations in the light-shielding reference signal level due to random noise are suppressed, it is possible to prevent the fluctuations in the light-shielding reference signal level from increasing even when the adjustment coefficient k is larger than 1.

The above is a description of exemplary embodiments of the present invention. The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the contents of a combination of the embodiments exemplarily described in the specification or the obvious embodiments are appropriately included in the embodiments of the present application.

For example, the dark exposure process S102 of the first embodiment is executed with the same shooting settings as the main exposure process S101 except that the shutter is closed, but the present invention is not limited to this. For example, the exposure time in the dark exposure process S102 may be set shorter than the exposure time in the main exposure process S101. As a result, the dark exposure process S102 can be shortened. In this case, since the magnitude of shading and dark current noise is different between the captured image data and the dark image data, it is appropriate to correct the offset contained in the captured image data and remove shading and dark current noise. May not be done. Therefore, when the exposure time is changed between the main exposure processing S101 and the dark exposure processing S102, the offset correction processing S103 and the dark image data subtraction processing S104 are performed by the adjustment coefficient k in the same manner as in the second embodiment. The magnitude of shading and dark current noise is adjusted.

Further, in the dark exposure processing S102 in the first embodiment, the approximate curve is calculated using the row representative values of all the rows of the image sensor 112, but the present invention is not limited to this. For example, when the approximate shape of shading is known in advance, the approximation curve may be calculated using only the row representative values of some rows. In this case, the row representative value of the row not used for calculating the approximate curve can be obtained by interpolating or extrapolating the approximate curve. For example, when the pixel signal has the shading shown in FIG. 5, the approximate curve of the row representative value may be calculated using the row representative value of the effective pixel region 130A from the (q+1)th row to the mth row. In this case, the row representative value of the reference pixel region 130B in the first row to the p-th row is calculated by extrapolating the approximate curve.

Further, in the second embodiment, compression processing may be performed on the dark image data (fixed dark image data and variable dark image data) stored in the memory 118. In the compression process, the high-frequency component of the dark image data is mainly removed to reduce the data capacity. By this compression processing, a random noise component having a relatively high frequency is removed from the dark image data. Since the dark current noise has a relatively low frequency, it is possible to suppress the loss of the dark current noise removed by the compression process. Further, when the dark image data is compressed and stored in the memory 118, when the dark image data is read from the memory 118 in processing steps S202 and S203, the dark image data is converted to the size of the captured image data. On the other hand, extension processing is performed.

Further, in the offset correction processing S202 and S203 of the second embodiment, the offset is calculated using the dark image data, but the present invention is not limited to this. For example, the image capturing apparatus 1 may store shading data representing the shape and size of shading in the memory 118 in advance. The shading data corresponds to an approximate curve of row representative values of pixel signals obtained from dark image data. By storing the shading data in advance, it is not necessary to obtain an approximate curve in the offset correction processing S202 and S203 each time the main exposure processing S201 is performed, and the processing load of the image capturing apparatus 1 can be reduced. Further, the dark image data includes information about the shading size of a plurality of pixels arranged two-dimensionally, whereas the shading data includes information about the shading size of each row of effective pixels and reference pixels. The data amount is smaller than that of the dark image data because it includes only the image data. Therefore, even if the shading data is stored in the memory 118 in advance, the storage area of the memory 118 is not pressed.

Further, in the second embodiment, the memory 118 may previously store a plurality of dark image data according to the shooting setting. The shading included in the captured image data or the dark image data may change depending on the capturing settings of the capturing device 1. Here, the shooting setting is, for example, a drive mode of the image sensor 112. When a CMOS image sensor is used for the image sensor 112, the drive mode is a global shutter method for simultaneously reading pixel signals from all pixels, and a rolling shutter for reading pixel signals for each area by dividing the pixels into a plurality of areas. There are some that can be switched between methods. Such switching of drive modes may change the position where shading occurs in a captured image, that is, the shape of shading. Therefore, by preliminarily storing a plurality of dark image data corresponding to the shooting setting (driving mode) in the memory 118, shading of the shot image data is performed no matter what shooting conditions the main exposure process S201 is performed. It is possible to correct the offset and remove the shading from the captured image data. Further, a plurality of dark image data may be stored in the memory 118 in advance for each shooting condition such as ISO sensitivity and exposure time, or for each shooting mode setting such as AE control and AF control.

1 Imaging Device 100 System Controller 102 Operation Unit 104 Aperture/Shutter Drive Circuit 106 Imaging Lens 108 Aperture 110 Shutter 112 Image Sensor 114 Image Sensor Drive Circuit 116 Image Processing Engine 118 Memory 120 Card Interface 122 LCD Control Circuit 124 LCD
126 ROM
130 Image plane 200 Memory card

Claims (10)

An imaging device having an effective pixel region comprising a plurality of effective pixels for outputting a signal corresponding to the light flux from the object, and a reference pixel region composed of a plurality of reference pixels is shielded against the light flux,
When the effective pixel exposure condition being exposed to the light beam, generates the photographed image data on the basis of a signal output from the effective pixel area, among the reference pixel region, shading occurs A captured image generating means for calculating the exposure reference signal level based on the signal output from the area ;
When the light shielding state in which the effective pixel is blinded to the light beam, generates the dark image data based on the signal output from the effective pixel area, among the effective pixel region and the reference pixel region A dark image generating means for calculating a light-shielding reference signal level based on a signal output from a region where shading has not occurred ,
Reference signal correction means for calculating a signal level difference between the light-shielding reference signal level and the exposure reference signal level, and correcting the signal level of the photographed image data based on the signal level difference,
A photographic image correction means for correcting the photographic image data after the signal level correction by using the light-shielding reference signal level;
With
Imaging device. The photographed image correction means corrects the photographed image data by subtracting the dark image data from the photographed image data after correction of the signal level at a predetermined ratio.
The image capturing apparatus according to claim 1. The dark image generating means determines the light-shielding reference signal level by calculating one of an average value, a median value, and a mode value of the signals output from the effective pixel area,
The image capturing apparatus according to claim 1 . The reference signal correction means,
When the plurality of effective pixels of the light shielding state, and calculates an approximate curve representing the distribution of the light blocking during the reference signal level of the effective pixel region by using the signal outputted from the effective pixel region,
Calculating the signal level difference using the approximation curve,
The image capturing apparatus according to claim 1 . The reference signal correction means,
When the plurality of effective pixels of the light shielding state, the effective pixel area and using the signal output from the reference pixel region, approximation representing the distribution of the signal level of the effective pixel region and the reference pixel region Calculate the curve,
Calculating the signal level difference using the approximation curve,
The image capturing apparatus according to claim 1 . The dark image generating means calculates the light-shielding reference signal level by using the approximate curve calculated by the reference signal correcting means,
The imaging device according to claim 4 or claim 5 . The photographed image generating means determines the exposure reference signal level by calculating one of an average value, a median value, and a mode value of the signals output from the reference pixel area.
Imaging device according to any one of claims 1 to 6. The dark image generating means changes an area for outputting a signal used for calculating the light-shielding reference signal level in the effective pixel area, based on a photographing condition when the effective pixel is in the exposure state,
Imaging device according to any one of claims 1 to 7. Further comprising dark image storage means for storing the dark image data,
The captured image correction means corrects the captured image data using the dark image data stored in advance in the dark image storage means,
Imaging device according to any one of claims 1 to 8. Run in the imaging apparatus including an imaging device having an effective pixel region comprising a plurality of effective pixels and the reference pixel region composed of a plurality of reference pixels is shielded against the light flux for outputting a signal corresponding to the light flux from the object The shooting method is
When the effective pixel exposure condition being exposed to the light beam, generates the photographed image data on the basis of a signal output from the effective pixel area, among the reference pixel region, shading occurs Calculating a reference signal level during exposure based on the signal output from the area ,
When the light shielding state in which the effective pixel is not exposed to the light beam, generates the dark image data based on the signal output from the effective pixel area, among the effective pixel region and the reference pixel region A step of calculating a light-shielding reference signal level based on a signal output from a region where shading has not occurred ,
Calculating a signal level difference between the light-shielding reference signal level and the exposure reference signal level, and correcting the signal level of the captured image data based on the signal level difference,
Correcting the captured image data after the correction of the signal level using the light-shielding reference signal level ,
including,
How to shoot.

Images