JP6100023B2 - Flicker detection device, flicker correction device, control method thereof, and control program - Google Patents

Description

  The present invention relates to a flicker detection apparatus, a flicker correction apparatus, a control method thereof, and a control program, and more particularly to a flicker correction apparatus that detects and corrects periodic luminance fluctuations included in a captured image.

  In general, in an imaging apparatus including an imaging device such as a CMOS image sensor, when a moving image of a subject is captured under fluorescent lamp illumination, periodic brightness and darkness may appear in an image obtained as a result of the imaging. This phenomenon is called fluorescent lamp flicker and occurs due to the difference between the commercial power supply frequency and the vertical synchronization frequency of the imaging device.

  FIG. 9 is a diagram illustrating a state of charge accumulation in the image sensor when a subject is continuously photographed under fluorescent lamp illumination.

  Here, it is assumed that the subject is continuously photographed by the imaging device under fluorescent lamp illumination in an area where the commercial power supply frequency is 50 Hz. An NTSC CMOS image sensor is used as an image sensor in the image pickup apparatus.

  In FIG. 9, the imaging apparatus accumulates electric charges in accordance with a vertical synchronization signal (VD) 901. At this time, the fluorescent lamp (phosphor) emits light from the commercial power source having the commercial power source frequency 902, and the fluorescent lamp emits light at the light emission period 903 corresponding to the commercial power source period 902.

  Now, assuming that the image pickup device accumulates electric charges with the same period (1/60 s) as that of the vertical synchronization signal 901, exposure for each pixel is sequentially performed in the line direction. Here, in accordance with the vertical synchronization signal, the first frame, the second frame, and the third frame are sequentially arranged from the left in the figure, and the accumulated charge amount of the first line of the first frame is b11 and the second line of the first frame. Let b12 be the accumulated charge amount. Similarly, the accumulated charge amount of the first line of the second frame is b21, and the accumulated charge amount of the second line of the second frame is b22. Further, the accumulated charge amount b31 of the first line in the third frame is assumed to be b32.

  In this case, the total amount of accumulated charge differs for each line in each frame. That is, in each frame, the total amount of accumulated charges for each line differs because it is proportional to the amount of light emitted from the light emitter integrated by the accumulation time. For this reason, the accumulation time for each line differs for each frame. That is, the relationship of t11 ≠ t12, t21 ≠ t22, t31 ≠ t32 is established.

  As a result, in the same frame, a phenomenon in which the luminance level differs for each line occurs, and even if the luminance levels of the lines at the same position (for example, the first line of each frame) are compared for each frame, a difference also occurs. (T11 ≠ t21 ≠ t31).

  FIG. 10 is a diagram for explaining an image signal obtained as a result of imaging by the imaging device. FIG. 10A is a diagram schematically showing an image signal obtained when a blinking light source such as a fluorescent lamp is present, and FIG. 10B is obtained when a fluctuation occurs in the commercial power supply frequency. It is a figure which shows the obtained image signal typically.

  In FIG. 10A, an image signal 1001 obtained in the presence of a blinking light source has a signal level 1002 modulated by the blinking light source, and the image signal 1001 has a periodic level fluctuation depending on the light emission period of the blinking light source. It can be seen that occurs.

  On the other hand, there is known an imaging apparatus that detects and corrects periodic luminance fluctuations caused by a blinking light source included in an image obtained as a result of photographing (see Patent Document 1).

JP 2004-222228 A

  By the way, as described above, the blinking cycle in the blinking light source, which is a flicker generation source, depends on the commercial power supply frequency. For this reason, when the power supply frequency fluctuates, the bright and dark stripe intervals due to flicker appearing in the image fluctuate.

  For example, when the commercial power supply frequency fluctuates (for example, when the frequency decreases), an image signal 1003 shown in FIG. 10B is obtained. At this time, the signal level 1004 of the image signal 1003 is modulated by the blinking light source. In FIG. 10B, the signal level 1002 when there is no fluctuation in the commercial power supply frequency is indicated by a broken line.

  As can be easily understood from FIG. 10B, the light / dark position of the image changes after the change compared to before the change of the commercial power supply frequency.

  On the other hand, in Patent Document 1, when correcting the periodic luminance fluctuation due to the blinking light source, the frequency serving as a reference for the frequency analysis processing is fixed. Therefore, when the flicker frequency changes, the luminance fluctuation is accurately detected. Therefore, it becomes difficult to correct. If the resolution for the frequency to be analyzed in the frequency analysis process is increased, the number of samplings is increased, which is not preferable in consideration of the calculation processing load.

  Accordingly, an object of the present invention is to provide a flicker detection device, a flicker correction device, a control method therefor, and a control program capable of accurately detecting and correcting luminance fluctuations even when the flicker frequency in a blinking light source varies. It is in.

In order to achieve the above object, a flicker detection apparatus according to the present invention is a flicker detection apparatus that detects flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image. Te, and a flicker detection means for detecting the illumination intensity change information indicating a change in illumination intensity in the image signal the image signal by frequency analysis processing according to the reference frequency predetermined as a flicker, the flicker detection The means is characterized in that the reference frequency is changed according to an error between the flicker frequency and the reference frequency, and the frequency analysis processing is performed according to the changed reference frequency.

A flicker correction apparatus according to the present invention is a flicker correction apparatus that detects and corrects flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image, and is predetermined. Flicker detection means for detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to the reference frequency, and the illumination intensity change information and the flicker frequency. Flicker correction value generation means for generating a flicker correction value for performing the flicker correction based on an error from the reference frequency, and flicker correction means for correcting flicker generated in the image signal based on the flicker correction value. It is characterized by having.

  According to the present invention, even if the flicker frequency in the blinking light source fluctuates, it is possible to accurately detect the luminance fluctuation and correct the flicker.

It is a block diagram which shows an example of an imaging device provided with the flicker detection correction apparatus by the 1st Embodiment of this invention. 3 is a flowchart for explaining flicker detection processing in the imaging apparatus shown in FIG. 1. It is a figure which shows the change of phase amount (DELTA) phi when a horizontal axis is made into a time axis. It is a block diagram which shows the imaging device provided with an example of the flicker correction apparatus by the 2nd Embodiment of this invention. It is a block diagram which shows the imaging device provided with the other example of the flicker correction apparatus by the 2nd Embodiment of this invention. It is a block diagram which shows the imaging device provided with an example of the flicker correction apparatus by the 3rd Embodiment of this invention. It is a figure which shows the relationship between the charge accumulation time of the image pick-up element shown in FIG. 6, and the magnitude | size of the flicker which appears in an image. It is a block diagram which shows the imaging device provided with the other example of the flicker correction apparatus by the 3rd Embodiment of this invention. It is a figure which shows the mode of accumulation | storage of the electric charge in an image pick-up element at the time of image | photographing a to-be-photographed object continuously under fluorescent lamp illumination. It is a figure for demonstrating the image signal obtained as a result of imaging with an image pick-up element, (a) is a figure which shows typically the image signal obtained when blinking light sources, such as a fluorescent lamp, exist, (b) FIG. 6 is a diagram schematically showing an image signal obtained when a fluctuation occurs in the commercial power supply frequency.

  Hereinafter, an example of a flicker detection apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram illustrating an example of an imaging apparatus including a flicker detection apparatus according to the first embodiment of the present invention. In the imaging apparatus shown in FIG. 1, only components for detecting flicker are shown, and for example, components for image processing of image signals are omitted.

  The illustrated imaging apparatus includes an imaging element 101 such as a CMOS image sensor, and the imaging element 101 includes a plurality of photoelectric conversion elements. The image sensor 101 is driven by a drive pulse under the control of the control unit 102 and converts a subject image (optical image) formed through a lens into an electric signal (analog signal) by photoelectric conversion. The image sensor 101 converts an analog signal into a digital signal by a built-in A / D converter (not shown), and outputs the digital signal as an image signal.

  If the image sensor 101 has an electronic shutter function, the exposure time may be adjusted under the control of the control unit 102.

  The control unit 102 controls the entire imaging apparatus. The control unit 102 controls, for example, the charge accumulation time in the image sensor 101. And the control part 102 outputs the information which shows the control state of the image pick-up element 101 to the error estimation part 104 mentioned later as control information. Further, the control unit 102 designates a flicker detection area for detecting flicker appearing in the input image signal to the flicker detection unit 103 described later.

  The flicker detection unit 103 receives an image signal from the image sensor 101 and detects flicker appearing in an image indicated by the image signal as illumination intensity by frequency analysis processing. Then, the flicker detection unit 103 outputs illumination intensity change information indicating a change in illumination intensity to the error estimation unit 104.

  The error estimation unit 104 estimates error frequency information indicating an error frequency between the reference frequency and the flicker frequency in the frequency analysis process according to the illumination intensity change information. Then, the error estimation unit 104 outputs the error frequency information to the flicker detection unit 103. In the frequency analysis process, the flicker detection unit changes the reference frequency based on the error frequency information, and sets the changed reference frequency.

  FIG. 2 is a flowchart for explaining flicker detection processing in the imaging apparatus shown in FIG.

  When the flicker detection process is started, the control unit 102 designates a flicker detection area to the flicker detection unit 103. Then, the flicker detection unit 103 acquires flicker occurring in the image indicated by the image signal in the flicker detection area, that is, illumination intensity change information indicating a change in illumination intensity (step S201).

  By the way, as a method of extracting a periodic luminance fluctuation component caused by a blinking light source or the like, for example, there is a method of converting a temporally changing image signal into a frequency domain. In general, data sampled in a predetermined period, such as an image signal, is converted into a frequency domain using a discrete Fourier transform. Therefore, in the following description, it is assumed that the flicker detection unit 103 obtains a change in illumination intensity by converting flicker of an image indicated by an image signal into a frequency domain using discrete Fourier transform.

  As described with reference to FIG. 10, flicker generated by the blinking light source appears in the form of light and dark in the vertical direction of the image according to the driving of the image sensor 101. Therefore, it is only necessary to perform sampling in the vertical direction in the image indicated by the image signal. For example, the image signal is averaged for each horizontal line, and is reduced to two-dimensional information on the time axis (time sampling) and the signal level to reduce flicker. To detect.

  In addition to the above method, for example, a flicker detection area is set in the center of the image, assuming that flicker occurring in the center of the image indicated by the image signal gives the user discomfort. Then, the flicker may be detected by averaging the image signals for each horizontal direction in the flicker detection area.

  In any case, flicker may be detected based on data obtained by degenerating an image signal into two-dimensional information of a time axis (time sampling) and a signal level.

  Further, since flicker generated in an image has luminance modulation properties, if the average signal level (line average level) for each line when flicker does not occur is different, the flicker amplitude is also different. Therefore, the flicker detection unit 103 normalizes the line average level when flicker occurs by dividing it by the line average level when flicker does not occur.

  When the image signal is NTSC and the frequency of the commercial power supply is 60 Hz, the flicker appearing in the image signal is 100 / 60≈1.7 cycles. For this reason, the line average level when no flicker occurs can be obtained with high accuracy by averaging the three frames. Further, in order to cancel the direct current component (DC component), 1 is subtracted from the normalized sampling data to obtain a frequency analysis parameter.

  In addition, as described above, flicker appearing in the image signal has a period of 1.7. Therefore, all or a part of one period or more in the vertical direction in the image signal may be sampled. When the image signal is PAL and the commercial power supply frequency is 50 Hz, the period is 120/50 = 2.4. Therefore, all or a part of one or more periods in the vertical direction may be similarly sampled in the image signal. .

  Here, an example of converting the sampling data obtained as described above into the frequency domain will be described.

  Assuming that the sampling parameter (that is, the sampling number) is N (N is an integer of 2 or more) and the sampled data is Xn (n = 0, 1,..., N−1), The discrete Fourier transform is performed in the frequency domain by the discrete Fourier coefficients indicated by

Here, k = 0, 1,..., (N−1).

  When N time samplings are performed for the time L (sec) and the sampling interval is a (sec), the Fourier coefficient after the discrete Fourier transform is a range of the discrete frequency fk represented by the equation (2). Can be obtained.

Here, fs is a sampling frequency in the time domain.

  Since the data sequence sampled from the image signal is a real-time signal, when the expression (1) is expanded, the data string can be separated into the real part Re and the imaginary part Im. The expressions (3) and (4) It is represented by

  From the definition of the discrete Fourier transform, the frequency component (amplitude value and phase information) at the order k can be calculated according to the real part Re and the imaginary part Im shown in Equation (4), and the frequency of the flicker to be detected can be calculated. If the corresponding order k is appropriately selected, illumination intensity change information can be obtained.

  As described above, when the illumination intensity change information is obtained by the flicker detection unit 103, the error estimation unit 104 determines the phase information Ωk according to the illumination intensity change information (the real part Re and the imaginary part Im that are frequency components of flicker). Is calculated (step S202).

  The calculation of the phase spectrum in the discrete Fourier transform is defined by equation (5).

  However, the values that can be taken in the equation (5) are −π / 2 to π / 2, which is inconvenient in the error estimation process performed by the error estimation unit 104 described later, and as shown in the equation (6), The possible values are expanded to 0-2π.

  Subsequently, the error estimation unit 104 performs error estimation processing by monitoring the transfer information Ωk with the time information from the control unit 102 (step S203).

  Of the phase information Ωk, the phase information Ωk at the order k corresponding to the frequency of the flicker to be detected indicates phase information when flicker appearing in the image is regarded as a periodic wave. For example, when the image signal is NTSC and the frequency of the commercial power supply is 60 Hz, the component of 100 Hz which is the fundamental frequency among the flickers appearing in the image signal is 100 / 60≈1.7 cycles. For this reason, the phase of the component of 100 Hz is repeated every three frames.

  Therefore, the phase information Ωk obtained for each frame includes a predetermined phase amount Δφ and a phase change amount θ due to repetition of three frames. (The phase amount Δφ is a phase shift amount with respect to a cosine wave (equation (4)) of discrete Fourier transform in a specific frame.

Therefore, with respect to the phase information Ωk, the phase amount Δφ can be calculated by Expression (7).

  For example, on the basis of a certain frame, θ = 0 is set for the phase information Ωk obtained in the reference frame, θ = 2π / 3 is set in the next frame, and θ = 4π / 3 is set in the next frame. The calculation is repeated every three frames. By this operation, the phase amount Δφ can be obtained for each frame.

  Here, as shown in Expression (6), since the possible value is 0 to 2π, 2π is added when the result of the calculation according to Expression (7) becomes negative.

  FIG. 3 is a diagram showing a change in the phase amount Δφ when the horizontal axis is the time axis. Since the phase amount Δφ is obtained in units of frames, the phase amount Δφ should take a discrete value, but in FIG. 3, it is drawn with a continuous value for convenience of explanation.

  In FIG. 3, a broken line (dotted line) 302 indicates a case where the flicker frequency appearing in the image matches the reference frequency of the frequency analysis processing in the flicker detection unit 103. In the dotted line 302, no change is observed in the value when viewed in the time direction, that is, in units of frames, and the phase amount Δφ becomes a constant value Δφ1.

  On the other hand, when the flicker frequency that appears in the image does not match the reference frequency of the frequency analysis processing in the flicker detection unit 103, that is, when the blinking cycle of the blinking light source fluctuates, as shown by the solid line 301, The phase amount Δφ varies in the range of 0 to 2π. In particular, when the cycle (frequency) of the flashing light source fluctuates with a constant error amount, the phase amount Δφ also increases or decreases at a constant change rate (FIG. 3 shows an example in which it increases. ).

  The error estimation unit 104 calculates an error frequency Δf between the flicker frequency appearing in the image and the reference frequency of the frequency analysis processing in the flicker detection unit 103 (step S204).

  Referring to FIG. 3 again, when there is a certain error in the flicker frequency appearing in the image, the phase amount Δφ changes according to the error amount as indicated by the solid line 301. At this time, since the phase amount Δφ increases or decreases in the range of 0 to 2π, the phase amount changes from the phase amount Δφ1 at the time t1 when the error estimation unit 104 starts monitoring, and a certain amount of time After the elapse of time, the phase amount returns to Δφ1 again. If the time at this time is t2, the error frequency Δf is calculated by the equation (8) from the time Δt required for the phase to change by 2π (rad).

However, since the sign is not taken into consideration in Expression (8), Expression (8) is corrected to Expression (9) to calculate the error frequency Δf.

  The error estimation unit 104 acquires from the control unit 102 the number of frames (that is, time) required for the phase amount Δφ obtained for each frame to make one round. Then, the error estimation unit 104 calculates an error frequency Δf between the flicker frequency appearing in the image and the reference frequency of the frequency analysis processing from the phase amount Δφ obtained in units of frames.

  This error frequency Δf is sent from the error estimation unit 104 to the flicker detection unit 103, and the flicker detection unit 103 changes the reference frequency in the frequency analysis processing according to the error frequency Δf.

  The reference frequency is a component of the fundamental wave of flicker generated in the image signal, and generally corresponds to a frequency twice as high as the power supply frequency. For example, in the case of frequency analysis processing by discrete Fourier transform, the frequency component obtained as apparent from Equation (2) becomes a discrete value fk, so changing the reference frequency changes the sampling parameter N. It corresponds to that.

  Since the error frequency Δf obtained by the error estimation unit 104 indicates the error amount of the fundamental wave component of the flicker generated in the image, the flicker detection unit 103 sets the error frequency Δf as shown in Expression (10). The new sampling parameter N ′ considered is taken as the analysis range of the frequency analysis processing.

  Here, f′k is a discrete frequency analyzed by a discrete Fourier transform that takes into account the error frequency Δf, and is accurate according to the component of the fundamental wave and the harmonic component of the flicker frequency appearing in the image. Can be detected.

  As described above, in the first embodiment of the present invention, in the frequency analysis processing when flicker is detected, the reference frequency is determined according to the error frequency between the flicker frequency appearing in the image and the reference frequency of the frequency analysis processing. Therefore, flicker can be detected with high accuracy.

  In the first embodiment, the error frequency Δf is calculated from the time Δt when the phase amount Δφ makes one revolution in the range of 0 to 2π and the change direction (increase or decrease) of the phase amount Δφ according to the equation (9). Calculated. Note that one round means returning to the original Δφ1 in the range 0 to 2π by repeating the increase or decrease from the phase amount Δφ1 at the start of monitoring shown in FIG.

  As another method, the error frequency Δf may be obtained as follows. For example, in FIG. 3, when the phase amount when a predetermined time t3 has elapsed from the phase amount Δφ1 at the time t1 when monitoring is started is Δφ2, the error frequency Δf is defined by the equation (11) from the definition of the angular velocity. Can be calculated.

  That is, the error frequency Δf can be calculated by calculating the amount of phase change per unit time, and as described above, the error frequency Δf can be estimated without waiting for the phase amount Δφ to make one round. Further, the phase change amount per unit time is calculated every frame or every several frames (unit frames), and whether the phase change amount per unit time is constant or not is monitored in the time direction similarly to the phase amount Δφ. . Thus, it can be determined whether or not the error frequency Δf is constant, and the reliability of the error frequency Δf can be improved.

  Further, the flicker detection unit 103 may change the frequency fk according to the equation (10) when the error frequency Δf exceeds a predetermined threshold Th. As a result, frequent switching of the frequency can be suppressed, and the calculation time required for changing the calculation parameter for frequency analysis can be suppressed.

[Second Embodiment]
Subsequently, an example of an imaging apparatus including the flicker correction apparatus according to the second embodiment of the present invention will be described.

  FIG. 4 is a block diagram illustrating an imaging apparatus including an example of a flicker correction apparatus according to the second embodiment of the present invention. In FIG. 4, the same components as those in the imaging apparatus shown in FIG.

  4, the illustrated imaging apparatus includes a flicker correction value generation unit 401 and a flicker correction unit 402, and the error frequency Δf obtained by the error estimation unit 104 is given to the flicker correction value generation unit 401. Under the control of the control unit 102, the flicker correction value generation unit 401 calculates a correction value (flicker correction value) for correcting image flicker based on the illumination intensity change information obtained by the flicker detection unit 103. Then, the flicker correction value generation unit 401 sends the flicker correction value to the flicker correction unit 402.

  Here, generation of the flicker correction value by the flicker correction value generation unit 401 will be described.

  The flicker correction value generation unit 401 holds, for example, a sine wave model represented by Expression (12) in order to generate a flicker correction value for correcting flicker appearing in an image indicated by the image signal.

In Equation (12), Ak is the flicker amplitude value calculated by Equation (13) according to the illumination intensity change information (real part Re and imaginary part Im).

  When generating the flicker correction value, the flicker correction value generation unit 401 selects the order k of the frequency analysis processing by the flicker detection unit 103, and the amplitude value Ak and the frequency fk in the order k (that is, the flicker to be corrected). Frequency).

  Θ shown in Expression (12) is a phase amount obtained from the phase information Ωk in the selected order k, and corresponds to an initial phase for each frame when flicker is regarded as a periodic wave.

  The flicker correction value generation unit 401 generates a flicker correction value for each line included in the image signal, and thus calculates the time t of Expression (12) according to the number of lines included in the frame, thereby flickering each line. A correction value is calculated. Specifically, the flicker correction value generation unit 401 receives the start timing of each line from the control unit 102 as control information. Then, the flicker correction value generation unit 401 calculates an elapsed time between lines according to the control information.

  If a plurality of orders k are selected, a flicker correction value related to a higher-order component of flicker appearing in the image can be generated. However, for convenience of explanation, the fundamental wave (that is, twice the power supply frequency) is generated. It is assumed that a flicker correction value related to the component is generated.

  Since fk (t) obtained by the equation (12) reproduces the flicker of the image signal, in order to correct the flicker of the image signal, 1 is added to fk (t), and the reciprocal thereof is obtained. If it is obtained, it becomes a flicker correction value.

  The flicker correction unit 402 multiplies the flicker correction value by the image signal that is the output of the image sensor 101 to correct flicker appearing in the image indicated by the image signal. Note that flickers appearing in the image have almost the same modulation amount in the same line, and therefore the same flicker correction amount can be used in the same line.

  In the imaging apparatus shown in FIG. 4, it is assumed that the reference frequency for generating the flicker correction value is, for example, fk = 100 Hz. This fk = 100 Hz is due to the fact that the flicker frequency (fundamental frequency) appearing in the image is twice the power supply frequency as described above, and is a generally known value.

  When the flicker frequency appearing in the image fluctuates due to an error in the power supply frequency, the flicker correction value generation unit 401 adds the error frequency Δf obtained from the error estimation unit 104 to fk shown in Expression (12). Thus, even if the flicker frequency changes, the flicker correction value generation unit 401 can generate the flicker correction value at the changed frequency, and the accuracy of flicker correction is improved.

  FIG. 5 is a block diagram showing an imaging apparatus provided with another example of the flicker correction apparatus according to the second embodiment of the present invention. In FIG. 5, the same components as those in the image pickup apparatus shown in FIG.

  In the imaging apparatus shown in FIG. 5, the error estimation unit 104 sends the error frequency Δf to the flicker detection unit 103 and the flicker correction value generation unit 401. The flicker detection unit 103 changes the reference frequency reference frequency of the frequency analysis processing according to the error frequency Δf as described in the first embodiment.

  As a result, flicker appearing in the image can be detected with high accuracy. That is, the accuracy of the illumination intensity change information (the real part Re and the imaginary part Im obtained by discrete Fourier transform) obtained by the flicker detection unit 103 is improved. As a result, the accuracy of the flicker amplitude value Ak calculated by the equation (13) is improved.

  Further, the flicker correction value generation unit 401 can generate the frequency fk of the flicker correction value at the changed frequency, and both the amplitude values Ak and fk in the equation (12) can correspond to the fluctuation of the flicker frequency. it can. Therefore, even when the frequency of flicker appearing in the image fluctuates, the accuracy of flicker correction can be further improved.

[Third Embodiment]
Next, an example of an imaging apparatus provided with the flicker correction apparatus according to the third embodiment of the present invention will be described.

  FIG. 6 is a block diagram illustrating an imaging apparatus including an example of a flicker correction apparatus according to the third embodiment of the present invention. In FIG. 6, the same constituent elements as those of the imaging apparatus shown in FIGS. 1, 4, and 5 are denoted by the same reference numerals, and the description thereof is omitted.

  The illustrated imaging apparatus includes an adjustment determination unit 601. The adjustment determination unit 601 determines whether or not to adjust the reference frequency with the error frequency Δf obtained by the error estimation unit 104 under the control of the control unit 102. judge. The control unit 102 sends an adjustment control signal to the adjustment determination unit 601. This adjustment control signal indicates the charge accumulation time in the image sensor 101. The control unit 102 generates an adjustment control signal according to the charge accumulation time obtained when controlling the charge accumulation of the image sensor 101.

  FIG. 7 is a diagram showing the relationship between the charge accumulation time of the image sensor 101 shown in FIG. 6 and the size of flicker appearing in the image. The flicker size refers to the degree of modulation of the luminance signal by the blinking light source.

  In FIG. 7, the degree of modulation on the vertical axis is obtained by normalizing the level fluctuation (amplitude) when flicker is generated by the blinking light source by the signal level when flicker is not generated. A characteristic curve 701 shown by a solid line indicates how much the signal level is modulated by flicker in each charge accumulation time. The shorter the charge accumulation time, the shorter the time for integrating the amount of light from the flashing light source, and the greater the fluctuation in the signal level for each line. For this reason, the modulation factor has the characteristics shown in the figure from the integration time and the period of the blinking light source.

  Now, the charge accumulation time in which the size of the flicker is larger than the threshold value Th determined by a predetermined standard is indicated by times tl and tr. The adjustment determination unit 601 determines whether the charge accumulation time ts indicated by the adjustment control signal sent from the control unit 102 is included in the time tl or tr by the following equations (14) and (15).

  Note that the threshold Th is determined according to the boundary condition in which flicker appearing in the image is recognized in the image, and it is desirable that the flicker is not visually recognized by the user if the flicker size is equal to or smaller than the threshold Th.

  If the adjustment determination unit 601 determines that the current charge accumulation time of the image sensor 101 is a charge accumulation time in which the flicker is larger than the threshold value Th, the error frequency Δf is detected by the flicker detection unit 103 and the flicker correction value generation unit. 401. On the other hand, if it is determined that the charge accumulation time of the image sensor 101 is not the charge accumulation time in which the flicker is larger than the threshold Th, the adjustment determination unit 601 sends the error frequency Δf to the flicker detection unit 103 and the flicker correction value generation unit 401. do not send.

  If it is determined that the charge accumulation time of the image sensor 101 is not the charge accumulation time in which the magnitude of the flicker is larger than the threshold Th, the adjustment control unit 601 sets the error frequency Δf to Δf = 0 and flicker detection unit 103 and flicker correction. You may make it send to the value production | generation part 401. FIG.

  6, the adjustment determination unit 601 adjusts the reference frequency fk of the flicker detection unit 103 and the reference frequency fk of the flicker correction value generation unit according to the charge accumulation time of the image sensor 101. Thus, frequent switching of the reference frequency can be suppressed. Thereby, it is possible to suppress the calculation time required for changing the calculation parameter of the frequency analysis processing and the calculation time required for generating the flicker correction value.

  Further, as can be easily understood from the characteristic curve 701 shown in FIG. 7, even if the error frequency Δf obtained by the error estimation unit 104 is the same, the flicker that appears in the image with respect to the charge accumulation time of the image sensor 101 is also shown. Since the size fluctuates, a difference occurs in an error component appearing in the image after flicker correction depending on the charge accumulation time of the image sensor 101. This error component is a residual component resulting from the remaining correction. For this reason, the detection accuracy can be further improved as compared with the case where the threshold determination is performed on the error frequency Δf described in the first embodiment.

  FIG. 8 is a block diagram showing an imaging apparatus provided with another example of the flicker correction apparatus according to the third embodiment of the present invention. In FIG. 8, the same constituent elements as those of the imaging apparatus shown in FIG.

  In the imaging apparatus illustrated in FIG. 8, the illumination intensity change (real part Re and imaginary part Im) obtained by the flicker detection unit 103 is input to the adjustment determination unit 601 as illumination intensity change information 801. The adjustment determination unit 601 calculates the flicker amplitude value Ak generated in the image according to the expression (13) according to the illumination intensity change information 801.

  The amplitude value Ak indicates the size of flicker during the charge accumulation time of the image sensor 101, that is, one point on the characteristic curve 701 shown in FIG. Then, as described with reference to FIG. 6, the adjustment determination unit 601 compares the predetermined threshold Th with the amplitude value Ak, and determines the error frequency Δf as the flicker detection unit 103 and the flicker correction value generation unit 401. It is determined whether or not to send to.

  As described above, the image pickup apparatus shown in FIG. 8 not only achieves the same effect as the image pickup apparatus shown in FIG. 6, but also has a more accurate frequency even if there is a difference in the characteristic curve due to the difference in the blinking light source. It is possible to suppress the calculation time required for changing the calculation parameter during the analysis process and the calculation time required for generating the flicker correction value.

  As described above, in the third embodiment of the present invention, the calculation time required for changing the calculation parameter of the frequency analysis processing and the calculation time required for generating the flicker correction value are suppressed, and flicker is detected with high accuracy. Correction can be performed.

  As is clear from the above description, in the examples shown in FIGS. 1, 4, 5, 6, and 8, the flicker detection unit 103 and the control unit 102 function as flicker detection means, and the error estimation unit 104 And the control part 102 functions as an error estimation means.

  Further, the flicker correction value generation unit 401 and the control unit 102 function as flicker correction value generation means. Furthermore, the adjustment determination unit 601 and the control unit 102 function as a determination unit.

  In the example illustrated in FIG. 1, the control unit 102, the flicker detection unit 103, and the error estimation unit 104 constitute a flicker detection device. In the example shown in FIGS. 4 and 5, the control unit 102, the flicker detection unit 103, the error estimation unit 104, the flicker correction value generation unit 401, and the flicker correction unit 402 constitute a flicker correction apparatus. Furthermore, in the example shown in FIGS. 6 and 8, the control unit 102, flicker detection unit 103, error estimation unit 104, flicker correction value generation unit 401, flicker correction unit 402, and adjustment determination unit 601 constitute a flicker correction apparatus. .

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and the control method may be executed by the flicker detection device or the flicker correction device. Alternatively, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the flicker detection device or the flicker correction device. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a flicker detection step, an error estimation step, and a change step. Each of the control method and the control program may include a flicker detection step, an error estimation step, a flicker correction value generation step, and a flicker correction step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Image pick-up element 102 Control part 103 Flicker detection part 104 Error estimation part 401 Flicker correction value generation part 402 Flicker correction part 601 Adjustment determination part

Claims (18)

A flicker detection apparatus that detects flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
Flicker detection means for detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
The flicker detection unit changes the reference frequency according to an error between the flicker frequency and the reference frequency, and performs the frequency analysis processing according to the changed reference frequency. .   The flicker detection apparatus according to claim 1, wherein the flicker detection unit detects a change in illumination intensity in a direction perpendicular to the line in a unit frame.   3. The flicker detection apparatus according to claim 1, wherein the flicker detection unit changes the reference frequency according to the error when the error is larger than a predetermined threshold.   4. The flicker detection apparatus according to claim 3, wherein when the reference frequency is changed, the flicker detection unit changes a sampling number in the frequency analysis processing.   5. The apparatus according to claim 1, further comprising: an error estimation unit configured to estimate an error frequency indicating the error according to the illumination intensity change information and time information indicating time. Flicker detection device. A flicker correction apparatus that detects and corrects flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
Flicker detection means for detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
Flicker correction value generation means for generating a flicker correction value for performing the flicker correction based on the illumination intensity change information and an error between the flicker frequency and the reference frequency;
Flicker correction means for correcting flicker generated in the image signal based on the flicker correction value;
A flicker correction device comprising:   The flicker correction apparatus according to claim 6, wherein the flicker detection unit changes the reference frequency according to the error, and performs the frequency analysis processing according to the changed reference frequency.   The flicker correction apparatus according to claim 6 or 7, wherein the flicker detection means detects a change in illumination intensity in a direction perpendicular to the line in a unit frame.   The flicker correction apparatus according to claim 6, further comprising a determination unit that determines whether or not to send the error to the flicker correction value generation unit in accordance with a charge accumulation time of the image sensor.   The flicker correction according to claim 7, further comprising: a determination unit that determines whether to send the error to the flicker detection unit and the flicker correction value generation unit according to a charge accumulation time of the image sensor. apparatus.   The flicker according to claim 6, further comprising: a determination unit that determines whether or not to send the error to the flicker correction value generation unit according to a flicker amplitude value obtained according to the illumination intensity change information. Correction device. In accordance with the amplitude value of the resulting flicker in response to the illumination intensity change information, to have a judging means for judging whether to send the error to the flicker detection means and before Symbol flicker correction value generator The flicker correction apparatus according to claim 7, wherein the flicker correction apparatus is characterized. Further, in response to said illumination intensity variation information and time indicating the time information, according to any one of claims 6 to 12, characterized in that it has an error estimation means for estimating the error frequency representing the error Flicker correction device. The illumination intensity change information is phase information of frequency components obtained by the frequency analysis process,
The flicker correction apparatus according to claim 13, wherein the error estimation unit estimates the error frequency based on a phase change amount per unit time according to the phase information. A control method of a flicker detection apparatus for detecting flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
A flicker detection step of detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
Changing the reference frequency according to an error between the flicker frequency and the reference frequency, and changing the frequency analysis processing according to the changed reference frequency in the flicker detection step;
A control method characterized by comprising: A control program used in a flicker detection apparatus for detecting flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
In the computer provided in the flicker detection device,
A flicker detection step of detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
Changing the reference frequency according to an error between the flicker frequency and the reference frequency, and changing the frequency analysis processing according to the changed reference frequency in the flicker detection step;
A control program characterized by causing A control method of a flicker correction device that detects and corrects flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
A flicker detection step of detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
A flicker correction value generating step for generating a flicker correction value for performing the flicker correction based on the illumination intensity change information and an error between the flicker frequency and the reference frequency;
A flicker correction step for correcting flicker generated in the image signal based on the flicker correction value;
A control method characterized by comprising: A control method used in a flicker correction apparatus for detecting and correcting flicker generated in an image signal output from an image sensor that photoelectrically converts an optical image formed on an optical image,
In the computer provided in the flicker correction device,
A flicker detection step of detecting, as flicker, illumination intensity change information indicating a change in illumination intensity in the image signal by performing frequency analysis processing on the image signal according to a predetermined reference frequency;
A flicker correction value generating step for generating a flicker correction value for performing the flicker correction based on the illumination intensity change information and an error between the flicker frequency and the reference frequency;
A flicker correction step for correcting flicker generated in the image signal based on the flicker correction value;
A control program characterized by causing