JP4236642B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus such as a digital camera provided with camera shake correction means.

  The still image stabilization technology is a technology for reducing camera shake in still image shooting, and is realized by detecting camera shake and stabilizing an image based on the detection result.

  Methods for detecting camera shake include a method using a camera shake sensor (angular velocity sensor) and an electronic method for analyzing and detecting an image. As a method for stabilizing an image, there are an optical method for stabilizing a lens and an image sensor, and an electronic method for removing blur caused by camera shake by image processing.

  On the other hand, a completely electronic camera shake correction technique, that is, a technique for generating an image free from camera shake by analyzing and processing only one photographed camera shake image has not reached a practical level. In particular, it is difficult to obtain an accurate camera shake signal obtained by a camera shake sensor by analyzing a single camera shake image.

  Therefore, it is realistic to detect camera shake using a camera shake sensor and remove camera shake blur by image processing using the camera shake data. Deblurring by image processing is called image restoration. In addition, a technique based on a camera shake sensor and image restoration will be referred to as electronic camera shake correction here.

  By the way, if the degradation process of an image such as camera shake or blur is clear, it is possible to reduce the degradation by using an image restoration filter called a Wiener filter or a general inverse filter. However, as a side effect, wavy deterioration called ringing occurs in the periphery of the edge portion of the image, or noise is emphasized.

As a method of reducing such side effects, a smoothing filter (median filter, Gaussian filter, etc.) is inserted before or after restoration by the image restoration filter, or ringing is removed or unsharpened from the image after restoration. There is a way to do masking. However, side effects vary greatly depending on imaging scenes such as night views, people, landscapes, and movements, so it is difficult to efficiently reduce side effects in all shooting scenes.
JP-A-8-265572 JP 2004-88567 A Japanese Patent Laid-Open No. 11-24122 JP 2003-274280 A

  An object of the present invention is to provide an imaging apparatus capable of efficiently reducing side effects due to camera shake correction regardless of the shooting scene.

  The invention described in claim 1 is based on a shooting scene detection unit that automatically detects a shooting scene, a camera shake correction unit that performs a camera shake correction process on an input image, and a shooting scene detected by the shooting scene detection unit. The image processing apparatus is characterized by comprising control means for controlling the intensity of camera shake correction by the camera shake correction means.

Shake correction means is to carry out camera shake correction using the inverse filter, the control means based on the photographic scene that is detected by the photographing scene detecting means to control the regularization coefficient used in generating an inverse filter The

The invention described in claim 2 removes ringing from a restored image obtained by a shooting scene detection unit that automatically detects a shooting scene, a camera shake correction unit that performs a camera shake correction process on an input image, and a camera shake correction unit. And a control means for controlling the intensity of the ringing removal by the ringing removal means based on the shooting scene detected by the shooting scene detection means.

Ringing removal means, the edge strength calculating means for calculating an edge strength of each pixel of the input image before camera shake compensation, the relationship between the edge intensity for each pixel calculated by the edge strength calculating unit, the weighted addition coefficient for the edge strength Based on the above, the coefficient calculation means for obtaining the weighted addition coefficient for each pixel and the weighted addition coefficient obtained for each image are used to weight the input image and the restored image obtained by the camera shake correction means for each pixel. A weighted average means for adding is provided. The edge strength and the weighted addition coefficient are such that when the edge strength is equal to or lower than the threshold value, the weighted ratio of the restored image increases as the edge strength increases. ratio is in the relationship as 100%, the control means based on the photographic scene that is detected by the photographing scene detecting means, controls the threshold value .

The invention described in claim 3 removes ringing from the first image obtained by the shooting scene detection means for automatically detecting the shooting scene, the camera shake correction means for performing the camera shake correction processing on the input image, and the camera shake correction means. Based on the shooting scene detected by the shooting scene detection means and the unsharp masking processing means for performing edge enhancement processing on the second image obtained by the ringing removal means Control means for controlling at least one of the edge emphasis width and the degree of edge emphasis by the masking processing means is provided.

According to a fourth aspect of the present invention, in the imaging apparatus according to the third aspect , the unsharp masking processing unit generates a smoothed image of the second image using a Gaussian filter, and the second image and the second image An edge strength image is generated by multiplying the difference from the smoothed image by a correction strength parameter, and an edge strength image is obtained by adding the edge strength image to the second image. Based on the photographic scene detected by the photographic scene detection means, a parameter representing dispersion in a two-dimensional Gaussian function used to generate a smoothed image, and the correction intensity parameter used to generate an edge-enhanced image And at least one of them is controlled.

The invention according to claim 5 is based on the shooting scene setting means for setting the shooting scene, the camera shake correction means for performing the camera shake correction process on the input image, and the shooting scene set by the shooting scene setting means. Control means for controlling the intensity of camera shake correction by the camera shake correction means is provided.

Shake correction means is to carry out camera shake correction using the inverse filter, the control means on the basis of the photographic scene is set by the photographing scene setting means, to control the regularization coefficient used in generating an inverse filter The

According to the sixth aspect of the present invention, a shooting scene setting unit for setting a shooting scene, a camera shake correction unit that performs a camera shake correction process on an input image, and a ringing is removed from a restored image obtained by the camera shake correction unit. And a control means for controlling the intensity of the ringing removal by the ringing removal means based on the shooting scene set by the shooting scene setting means.

Ringing removal means, the edge strength calculating means for calculating an edge strength of each pixel of the input image before camera shake compensation, the relationship between the edge intensity for each pixel calculated by the edge strength calculating unit, the weighted addition coefficient for the edge strength Based on the above, the coefficient calculation means for obtaining the weighted addition coefficient for each pixel and the weighted addition coefficient obtained for each image are used to weight the input image and the restored image obtained by the camera shake correction means for each pixel. A weighted average means for adding is provided. The edge strength and the weighted addition coefficient are such that when the edge strength is equal to or lower than the threshold value, the weighted ratio of the restored image increases as the edge strength increases. ratio is in the relationship as 100%, the control means on the basis of the photographic scene is set by the photographing scene setting means, to control the threshold value .

The invention described in claim 7 removes ringing from the first image obtained by the shooting scene setting means for setting the shooting scene, the camera shake correction means for performing the camera shake correction processing on the input image, and the camera shake correction means. Unsharp masking based on the shooting scene set by the shooting scene setting means and the unsharp masking processing means for performing edge enhancement processing on the second image obtained by the ringing removal means Control means for controlling at least one of an edge emphasis width and an edge emphasis degree by the processing means is provided.

According to an eighth aspect of the present invention, in the imaging apparatus according to the seventh aspect , the unsharp masking processing unit generates a smoothed image of the second image using a Gaussian filter, and the second image and the second image An edge strength image is generated by multiplying the difference from the smoothed image by a correction strength parameter, and an edge strength image is obtained by adding the edge strength image to the second image. Based on the photographic scene detected by the photographic scene setting means, a parameter representing dispersion in a two-dimensional Gaussian function used to generate a smoothed image, and the correction intensity parameter used to generate an edge enhanced image And at least one of them is controlled.

  According to the present invention, side effects due to camera shake correction can be efficiently reduced regardless of the shooting scene.

  Hereinafter, embodiments of the present invention applied to a digital camera will be described with reference to the drawings.

The first embodiment will be described below.
[1] Configuration of Camera Shake Correction Processing Circuit FIG. 1 shows a configuration of the camera shake correction processing circuit provided in the digital camera.

  Reference numerals 1a and 1b denote angular velocity sensors for detecting the angular velocity. One angular velocity sensor 1a detects the angular velocity in the pan direction of the camera, and the other angular velocity sensor 1b detects the angular velocity in the tilt direction of the camera. Reference numeral 2 denotes an image restoration filter calculation unit that calculates the coefficient of the image restoration filter based on the biaxial angular velocities detected by the angular velocity sensors 1a and 1b. Reference numeral 3 denotes an image restoration processing unit that performs image restoration processing on a captured image (camera shake image) based on the coefficients calculated by the image restoration filter calculation unit 2.

  Reference numeral 4 denotes a shooting scene detection unit that automatically detects a shooting scene. Reference numeral 5 denotes a correction intensity parameter control unit for changing the correction intensity parameter used in the image restoration filter calculation unit 2 in accordance with the shooting scene detected by the shooting scene detection unit 4.

  In this embodiment, the “camera shake correcting means” described in the claims is based on the speed sensors 1a and 1b, the image restoration filter calculation unit 2, and the filter processing units 31 and 32 in the image restoration processing unit 3. It is configured. That is, the restored image (v_fukugen) restored by the image restoration filter (inverse filter) is an image obtained by the “camera shake correcting unit”.

  Hereinafter, the image restoration filter calculation unit 2, the image restoration processing unit 3, the shooting scene detection unit 4, and the correction intensity parameter control unit 5 will be described.

[2] Description of Image Restoration Filter Calculation Unit 2 The image restoration filter calculation unit 2 includes a camera shake signal / motion vector conversion processing unit 21 that converts angular velocity data (camera shake signals) detected by the angular velocity sensors 1a and 1b into motion vectors. A motion vector / shake function conversion processing unit 22 and a motion vector / shake function conversion for converting the motion vector obtained by the hand shake signal / motion vector conversion processing unit 21 into a hand shake function (PSF: Point Spread Function) representing blur of an image. A camera shake function / general inverse filter conversion processing unit 23 that converts the camera shake function obtained by the processing unit 22 into a general inverse filter (image restoration filter) is provided.

[2-1] Description of Camera Shake Signal / Motion Vector Conversion Processing Unit 21 The original data of camera shake is output data from the angular velocity sensors 1a and 1b from the start of shooting to the end of shooting. By synchronizing with the exposure time of the camera using the angular velocity sensors 1a and 1b, the angular velocity in the pan direction and the tilt direction is measured at a predetermined sampling interval dt [sec] at the start of photographing, and data until the end of photographing is obtained. The sampling interval dt [sec] is, for example, 1 msec.

As shown in FIG. 2, for example, the angular velocity θ ′ [deg / sec] in the pan direction of the camera is amplified by the amplifier 101 after being converted into the voltage V _g [mV] by the angular velocity sensor 1 a. Voltage V _a [mV] output from the amplifier 101 Is converted to a digital value D _L [step] by the A / D converter 102. In order to convert the data obtained as a digital value into an angular velocity, calculation is performed using sensor sensitivity S [mV / deg / sec], amplifier magnification K [times], and A / D conversion coefficient L [mV / step]. An amplifier and an A / D converter are provided for each angular velocity sensor 1a, 1b. These amplifiers and A / D converters are provided in the camera shake signal / motion vector conversion processing unit 21.

The voltage value V _g [mV] obtained by the angular velocity sensor 1a is proportional to the value of the angular velocity θ ′ [deg / sec]. Since the proportionality constant at this time is sensor sensitivity, V _g [mV] is expressed by the following equation (1).

V _g = Sθ ′ (1)

Since the amplifier 101 only amplifies the voltage value, the amplified voltage V _a [mV] Is represented by the following equation (2).

V _a = KV _g (2)

Voltage value V _a [mV] amplified by the amplifier 101 Is A / D converted and expressed using a digital value D _L [step] of n [step] (for example, −512 to 512). When the A / D conversion coefficient is L [mV / step], the digital value D _L [step] is expressed by the following equation (3).

D _L = V _a / L (3)

  By using the above equations (1) to (3), the angular velocity can be obtained from the sensor data as shown in the following equation (4).

θ ′ = (L / KS) D _L (4)

  It is possible to calculate how much blur has occurred on the photographed image from the angular velocity data being photographed. This apparent motion on the image is called a motion vector.

  The rotation amount generated in the camera from one sample value to the next sample value of the angular velocity data is defined as θ [deg]. During this time, assuming that the camera rotates at a constant angular velocity and the sampling frequency is f = 1 / dt [Hz], θ [deg] is expressed by the following equation (5).

θ = θ ′ / f = (L / KSf) D _L (5)

  As shown in FIG. 3, when r [mm] is a focal length (35 [mm] film equivalent), the moving amount d [mm] on the screen is calculated from the rotation amount θ [deg] of the camera by the following equation (6). Desired.

  d = rtan θ (6)

  The amount of movement d [mm] obtained here is the size of camera shake when converted to 35 [mm] film, and its unit is [mm]. In actual calculation processing, the size of the image must be considered in the unit [pixel] of the image size of the digital camera.

  Since the 35 [mm] film equivalent image and the [pixel] unit image taken with the digital camera have different aspect ratios, the calculation is performed as follows. As shown in FIG. 4, at the time of 35 [mm] film conversion, the horizontal size and vertical size of the image size are determined to be 36 [mm] × 24 [mm]. If the size of an image taken with a digital camera is X [pixel] x Y [pixel], the horizontal blur (pan direction) is x [pixel] and the vertical blur (tilt direction) is y [pixel]. The conversion equations are the following equations (7) and (8).

x = d _x (X / 36) = rtan θ _x (X / 36) (7)
y = d _y (Y / 24) = r tan θ _y (Y / 24) (8)

  In the above formulas (7) and (8), the subscripts x and y are used for d and θ. The subscript x is a horizontal value, and the subscript y is a vertical value. Is shown.

  Summarizing the above formulas (1) to (8), the horizontal direction (pan direction) blur x [pixel] and the vertical direction (tilt direction) blur y [pixel] are expressed by the following formulas (9) and (10). expressed.

x = rtan {(L / KSf) D _Lx } X / 36 (9)
y = rtan {(L / KSf) D _Ly } Y / 24 (10)

  By using the conversion equations (9) and (10), it is possible to determine the amount of motion blur (motion vector) from the angular velocity data of each axis of the camera obtained as a digital value.

  Motion vectors during shooting can be obtained by the number of angular velocity data obtained from the sensor (by the number of sample points). When the start point and end point are connected in order, the camera shake trajectory on the image become. Also, by looking at the size of each vector, the speed of camera shake at that time can be determined.

[2-2] Motion Vector / Shake Function Conversion Processing Unit 22 Hand shake can be expressed using a spatial filter. A spatial filter process is performed by adding a weight to the operator's element in accordance with the hand movement locus shown in the left diagram of FIG. 5 (the locus drawn by one point on the image when the camera is shaken, the amount of image blur). In the filtering process, since the gray value of the pixel takes into account only the gray value of the neighboring pixel corresponding to the locus of the camera shake, it is possible to create a camera shake image.

  The operator weighted according to this trajectory is called Point Spread Function (PSF) and used as a mathematical model for camera shake. The weight of each element of the PSF is a value proportional to the time during which the hand movement trajectory passes through the element, and is a value normalized so that the sum of the weights of each element is 1. That is, the weight is proportional to the inverse of the magnitude of the motion vector. This is because, when considering the effect of camera shake on the image, the slower moving part has a greater effect on the image.

  The center diagram in FIG. 5 represents the PSF when the motion of the camera shake is assumed to be constant, and the diagram on the right side of FIG. 5 represents the PSF when the magnitude of the motion of the actual camera shake is considered. Yes. In the diagram on the right side of FIG. 5, elements with low PSF weight (large motion vector magnitude) are displayed in black, and elements with high weight (small motion vector magnitude) are displayed in white.

  The motion vector (image blurring amount) obtained in [2-1] above has a camera shake trajectory and a trajectory speed as data.

  In order to create a PSF, first, an element to which a PSF weight is applied is determined from the locus of camera shake. Then, the weight applied to the element of the PSF is determined from the speed of camera shake.

  By connecting the series of motion vectors obtained in [2-1] above, a hand movement locus approximated by a polygonal line is obtained. This trajectory has a precision below the decimal point, but by converting it into an integer, an element to be weighted in the PSF is determined. To this end, in this embodiment, elements to be weighted in the PSF are determined using Bresenham's straight line drawing algorithm. Bresenham's straight line drawing algorithm is an algorithm that selects the optimal dot position when it is desired to draw a straight line passing through two arbitrary points on a digital screen.

  The Bresenham straight line drawing algorithm will be described with reference to the example of FIG. In FIG. 6, a straight line with an arrow indicates a motion vector.

(A) Starting from the origin (0, 0) of the dot position, the horizontal element of the motion vector is increased by one.
(B) The vertical position of the motion vector is confirmed, and when the vertical position is larger than 1 compared to the vertical position of the previous dot, the vertical direction of the dot position is increased by one.
(C) The horizontal element of the motion vector is increased by one again.

  By repeating such processing to the end point of the motion vector, a straight line through which the motion vector passes can be expressed by the dot position.

  The weight applied to the elements of the PSF is determined by utilizing the fact that the vector size (speed component) is different for each motion vector. The weight is the reciprocal of the magnitude of the motion vector, and the weight is substituted into the element corresponding to each motion vector. However, the weights of the elements are normalized so that the sum of the weights of the elements becomes 1. FIG. 7 shows the PSF obtained from the motion vector of FIG. Where the speed is high (where the motion vector is long), the weight is small, and where the speed is low (where the motion vector is short), the weight is large.

[2-3] Camera Shake Function / General Inverse Filter Conversion Processing Unit 23 It is assumed that an image is digitized with a resolution of N _x pixels in the horizontal direction and N _y pixels in the vertical direction. The value of the pixel at the i-th position in the horizontal direction and the j-th position in the vertical direction is represented by p (i, j). The image conversion by the spatial filter is to model the conversion by convolution of neighboring pixels of the target pixel. Let the convolution coefficient be h (l, m). Here, for the sake of simplicity, if −n <l and m <n, the conversion of the pixel of interest can be expressed by the following equation (11). Further, h (l, m) itself is called a spatial filter or a filter coefficient. The nature of the conversion is determined by the coefficient value of h (l, m).

  When a point light source is observed with an imaging device such as a digital camera, assuming that there is no deterioration in the image formation process, the image observed on the image has a pixel value other than 0, and other than that, The pixel value of becomes zero. Since an actual imaging device includes a deterioration process, even if a point light source is observed, the image is not a single point but a widened image. When camera shake occurs, the point light source generates a locus on the screen according to the camera shake.

  A spatial filter having a coefficient proportional to the pixel value of the observation image with respect to the point light source and the sum of the coefficient values being 1 is called a point spread function (PSF). In this embodiment, the PSF obtained by the motion vector / camera shake function conversion processing unit 22 is used as the PSF.

  When the PSF is modeled by a vertical and horizontal (2n + 1) × (2n + 1) spatial filter h (l, m), −n <l, m <n, the pixel value p (i, j of the image without blur for each pixel. ) And the pixel value p ′ (i, j) of the blurred image have the relationship of the above equation (11). Here, what can actually be observed is the pixel value p ′ (i, j) of the blurred image, and the pixel value p (i, j) of the image without blur needs to be calculated by some method.

  When the above equation (11) is written for all the pixels, the following equation (12) is obtained.

  These equations can be collectively expressed as a matrix, and the following equation (13) is obtained. Here, P is a unified original image in the raster scan order.

  P ′ = H × P (13)

If there is an inverse matrix H ^{−1 of} H, it is possible to obtain a non-degraded image P from the degraded image P ′ by calculating P = H ⁻¹ × P. There is no matrix. In contrast to a matrix that does not have an inverse matrix, there is a so-called general inverse matrix or pseudo inverse matrix. The following equation (14) shows an example of a general inverse matrix.

H ^* = (H ^t · H + γ · I) ⁻¹ · H ^t (14)

Here, H ^* is a general inverse matrix of H, H ^t is a transposed matrix of H, γ is a scalar, and I is a unit matrix having the same size as H ^t · H. By calculating the following equation (15) using H ^* , it is possible to obtain an image P in which camera shake is corrected from the observed camera shake image P ′. γ is a regularization coefficient (correction intensity parameter) for adjusting the intensity of correction. If γ is small, strong correction processing is performed, and if γ is large, weak correction processing is performed. In the first embodiment, as will be described later, the value of the correction intensity parameter γ is determined according to the imaging scene.

P ′ = H ^* × P (15)

When the image size is 640 × 480, P in the above equation (15) is a 307,200 × 1 matrix, and H ^* is a 307,200 × 307,200 matrix. Since such a very large matrix is used, it is not practical to directly use the equations (14) and (15). Therefore, the size of the matrix used for the calculation is reduced by the following method.

First, in the above equation (15), the size of the image that is the source of P is set to a comparatively small size such as 63 × 63. In the case of a 63 × 63 image, P is a 3969 × 1 matrix and H ^* is a 3969 × 3969 matrix. H ^* is a matrix for converting the entire blurred image into the corrected image, and the product of each row of H ^* and P corresponds to an operation for correcting each pixel. The product of the middle row of H ^* and P corresponds to the correction of the middle pixel of the original image of 63 × 63 pixels. Since P is an original image that is unified in the order of raster scanning, conversely, a spatial filter having a size of 63 × 63 can be configured by two-dimensionalizing the middle row of H ^* by reverse raster scanning. . The spatial filter configured in this way is called a general inverse filter (hereinafter referred to as an image restoration filter).

  The blurred image can be corrected by sequentially applying the spatial filter of the practical size created in this way to each pixel of the entire large image. The blur image restoration filter obtained by the above procedure also has a parameter for adjusting the restoration strength represented by γ.

[3] Image Restoration Processing Unit 3 As shown in FIG. 1, the image restoration processing unit 3 includes a filter processing unit 31, a filter processing unit 32, a ringing removal processing unit 33, and an unsharp masking processing unit 34. The filter processing unit 31 performs filter processing using a median filter. The filter processing unit 32 performs filter processing using the image restoration filter obtained by the image restoration filter calculation unit 2.

  The camera shake image photographed by the camera is sent to the filter processing unit 31, where filter processing using a median filter is performed, and noise is removed. The image obtained by the filter processing unit 31 is sent to the filter processing unit 32. The filter processing unit 32 performs filter processing using the image restoration filter generated by the camera shake function / general inverse filter conversion processing unit 23, and restores an image free from camera shake from the camera shake image. The image obtained by the filter processing unit 32 is sent to the ringing removal processing unit 33, where the ringing is removed. The image obtained by the ringing removal processing unit 33 is sent to the unsharp masking processing unit 34, and processing for enhancing the edges is performed.

[3-1] Description of Ringing Removal Processing Unit 33 The ringing removal processing unit 33 includes an edge strength calculation unit 61, a weighted average coefficient calculation unit 62, and a weighted average processing unit 63 as shown in FIG.

  The camera shake image (v_tebre) photographed by the camera is sent to the edge strength calculation unit 61, and the edge strength is calculated for each pixel. A method for obtaining the edge strength will be described.

  As shown in FIG. 9, a 3 × 3 region centered on the target pixel v22 is assumed. A horizontal edge component dh and a vertical edge component dv are calculated for the target pixel v22. For the calculation of the edge component, for example, a Prewitt edge extraction operator shown in FIG. 10 is used. FIG. 10A shows a horizontal edge extraction operator, and FIG. 10B shows a vertical edge extraction operator.

  The horizontal edge component dh and the vertical edge component dv are obtained by the following equations (16) and (17).

dh = v11 + v12 + v13 −v31 −v32 −v33 (16)
dv = v11 + v21 + v31−v13−v23−v33 (17)

  Next, the edge strength v_edge of the target pixel v22 is calculated from the horizontal edge component dh and the vertical edge component dv based on the following equation (18).

  v _edge = sqrt (dh × dh + dv × dv) (18)

  Note that abs (dh) + abs (dv) may be used as the edge strength v_edge of the target pixel v22. Further, a 3 × 3 noise removal filter may be further applied to the edge intensity image obtained in this way.

  The edge strength v_edge of each pixel calculated by the edge strength calculation unit 61 is given to the weighted average coefficient calculation unit 62. The weighted average coefficient calculation unit 62 calculates the weighted average coefficient k of each pixel based on the following equation (19).

If v _edge> th then k = 1
If v_edge ≦ th then k = v_edge / th (19)

 th is a threshold value for determining that the edge is sufficiently strong. That is, the relationship between v_edge and the weighted average coefficient k is as shown in FIG.

  The weighted average coefficient k of each pixel calculated by the weighted average coefficient calculating unit 62 is given to the weighted average processing unit 63. When the pixel value of the restored image obtained by the filter processing unit 32 is v_fukugen and the pixel value of the camera shake image captured by the camera is v_tebre, the weighted average processing unit 62 is expressed by the following equation (20). By performing the calculation, the pixel value v_fukugen of the restored image and the pixel value v_tebre of the camera shake image are weighted averaged.

  v = k * v_fukugen + (1-k) * v_tebre (20)

  That is, for the pixel having the edge strength v_edge larger than the threshold value th, the ringing of the restored image corresponding to the position is inconspicuous, and thus the pixel value v_fukugen of the restored image obtained by the image restoration processing unit 3 is output as it is. . For pixels whose edge strength v_edge is less than or equal to the threshold th, the smaller the edge strength v_edge, the more conspicuous the ringing of the restored image, so that the degree of the restored image is weakened and the degree of the camera shake image is increased.

[3-2] Description of Unsharp Masking Processing Unit 34

  Unsharp masking is an image processing technique that performs edge enhancement. An image sent from the ringing removal processing unit 33 to the unsharp masking processing unit 34 is referred to as an original image. The unsharp masking processing unit 34 first generates a smoothed image by smoothing the original image using a Gaussian filter, takes a difference between the original image and the smoothed image, and multiplies the difference by a correction strength parameter. Thus, an edge strength image is obtained. Then, an edge-enhanced image is obtained by adding the edge strength image to the original image.

  If the original image is I and the image obtained by the unsharp masking process is I ′, I ′ is expressed by the following equation (21).

  I '= I + K (IG (I)) (21)

  In the above equation (21), G (I) is a smoothed image of the original image, and K is a correction intensity parameter.

  There are two types of parameters required when performing unsharp masking processing. One of them is the correction intensity parameter K of the above equation (21), which is a parameter for adjusting the degree of edge enhancement. The other is a parameter used when smoothing using a Gaussian filter.

An image G (I) smoothed using a Gaussian filter is a two-dimensional Gaussian expressed by the following equation (22), where the average is 0 and the variance is σ ² with respect to the original image I (x, y). It is calculated by folding the function.

  That is, G (I) is expressed by the following equation (23).

  In the above formula (23), the symbol * represents convolution. A parameter σ representing dispersion is an adjustment parameter.

  This parameter σ indicates the edge enhancement width. FIG. 12 shows the difference in edge enhancement effect when σ is small and when σ is large. In FIG. 12, a one-dimensional signal is used for easy understanding.

  A indicates an edge signal (original signal). B indicates a signal obtained by smoothing the edge signal A. C represents an edge intensity signal obtained by taking the difference (A−B) between the edge signal A and the smoothed signal B. D indicates a signal obtained by adding the edge strength signal C to the edge signal A.

  When σ is reduced, the edge enhancement width a appearing in the signal D in FIG. 12 is reduced. When the edge emphasis width a is narrow, a satisfactory emphasis effect can be obtained for a steep edge with a narrow edge width. However, for a gentle edge with a wide edge width, there is not enough width to be emphasized with respect to the edge width of the original signal, so that the obtained image is an image with a weak enhancement effect.

  When σ is increased, the edge emphasis width a is increased. When the edge emphasis width a is wide, a satisfactory emphasis effect is obtained for a gentle edge with a wide edge width, contrary to the case where σ is small. However, for a steep edge with a narrow edge width, the width to be emphasized is wide with respect to the edge width of the original signal, and thus the obtained image becomes an unnatural image.

  The parameter K is a parameter for adjusting the height b of the edge emphasis portion appearing in the signal C in FIG. When K is increased, the edge is enhanced more, but noise included in the image is also enhanced. When K is decreased, the edge enhancement effect is decreased.

[4] Description of Shooting Scene Detection Unit 4 The shooting scene detection unit 4 automatically detects a shooting scene. As a method for automatically detecting a photographic scene, for example, a technique disclosed in Japanese Patent Laid-Open No. 2003-274280 can be used.

  Specifically, the shooting scene is determined based on the determination criteria shown in Table 1.

  The types of shooting scenes to be identified include “portrait”, “far view”, “night view”, and “sport”. The determination parameters include subject distance, focal length, luminance, light source, and moving object. The subject distance represents the distance to the subject and is obtained based on the autofocus control value. The focal length is a zoom magnification and is obtained based on the zoom magnification value. The brightness represents the brightness of the image, and it is determined whether the image is bright or dark based on the brightness value of the image. The light source represents the color of the light source and is determined based on the white balance evaluation value. The moving object indicates whether the subject is moving or stationary, and is determined based on a feature point or a color change (movement).

  Hereinafter, the reason why the determination criteria are set as shown in Table 1 for each shooting scene will be described.

(1) Portrait When shooting a person, the subject distance is shorter (closer) than in other shooting scenes. In addition, since a close subject is often photographed by slightly telephoto, the focal length setting is slightly increased during shooting. In addition, there is a high probability that shooting is performed in a bright place, and there is a high probability that the light source is daylight color or room light (illumination light close to natural light). Also, the subject often stops.

(2) Distant view When photographing a distant view, the subject distance becomes large (far). In addition, since the entire image is often shot on the wide-angle side, the focal length setting is reduced during shooting. In addition, photographing is performed outdoors in a bright environment, and there is a high probability that the light source is a daylight color. Also, the subject often stops.

(3) Night view When shooting a night view, the subject distance becomes large (far). In addition, since the entire image is often shot on the wide-angle side, the focal length setting is reduced during shooting. In addition, photographing is performed in a dark state, and the light source has a lot of illumination light from an instrument such as a fluorescent lamp, a neon tube, or a sodium tube.

(4) Sports When shooting a moving subject, the subject distance often becomes large (far). Furthermore, since the subject that is moving at high speed is often captured from a distance in order to fit the subject within the screen, the setting of the focal length becomes large at the time of shooting. Further, photographing is performed in a bright place, and there is a high probability that the light source is daylight color or room light (illumination light close to natural light). The subject is moving.

[5] Explanation of the correction intensity parameter control unit 5

  The correction intensity parameter control unit 5 determines a correction intensity parameter (regularization) used in the camera shake function / general inverse filter conversion processing unit 23 in the image restoration filter calculation unit 2 according to the shooting scene detected by the shooting scene detection unit 4. Coefficient) controls the value of γ. As described above, when γ is small, strong camera shake correction processing is performed, and when γ is large, weak camera shake correction processing is performed.

  Table 2 shows the relationship between the shooting scene detected by the shooting scene detection unit 4 and the value of the correction intensity parameter γ determined by the correction intensity parameter control unit 5.

  The correction intensity parameter control unit 5 sets γ to a standard value when the shooting scene detected by the shooting scene detection unit 4 is a distant view.

  When the shooting scene detected by the shooting scene detection unit 4 is a night scene, the correction intensity parameter control unit 5 sets γ to a value larger than the standard value in order to weaken camera shake correction. This is because, when the shooting scene is a night view, there is a high probability of shooting an image with a small amount of light and significant noise, so increasing the camera shake correction increases the side effects, so it is preferable to reduce the camera shake correction. It is.

  When the shooting scene detected by the shooting scene detection unit 4 is a portrait, the correction strength parameter control unit 5 sets γ to a value smaller than the standard value in order to increase the camera shake correction effect. This is because, when the shooting scene is a portrait, there is a high probability of shooting a subject with few edge components, so even if the camera shake correction intensity is increased, the side effects are small, so it is preferable to increase the camera shake correction effect. Because.

  When the shooting scene detected by the shooting scene detection unit 4 is a sport, the correction intensity parameter control unit 5 does not perform camera shake correction because there is a high probability that a subject shake different from camera shake occurs. In this case, in FIG. 1, after the input image (v_tebre) is filtered by the filter processing unit 31, unsharp masking processing is performed by the unsharp masking processing unit 34, and the obtained image is output as a restored image. .

  The second embodiment will be described below.

  FIG. 13 shows a configuration of a camera shake correction processing circuit provided in the digital camera. In FIG. 13, the same components as those in FIG.

  In the camera shake correction processing circuit of FIG. 13, the threshold th used in the weighted average coefficient calculation unit 62 in the ringing removal processing unit 33 instead of the correction intensity parameter control unit 5 of the camera shake correction processing circuit of FIG. 1 (see FIG. 11). A threshold control unit 5A for controlling the above is used.

  The threshold control unit 5A controls the threshold th used by the weighted average coefficient calculation unit 62 in the ringing removal processing unit 33 according to the shooting scene detected by the shooting scene detection unit 4.

  K shown in FIG. 11 indicates a mixture ratio (weighted average coefficient) between the pixel value v_fukugen of the restored image and the pixel value v_tebre of the camera shake image. As k increases, the ratio of the pixel value v_fukugen of the restored image increases. k increases in proportion to the edge strength v_edge when the edge strength v_edge is less than or equal to the threshold th. K is 1 when the edge strength v_edge is equal to or greater than the threshold th.

  For pixels whose edge strength v_edge is greater than the threshold th, ringing of the restored image corresponding to the position is not noticeable, and thus the pixel value v_fukugen of the restored image obtained by the image restoration processing unit 3 is output as it is. For pixels whose edge strength v_edge is less than or equal to the threshold th, the smaller the edge strength v_edge, the more conspicuous the ringing of the restored image, so that the degree of the restored image is weakened and the degree of the camera shake image is increased.

  When the threshold th is increased, the ratio of the image blur image v_tebre increases, so that the camera shake correction effect is reduced, but the ringing removal effect is increased. Further, noise amplified by camera shake correction is reduced. Note that when the camera shake is large, ringing is likely to occur. Therefore, it is preferable to increase the threshold th in order to improve the ringing removal effect.

  Conversely, when the threshold th is reduced, the ratio of the restored image v_fukugen increases, so that the camera shake correction effect is increased, but the ringing removal effect is decreased. Further, the noise amplified by the camera shake correction becomes high.

  Table 3 shows the relationship between the shooting scene detected by the shooting scene detection unit 4 and the threshold value th determined by the threshold control unit 5A.

  When the shooting scene detected by the shooting scene detection unit 4 is a night scene, the threshold control unit 5A sets the threshold th to a value larger than the standard value. When the shooting scene is a night view, the probability of shooting an image with a small amount of light and significant noise is high. In addition, since the exposure time is long because the amount of light is small, there is a high probability that camera shake will increase. That is, when the camera shake correction effect is enhanced, ringing and noise are likely to occur. Therefore, in order to increase the ringing removal effect and the noise removal effect, the threshold value th is set to a value larger than the standard value.

  The threshold value control unit 5A sets the threshold value th to a value smaller than the standard value when the shooting scene detected by the shooting scene detection unit 4 is a distant view. When the shooting scene is a distant view, ringing is unlikely to occur because the probability of large camera shake is low. Also, since shooting is performed in a bright place, it is considered that there is little noise. In other words, even if the camera shake correction effect is enhanced, ringing and noise are less likely to occur, so the threshold th is set to a value smaller than the standard value.

  The threshold value control unit 5A sets the threshold value th to a standard value when the shooting scene detected by the shooting scene detection unit 4 is a portrait. When the shooting scene is a portrait, the focal length is slightly larger than in the distant view mode (see Table 1). For this reason, camera shake is more likely to occur than in the distant view mode. However, camera shake is unlikely to occur compared to night views. Therefore, the threshold value th is set to a standard value that is larger than that for a distant view and smaller than that for a night view.

  When the shooting scene detected by the shooting scene detection unit 4 is a sport, the threshold control unit 5A does not perform camera shake correction because there is a high probability that a subject shake different from camera shake will occur. For this reason, ringing removal processing is not performed.

  The third embodiment will be described below.

  FIG. 14 shows a configuration of a camera shake correction processing circuit provided in the digital camera. In FIG. 14, the same components as those in FIG.

  In the camera shake correction processing circuit of FIG. 14, a parameter control unit 5B for controlling the parameters σ and K used in the unsharp masking processing unit 34 is provided instead of the correction intensity parameter control unit 5 of the camera shake correction processing circuit of FIG. It is used.

  The parameter control unit 5B controls the parameters σ and K used in the unsharp masking processing unit 34 according to the shooting scene detected by the shooting scene detection unit 4.

  As described above, it is preferable to decrease σ for steep edges having a narrow edge width, and it is preferable to increase σ for gentle edges having a wide edge width. Further, when K is increased, the edge is enhanced more, but noise included in the image is enhanced, and when K is decreased, the edge enhancement effect is decreased.

  Table 4 shows the relationship between the shooting scene detected by the shooting scene detection unit 4 and the values of the parameters σ and K determined by the parameter control unit 5B.

  When the photographic scene detected by the photographic scene detection unit 4 is a portrait, the parameter control unit 5B sets σ to a value larger than the standard value and sets K to a value larger than the standard value. When the shooting scene is a portrait, since a person is mainly shot, the probability of shooting an image with a gentle edge is high, so σ is set to a value larger than the standard value. Also, since shooting is performed in a bright place, noise is unlikely to occur, so K is set to a value larger than the standard value.

  When the photographic scene detected by the photographic scene detection unit 4 is a distant view, the parameter control unit 5B sets σ to a value smaller than the standard value and sets K to a value larger than the standard value. When the shooting scene is a distant view, σ is set to a value smaller than the standard value because there is a higher probability of shooting an image with a relatively steep edge such as a building or a vehicle compared to a portrait. Also, since shooting is performed in a bright place, noise is unlikely to occur, so K is set to a value larger than the standard value.

  When the photographic scene detected by the photographic scene detection unit 4 is a night scene, the parameter control unit 5B sets σ to a value smaller than the standard value and sets K to a value smaller than the standard value. When the shooting scene is a night view, the probability of shooting an image with a relatively steep edge such as a building or a vehicle is higher than that of a portrait, so σ is set to a value smaller than the standard value. Also, since shooting is performed in a dark place, noise is likely to occur, so K is set to a value smaller than the standard value.

  When the photographic scene detected by the photographic scene detection unit 4 is a sport, the parameter control unit 5B sets σ to a standard value and sets K to a value smaller than the standard value. When the shooting scene is a sport, the subject is blurred, and both a sharp edge and a gentle edge can be formed, so σ is set to a standard value. In addition, since the subject is usually photographed, the shutter speed is set to a fast value at the time of photographing, the amount of light is reduced, and noise is likely to occur at the time of photographing. Therefore, K is set to a value smaller than the standard value. .

  Note that only one of σ and K may be controlled.

  In the first to third embodiments, the photographic scene is automatically detected by the photographic scene detection means 4, but setting means for allowing the user to set the photographic scene may be provided in the camera. . In this case, the correction intensity parameter γ, the threshold th, and the parameters σ and K are controlled based on the shooting scene set by the setting unit.

1 is a block diagram illustrating a configuration of a camera shake correction processing circuit according to a first embodiment. FIG. It is a block diagram which shows the amplifier which amplifies the output of the angular velocity sensor 1a, and the A / converter which converts an amplifier output into a digital value. It is a schematic diagram which shows the relationship between the rotation amount (theta) [deg] of a camera, and the moving amount d [mm] on a screen. It is a schematic diagram which shows the image size of 35 [mm] film conversion, and the image size of a digital camera. It is a schematic diagram which shows the spatial filter (PSF) expressing camera shake. It is a schematic diagram for demonstrating Bresenham's straight line drawing algorithm. It is a schematic diagram which shows PSF obtained by the motion vector of FIG. FIG. 3 is a block diagram illustrating a configuration of a ringing removal processing unit 33 in FIG. 1. It is a schematic diagram which shows the 3 * 3 area | region centering on the attention pixel v22. It is a schematic diagram which shows the edge extraction operator of Prewitt. It is a graph which shows the relationship between edge strength v_edge and the weighted average coefficient k. FIG. 12 is a waveform diagram for showing a difference in effect when σ is small and when σ is large. It is a block diagram which shows the structure of the camera-shake correction process circuit which is 2nd Example. It is a block diagram which shows the structure of the camera-shake correction process circuit which is 3rd Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1a, 1b Angular velocity sensor 2 Image restoration filter calculation part 3 Image restoration process part 4 Shooting scene detection part 5 Correction intensity parameter control part 5A Threshold control part 5B Parameter control part 21 Shake signal / motion vector conversion process part 22 Motion vector / shake function Conversion processing unit 23 Camera shake function / general inverse filter conversion processing unit 31 Filter processing unit 32 Filter processing unit 33 Ringing removal processing unit 34 Unsharp masking processing unit 61 Edge strength calculation unit 62 Weighted average coefficient calculation unit 63 Weighted average processing unit

Claims (8)

Shooting scene detection means for automatically detecting shooting scenes,
Camera shake correction means for performing camera shake correction processing on the input image, and control means for controlling the intensity of camera shake correction by the camera shake correction means based on the shooting scene detected by the shooting scene detection means;
Bei to give a,
The camera shake correction means performs camera shake correction using an inverse filter, and the control means controls a regularization coefficient used when generating the inverse filter based on the shooting scene detected by the shooting scene detection means. imaging device, characterized in that it. Shooting scene detection means for automatically detecting shooting scenes,
Image stabilization means for performing image stabilization processing on the input image;
Ringing removal means for removing ringing from the restored image obtained by the image stabilization means, and
Control means for controlling the intensity of ringing removal by the ringing removal means based on the shooting scene detected by the shooting scene detection means;
With
The ringing removing means includes an edge strength calculating means for calculating edge strength for each pixel of the input image before camera shake correction, a relationship between the edge strength for each pixel calculated by the edge strength calculating means, and a weighted addition coefficient for the edge strength. Based on the above, the coefficient calculation means for obtaining a weighted addition coefficient for each pixel, and the weighted addition coefficient for each pixel using the weighted addition coefficient obtained for each image, the input image and the restored image obtained by the camera shake correction means Weighted average means to
The edge strength and the weighted addition coefficient have such a relationship that the weighted ratio of the restored image increases as the edge strength increases when the edge strength is less than or equal to the threshold, and the weighted ratio of the restored image becomes 100% when the edge strength exceeds the threshold. Yes,
An image pickup apparatus characterized in that the control means controls the threshold based on the shooting scene detected by the shooting scene detection means. Shooting scene detection means for automatically detecting shooting scenes,
Image stabilization means for performing image stabilization processing on the input image;
Ringing removal means for removing ringing from the first image obtained by the image stabilization means;
Unsharp masking processing means for performing edge enhancement processing on the second image obtained by the ringing removal means, and
Control means for controlling at least one of the edge enhancement width and the edge enhancement degree by the unsharp masking processing means based on the photographing scene detected by the photographing scene detection means;
An imaging apparatus comprising: The unsharp masking processing means generates a smoothed image of the second image using a Gaussian filter, and multiplies the difference between the smoothed image of the second image and the second image by the correction strength parameter to generate the edge strength image. Generating and adding an edge strength image to the second image to obtain an image with enhanced edges,
The control means is used to generate a parameter representing dispersion in a two-dimensional Gaussian function used for generating a smoothed image and an edge enhanced image based on the shooting scene detected by the shooting scene detection means. The imaging apparatus according to claim 3, wherein at least one of the correction intensity parameters is controlled. Shooting scene setting means for setting a shooting scene;
Image stabilization means for performing image stabilization processing on the input image, and
Control means for controlling the strength of camera shake correction by the camera shake correction means based on the shooting scene set by the shooting scene setting means;
With
The camera shake correction means performs camera shake correction using an inverse filter, and the control means controls a regularization coefficient used when generating the inverse filter based on the shooting scene set by the shooting scene setting means. An imaging device characterized by being: Shooting scene setting means for setting a shooting scene;
Image stabilization means for performing image stabilization processing on the input image;
Ringing removal means for removing ringing from the restored image obtained by the image stabilization means, and
Control means for controlling the intensity of ringing removal by the ringing removal means based on the shooting scene set by the shooting scene setting means;
With
The ringing removing means includes an edge strength calculating means for calculating edge strength for each pixel of the input image before camera shake correction, a relationship between the edge strength for each pixel calculated by the edge strength calculating means, and a weighted addition coefficient for the edge strength. Based on the above, the coefficient calculation means for obtaining a weighted addition coefficient for each pixel, and the weighted addition coefficient for each pixel using the weighted addition coefficient obtained for each image, the input image and the restored image obtained by the camera shake correction means Weighted average means to
The edge strength and the weighted addition coefficient have such a relationship that the weighted ratio of the restored image increases as the edge strength increases when the edge strength is less than or equal to the threshold, and the weighted ratio of the restored image becomes 100% when the edge strength exceeds the threshold. Yes,
An imaging apparatus characterized in that the control means controls the threshold value based on the shooting scene set by the shooting scene setting means. Shooting scene setting means for setting a shooting scene;
Image stabilization means for performing image stabilization processing on the input image;
Ringing removal means for removing ringing from the first image obtained by the image stabilization means;
Unsharp masking processing means for performing edge enhancement processing on the second image obtained by the ringing removal means, and
Control means for controlling at least one of the edge enhancement width and the edge enhancement degree by the unsharp masking processing means based on the photographing scene set by the photographing scene setting means;
An imaging apparatus comprising: The unsharp masking processing means generates a smoothed image of the second image using a Gaussian filter, and multiplies the difference between the smoothed image of the second image and the second image by the correction strength parameter to generate the edge strength image. Generating and adding an edge strength image to the second image to obtain an image with enhanced edges,
The control means is used to generate a parameter representing a variance in a two-dimensional Gaussian function used for generating a smoothed image and an edge enhanced image based on the shooting scene detected by the shooting scene setting means. The imaging apparatus according to claim 7, wherein at least one of the correction intensity parameters is controlled.

Images