JP2015102794A - Compound-eye imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compound-eye imaging apparatus which is thin and has a high variable power ratio and further is advantageous to acquire peripheral space information about a subject space to be imaged.SOLUTION: In a compound-eye imaging apparatus (1) including a plurality of optical systems, the plurality of optical systems include at least a first optical system and a second optical system (101a and 101d) having the widest first angle of view among the plurality of optical systems and a third optical system (101b) having a second angle of view narrower than the first angle of view, and an imaging device (102) having a first imaging area corresponding to the first optical system, a second imaging area corresponding to the second optical system, and a third imaging area corresponding to the third optical system is provided, and the plurality of optical systems are so arranged that base line lengths of the first optical system and the second optical system are longest.

Description

  The present invention relates to a compound eye imaging device in which a plurality of optical systems are arranged.

  Conventionally, a “compound eye” imaging device has been proposed that realizes a compact optical system by dividing the optical system into a plurality of parts. “Composite eye” uses the structure of an insect's eye. For example, a lens array consisting of a plurality of lens units constitutes an optical system, and each lens unit has a smaller diameter and a shorter focal length, thereby reducing the size of the optical system. It has been known. However, the conventional compound-eye imaging device adds an optical zoom function that makes the shooting angle of view variable by a method of moving the position of the lens constituting the optical system because the configuration of the imaging system becomes large. It was difficult. Therefore, for example, Patent Document 1 proposes a configuration in which a short-focus lens unit and a long-focus lens unit having different angles of view are arranged and an image is captured so as to include the same portion of the subject. In other words, by fitting a zoom-up image obtained by an image sensor corresponding to a long focus lens into a part of a wide image obtained by an image sensor corresponding to a short focus lens, the resolution of the part is high, and other Although the resolution of the portion is low, an image with a wide angle of view can be obtained.

  On the other hand, a configuration has been proposed for acquiring peripheral space information on the subject side that has been difficult to acquire with a conventional general imaging system by applying a configuration having a plurality of optical systems (compound eye optical systems). Yes. Here, the peripheral space information on the subject side is a general term for various information for expressing the subject space such as subject distance information, position information, configuration information, light source information, subject spectral characteristics, and scattering characteristics in the subject space. For example, Patent Document 2 proposes a configuration for acquiring each parallax image together with an image with a wide angle of view and an image with a narrow angle of view by adopting a configuration having a pair of short focus lenses and a pair of long focus lenses. Yes. That is, the distance information of the subject can be calculated using the principle of triangulation together with images having different angles of view. The subject distance information is very useful information, for example, when 3D modeling of the subject is performed.

JP 2005-303694 A JP 2009-117976 A

  However, since the conventional techniques disclosed in Patent Documents 1 and 2 each include only an optical system having two different types of field angles, an image with a specific wide field angle and an image with a specific narrow field angle are used. It can only be acquired. Conventional imaging devices such as video cameras and digital cameras are required to have a continuous zoom function in order to capture an image of the angle of view desired by the photographer. There is a problem that the degree of freedom of the angle of view that can be taken is too small.

  Furthermore, regarding acquisition of subject distance information, which is one of the peripheral space information, as described later, it is important to increase the base length in order to improve the accuracy of subject distance information in a compound eye imaging apparatus. There is no description about improving the accuracy of the distance information regarding the baseline length. In addition, there is no description relating to achieving both higher magnification of the continuous zoom function and improvement of distance information acquisition accuracy in a subject space range that can be photographed.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a compound eye imaging apparatus that is advantageous in acquiring peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio.

  A compound eye imaging device according to one aspect of the present invention is a compound eye imaging device including a plurality of optical systems, wherein the plurality of optical systems has a first angle of view that is at least the widest of the plurality of optical systems. First optical system and second optical system, and a third optical system having a second field angle narrower than the first field angle, and corresponding to the first optical system. An imaging device having an imaging region, a second imaging region corresponding to the second optical system, and a third imaging region corresponding to the third optical system, wherein the plurality of optical systems includes: The first optical system and the second optical system are arranged so that the base line length is maximized.

  According to the present invention, it is possible to provide a compound eye imaging device that is advantageous in acquiring peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio.

1 is a configuration diagram of a compound eye imaging apparatus according to Embodiment 1. FIG. It is explanatory drawing of the picked-up image concerning Example 1. FIG. FIG. 6 is a flowchart of continuous zoom function processing according to the first embodiment. It is a distance information calculation flow chart of Example 1. FIG. 6 is a configuration diagram of a compound eye imaging apparatus according to Example 2. FIG. 6 is a configuration diagram of a compound eye imaging apparatus according to Example 3; It is explanatory drawing of the best shot mode by a compound eye imaging device. It is a photographed image explanatory view for the continuous zoom function. It is explanatory drawing of a stereo image imaging | photography model. It is explanatory drawing of a corresponding point extraction method.

  The main viewpoint of the present invention is to improve the problem of easily acquiring advanced peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio in a compound-eye imaging device.

  Here, a method for realizing a continuous zoom function in a compound eye imaging apparatus will be described first. By trimming a part of the photographed image with respect to the image photographed by the image pickup device and expanding the trimmed range to a predetermined size, the same effect as that obtained by pseudo zooming can be obtained (hereinafter referred to as “zoom”). Digital zoom) method is known. Conventionally, it has been known to realize a higher zoom ratio by combining digital zoom and optical zoom. For example, by applying this technique, the compound eye imaging apparatus is provided with an imaging optical system having different angles of view, and the same effect as that obtained by performing pseudo zooming by interpolating between the different angles of view using the digital zoom technique described above. Can be obtained. However, since the conventional digital zoom method uses linear interpolation (bilinear method), an image in the digital zoom region may be deteriorated. The bilinear method is based on interpolation based on a sinc function based on the sampling theorem. In the bilinear method, an interpolation function approximating a sinc function is convoluted with sample points of an original image in order to reduce computational load, thereby interpolating between sample points and increasing or decreasing the number of pixels. In addition, there is an advantage that jaggies are not conspicuous due to the smoothing effect, but the smoothing effect results in a smoothed image centering on the edge portion that does not meet the assumptions at the time of interpolation, for example, and the image is totally blurred It becomes feeling. Furthermore, there is a known problem that the image quality (resolution) of an image decreases as the digital zoom magnification increases. There is a need to improve the reduction in resolution associated with this digital zoom technique and create zooming image data that maintains high resolution. Various techniques for super-resolution techniques have been proposed in the past as techniques for canceling resolution degradation due to digital zoom processing in response to such requirements. Examples of the super-resolution technique include an ML (Maximum-Likelihood) method, a MAP (Maximum A Posterior) method, and a POCS (Projection Onto Convex Set) method. Further, there are an IBP (Iterative Back Projection) method, an LR (Lucy-Richardson) method, and the like. For example, in the LR method, the distribution of illuminance in an original image and the distribution of illuminance in a degraded image are normalized and regarded as a probability density function distribution. Considering the above, the point spread function (PSF), which is the transfer characteristic of the optical system, can be regarded as a distribution of a probability density function that is conditionally established. The distribution of the original image is estimated by iterative calculation using maximum likelihood estimation based on the Bayesian statistics using each of the distribution of the degraded image and the distribution of the PSF.

  Next, an image restoration method based on Bayesian statistics will be described. Hereinafter, a case of a monochrome one-dimensional image will be described for simplification of description. Here, the original image of the subject, the image captured by the imaging device, or an image obtained by electrically enlarging the image is deteriorated, and the image restoration method is used to restore the image close to the original image using the deteriorated image. This is called a resolution technique, and the restored image is called a high-resolution image (recovered image).

  Assuming that a degraded image represented by a one-dimensional vector is g (x) and an original image represented by a one-dimensional vector for the degraded image is f (x), the two images satisfy the relationship represented by the following equation: .

g (x) = h (x) * f (x) (1)
Here, h (x) is PSF which is a transfer characteristic of the optical system.

  On the other hand, with Bayesian statistics, when f (x) is an original image and g (x) is a degraded image, the reverse process is established using the Bayes formula from the forward process in which the original image is converted to the degraded image, that is, This constitutes post-establishment, and based on this, the original image is estimated from the degraded image. Here, P (f (x)) is the probability density function of the event in which the original image f exists, P (g (x)) is the probability density function of the event in which the degraded image g occurs, and P (g (x) | f If (x)) is a conditional probability density function of the deteriorated image g when the original image f is given,

Is a relational expression called Bayes formula. Here, P (f (x) | g (x)) is a conditional probability density distribution for the original image f under the condition that the deteriorated image g is given, and is called a posterior probability density function.

Here, when the Bayesian formula based on the above Bayesian statistics holds for f and g, it is assumed that f and g are normalized, and f and g can be treated as a probability density function. When an event in which a point light source exists at a coordinate x ₁ in the original image is f (x ₁ ), and an event in which an image is formed at a coordinate x ₂ in the degraded image is g (x ₂ ),
P (f (x ₁ )) = f (x ₁ ) (3)
P (g (x ₂ )) = g (x ₂ ) (4)
Can be expressed as

Furthermore, P (g (x ₂ ) | f (x ₁ )) is h using the PSF of the optical system,
P (g (x ₂ ) | f (x ₁ )) = h (x ₂ −x ₁ ) (5)
Can be expressed as

In other words, the distribution of the original image that forms an image at a certain coordinate x ₂ in the deteriorated image is expressed by equations 2, 3, 4, and 5.

It can be estimated from

  From the definition of surrounding establishment here,

Holds.

  In other words, Equation 6 further

Can be expressed as Here, when integrating with P (g (x ₂ )) = g (x ₂ ) on both sides, the left side of Equation 8 is

Also, the right side of Equation 8 is

It becomes.

  The above relationship holds when f (x) is a true original image. That is, calculating the above f (x) corresponds to restoration of a deteriorated image.

Here, f (x) in Equation 9 is f _{k + 1} (x), and f (x) in Equation 10 is f _k (x).

Equation 11 above is obtained.

By performing the iterative calculation related to Equation 11 above, the convergence value of f _k , that is, the distribution of the original image can be obtained.

  By using the restoration method based on the Bayesian statistics described above, it is understood that an unknown original image can be restored if the transfer characteristics of the degraded image and the optical system are known. Further, based on the same principle as described above, it is possible to restore the transfer characteristics of the optical system if the deteriorated image and the original image corresponding thereto are known. In the above description, PSF is considered as the transfer characteristic of the optical system. However, by using Fourier transform, it is possible to restore the OTF taking into account the accurate phase characteristic.

  As another method, when the MAP method is used, a method for obtaining f (x) that maximizes the a posteriori density as shown below is used.

Here, when it is assumed that Gaussian noise n is added to the degraded image, and h (x) given as the PSF of the optical system is an m × m convolution matrix C that linearly acts, the degraded image and The two images of the original image can also be expressed as follows:
g (x) = C × f (x) + n (13)
Here, the newly given matrix C may include not only the PSF but also a deterioration factor caused by the imaging system.

In consideration of the above assumptions, f (x) at which the posterior probability density is maximum in Expression 12 is obtained from the proportional expression of Expression 12 as f (x) at which the following evaluation function is minimum.
T (f) = ‖g (x) −C × f (x) ‖ ² + αZ (f) (14)
Here, Z (f) is a constraint function including a constraint term from the smoothness of the image and additional conditions, and α is a weighting factor. A conventional steepest descent method or the like can be used to minimize the evaluation function.

  Calculation of f (x) that minimizes the above expression 14 corresponds to restoration of a deteriorated image.

  In order to restore a deteriorated image from the estimation equations and evaluation functions of Equations 11 and 14 described above, it is necessary to set the initial estimated distribution f, and the initial estimated distribution is a deteriorated image having the same imaging magnification as the recovered image. It is common to use g. In addition, it is understood that the constraint term obtained from the transfer characteristic of the optical system such as PSF or OTF and the additional condition or constraint condition is important. However, the transfer characteristics of the optical system depend on parameters such as lens aberration, illumination wavelength, and image sensor aperture, and it is generally difficult to evaluate accurately. For this reason, a Gaussian distribution or the like is simply used as the PSF of the initial condition. However, in an actual imaging system, it is rare that the PSF matches the Gaussian distribution, and in most cases it causes an increase in estimation error. In addition, although it is conceivable to estimate from the degraded image to the PSF based on the above-described principle, it is difficult to estimate an accurate PSF because the degraded image lacks much information. Therefore, it is considered to improve the accuracy of the super-resolution technique by adding a new strong constraint condition to the super-resolution technique. As the above strong constraint condition in the present invention, a high resolution image having a different imaging magnification from the deteriorated image to be restored is set as an additional condition. As shown in FIG. 8, an image with a different imaging magnification is an enlarged deteriorated image in which an image 120b of a partial region of an image obtained by photographing the subject 120a is desired to be restored. In such a case, an image having a relationship such as an image 120c photographed by enlarging the inside of a broken line in the subject 120a with different angles of view with respect to the same subject. As described above, by using high-resolution images with different imaging magnifications, it is possible to acquire detailed information in a partial area in the degraded image. Therefore, for example, the PSF in the central area of the deteriorated image where the main subject exists is estimated more accurately based on the high-resolution image, and iterative calculation of Expression 11 enables more accurate restoration than in the conventional method. . Further, since details of the degraded image partial region are acquired in advance, the correlation function using the correlation between the high-resolution image and the degraded image partial region as an evaluation value is added to the constraint function of Equation 14 for higher accuracy. Recovery is possible. In addition, from the above principle, it is clear that acquiring detailed information about as wide a range as possible in the degraded image enables more accurate restoration. In other words, in order to realize a continuous zoom function in a compound-eye imaging device, it is important to provide a plurality of imaging optical systems having different angles of view.

  Next, the principle of subject distance calculation in the compound eye imaging apparatus will be described. FIG. 9 is a diagram for explaining a conventional compound eye photography model. The coordinates are the center of the left and right cameras, the x axis in the horizontal direction, and the y axis in the depth direction. The height direction is omitted for simplification. Assume that the principal points of the imaging optical systems of the left and right cameras are arranged at (−Wc, 0) and (Wc, 0), respectively. Here, the focal length of the imaging optical system of the left and right cameras is assumed to be f. In this state, when the subject A on the y-axis (0, y1) is photographed by the respective cameras, the deviation amount of the subject A image from the sensor center of the left and right cameras is taken as the photographing parallax, and is expressed by the following equations as Plc and Prc, respectively. I can do it.

  By photographing the same subject from different viewpoints based on the above principle, it is possible to acquire right and left parallax images each having a shift amount represented by the above formulas (15) and (16) in the principal point position shift (baseline) direction. The distance y1 from the deviation amount to the subject A can be calculated by the following equation.

Here, since the shooting parallax amounts Plc and Prc are actually acquired with the resolution of the pixel size of the imaging system, it can be understood that the error in the subject distance calculation increases as the shooting parallax amount approaches the pixel size. That is, in order to improve the accuracy of subject distance calculation in the compound-eye imaging device, it is important to configure the shooting parallax amount to be as large as possible.

  In view of this, the present invention proposes a new imaging apparatus that includes an optical system having at least three different angles of view, and in which an optical system pair having the widest angle of view has the maximum baseline length.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  A compound eye imaging device according to a first embodiment of the present invention will be described below with reference to the drawings.

  FIG. 1 is a configuration diagram of a compound eye imaging apparatus 1 of the first embodiment. The imaging optical systems 101a, 101b, 101c, and 101d are imaging optical systems (compound eye optical systems) constituting a compound eye. As shown in FIG. 1, the image sensor 102 has image areas 102 a, 102 b, 102 c, and 102 d. The imaging regions 102a to 102d are regions corresponding to the imaging optical systems 101a to 101d, respectively. The imaging regions 102a to 102d convert optical images that have reached the element surface via the imaging optical systems 101a to 101d, respectively, into electrical signals. The A / D converter 103 converts the analog signal output from the image sensor 102 into a digital signal and supplies the digital signal to the image processing unit 104. The image processing unit 104 performs predetermined pixel interpolation processing, color conversion processing, and the like on each image data from the A / D converter 103. Further, the image processing unit 104 (image restoration unit) also performs image restoration processing for restoring a deteriorated image as will be described later. In the image processing unit 104, a predetermined calculation process is performed using each captured image data. The obtained calculation result is supplied to the system controller 108. The information input unit 106 detects information input by the user by selecting a desired shooting condition and supplies data to the system controller 108. The system controller 108 controls the imaging drive control unit 107 based on the supplied data, and acquires a necessary image.

  The image recording medium 105 is a recording unit for storing a plurality of captured still images and moving images, as well as storing a file header when configuring an image file. The display unit 200 is composed of a display device composed of, for example, a liquid crystal display element. The reference image selection unit 110 selects a reference image from a plurality of parallax images imaged by the imaging optical systems 101a, 101b, 101c, and 101d, and the corresponding point extraction unit 111 extracts corresponding pixels in the parallax image. . Also, the parallax amount calculation unit 112 calculates the parallax amounts of all the corresponding points extracted above, and the distance information calculation unit 113 (calculation unit) calculates subject distance information in the image from the calculated parallax amount. To do.

  As shown in FIG. 1, the compound eye imaging apparatus 1 includes four imaging optical systems 101a, 101b, 101c, and 101d. The four imaging optical systems 101a to 101d are arranged so that their optical axes are substantially parallel. Here, “substantially parallel” means that the case is completely parallel and the case where it is deviated from the case of being completely parallel within the allowable error range. The same applies to the following embodiments. The four imaging optical systems 101a to 101d are arranged so that the base lines connecting the optical axes of the optical systems intersect (orthogonal). Further, the imaging optical systems 101a and 101d (first optical system and second optical system) arranged at diagonal vertex positions in the diagonal direction have the widest shooting angle of view (first optical system among the four imaging optical systems). 1 angle of view) and a wide-angle imaging optical system pair having 4θ. Further, the imaging optical system 101b (third optical system) has a photographic field angle half that of the imaging optical systems 101a and 101d (second field angle narrower than the first field angle) 2θ. It is configured. Further, the imaging optical system 101c (fourth optical system) is configured so that the shooting field angle is half that of the imaging optical system 101b (third field angle narrower than the second field angle) θ. Has been. For simple explanation, FIG. 2 shows captured images 10a, 10b, 10c, and 10d corresponding to the respective imaging optical systems 101a to 101d when the imaging optical system arrangement is used. Note that the captured image 10a is a first image obtained from the imaging region 102a (the first imaging region corresponding to the first optical system) of the imaging element 102. The captured image 10d is a second image obtained from the imaging region 102d (second imaging region corresponding to the second optical system) of the imaging element 102. The captured image 10b is a third image obtained from the imaging area 102b (third imaging area corresponding to the third optical system) of the imaging element 102. The captured image 10c is a fourth image obtained from the imaging region 102c (fourth imaging region corresponding to the fourth optical system) of the imaging element 102. As shown in FIG. 2, the captured images 10a and 10d corresponding to the imaging optical systems 101a and 101d capture the widest subject space, and the captured images 10b and 10c corresponding to 101b and 101c correspond to the angle of view. The subject space to be imaged is narrow.

  Hereinafter, the details of the photographing operation for realizing the continuous zoom function in this compound eye imaging apparatus will be described with reference to the flowchart of FIG. First, in S301, when a photographing signal is input from the user, a wide-angle image (first image) obtained by imaging the widest subject space and a telephoto image (first image obtained by imaging a subject space narrower than the wide-angle image). 3 images) at the same time. Here, for example, the wide-angle image 10a (first image) and the telephoto image 10b (third image) are acquired simultaneously. The first and third images 10a and 10b are wide-angle and telephoto images with guaranteed optical performance, and are sufficiently high-resolution images. The first image 10a includes at least a part (preferably all) of the third image 10b.

  Next, in S302, an arbitrary angle of view desired by the user is cut out from the first image and enlarged. Here, the enlarged image (enlarged image) becomes a deteriorated image because it is enlarged using linear interpolation which is a conventional digital zoom technique. Note that this deteriorated image (enlarged image) is an image having an angle of view that is between the first angle of view and the second angle of view in this embodiment. In other words, the deteriorated image includes at least a part (preferably all) of the third image 10b. Next, in S303, the distribution g of the deteriorated image is specified. Next, in S304, the same subject area as the reference image 10b which is the third image in the deteriorated image is specified. For specifying the same area of the subject, a block matching method described later may be used. For specifying the same subject area, the reference image 10b as the third image may be reduced to the same size as the deteriorated image, or the deteriorated image may be enlarged and compared.

Next, super-resolution processing is performed based on the above-described super-resolution principle. Here, in S305, the evaluation function T (f) shown in Expression 14 is created. In the present embodiment, by adding a correlation function that decreases in value when the correlation between the deteriorated image and the same subject area of the reference image 10b is high in the term Z (f) in the evaluation function T (f). Therefore, it is possible to restore an image with higher accuracy than in the past. Further, the deteriorated image and the reference image 10b are images of the same subject, and when the respective imaging magnifications are β1 and β2,
β1 <β2
Meet the relationship. Since the reference image having such a relationship can acquire even a higher frequency component of the subject, it is possible to restore the image to the higher frequency component with high accuracy.

  Next, in S306, the estimated distribution f that minimizes the evaluation function is calculated using the steepest descent method or the like.

  Next, in S307, the image having the calculated estimated distribution f is stored as a restored image, and this flow is completed.

  The display unit 200 may display an image that has been subjected to predetermined display processing on the image after image processing (after image recovery), or may not perform image processing or perform simple correction processing. The performed image may be displayed. In addition, regarding the display during shooting, the first image may be displayed as it is, and after the shooting, the user may specify the desired angle of view and start the above-described processing, or the desired angle of view during shooting. May be specified. Further, the correction amount is determined by how to set an allowable amount for implementation, and may be determined according to the purpose of the image quality level to be obtained and the processing load amount.

Similarly, by using the telephoto image 10b and the further telephoto image 10c as the first image and the third image, it is possible to realize continuous zoom for a higher magnification region. As described above, in this embodiment, in the imaging optical system having consecutive different angles of view, the ratio of the angle of view on the wide angle side and the angle of view on the telephoto side is set to 2, which will be described. It is an example for simplifying. Desirably, an angle of view of an arbitrary image forming optical system constituting the plurality of image forming optical systems is W, and an angle of view of an image forming optical system having the next narrowest angle of view after the arbitrary image forming optical system is Wn. When, it is desirable to satisfy the following conditional expression (18).
1.1 <W / Wn <3 (18)
If the lower limit of conditional expression (18) is not reached, a large number of imaging optical systems are required to achieve a high zoom ratio of the imaging apparatus, and the apparatus becomes large. If the upper limit of conditional expression (18) is exceeded, it will be difficult to restore the image with high accuracy even if the super-resolution technique is used, and the image quality after zooming will deteriorate.

  As in this embodiment, when a compound eye lens having a plurality of different angles of view is used, continuous zooming with resolution is possible, and a configuration that does not require a zoom driving device or the like can be realized. Can contribute.

  Next, the subject distance information recording operation of the compound eye imaging apparatus 1 of the present embodiment will be described in detail with reference to the flowchart of FIG. Here, first, an operation in the case of using a parallax image obtained by the imaging optical system of the wide-angle optical systems 101a and 101d that captures the widest subject space will be described. First, when an imaging signal is input from the user, the system controller 108 starts drive control of the entire imaging apparatus 1 in S401. Next, in S402, an optical image via the imaging optical systems 101a and 101d is photoelectrically converted by the image sensor 102, transferred to the image processing unit 104 through the A / D converter 103, and a predetermined calculation process is performed. Obtained as image data (parallax image).

  Next, in S403, one of the parallax image data acquired by the reference image selection unit 110 is selected as a reference image for calculating the parallax amount. In this embodiment, an image obtained by the imaging optical system 101a is selected as a reference image (first image).

  Next, in step S404, the corresponding point extraction unit 111 detects the corresponding pixel by using the image obtained from the imaging optical system 101d as the reference image (second image) for the selected standard image. Here, the corresponding pixel is, for example, each pixel corresponding to the same subject A in the parallax image data obtained for the point image subject A in the shooting model. The corresponding point extraction method will be described in detail. Here, the image coordinates (X, Y) are used.

  The image coordinates (X, Y) are defined with the upper left corner of each pixel group in FIG. 10 as the origin, the horizontal direction being the X axis, and the vertical direction being the Y axis. Also, the luminance of the image coordinates (X, Y) of the standard image data 501 will be described as F1 (X, Y), and the luminance of the reference image data 502 will be described as F2 (X, Y).

  The pixel of the reference image data corresponding to an arbitrary coordinate (X, Y) in the standard image data has the luminance of the reference image data most similar to the luminance F1 (X, Y) of the standard image data at the coordinate (X, Y). It can be obtained by searching. In FIG. 10, arbitrary coordinates (X, Y) in the standard image data are set as vertical line pixels in 501 in FIG. 10, and pixels of reference image data corresponding to the coordinates (X, Y) are in 502 in FIG. 10. Let it be a vertical line pixel. However, in general, it is difficult to search for a pixel that is most similar to an arbitrary pixel. Therefore, a pixel near the image coordinates (X, Y) is also used to search for a similar pixel by a technique called block matching.

For example, a block matching process when the block size is 3 will be described. The luminance values of a total of three pixels, that is, a pixel at an arbitrary coordinate (X, Y) of the reference image data and two pixels before and after (X−1, Y) and (X + 1, Y), respectively,
F1 (X, Y), F1 (X-1, Y), F1 (X + 1, Y)
It becomes.

On the other hand, the luminance values of the pixels of the reference image data shifted from the coordinates (X, Y) by k in the X direction are respectively
F2 (X + k, Y), F2 (X + k-1, Y), F2 (X + k + 1, Y)
It becomes.

  In this case, the similarity E with the pixel at the coordinates (X, Y) of the reference image data is defined by the following equation (19).

  In this equation (19), the value of the similarity E is calculated by successively changing the value of k, and (X + k, Y) giving the smallest similarity E is a corresponding point with respect to the coordinates (X, Y) of the reference image data It is. Here, for the sake of simplicity, a parallax image having a base line in the horizontal direction has been described, but it is possible to detect corresponding points in the vertical direction and the diagonal direction using the same principle.

  In step S <b> 405, the parallax amount calculation unit 112 calculates the parallax amount for each corresponding point extracted above. As a calculation method, the pixel position difference is calculated from each pixel of the reference image data corresponding to each pixel of the standard image data obtained by the block matching method. In step S406, the distance information calculation unit 113 calculates Equation 17 from the parallax amount calculated above, the focal length of the imaging optical system, which is known information, and the baseline length data of the imaging optical systems 101a and 101d. The distance information for the photographic subject is calculated. Here, as described above, in order to improve the accuracy of subject distance calculation, it is important to configure the shooting parallax amount to be as large as possible. For the photographer, distance information for the entire subject space to be photographed is important. For this reason, in this embodiment, the wide-angle optical systems 101a and 101d for imaging the widest subject space in the imaging optical system are arranged diagonally so that the base line length becomes the longest. Although the distance information calculation in the case of using the imaging optical systems 101a and 101d has been described here, the distance can be obtained even if another imaging optical system pair (for example, a pair of the optical systems 101a and 101b) is used according to the same principle. It is possible to calculate information. Here, when the above method is used for images with different angles of view, a method of extracting a corresponding point by cutting out a portion corresponding to an image with a narrow angle of view from an image with a wide angle of view is more desirable. Next, in S407, the subject distance information calculated together with the acquired image data is recorded, and this flow is completed.

  As described above, a continuous zoom function that does not require a zoom drive device or the like can be realized by providing three types of imaging optical systems having different angles of view. Furthermore, in this embodiment, a pair of wide-angle optical systems that image the widest subject space among the imaging optical systems are provided as a pair, and the optical system pairs are arranged diagonally so that the base line length becomes the longest. Therefore, it improves both distance information acquisition accuracy. That is, with the configuration of the present invention, an imaging apparatus such as a video camera or a digital camera can be easily acquired high-level peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio. It becomes possible to develop.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  A compound eye imaging device according to a second embodiment of the present invention will be described below with reference to the drawings.

  FIG. 5 is a configuration diagram of the compound eye imaging apparatus 2 of the second embodiment. The imaging optical systems 201a, 201b, 201c, 201d, and 201e are imaging optical systems (compound eye optical systems) that constitute a compound eye. The imaging element 202 has imaging areas 202a, 202b, 202c, 202d, and 202e. The imaging regions 202a to 202e are regions corresponding to the imaging optical systems 201a to 201e, respectively. The imaging regions 202a to 202e convert optical images that have reached the element surface via the imaging optical systems 201a to 201e, respectively, into electrical signals. The other parts having the same numbers are the same as those in the first embodiment, and hence the description thereof is omitted.

  As shown in FIG. 5, the compound-eye imaging device 2 includes five imaging optical systems 201a, 201b, 201c, 201d, and 201e. The five imaging optical systems 201a to 201e are arranged so that their optical axes are substantially parallel to each other. Further, the five imaging optical systems 201a to 201e are arranged so that the base lines connecting the optical axes of the respective optical systems intersect (partially orthogonal). Further, the imaging optical systems 201a and 201d arranged in the diagonal direction are configured as a wide-angle imaging optical system pair having the widest shooting angle of view 8θ among the five imaging optical systems. Further, the imaging optical system 201b has 4θ which is a half of the field angle of view compared to 201a and 201d, the imaging optical system 201e further has 2θ which is a half of the field angle of view, and the imaging optical system 201c further takes a photographed image. The angle is half of θ. The means for realizing the continuous zoom function and the configuration reason for improving the accuracy of distance information acquisition are the same as in the first embodiment, and thus detailed description thereof is omitted here.

  As described above, a continuous zoom function that does not require a zoom drive device or the like can be realized by providing four types of imaging optical systems having different angles of view. Furthermore, in this embodiment, a pair of wide-angle optical systems that image the widest subject space among the imaging optical systems are provided as a pair, and the optical system pairs are arranged diagonally so that the base line length becomes the longest. Therefore, it improves both distance information acquisition accuracy. That is, with the configuration of the present invention, an imaging apparatus such as a video camera or a digital camera can be easily acquired high-level peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio. It becomes possible to develop.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  A compound eye imaging device according to a third embodiment of the present invention will be described below with reference to the drawings.

  FIG. 6 is a configuration diagram of the compound eye imaging apparatus 3 of the third embodiment. The imaging optical systems 301a, 301b, 301c, 301d, 301e, 301f, 301g, 301h, and 301i are imaging optical systems (compound eye optical systems) constituting a compound eye. The imaging element 302 has imaging areas 302a, 302b, 302c, 302d, 302e, 302f, 302g, 302h, and 302i. The imaging regions 302a to 302i are regions corresponding to the imaging optical systems 301a to 301i, respectively. The imaging regions 302a to 302i convert optical images that have reached the element surface via the imaging optical systems 301a to 301i, respectively, into electrical signals. The other parts having the same numbers are the same as those in the first embodiment, and hence the description thereof is omitted.

  As shown in FIG. 6, the compound eye imaging device 3 includes nine imaging optical systems 301a, 301b, 301c, 301d, 301e, 301f, 301g, 301h, and 301i. The nine imaging optical systems 301a to 301i are arranged so that their optical axes are substantially parallel to each other. Further, the nine imaging optical systems 301a to 301i are arranged so that the base lines connecting the optical axes of the respective optical systems intersect (orthogonal). Further, the imaging optical systems 301a and 301i arranged in the diagonal direction are configured as a wide-angle imaging optical system pair having the widest shooting angle of view 8θ among the nine imaging optical systems. Further, the imaging optical system 301b is 7θ, the imaging optical system 301c is 6θ, and the imaging optical system 301f is 5θ. Further, the imaging optical system 301e is 4θ, the imaging optical system 301d is 3θ, the imaging optical system 301g is 2θ, and the imaging optical system 301h is configured to be θ. The means for realizing the continuous zoom function and the configuration reason for improving the accuracy of distance information acquisition are the same as in the first embodiment, and thus detailed description thereof is omitted here. As described above, in this embodiment, the imaging optical system having consecutive different angles of view is configured such that the ratio of the angle of view on the wide angle side and the angle of view on the telephoto side is 1.14 to 2. This is an example for ease of explanation. Desirably, an angle of view of an arbitrary image forming optical system constituting the plurality of image forming optical systems is W, and an angle of view of an image forming optical system having the next narrowest angle of view after the arbitrary image forming optical system is Wn. When, it is desirable to satisfy the above conditional expression (18).

  If the lower limit of conditional expression (18) is not reached, a large number of imaging optical systems are required to achieve a high zoom ratio of the imaging apparatus, and the apparatus becomes large. If the upper limit of conditional expression (18) is exceeded, it will be difficult to restore the image with high accuracy even if the super-resolution technique is used, and the image quality after zooming will deteriorate.

  As described above, by providing eight types of image forming optical systems having different angles of view, it is possible to realize a continuous zoom function that does not require a zoom drive device or the like. Furthermore, in this embodiment, a pair of wide-angle optical systems that image the widest subject space among the imaging optical systems are provided as a pair, and the optical system pairs are arranged diagonally so that the base line length becomes the longest. Therefore, it improves both distance information acquisition accuracy. That is, with the configuration of the present invention, an imaging apparatus such as a video camera or a digital camera can be easily acquired high-level peripheral space information about a subject space to be photographed while being thin and having a high zoom ratio. It becomes possible to develop.

  Furthermore, other surrounding space information acquisition modes or surrounding space information application modes of the imaging devices 1, 2 and 3 configured as described above will be described. In the high dynamic range mode, each unit constituting the compound eye performs shooting while changing the exposure condition, and obtains image information with a wide dynamic range by combining the captured images. In the blur addition mode, an image that emphasizes the main subject is obtained by adding blur to the background based on the subject distance information calculated as described above. In the background removal mode, an image in which the background other than the main subject is removed based on the subject distance information calculated as described above is obtained. In the stereoscopic image capturing mode, right and left parallax images are acquired by the respective units constituting the compound eyes arranged in the horizontal direction, and an image with one narrow angle of view and a part of the image region with the other wide angle of view corresponding thereto are obtained. Use to save images as stereoscopic images. Thus, the compound eye imaging device of the present invention can also function as a stereoscopic image imaging device. In the shooting composition selection mode, an image with a plurality of angles of view is displayed on the display unit 200 as shown in FIG. 2, thereby allowing the photographer to select a desired angle of view image. Next, the best shot mode will be briefly described with reference to FIG. Here, FIG. 7 is an image of an angle of view range corresponding to the imaging optical system 101a of the compound eye imaging apparatus 1, and a broken line portion 131 is an angle of view range corresponding to the imaging optical system 101b. When the photographer operates to photograph the moving person 130 at the angle of view corresponding to the imaging optical system 101b, the image processing is performed when the moving object (person 130) that is the object of photographing enters the area of the broken line portion 131. Calculated by the unit 104. The timing can be calculated from an image in a wide angle range corresponding to the imaging optical system 101a using a motion vector of a moving object. In the shooting area change mode, each image forming optical system can be displaced in a plane perpendicular to each optical axis, so that the subject space area corresponding to each image forming optical system is changed and each image is displayed. save. Thus, this invention may be provided with the displacement means which displaces each of the some imaging optical system as described in Examples 1-3 in the surface perpendicular | vertical to an optical axis. For example, an arbitrary area in the screen corresponding to the wide-angle imaging optical system 101a can be enlarged and displayed by displacing the telephoto imaging optical system 101c.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The compound eye imaging device of the present invention can be suitably used for an optical device such as a video camera or a compact digital camera.

DESCRIPTION OF SYMBOLS 1 Compound eye imaging device 101a, 101b, 101d Imaging optical system 102 Imaging element

Claims (10)

In a compound eye imaging device having a plurality of optical systems,
The plurality of optical systems include a first optical system and a second optical system having the widest first field angle among the plurality of optical systems, and a second field angle narrower than the first field angle. A third optical system having
A first imaging area corresponding to the first optical system; a second imaging area corresponding to the second optical system; and a third imaging area corresponding to the third optical system. With an image sensor
The compound-eye imaging device, wherein the plurality of optical systems are arranged so that a baseline length of the first optical system and the second optical system is maximized.   Based on the first image obtained from the first imaging area and the third image obtained from the third imaging area, between the first angle of view and the second angle of view. The compound-eye imaging apparatus according to claim 1, further comprising an image restoration unit that restores an image having a certain angle of view. Based on the first image obtained from the first imaging area and the second image obtained from the second imaging area,
The compound-eye imaging apparatus according to claim 1, further comprising a calculation unit that calculates subject distance information in the images of the first image and the second image.   4. The compound eye imaging apparatus according to claim 1, wherein the plurality of optical systems are arranged such that base lines connecting optical axes of the optical systems intersect each other. 5.   5. The compound eye imaging according to claim 1, wherein the plurality of optical systems includes a fourth optical system having a third field angle narrower than the second field angle. apparatus. The plurality of optical systems are arranged so that the base lines connecting the optical axes of the respective optical systems are orthogonal to each other,
6. The compound eye imaging apparatus according to claim 1, wherein the first optical system and the second optical system are arranged at diagonal vertex positions. 6.   The image restoration means restores the enlarged image by super-resolution processing based on an enlarged image obtained by enlarging a part of the first image and the third image. The compound eye imaging device described in 1. When the angle of view of an arbitrary optical system among the plurality of optical systems is W, and the angle of view of an optical system having the next narrowest angle of view after the arbitrary optical system is Wn,
1.1 <W / Wn <3
The compound eye imaging device according to claim 1, wherein:   The compound-eye imaging apparatus according to claim 1, wherein optical axes of the plurality of optical systems are parallel to each other.   The displacement mechanism for displacing each of the plurality of optical systems in a plane perpendicular to the optical axis is provided in the plurality of optical systems. Compound eye imaging device.