JP2019066186A - Imaging apparatus and method for calculating distance of imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus and a distance calculation method of the imaging apparatus with which it is possible to measure a distance with high accuracy.SOLUTION: The imaging apparatus comprises a TOF camera, a color opening camera, a third distance calculation unit 22, and distance calculation means. The TOF camera includes a light-emitting unit 10a for irradiating light, a first imaging unit 13a for receiving reflected light and acquiring a first image, and a first distance calculation unit 14a for calculating a first distance to a subject of the first image on the basis of a time from when the light-emitting unit irradiates light to when the first imaging unit receives reflected light. The color opening camera includes a second imaging unit 18b, juxtaposed with the TOF camera, for receiving light and acquiring a second image, and a second distance calculation unit 19b for calculating a second distance to a subject of the second image on the basis of the received light. The third distance calculation unit calculates a third distance to the subject on the basis of parallax between the first image and the second image. The distance calculation means finds a fourth distance to the subject on the basis of the first distance, the second distance and the third distance.SELECTED DRAWING: Figure 2

Description

Embodiments of the present invention relate to an imaging device for acquiring a distance image to a subject and a distance calculation method of the imaging device.

Generally, a three-dimensional camera such as a TOF (Time Of Flight) camera capable of measuring the distance to an object, and a color aperture camera is known. The TOF camera is a technology that measures the distance based on the time it takes for the illumination light to hit the subject and return. In the case where the object is a thin cylindrical object or the like having a small number of flat surfaces, the light emitted by the TOF camera is diffused by the object, and the amount of received reflected light decreases, so there is a problem that the distance can not be measured with sufficient accuracy.

Also, the color aperture camera is a camera equipped with color aperture imaging technology, and is described in Non-Patent Document 1
As described, the color filter divided into two colors is interpolated at the aperture position (aperture) of the camera, and the blur of the image changes differently for each color according to the distance of the subject, and the distance is measured according to the degree It is a technology. Therefore, although the color aperture camera can measure the distance with high accuracy at the edge portion of the subject, there is a problem that the distance can not be measured with sufficient accuracy for a plane where blurring is not likely to occur.

Furthermore, a stereo camera is known as a method of measuring the distance to an object. A stereo camera includes two cameras, that is, two camera lens mechanisms and a shutter mechanism, captures an image of an object from different directions, and measures the distance to the object using parallax of images obtained from the respective cameras. . However, stereo cameras have a problem that they can not measure the distance in a particular direction. For example, when two cameras are arranged side by side, there is a problem that the distance can not be measured with sufficient accuracy for the lateral edge.

JP 2003-75137

Toshiba Corp., "Development of imaging technology that can simultaneously acquire color image and distance image from one image taken with single camera," [online], June 2016, [September 19, 2017 search], Internet <URL: https://www.toshiba.co.jp/rdc/detail/1606_01.htm>

The problem to be solved by the present invention is to provide an imaging device capable of measuring the distance to an object with high accuracy and a method of calculating the distance of the imaging device.

According to the embodiment, the imaging device includes a TOF camera, a color aperture camera, a third distance calculation unit, and a distance calculation unit. The TOF camera includes a light emitting unit that emits light to a subject, a first imaging unit that receives reflected light from the subject of the light emitted by the light emitting unit, and obtains a first image, and the light emitting unit A first distance to the subject is calculated for each pixel of the first image based on a time from when the light is irradiated to when the first imaging unit receives the reflected light from the subject. And a distance calculation unit of 1. The color aperture camera is provided on the TOF camera, receives the light from the subject, and acquires a second image. The second imaging unit and the light received by the second imaging unit. The second to the subject for each pixel of the second image
And a second distance calculation unit that calculates the distance of A third distance calculation unit calculates a third distance to the subject for each pixel based on the parallax between the first image and the second image. The distance calculating means determines a fourth distance to the subject based on the first distance, the second distance, and the third distance.

BRIEF DESCRIPTION OF THE DRAWINGS The perspective view which shows an example of the external appearance of the imaging device which concerns on embodiment. FIG. 1 is a block diagram showing an example of a device configuration of an imaging device according to an embodiment. BRIEF DESCRIPTION OF THE DRAWINGS The perspective view which shows an example of the imaging condition using the imaging device which concerns on embodiment. The figure which shows an example of the pixel of the 1st distance image which the imaging device which concerns on embodiment acquires. The figure which shows an example of the pixel of the 2nd distance image which the imaging device which concerns on embodiment acquires. The figure which shows an example of the pixel of the 3rd distance image which the imaging device which concerns on embodiment acquires. The figure which shows an example of the relationship of the combination of the result which the imaging device which concerns on embodiment measured, and 4th distance. The flowchart which shows an example of the flow from which the control part which the imaging device concerning an embodiment has calculates the 4th distance.

Hereinafter, an imaging device according to an embodiment will be described using the drawings.
FIG. 1 is a perspective view showing an example of the appearance of an imaging device according to the embodiment. The imaging device 1 includes a TOF camera 2 having a light emitting unit 10 a that emits light to a subject, and a color aperture camera 3. In addition, the imaging device 1 also has a function as a stereo camera 5 configured by the TOF camera 2 and the color aperture camera 3. The imaging apparatus 1 is configured to be able to independently measure the distance to the subject measured by the TOF camera 2, the distance to the subject measured by the color aperture camera 3, and the distance to the subject measured by the stereo camera 5. .

The TOF camera 2 is configured to be able to measure the distance to the subject for each pixel based on the time until the irradiation light from the light emitting unit 10 a of the TOF camera 2 strikes the subject and returns. In addition, the TOF camera 2 can generate an image obtained by imaging a subject, and can further generate a distance image representing information of the measured distance for each pixel.

The color aperture camera 3 is annexed to the TOF camera 2. Also, the color aperture camera 3
The color filter 17b divided into two in two colors is inserted into the aperture position (opening 11b) of the camera, and the blur of the image uses different changes for each color according to the distance to the subject, and the distance according to the degree It is possible to measure In the present embodiment, as shown in the figure,
The color filter 17b is divided into two in the vertical direction. Further, the TOF camera 2 and the color aperture camera 3 are juxtaposed in the direction orthogonal to the two division direction of the color filter 17 b. That is, if the division direction of the color filter 17b is the vertical direction, the TOF camera 2 and the color aperture camera 3 are arranged in the horizontal direction which is a direction orthogonal to the vertical direction. In addition, the color aperture camera 3 can generate an image obtained by capturing an object, and can further generate a distance image representing information of the measured distance for each pixel.

In the following, as shown in FIG. 1, the arrangement direction of the TOF camera 2 and the color aperture camera 3 is defined as the horizontal direction, and the division direction of the color filter 17b is defined as the vertical direction.

The stereo camera 5 can calculate the distance to the subject for each pixel based on the parallax of the image acquired by the TOF camera 2 and the color aperture camera 3 respectively.
The imaging device 1 calculates the distance for each pixel with higher accuracy based on the distance to the subject calculated by the TOF camera 2, the distance to the subject calculated by the color aperture camera 3, and the distance calculated by the stereo camera 5. It is possible.

Next, the device configuration of the imaging device 1 will be described using FIG. FIG. 2 is a block diagram showing an example of the apparatus configuration of the imaging apparatus 1 according to the embodiment.

The TOF camera 2 includes a light emitting unit 10a, an opening 11a, a lens 12a, a first imaging unit 13a, and a first distance calculating unit 14a.

The light emitting unit 10a has a light source (not shown) and emits light toward the subject. Light emitting unit 10a
The light emitted by may be infrared light or visible light. Further, the light emitting unit 10a may have a plurality of LED light sources in order to increase the amount of light irradiated to the subject or the irradiation intensity.

The opening 11 a is a stop for adjusting the amount of light incident on the lens 12 a. In addition, the lens 12a condenses the light from the subject on the first imaging unit 13a. The lens 12a is not limited to one lens, and may be, for example, a focus lens or a zoom lens by combining a plurality of lenses.

The first imaging unit 13a receives the reflected light from the subject of the light emitted by the light emitting unit 10a, and acquires a first image. For example, the first imaging unit 13a is an image sensor, and images a subject based on the received reflected light.

The first distance calculation unit 14a performs a process of calculating a first distance based on the result of imaging by the first imaging unit 13a. That is, the distance of each pixel to the subject is calculated by the processing in the first distance calculation unit 14a. The first distance calculation unit 14a is a first distance-to-subject for each pixel based on the time from when the light emitting unit 10a emits light to when the first imaging unit 13a receives reflected light from the subject. Calculate the distance.

The color aperture camera 3 includes an aperture 11 b, a color filter 17 b, and a lens 12 b.
A second imaging unit 18 b and a second distance calculation unit 19 b are provided. In addition, the broken line shown in FIG.
An example of an optical path which each 1st image pick-up part 13a and 2nd image pick-up part 18b light-receive is shown.

In addition, since the opening 11 b and the lens 12 b included in the color aperture camera 3 have the same functions as the opening 11 a and the lens 12 a included in the TOF camera 2, the description will be omitted.

The color filter 17 b is a filter divided into two colors that divides the opening 11 b into two.
The color filter 17b is attached to the opening 11b of the lens 12b and causes blurring and color shift in the image based on the distance to the subject. The color filter 17b is divided into, for example, two colors of cyan and yellow, and the cyan portion is G (green) and B among RGB components.
The light of (blue) is transmitted, and the portion of yellow transmits R (red) and G (green). Therefore, the bias direction of the blur of the R image and the G image is changed before and after the focus position. That is, it becomes possible to cause color shift, and the position of the subject can be uniquely determined. For example, if the subject is located at the in-focus position, an image is formed without blurring, but if the subject is not located at the in-focus position, blurring occurs. I'm sorry.

The second imaging unit 18 b receives the light from the subject through the lens 12 b and the color filter 17 b, and acquires a second image. For example, the second imaging unit 18 b is an image sensor, and images a subject. In addition, the second distance calculation unit 19 b performs a process of calculating a second distance based on the result of imaging by the second imaging unit 18 b. The second distance calculation unit 19 b
The second distance to the subject is calculated for each pixel based on the light received by the imaging unit 18b. That is, by the processing in the second distance calculation unit 19b, the distance for each pixel to the subject is calculated. The second distance calculation unit 19b calculates the distance to the subject for each pixel based on the color shift of the blur generated by the color filter 17b and the size of the blur.

The control unit 4 includes a CPU and a storage unit (not shown), and centrally controls the operation of each component of the imaging apparatus 1. Further, the control unit 4 includes a third distance calculation unit 22 and a distance calculation unit 15.

The third distance calculation unit 22 calculates a third distance based on the first image and the second image. The third distance calculation unit 22 calculates a third distance to the subject based on the parallax between the subject shown in the first image and the subject shown in the second image. In the first and second images,
Due to the difference in position between the TOF camera 2 and the color aperture camera 3, the viewing direction of the subject is different. That is, the parallax is the difference between the position of a point on the subject in the first image and the position of the point on the subject in the second image. Based on the parallax calculated by the measurement, the focal distances of the TOF camera 2 and the color aperture camera 3 which are known, and the separation distance between the TOF camera 2 and the color aperture camera 3 which is known, for example, the subject using trigonometry It is possible to calculate a third distance up to.

The distance calculating unit 15 calculates the first distance calculated by the first distance calculating unit 14a, the second distance calculated by the second distance calculating unit 19b, and the third distance calculated by the third distance calculating unit 22. The fourth distance for each pixel to the subject is calculated based on That is, the distance calculation unit 15 generates the first distance,
It functions as distance calculation means for calculating a fourth distance to the subject based on the second distance and the third distance.

First, the distance calculation unit 15 extracts feature points for each of the first distance image, the second distance image, and the third distance image. For example, the distance calculation unit 15 associates the same feature by associating a certain feature point in the first range image, a feature point in the corresponding second range image, and a feature point in the corresponding third range image. Processing is performed so that the points are at the same pixel position. This process makes it possible to compare the first distance, the second distance, and the third distance for each pixel. Next, the distance calculation unit 15 calculates a fourth distance to the subject for each pixel based on the first distance, the second distance, and the third distance.

Here, the result of imaging using the imaging device 1 will be described with reference to FIGS. 3 to 6.
FIG. 3 is a view showing an example of an imaging situation using the imaging device 1 according to the embodiment. Figure 3 shows
A situation is shown in which a rectangular object 21 and a cylindrical fence 20 between the imaging device 1 and the object 21 are present in front of the imaging device 1. It is assumed that the object 21 is positioned at an angle at which only one surface of the object 21 can be viewed when viewed from the imaging device 1.

FIG. 4 is a view showing an example of a pixel 30 of a first distance image in which the first distance of the TOF camera 2 acquired by the imaging device 1 according to the embodiment is represented for each pixel. FIG. 5 is a view showing an example of a pixel 40 of a second distance image representing the second distance of the color aperture camera 3 acquired by the imaging device 1 according to the embodiment for each pixel. FIG. 6 is a view showing an example of a pixel 50 of a third distance image in which the third distance of the stereo camera 5 acquired by the imaging device 1 according to the embodiment is represented for each pixel.

The pixel 30 in the first distance image, the pixel 40 in the second distance image, and the pixel 50 in the third distance image shown in FIGS. 4 to 6 are examples of pixels captured in the imaging situation of FIG. 3. . Note that
In order to simplify the description, alphabets and numbers are given to the rows and columns of the pixels shown in FIGS. 4 to 6, respectively. For example, the pixel located at the top left is called A row 1 column. Further, it is assumed that calculation processing is performed so that the pixels 30 of the first distance image, the pixels 40 of the second distance image, and the pixels 50 of the third distance image have the same positional relationship. For example, pixel 3 of the first distance image
Row B, row 4 of 0, row B, row 4 of the pixel 40 of the second distance image, and row B, row 4 of the pixel 50 of the third distance image are pixels obtained by imaging the same position of the object 21.

Further, the numerical value described in each pixel means the distance to the subject, and the larger the number, the farther the distance to the subject. However, a pixel with a numerical value of 0 means that a valid distance could not be obtained. For example, if the effective distance can not be acquired, meaning that a pixel with a numerical value of 0 is too far from the subject, too little light is received, or too much light is received, the imaging unit It becomes saturated and so on.

B rows 4 except 5 rows and 8 rows of pixels 30 of the first distance image of the TOF camera 2 shown in FIG. 4
The numerical value 6 is shown in the range of the rectangle which makes the pixel of column B row 9 column F row 4 column F row 9 column a vertex. This indicates that the first distance to the plane of the object 21 is shown, and the five rows and eight rows indicate that the rectangular parallelepiped is blocked by the fence 20. Although the TOF camera 2 is advantageous for measuring the distance to the plane, it exemplifies that it is disadvantageous for measuring the distance to the fence 20 which is the edge, and the difference from the color aperture camera 3 and the stereo camera 5 Is an extreme example to briefly explain. Therefore, the pixel corresponding to the edge indicates the numerical value 0.

B rows 4 columns B rows 6 columns of pixels 40 of the second distance image of the color aperture camera 3 shown in FIG.
A numerical value 6 is shown in B row 7 column, B row 9 column, F row 4 column, F row 6 column, F row 7 column, and F row 9 column. This indicates the second distance to the lateral edge of the two sides of the object 21. The color aperture camera 3 is advantageous for measuring the distance to the edge in the direction perpendicular to the split direction of the color filter 17b inserted in the color aperture camera 3, but the distance to the plane is less than that of the color filter 17b. This is an extreme example for illustrating the disadvantage in measuring the distance to the edge in the direction parallel to the 2-division direction, and for simply explaining the difference between the TOF camera 2 and the stereo camera 5. Therefore, the pixel corresponding to the plane of the object 21 and the edge in the vertical direction indicates the numerical value 0.

Two rows, five rows, eight rows, 11 of pixels 50 of the third distance image of the stereo camera 5 shown in FIG.
In the column, the number 4 is shown. This indicates the third distance to the fence 20. Also,
A numerical value 6 is shown in rows B to 4 and rows F and 4 and rows B and 9 to rows F and 9. This indicates the third distance to the vertical edges of the two sides of the object 21. The stereo camera 5 is advantageous for measuring the distance to the edge in the direction perpendicular to the direction in which the two cameras TOF camera 2 and the color aperture camera 3 are juxtaposed, but the distance to the plane, The TOF camera 2 and color aperture camera are illustrated as being disadvantageous for measuring the distance to the edge parallel to the direction in which the two cameras TOF camera 2 and color aperture camera 3 are juxtaposed. This is an extreme example to briefly explain the difference from 3. Thus, the plane of the object 21,
The pixels corresponding to the horizontal direction edge have a numerical value of zero.

Here, a distance calculation method for calculating the fourth distance will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a relationship between a combination of results measured by the imaging device 1 according to the embodiment and a fourth distance.
In the column of numbers shown in FIG. 7, a serial number attached to the combination is described, and a first distance, a second distance,
In the third distance column, indicate whether it is possible to obtain an effective distance to the subject with a circle and a cross,
The column of expected objects describes the objects expected from the combination of results, and the column of fourth distances describes the distances suitable for adoption as the fourth distance estimated from the expected objects. .

The number 1 is the case where only the first distance is acquired. Since the TOF camera 2 which acquired the first distance is advantageous with respect to the plane, the assumed object is a plane. Therefore, the first distance is adopted as the fourth distance.

The number 2 is the case where only the second distance is acquired. Since the color aperture camera 3 having acquired the second distance is advantageous for the edge in the direction orthogonal to the direction of division of the color filter 17b, the assumed object is a lateral edge. That is, since the color filter 17b is divided in the vertical direction, it is advantageous for the edge in the horizontal direction. Therefore, the second distance is adopted as the fourth distance.

The number 3 is the case where only the third distance is acquired. Since the stereo camera 5 having acquired the third distance is advantageous to the edge in the direction orthogonal to the direction in which the TOF camera 2 and the color aperture camera 3 which are two cameras are juxtaposed, the assumed object Is the vertical edge. Therefore, the third distance is adopted as the fourth distance.

The number 4 is a case where the first distance and the second distance are acquired and the third distance can not be acquired. That is, from the assumed objects of No. 1 and No. 2, the assumed object of No. 4 is the horizontal line on the plane or the boundary between the horizontal edge and the plane. Therefore, the second distance is adopted as the fourth distance because the edge ratio is expected to be higher than the plane.

The number 5 is a case where the second distance and the third distance are acquired and the first distance can not be acquired. That is, from the assumed objects of the numbers 2 and 3, the assumed object of the number 5 is an edge in the diagonal direction. Therefore, the third distance is adopted as the fourth distance.

The number 6 is a case where the first distance and the third distance are acquired and the second distance can not be acquired. That is, from the assumed objects of No. 1 and No. 3, the assumed object of No. 6 is a vertical line on a plane or a boundary between a vertical edge and a plane. Therefore, the third distance is adopted as the fourth distance because the edge ratio is expected to be higher than the plane.

The number 7 is the case where both the first distance, the second distance, and the third distance are acquired. That is, from the assumed objects of the numbers 1, 2 and 3, the assumed object of the number 7 is a diagonal line on the plane. Therefore, the first distance is adopted as the fourth distance because the proportion of the plane is expected to be high.

The number 8 is a case where the first distance, the second distance, and the third distance can not be obtained. That is, the distance to the subject is too far for imaging (infinity), or the assumed object is a black plane with low light reflectance. Therefore, the fourth distance can not be calculated (numerical value 0).

As described above, the distance measured by the camera suitable for the subject can be adopted by considering which camera can obtain the effective distance to the subject, and the distance to the subject can be further increased. It is possible to measure to an accuracy.

Next, the flow of calculating the fourth distance will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of a flow in which the control unit 4 of the imaging device 1 according to the embodiment calculates the fourth distance.

First, the control unit 4 selects one arbitrary pixel (step S01). The control unit 4 determines whether or not a plurality of effective distances having nonzero numerical values among the first distance, the second distance, and the third distance corresponding to the selected pixel can be acquired (step S03).

When the control unit 4 can not acquire a plurality of effective distances where the numerical value is not 0 (step S03)
, No), It is determined whether the acquired numerical value is a valid distance other than 0 (step S05).
If the obtained numerical value is a valid distance other than 0 (Yes at Step S05), the control unit 4
The distance is adopted as the fourth distance (step S07). Next, the control unit 4 determines whether the process has been performed on all the pixels (step S11), and the process has not been performed on all the pixels (step S11).
In step S11, No), the process returns to step S01 to perform the process on the next pixel.

Further, when the effective distance can not be acquired in step S05, that is, when the acquired numerical value is 0 (No in step S05), the control unit 4 adopts the numerical value 0 as the fourth distance (step S09). The process proceeds to step S11.

In addition, when the control unit 4 can obtain a plurality of effective distances where the numerical value is not 0 (step S03)
, Yes), and determines whether the first distance, the second distance, and the third distance are all valid distances (step S13).

When the first distance, the second distance, and the third distance are all effective distances (Yes in step S13), the control unit 4 adopts the first distance as the fourth distance (step S15). The process proceeds to step S11.

In addition, when the first distance, the second distance, and the third distance include an ineffective distance (Step S13, No), the control unit 4 determines whether the third distance is an effective distance. It determines (step S17). If the third distance is an effective distance (step S17, Ye
s) The third distance is adopted as the fourth distance (step S19), and the process proceeds to step S11.

When the third distance is not a valid distance (No at step S17), the control unit 4 adopts the second distance as the fourth distance (step S21), and advances the process to step S11. Finally, when all the pixels have been processed (step S11, Yes), the control unit 4 ends the processing.

By executing the operation according to the above flow, the imaging device 1 according to the embodiment has a disadvantage that the distance of the edge portion of the TOF camera 2 can not be measured with sufficient accuracy. The plane portion of the color aperture camera 3 and the color filter 17b The drawback is that the distance to the edge in the direction orthogonal to the 2-division direction can not be measured with sufficient accuracy. Furthermore, the flat portion of the stereo camera and the TOF camera 2 and the color aperture camera 3 which are two cameras are juxtaposed It is possible to measure the distance to the subject with higher precision, compensating for the drawback that the distance to the edge in the direction parallel to the direction can not be measured with sufficient accuracy.

Here, some modified examples of the embodiment described above will be described. First, the first modification will be described. The description of the contents common to the embodiment and the modification will be appropriately omitted.

(First modification)
In the embodiment, as described with reference to FIG. 7 and FIG. 8, the distance to be adopted as the fourth distance is calculated based on which camera can obtain the effective distance to the subject. On the other hand
In the modification of the above, the fourth distance is calculated by weighting the distance adopted for the fourth distance shown in FIG. 7 so as to set the priority high. That is, for each of the first distance, the second distance, and the third distance, the fourth distance is calculated using weighted numerical values. For example, in the case of No. 7 in FIG. 7, a numerical value obtained by adding 80% of the first distance, 10% of the second distance, and 10% of the third distance is calculated as the fourth distance. The weighting ratio is set for each combination shown in FIG.

As described above, in the first modification, the distance to the subject can be measured more accurately by using a numerical value weighted in consideration of which camera can obtain an effective distance to the subject. Is possible.

(Second modification)
Next, a second modification will be described. In the embodiment, as described using FIG.
Although the color filter 17b inserted in the color aperture camera 3 is inserted in the horizontal direction (divided in the vertical direction), in the second modification, the color filter 17b is divided in the vertical direction (divided in the horizontal direction)
Insert into At this time, the color aperture camera 3 and the TOF camera 2 are juxtaposed in the vertical direction.

Thus, in the case of the second variant, the color aperture camera 3 is advantageous for measuring longitudinal edges rather than lateral ones. Also, stereo cameras are advantageous for measuring lateral rather than longitudinal edges.

That is, in the second modification, the distance to be adopted as the fourth distance along with the relationship between the horizontal and vertical directions of the assumed object described in FIG. 6 being reversed and the relationship between the directions being reversed. The relationship between the second distance and the third distance is reversed, but the same effect as that of the embodiment can be obtained.

In addition, the process in the imaging device 1 of the above-described embodiment and modification can be executed by some software. Therefore, the process can be easily realized only by installing the program into the imaging apparatus 1 through a computer readable storage medium storing a number of programs for executing the procedure of the process and executing the program.
For example, the imaging device 1 can download the program via the network, store the downloaded program, and complete the installation of the program. Alternatively, the imaging device 1 can read the program from the information storage medium, store the read program, and complete the installation of the program.

While some embodiments and modifications of the present invention have been described, these embodiments and modifications are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and modifications can be implemented in other various forms, and various omissions, replacements and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

Reference Signs List 1 imaging device 2 TOF camera 3 color aperture camera 4 control unit 5 stereo camera 10a light emitting unit 11a, 11b aperture 12a, 12b lens 13a first imaging unit 14a first distance calculation unit 15 distance calculation unit 17b color filter 18b 2 imaging unit 19b second distance calculation unit 20 fence 21 object 22 third distance calculation unit 30 pixel of first distance image pixel of second distance image pixel 50 of third distance image

Claims (6)

A light emitting unit for emitting light to a subject, a first imaging unit for receiving a reflected light from the subject of light emitted by the light emitting unit, and acquiring a first image, and the light emitting unit for emitting light Then the first
And a first distance calculation unit for calculating a first distance to the subject for each pixel of the first image based on a time until the image pickup unit receives the reflected light from the subject. TOF
With the camera,
The second image pickup unit provided adjacent to the TOF camera, receiving light from the subject, and acquiring a second image; and the light of the second image based on the light received by the second image pickup unit. A second distance calculation unit that calculates a second distance to the subject for each pixel;
A third distance calculation unit that calculates a third distance to the subject for each pixel based on a parallax between the first image and the second image;
Distance calculation means for calculating a fourth distance to the subject based on the first distance, the second distance, and the third distance;
An imaging device comprising: The color aperture camera is attached to an opening of a lens that collects light from the subject, and has a color filter divided into two to cause blurring and color deviation in an image based on the distance to the subject.
The TOF camera and the color aperture camera are juxtaposed in a direction perpendicular to the division direction of the color filter.
The imaging device according to claim 1. The distance calculation means
The first distance is calculated as a fourth distance if any of the first distance, the second distance, and the third distance is a valid non-zero value.
The imaging device according to claim 1 or 2. The distance calculation means
If the numerical value of either the first distance or the second distance is a non-zero effective number and the third distance is an non-zero effective number, the third distance Calculated as the distance of
The imaging device according to any one of claims 1 to 3. The distance calculation means
The fourth distance is calculated using weighted values for each of the first distance, the second distance, and the third distance.
The imaging device according to claim 1 or 2. A TOF camera including: a light emitting unit that emits light to a subject; and a first imaging unit that receives reflected light from the subject of the light emitted by the light emitting unit and acquires a first image.
In the direction along with the TOF camera, which causes blur and color shift in the image based on the distance to the subject at the opening of the lens that is juxtaposed to the TOF camera and focuses the light from the subject An imaging apparatus having a color aperture camera having a color filter divided into two and a second imaging unit for acquiring a second image;
A first distance to the subject for each pixel of the first image based on a time from when the light emitting unit emits light to when the first imaging unit receives reflected light from the subject Calculate
A second distance to the subject is calculated for each pixel of the second image based on the light received by the second imaging unit,
A third distance to the subject is calculated for each pixel based on the parallax between the first image and the second image,
A fourth distance to the subject is calculated based on the first distance, the second distance, and the third distance.
The distance calculation method of an imaging device.

Images