JP2015194388A - Imaging device and imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging technology that is hard to be affected by environmental factors.SOLUTION: An imaging device A according to an embodiment of the present application comprises: a lens optical system L that has first to third optical areas D1, D2 and D3; an image pickup element N that has a plurality of first to third pixels arranged so as to receive light passing through the lens optical system L; and an array-shape optical element K that is arranged between the lens optical system L and the image pickup element N. The first area D1 is configured so as to cause visible light to transmit, the second area D2 is configured to cause red light and a near infrared-ray to transmit, and the third area D3 is configured to cause light oscillating in a direction of a specific polarization axis to transmit. The array-shape optical element K is configured to cause light passing through the first optical area D1 to be incident upon the plurality of first pixels, cause light passing through the second optical area D2 to be incident upon the plurality of second pixels, and cause light passing through the third optical area D3 to be incident upon the plurality of third pixels.

Description

  The present application relates to an imaging apparatus and an imaging system.

  In the field of in-vehicle cameras, an imaging device is known in which a polarizer having a different polarization axis direction is provided for each imaging element in an optical path of a compound eye camera in order to detect a road surface state or a lane (for example, Patent Document 1). .

JP 2010-25915 A

  In the prior art, there is a problem that an appropriate image may not be acquired due to environmental factors such as nighttime, western sun, and rainy weather.

  The present disclosure provides an imaging technique that is less susceptible to such environmental factors.

  An imaging device according to one embodiment of the present invention is arranged to receive a lens optical system having a first optical region, a second optical region, and a third optical region, and light that has passed through the lens optical system. An imaging element having a plurality of first pixels, a plurality of second pixels, and a plurality of third pixels, and an array-like optical element disposed between the lens optical system and the imaging element. . The first to third optical regions are a region that transmits visible light, a region that transmits red light and near infrared light, a region that transmits light that vibrates in the direction of a specific polarization axis, and a predetermined ratio of visible light. These are three areas selected from the four areas that are attenuated and transmitted through the area. The arrayed optical element causes light transmitted through the first optical region to enter the plurality of first pixels, and allows light transmitted through the second optical region to enter the plurality of second pixels. The light transmitted through the third optical region is incident on the plurality of third pixels.

  According to the imaging apparatus according to one embodiment of the present invention, it is possible to perform imaging that is not easily influenced by environmental factors such as nighttime, western sun, and rainy weather.

It is a mimetic diagram showing imaging device A of Embodiment 1 of this indication. It is a front view when the first to third optical regions D1 to D3 are viewed from the subject side. 2 is a perspective view showing a structure of a lenticular lens K. FIG. (A) is a figure which expands and shows the lenticular lens K and the image pick-up element N which are shown in FIG. 1, (b) is a figure which shows the arrangement | sequence of the pixel on the image pick-up surface Ni. It is a figure which shows an example of the vehicle Ve carrying the imaging device A of Embodiment 1 of the present disclosure. It is a figure for demonstrating the example of the recognition of the brake lamp and red signal which utilized 1st image information and 2nd image information. It is a mimetic diagram showing imaging device A of Embodiment 2 of this indication. It is a front view when the first to fourth optical regions D1 to D4 are viewed from the subject side. (A) is a figure which expands and shows the lenticular lens K and the image pick-up element N which are shown in FIG. 7, (b) is a figure which shows the arrangement | sequence of the pixel on the image pick-up surface Ni. It is the schematic of the distance measuring device using two imaging devices. 10 is a flowchart illustrating an example of a synthesis process of first to third image information by a signal processing unit. It is a figure for demonstrating SAD calculation. (A) And (b) is a figure which shows the example which integrated the lenticular lens and the image pick-up element N. FIG. It is a figure which shows the structural example at the time of using a microlens array instead of a lenticular lens. (A) shows the configuration of the optical element, and (b) shows the configuration of the microlens array K ′. It is a figure which shows the pixel arrangement | sequence on the imaging surface Ni in the structure of FIG.

  The outline of the embodiment of the present disclosure is as follows.

  (1) An imaging device according to one embodiment of the present invention receives a lens optical system having a first optical region, a second optical region, and a third optical region, and light that has passed through the lens optical system. An image sensor having a plurality of first pixels, a plurality of second pixels, and a plurality of third pixels, and an array-like optical element disposed between the lens optical system and the image sensor With. The first to third optical regions are a region that transmits visible light, a region that transmits red light and near infrared light, a region that transmits light that vibrates in the direction of a specific polarization axis, and a predetermined ratio of visible light. These are three areas selected from the four areas that are attenuated and transmitted through the area. The arrayed optical element causes light transmitted through the first optical region to enter the plurality of first pixels, and allows light transmitted through the second optical region to enter the plurality of second pixels. The light transmitted through the third optical region is incident on the plurality of third pixels.

  (2) In one embodiment, the first to third optical regions are arranged in a first direction perpendicular to an optical axis direction of the lens optical system, and the arrayed optical element includes the first optical element. This is a lenticular lens in which a plurality of cylindrical lenses are arranged in the direction of.

  (3) In one embodiment, the first optical region is configured to transmit red light and near-infrared light, and the second optical region is configured to transmit visible light, and the third optical region The optical region is configured to transmit light that vibrates in the direction of a specific polarization axis.

  (4) In one embodiment, the lens optical system further includes a fourth optical region that attenuates and transmits visible light at a predetermined rate, and the imaging element further includes a plurality of fourth pixels. The arrayed optical element is further configured to make light transmitted through the fourth optical region incident on the plurality of fourth pixels.

  (5) In one embodiment, the first to fourth optical regions are arranged in a first direction perpendicular to an optical axis direction of the lens optical system, and the arrayed optical element includes the first optical element. This is a lenticular lens in which a plurality of cylindrical lenses are arranged in the direction of.

  (6) In an embodiment, the arrayed optical element and the imaging element are integrated.

  (7) A distance measuring device according to another aspect of the present invention outputs two imaging devices, each of which is the imaging device according to any one of (1) to (6) above, and the two imaging devices. And a distance calculation circuit for generating distance information of the subject based on the received signal.

  (8) In one embodiment, each of the two imaging devices includes a composite image based on signals output from the plurality of first pixels, the plurality of second pixels, and the plurality of third pixels. The distance calculation circuit generates the distance information by performing pattern matching between the two images output from the two imaging devices.

  (9) In an embodiment, each of the two imaging devices is the imaging device according to (4) or (5), wherein the plurality of first pixels, the plurality of second pixels, A composite image based on a plurality of third pixels and signals output from the plurality of fourth pixels is generated and output, and the distance calculation circuit is configured to output the two images output from the two imaging devices. The distance information is generated by performing pattern matching in step (b).

  (10) An imaging system according to another aspect of the present invention is configured such that, in the imaging apparatus according to any one of (1) to (6), the lens optical system transmits red light and near infrared light. An imaging device having an optical region and an illumination device that emits near infrared rays.

  (11) A vehicle according to another aspect of the present invention provides an imaging device according to any one of (1) to (6), a distance measuring device according to any one of (7) to (9), and At least one of the imaging systems according to (10) above is provided.

  Hereinafter, more specific embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same reference numerals are assigned to the same or similar components.

(Embodiment 1)
FIG. 1 is a schematic diagram illustrating an imaging apparatus A according to the first embodiment. The imaging apparatus A according to the present embodiment includes a lens optical system L having an optical axis V, a lenticular lens K disposed near the focal point of the lens optical system L, an imaging element N, and a signal processing unit C.

  The lens optical system L has a diaphragm S on which light from a subject (not shown) is incident, an optical element L1 that is arranged so that light that has passed through the diaphragm S enters, and light that has passed through the optical element L1 is incident. And a lens L2 arranged so as to do so. The lens optical system L has a first optical region D1, a second optical region D2, and a third optical region D3.

  The lens L2 may be configured by a single lens or may be configured by a plurality of lenses. Further, a configuration may be adopted in which a plurality of sheets are arranged in front of or behind the diaphragm S. FIG. 1 shows a configuration example in which only one sheet is arranged behind the diaphragm S.

  The optical element L1 is disposed in the vicinity of the stop S and includes a portion located in the first optical region D1, a portion located in the second optical region D2, and a portion located in the third optical region D3. . In the first optical region D1, a spectral filter that transmits light in the red wavelength region (hereinafter referred to as “red light”) and light having a wavelength in the near infrared region (near infrared) is disposed. Yes. A spectral filter that transmits visible light is disposed in the second optical region D2. In the third optical region D3, a polarizing filter that transmits light whose electric field vibrates in a direction of a specific polarization axis (transmission axis) is disposed. Here, “red light” is, for example, light having a wavelength in the range of 600 nm to 700 nm, and near infrared light is light (electromagnetic wave) having a wavelength in the range of approximately 700 nm to 2.5 μm.

  In the present embodiment, the light that has passed through the three optical regions D1, D2, and D3 enters the lenticular lens K after passing through the lens L2. The lenticular lens K images the light that has passed through the optical region D1 to the plurality of pixels P1 in the imaging device N, the light that has passed through the optical region D2 to the plurality of pixels P2 in the imaging device N, and the light that has passed through the optical region D3. It is designed to enter a plurality of pixels P3 in the element N. The image sensor N converts light incident on the pixels P1, P2, and P3 into a pixel signal corresponding to the intensity. Here, the “pixel signal” refers to a signal indicating a luminance value (pixel value) generated by photoelectric conversion in each of the pixels P1, P2, and P3. The signal processing unit C receives a pixel signal from the image sensor N, generates image information of an image using red light and near infrared rays from luminance information of the pixel group of the pixel P1, and outputs the image information. Further, image information of an image by visible light is generated and output from the luminance information of the pixel group of the pixel P2. Further, the image information of the image by the polarized light whose electric field vibrates in the direction of the specific polarization axis is generated and output from the luminance information of the pixel group of the pixel P3. The signal processing unit C may be configured by an electronic circuit such as a digital signal processor (DSP), or may be configured by a combination of an arithmetic device such as a CPU and a memory storing a program.

  In FIG. 1, light beams B1, B2, and B3 are light beams that pass through the first to third optical regions D1, D2, and D3 on the optical element L1, respectively. The light beams B1, B2, and B3 pass through the stop S, the optical element L1, the lens L2, and the lenticular lens K in this order, and reach the imaging surface Ni (shown in FIG. 4) on the imaging element N.

  FIG. 2 is a front view when the first to third optical regions D1 to D3 are viewed from the subject side. All of the first to third optical regions D1 to D3 in the optical element L1 have the same area and are arranged in a direction perpendicular to the optical axis V (y direction in FIG. 2). In FIG. 2, the broken line s indicates the opening area of the stop S. The x direction and the y direction shown in FIG. 2 correspond to the horizontal direction and the vertical direction when the imaging device is used, respectively. Further, the optical regions D1 to D3 need not all have the same shape, and the arrangement of the optical regions D1 to D3 may be different from the example of FIG.

  FIG. 3 is a perspective view showing the structure of the lenticular lens K. FIG. In the present embodiment, the lenticular lens K is used as an arrayed optical element. The lenticular lens K has a plurality of cylindrical lenses M1. Each cylindrical lens M1 has a shape extending in the x direction, and a plurality of cylindrical lenses M1 are arranged in the y direction. The cross section perpendicular to the x direction of each cylindrical lens M1 has a curved shape protruding toward the image sensor N side. In the present embodiment, the x direction and the y direction are orthogonal to each other.

  As shown in FIG. 1, the lenticular lens K is disposed in the vicinity of the focal point of the lens optical system L, and is disposed at a position away from the imaging surface Ni by a predetermined distance.

  4A is an enlarged view showing the lenticular lens K and the image sensor N shown in FIG. 1, and FIG. 4B is a diagram showing an arrangement of pixels on the imaging surface Ni. As described above, the lenticular lens K is disposed so that the surface on which the cylindrical lens M1 is formed faces the imaging surface Ni side.

  The imaging element N includes an imaging surface Ni and a plurality of pixels P. As shown in FIG. 4B, the plurality of pixels P are two-dimensionally arranged in the x direction and the y direction. When the arrays in the y direction and the x direction are called rows and columns, respectively, the plurality of pixels are arranged in m rows and 1 columns (l and m are integers of 2 or more) on the imaging surface Ni, for example. That is, a group of pixels in which one pixel is arranged in the x direction is arranged in m rows from the 1st to mth rows in the y direction.

  The plurality of pixels P are each arranged in the x direction, and are divided into a plurality of pixels P1, a plurality of pixels P2, and a plurality of pixels P3 constituting a row. In the y direction, the row of pixels P1, the row of pixels P2, and the row of pixels P3 are repeatedly arranged in this order. The lenticular lens K is arranged so that one of the cylindrical lenses M1 corresponds to a pixel group of three rows of pixels P1 to P3 on the imaging surface Ni. On the imaging surface Ni, a microlens Ms is provided so as to cover each of the plurality of pixels P1 to P3.

  In the present embodiment, the plurality of pixels P1 to P3 all have the same shape. For example, the plurality of first pixels P1, the plurality of second pixels P2, and the plurality of third pixels P3 have the same rectangular shape and have the same area.

  Each of the pixels P1, P2, and P3 includes a light detection cell (for example, a photodiode) that outputs an electrical signal corresponding to the intensity of light received by photoelectric conversion. The output of each photodetecting cell is sent to the signal processing unit C and used to generate image information. In this embodiment, an element such as an optical filter that selectively transmits light is not provided at a position facing each photodetecting cell. However, each of the plurality of pixels P1 is provided with an optical filter having a transmission characteristic equivalent to that of the optical region D1, and each of the plurality of pixels P2 is provided with an optical filter having a transmission characteristic equivalent to that of the optical region D2. Each of these may be provided with an optical filter having transmission characteristics equivalent to those of the optical region D3. Thereby, unnecessary light can be prevented from entering each pixel.

  In the lenticular lens K, most of the light beam B1 that has passed through the optical region D1 (shown in FIGS. 1 and 2) on the optical element L1 reaches the pixel P1 on the imaging surface Ni, and the light beam B2 that has passed through the optical region D2. Is most likely to reach the pixel P2 on the imaging surface Ni, and most of the light beam B3 transmitted through the optical region D3 reaches the pixel P3 on the imaging surface Ni. Specifically, the above configuration is realized by appropriately setting parameters such as the refractive index of the lenticular lens K, the distance from the imaging surface Ni, and the radius of curvature of the surface of the cylindrical lens M1.

  In FIG. 1, each component is arranged so that light that has passed through the diaphragm S is directly incident on the optical element L1 (without passing through another optical member). The optical element L1 is not limited to the configuration illustrated in FIG. In this case, the light that has passed through the optical element L1 enters the diaphragm S directly (without passing through another optical member).

  The lenticular lens K has a function of distributing the emission direction according to the incident angle of the light beam. Therefore, the luminous flux can be distributed to the pixels on the imaging surface Ni so as to correspond to the first to third optical regions D1, D2, and D3 divided in the vicinity of the stop S.

  With the above configuration, using the luminance information of each pixel group of the pixels P1, P2, and P3, the first image information having the intensity information of red light and near infrared light, and the second image having the intensity information of visible light Information and third image information having intensity information of polarized light oscillating in the direction of a specific polarization axis (transmission axis) can be acquired at the same time.

  The imaging device of the present embodiment can be suitably used as, for example, an in-vehicle camera. FIG. 5 is a diagram illustrating an example of a vehicle Ve equipped with the imaging device A of the present embodiment. The vehicle Ve includes an imaging device A and an illumination device (headlight) Li that emits visible light and near infrared light. The illumination device Li is configured to emit visible light (low beam) at a relatively low angle and emit near-infrared light (high beam) at a relatively high angle. As shown in FIG. 5, the range irradiated by the low beam, the range irradiated by the high beam, and the shootable range (camera field of view) of the imaging apparatus A are all different.

  When the imaging apparatus A of the present embodiment is applied as an in-vehicle camera, a visible monochrome image can be obtained from the second image information, which can be used for image recognition such as lane discrimination. Moreover, since the near-infrared ray reflected from the front object can be recognized by the first image information, a distant person or an obstacle can be detected at night. Since the red light can be recognized by using the first image information and the second image information, the brake lamp and red signal of the preceding vehicle can also be detected. Furthermore, by using the third image information, it is possible to obtain an image in which light reflection by the road surface is suppressed, or to perform high dynamic range (HDR) synthesis in combination with the second image information. Hereinafter, these utilization examples will be described.

  FIG. 6 is a diagram for explaining an example of recognition of a brake lamp and a red signal using the first image information and the second image information. FIGS. 6A and 6B show examples of images obtained from the first and second image information, respectively, when the brake lamp is turned off and the signal is blue. FIGS. 6C and 6D show examples of images obtained from the first and second image information, respectively, when the brake lamp is lit and the signal is red.

  When the brake lamp of the preceding vehicle Ob1 is turned off or when the traffic light Ob2 is not red, as shown in FIGS. 6A and 6B, the location corresponding to the brake lamp of the preceding vehicle Ob1 and the red color of the traffic light Ob2 The portion corresponding to is dark in both the first image information and the second image information. On the other hand, when the brake lamp of the preceding vehicle is lit or when the traffic light is red, as shown in FIGS. 6C and 6D, the location corresponding to the brake lamp of the preceding vehicle and the signal red The portion corresponding to is bright in both the first image information and the second image information. Therefore, it is possible to determine whether or not the brake lamp or the traffic light of the preceding vehicle emits red light by comparing the first image information and the second image information after recognizing the preceding vehicle and the traffic signal. Can do. If it is possible to determine whether or not these lights emit red, the determination result can be used for driving assistance such as braking.

  Since the first image information includes near-infrared information, a high-beam near-infrared projector is mounted on the headlight Li as shown in FIG. It is possible to detect obstacles and persons in the area. Thereby, when an obstacle or a person is far away, the driver can be promptly alerted by voice or display.

  Since the second image information includes information in the visible light band, the second image information can be regularly used for daytime image recognition and for nighttime low beam irradiation region image recognition.

  Since the third image information is information of an image taken through a polarization filter, it corresponds to an image from which a specific polarization component of incident light has been removed. Therefore, by setting the polarization transmission axis of the third optical region D3 in a direction perpendicular to the road surface, it is possible to reflect light on the road surface when facing in the rain, snow, or on the west, or light on the water surface. It is possible to acquire an image in which the influence of reflection is suppressed. The third image information makes it easy to distinguish road lanes under conditions such as rainy weather or in the sun. Can be difficult. Moreover, when the imaging device A is installed inside the windshield of the vehicle, the reflection of the dashboard can be reduced, and an image suitable for image recognition can be obtained.

  Furthermore, since the third image information is image information taken through a polarizing filter, an image with an exposure amount reduced by half compared to the second image information can be acquired. Therefore, an image having a wide dynamic range can be generated by combining the second image information and the third image information. For example, in the second image information, a pixel whose luminance value has reached the upper limit value (255 in the case of 8 bits) is replaced with a value obtained by gain adjustment of the luminance value of the corresponding pixel in the third image information. A composite image having a wide dynamic range can be generated. The gain adjustment may be performed by multiplying 1 / r when the ratio of the light transmittance in the third optical region D3 to the light transmittance in the second optical region D2 is r (for example, 0.4). HDR synthesis is not limited to this method, and widely known methods can be applied. By HDR synthesis, it is possible to suppress the occurrence of whiteout or blackout in the preceding vehicle or road lane near the entrance of the tunnel.

  As described above, in the present embodiment, it is possible to perform imaging that is not easily influenced by daytime and weather conditions. For this reason, the reliability of the safe driving support using image recognition can be improved.

  In the present embodiment, the first optical region D1 is configured to transmit red light and near-infrared light, and the second optical region D2 is configured to transmit visible light, and the third The optical region D3 is configured to transmit light oscillating in the direction of a specific polarization axis, but it is not always necessary to have such a configuration. For example, one of the first to third optical regions D1, D2, and D3 may be a region that attenuates and transmits visible light at a predetermined rate. Such a region can be realized by using, for example, an ND (Neutral Density) filter having a visible light transmittance lower than that of the polarizing filter in the present embodiment. By using such an ND filter, the dynamic range can be further widened. As described above, when three optical regions are provided in the optical element as in the present embodiment, the first to third optical regions are a region that transmits visible light, a region that transmits red light and near infrared light, and a specific region. The three regions selected from the four regions of the region that transmits light oscillating in the direction of the polarization axis and the region that transmits visible light after being attenuated at a predetermined ratio may be used.

(Embodiment 2)
The second embodiment is different from the first embodiment in that the number of divisions of the optical region is four and that four rows of pixels correspond to one cylindrical lens M1 in the lenticular lens K. The following description will focus on the differences from the first embodiment, and a detailed description of the same contents will be omitted.

  FIG. 7 is a schematic diagram illustrating the imaging apparatus A according to the present embodiment. In the present embodiment, the optical element L1 in the lens optical system L has a first optical region D1, a second optical region D2, a third optical region D3, and a fourth optical region D4. The first to third optical regions D1, D2, and D3 are the same as those in the first embodiment. The fourth optical region D4 is an ND filter that attenuates and transmits visible light at a predetermined rate. In the following description, the transmittance of the ND filter is 12.5%, but this transmittance may be changed as necessary.

  In the present embodiment, the light that has passed through the four optical regions D1 to D4 passes through the lens L2, and then enters the lenticular lens K. The lenticular lens K images the light that has passed through the optical region D1 to the plurality of pixels P1 in the imaging device N, the light that has passed through the optical region D2 to the plurality of pixels P2 in the imaging device N, and the light that has passed through the optical region D3. It is designed such that light that has passed through the optical region D4 is incident on the plurality of pixels P4 in the image sensor N on the plurality of pixels P3 in the element N. The imaging element N converts light incident on the pixels P1, P2, P3, and P4 into a pixel signal corresponding to the intensity. The signal processing unit C receives a pixel signal from the image sensor N, generates image information of an image using red light and near infrared rays from luminance information of the pixel group of the pixel P1, and outputs the image information. Image information of an image by visible light is generated and output from the luminance information of the pixel group of the pixel P2. From the luminance information of the pixel group of the pixel P3, image information of an image by polarized light whose electric field vibrates in the direction of a specific polarization axis is generated and output. Furthermore, the image information of the image by the visible light attenuated by the ND filter is generated and output from the luminance information of the pixel group of the pixel P4.

  In FIG. 7, light beams B1, B2, B3, and B4 are light beams that pass through the first to fourth optical regions D1 to D4 on the optical element L1, respectively. The light beams B1 to B4 pass through the stop S, the optical element L1, the lens L2, and the lenticular lens K in this order, and reach the imaging surface Ni (shown in FIG. 9) on the imaging element N.

  FIG. 8 is a front view when the first to fourth optical regions D1 to D4 are viewed from the subject side. The first to fourth optical regions D1 to D4 in the optical element L1 have a rectangular shape and are arranged in a direction perpendicular to the optical axis V (y direction in FIG. 8). The shapes and areas of the optical regions D1 to D4 need not all be the same. The arrangement of the optical regions D1 to D4 may be different from the example of FIG.

  FIG. 9A is an enlarged view of the lenticular lens K and the image sensor N shown in FIG. 7, and FIG. 9B is a diagram showing an arrangement of pixels on the image pickup surface Ni. The plurality of pixels P are each arranged in the x direction, and are divided into a plurality of pixels P1, a plurality of pixels P2, a plurality of pixels P3, and a plurality of pixels P4 constituting a row. In the y direction, the row of the pixel P1, the row of the pixel P2, the row of the pixel P3, and the row of the pixel P4 are repeatedly arranged in this order. The lenticular lens K is arranged so that one of the cylindrical lenses M1 corresponds to a pixel group of four rows of pixels P1 to P4 on the imaging surface Ni. On the imaging surface Ni, a microlens Ms is provided so as to cover each of the plurality of pixels P1 to P4.

  Each of the pixels P1 to P4 has a light detection cell that outputs an electrical signal corresponding to the intensity of light received by photoelectric conversion. The output of each photodetecting cell is sent to the signal processing unit C and used to generate image information. In this embodiment, an element such as an optical filter that selectively transmits light is not provided at a position facing each photodetecting cell. However, each of the plurality of pixels P1 is provided with an optical filter having a transmission characteristic equivalent to that of the optical region D1, and each of the plurality of pixels P2 is provided with an optical filter having a transmission characteristic equivalent to that of the optical region D2. May be provided with an optical filter having a transmission characteristic equivalent to that of the optical region D3, and an optical filter having a transmission characteristic equivalent to that of the optical region D4 may be provided for each of the plurality of pixels P4. Thereby, unnecessary light can be prevented from entering each pixel.

  In the lenticular lens K, most of the light beam B1 that has passed through the optical region D1 on the optical element L1 reaches the pixel P1 on the imaging surface Ni, and most of the light beam B2 that has passed through the optical region D2 is on the imaging surface Ni. Most of the light beam B3 reaching the pixel P2 and transmitted through the optical region D3 reaches the pixel P3 on the imaging surface Ni, and most of the light beam B4 transmitted through the optical region D4 reaches the pixel P4 on the imaging surface Ni. Designed to be. Specifically, the above configuration is realized by appropriately setting parameters such as the refractive index of the lenticular lens K, the distance from the imaging surface Ni, and the radius of curvature of the surface of the cylindrical lens M1.

  With the above configuration, in addition to the first to third image information described in the first embodiment, the fourth image information based on the luminance information of the pixel group of the pixel P4 can be simultaneously acquired. The fourth image information corresponds to an image having an exposure amount of 12.5% with respect to the second image information. In the first embodiment, an image with a wide dynamic range can be generated using the second image information and the third image information. However, in this embodiment, the dynamic range can be further expanded. Since the transmittance of the polarizing filter is about 40%, the exposure amount ratio between the second image information and the third image information is about 1: 0.4. On the other hand, in the present embodiment, since the exposure amount ratio between the second image information and the fourth image information is 1: 0.125, an image having a wider dynamic range than that of the first embodiment is acquired. be able to. As a result, even when facing the entrance of a tunnel or facing the west, it is possible to further reduce whiteout and blackout, thereby further improving the reliability of safe driving support using image recognition.

  Instead of the ND filter having a transmittance of 0.125, a polarization filter having a transmittance of about 0.25 for light vibrating in the direction of a specific polarization axis may be provided in the fourth optical region D4. Even in such a configuration, since the light transmittance of the fourth optical region D4 is about 0.125, the same effect as the present embodiment can be obtained.

(Embodiment 3)
The third embodiment relates to a distance measuring device using a plurality of imaging devices of the first embodiment. FIG. 10 is a schematic diagram of a distance measuring device using two imaging devices. The distance measuring device includes two imaging devices A and a distance calculation circuit C2 that generates and outputs subject distance information. The distance calculation circuit C2 calculates parallax by performing pattern matching between two images generated by the two imaging devices A, and calculates the distance to the subject by calculation based on the principle of triangulation based on the calculated parallax. To do. The distance calculation circuit C2 may be configured by an electronic circuit such as a DSP, or may be configured by a combination of a calculation device such as a CPU and a memory storing a program. In the present embodiment, the arrangement direction E of the cylindrical lenses in the lenticular lens K of each imaging device is orthogonal to the baseline direction B of the distance measuring device. Therefore, the resolution of parallax calculation can be increased as compared with the case where the arrangement direction E of the lenticular cylindrical lenses is the same as the base line direction B of the distance measuring device.

  The signal processing unit C (FIG. 1) in each imaging apparatus A according to the present embodiment synthesizes the first to third image information by the light transmitted through the first to third optical regions D1 to D3, respectively. Image information (referred to as “composite image information”). The image indicated by each composite image information is an image that firmly captures the object regardless of daytime or nighttime or weather conditions. The distance calculation circuit C2 generates parallax information by performing pattern matching on the basis of the output composite image information, and generates distance information to the target object by calculation based on the principle of triangulation from the parallax information. . Here, the “distance information” is not limited to information indicating the value of the distance itself, but may be information indicating the relative positional relationship between the distance measuring device and the object, such as coordinate information of the object. .

  With the above configuration, it is possible to realize a distance measuring device that can acquire a distance to an object regardless of day and night or weather conditions.

  Hereinafter, an example of a method for synthesizing the first to third image information by the signal processing unit C of each imaging apparatus A will be described. In the following description, an image based on the first image information is referred to as a “near infrared image”, an image based on the second image information is referred to as a “visible image”, and an image based on the third image information is referred to as a “polarized image”. .

  FIG. 11 is a flowchart illustrating an example of the synthesis process of the first to third image information by the signal processing unit C of each imaging apparatus A. In step S100, the signal processing unit C determines whether the environment at the time of shooting is daytime or nighttime. This determination can be made based on whether the headlight switch is on or off, for example. Or you may perform based on other information, such as the average brightness | luminance of a visible image, the luminance ratio of a visible image and a near-infrared image, and time information. When the signal processing unit C determines that the photographing time is daytime, the signal processing unit C executes the processing of steps S101 to S104, and when it determines that the shooting time is nighttime, the signal processing unit C executes the processing of steps S111 and S112.

  First, the process when it is determined in step S100 that it is daytime will be described.

  In step S101, a new polarization image (referred to as a “gain-adjusted polarization image”) is generated by dividing the polarization image by the transmittance of the polarization filter in the optical region D3.

  In step S102, the “visible image” and the “gain-adjusted polarization image” are compared for each pixel, and an image having the lower luminance value as the luminance value of the pixel (referred to as “synthesized image 1”) is generated. To do. The composite image 1 is an image in which the influence of light reflection from the road surface and the reflection of the dashboard on the windshield are suppressed.

  In step S103, a new composite image (referred to as “composite image 2”) is generated by HDR composite of “composite image 1” and “polarized image”. The HDR synthesis method is, for example, the method described in Embodiment 1 or other known methods.

  In step S104, the generated composite image 2 is sent to the distance calculation circuit C2.

  Through the above processing, the distance calculation circuit C2 can perform pattern matching using the two composite images 2 respectively transmitted from the two imaging devices A, and calculate parallax information and distance information.

  Next, a process when it is determined that it is nighttime in step S100 will be described.

  In step S111, the “visible image” and the “near infrared image” are averaged for each pixel to generate a composite image (referred to as “composite image 3”).

  In step S112, “composite image 3” is sent to the distance calculation circuit.

  Through the above processing, the distance calculation circuit C2 can perform pattern matching using the two composite images 3 respectively transmitted from the two imaging devices A, and calculate parallax information and distance information.

  Note that the above synthesis method is an example, and the present disclosure is not limited to the above synthesis method.

Next, an example of a parallax calculation method by the distance calculation circuit C2 will be described. The distance calculation circuit C2 first extracts an image shift generated between a predetermined image block (reference image) in one of the two composite images to be compared and a predetermined image block (reference image) in the other by pattern matching. To do. The correlation degree of pattern matching can be obtained, for example, by an evaluation function SAD (Sum of Absolute Difference) that is a sum of luminance differences (absolute values) of pixels between the base image and the reference image. Here, if the calculation block size is m × n pixels, SAD can be obtained by (Equation 1).

  In (Equation 1), x and y are the coordinates of the imaging surface, and I0 and I1 are the luminance value of the standard image and the luminance value of the reference image, respectively, at the coordinates shown in parentheses. FIG. 12 is a diagram for explaining the SAD calculation. In the SAD calculation, the position of the search block area of the reference image is shifted by dx in the baseline direction as shown in FIG. 12 with respect to the base block area of the base image, and dx at which the SAD becomes the minimum value becomes the parallax Px. Since SAD can be calculated with arbitrary coordinates, the parallax of the entire region within the imaging field of view can be extracted. When extracting the parallax of a specific subject in the field of view, the region may be limited. For example, the distance calculation circuit C2 extracts the parallax Px of the area corresponding to the predetermined area of the subject.

  Although SAD is used as the evaluation function here, the parallax may be calculated using another evaluation function such as SSD (Sum of Squared Difference).

  The distance calculation circuit C2 generates information indicating the distance from the distance measuring device to the object by performing a calculation based on the principle of triangulation from the parallax information thus obtained. Thereby, since the distance information of an object can be obtained regardless of day and night, a vehicle-mounted distance measuring device capable of forward monitoring can be realized.

  In the present embodiment, two imaging devices A in the first embodiment are used, but two imaging devices A in the second embodiment may be used instead. Further, a configuration in which any one of the first to third optical regions D1 to D3 in the imaging apparatus A according to Embodiment 1 is replaced with an ND filter may be employed. Hereinafter, an example of synthesis processing by the signal processing unit C in these configurations will be described.

<Example 1>
First, the first to third optical regions D1 to D3 are regions that transmit visible light (for example, a near-infrared cut filter), regions that transmit visible light after being attenuated at a predetermined rate (for example, a transmittance of 25%) An example in the case of a combination of an ND filter) and a region that transmits light oscillating in the direction of a specific polarization axis (for example, a polarization filter) will be described. In the following description, an image by light transmitted through the ND filter is referred to as an “ND filter image”. In this example, the signal processing circuit C executes different processing in fine weather and rainy weather.

  In fine weather, the signal processing circuit C sends an image (referred to as “composited image 4”) generated by HDR combining the “visible image” and the “ND filter image” to the distance calculation circuit C2. Thereby, the distance calculation circuit C2 can generate parallax information and distance information using the two composite images 4 sent from the two imaging devices.

  In the rain, the signal processing circuit C first generates a “gain-adjusted polarization image” by dividing the polarization image by the transmittance of the polarization filter. Next, the “visible image” and the “gain-adjusted polarization image” are compared for each pixel, and a “synthesized image 5” having the lower luminance value as the luminance value of the pixel is generated to the distance calculation circuit C2. Send it out. Thereby, the distance calculation circuit C2 can generate parallax information and distance information using the two composite images 5 sent from the two imaging devices.

  Here, whether the weather during shooting is clear or rainy can be determined based on, for example, whether the wiper switch is on or off. Alternatively, the weather may be determined based on the characteristics of the image. For example, if it is estimated that the road surface is wet based on the difference between the “visible image” and the “gain-adjusted polarized image”, it may be determined that it is raining.

<Example 2>
Next, as in the second embodiment, the four optical regions D4 are regions that transmit visible light (for example, near-infrared cut filters), regions that transmit red light and near-infrared light (for example, spectral filters), In the case of a combination of a region where visible light is attenuated and transmitted at a predetermined ratio (for example, an ND filter having a transmittance of 25%) and a region which transmits light oscillating in the direction of a specific polarization axis (for example, a polarizing filter) An example will be described. In this example, the signal processing circuit C executes different processing depending on whether the weather is sunny or rainy and whether it is daytime or nighttime.

  In the rainy night, the signal processing circuit C generates a “composite image 6” by averaging the above-mentioned “composite image 5” and “near-infrared image” for each pixel, and the distance calculation circuit C2 To send. Thereby, the distance calculation circuit C2 can generate the parallax information and the distance information using the “composite image 6” sent from the two imaging devices.

  During the rainy daytime, the signal processing circuit C generates the above-mentioned “composite image 5” and sends it to the distance calculation circuit C2. Thereby, the distance calculation circuit C2 can generate the parallax information and the distance information using the “composite image 5” sent from the two imaging devices.

  During the daytime in fine weather, the signal processing circuit C generates the above-mentioned “composite image 4” and sends it to the distance calculation circuit C2. Thereby, the distance calculation circuit C2 can generate the parallax information and the distance information using the “composite image 4” sent from the two imaging devices.

  At night during fine weather, the signal processing circuit C generates the above-mentioned “composite image 3” and sends it to the distance calculation circuit C2. Thereby, the distance calculation circuit C2 can generate the parallax information and the distance information using the “composite image 3” sent from the two imaging devices.

  Note that whether the weather at the time of shooting is sunny or rainy, and whether it is daytime or nighttime can be determined by any of the methods described above.

(Other embodiments)
The first to third embodiments have been described above. However, the technology of the present disclosure is not limited to these embodiments, and various modifications may be made, or different embodiments may be combined to form new embodiments. Hereinafter, other embodiments will be exemplified.

  FIG. 13 is a diagram illustrating an example in which the lenticular lens and the image sensor N are integrated. FIGS. 13A and 13B are views showing the lenticular lens K and the image sensor N in an enlarged manner, and show only light rays that have passed through one optical region. In the present embodiment, the lenticular lens Md1 is formed on the imaging surface Ni of the imaging element N. Pixels P are arranged in a matrix on the imaging surface Ni, as in the first embodiment. Among these pixels P, a cylindrical lens of one lenticular lens Md1 corresponds to a pixel group having the number of rows corresponding to the number of divisions of the optical region.

  In the configuration shown in FIG. 13B, the lenticular lens Md2 (or microlens) is formed so as to cover the pixel P, and the low refractive index layer W is interposed on the surface of the lenticular lens Md2 (or microlens). A lenticular lens Md1 is stacked. In the configuration shown in FIG. 13B, the light collection efficiency can be increased as compared with the configuration in FIG. When the lenticular lens Md2 is used, a cylindrical lens of one lenticular lens Md2 is arranged corresponding to each row of the pixel group. In the case of using a microlens, one pixel is arranged to correspond to one microlens.

  When the lenticular lens K is separated from the image sensor N as in the first embodiment, it is difficult to align the lenticular lens K and the image sensor N. However, as in the present embodiment, the lenticular lens Md1 is imaged. With the structure formed on the element N, alignment can be performed by a wafer process. Therefore, alignment can be facilitated and alignment accuracy can be improved.

  In the present embodiment, instead of the lenticular lens Md2 provided on the pixels in each row of the image sensor N or the microlens provided on each pixel, an optical element having another shape may be provided in the image sensor. For example, you may use the refractive index distribution element which condenses light by distribution of the material from which refractive index differs as disclosed by Unexamined-Japanese-Patent No. 2008-10773. The entire disclosure of JP 2008-10773 is incorporated herein by reference. The diffractive optical element having such a structure can be manufactured using, for example, a semiconductor photolithography technique. For example, since a conventional microlens having a lens surface is manufactured by thermally deforming a resin, it is difficult to make the curved surfaces of the lens surfaces different between a plurality of microlenses provided in a plurality of pixel shapes of an image sensor. there were. On the other hand, when a diffractive optical element is used, the optical characteristics can be changed by making the dimensions of the above-mentioned optical member different between a plurality of pixels of the imaging element. Therefore, even when a light beam is incident on the pixel of the image sensor N from an oblique direction by the optical system L or the lenticular lens K, the light can be efficiently collected on the pixel.

  Instead of the lenticular lens in each of the above embodiments, a microlens array may be used as an arrayed optical element. FIG. 14 shows an example of such a configuration.

  In the example shown in FIG. 14A, the optical element L1p includes a first optical region D1, a second optical region D2, a third optical region D3, and a fourth optical region D4. The first optical region D1, the second optical region D2, the third optical region D3, and the fourth optical region D4 pass through the optical axis V and are in the horizontal direction H (y direction on the imaging surface Ni of the imaging device) and These are four regions separated by border lines parallel to the direction perpendicular to the horizontal direction. The first to fourth optical regions D1 to D4 in this example are the same as the corresponding regions in the second embodiment.

  FIG. 14B is a perspective view of the microlens array K ′. The microlens array K ′ includes a plurality of microlenses M2 arranged in the x direction and the y direction. The cross section of each microlens M2 (each cross section perpendicular to the x direction and the y direction) has a curved surface shape and protrudes toward the image sensor N side. Each microlens M2 is arranged corresponding to a plurality of pixels arranged in the x direction and the y direction among a plurality of pixels arranged in the x direction and the y direction on the imaging surface Ni of the imaging element N.

  FIG. 15A is a diagram illustrating a pixel array on the imaging surface Ni in the present configuration example. As shown in FIG. 15A, the plurality of pixels arranged on the imaging surface Ni includes pixels P1, P2, P3, and P4. Each microlens M2 of the microlens array K ′ causes the light that has passed through the first optical region D1 to be incident on the pixel P1, and the light that has passed through the second optical region D2 is incident on the pixel P2. The light that has passed through the region D3 is incident on the pixel P3, and the light that has passed through the fourth optical region D4 is incident on the pixel P4.

  Accordingly, the light transmitted through each of the first optical region D1, the second optical region D2, the third optical region D3, and the fourth optical region D4 is detected by the pixels P1 to P4. Thereby, signal processing similar to that of the second embodiment is possible.

  In this configuration example, the combination of the optical regions of the second embodiment is used, but the combination of the optical regions of the first embodiment may be used. In that case, an optical region having the same characteristics as any of the first to third optical regions may be arranged instead of the optical region D4.

  FIG. 15B is a diagram illustrating a pixel array on the imaging surface Ni in another configuration example. As shown in FIG. 15B, the plurality of pixels arranged on the imaging surface Ni includes a plurality of pixel groups each including a group of four pixels arranged in two in the x direction and in the y direction. It includes a pixel P1, a plurality of pixels P2, a plurality of pixels P3, and a plurality of pixels P4. At this time, in each of the first to fourth optical regions D1, D2, D3, and D4 shown in FIG. 14, a filter that transmits light in the near-infrared band, a filter that transmits light in the visible light band, A polarizing filter that transmits light whose electric field vibrates in a direction of a specific polarization axis (transmission axis) and an ND filter with a transmittance of 12.5% are arranged.

  The plurality of pixels P1 includes four pixels that are not provided with a spectral filter, and the plurality of pixels P2, P3, and P4 include a pixel R that includes a spectral filter that mainly transmits red band light. A total of four pixels, one with two pixels G provided with a spectral filter that mainly transmits light in the green band and one pixel B provided with a spectral filter that mainly transmits light in the blue band It consists of

  Each microlens M2 of the microlens array K ′ enters the light passing through the first optical region D1 into the pixel group of the pixel P1 in the corresponding unit region (a region including 16 pixels surrounded by a solid line in the drawing). The light that has passed through the second optical region D2 is incident on the pixel group of the plurality of pixels P2 in the corresponding unit region, and the light that has passed through the third optical region D3 is the plurality of pixels in the corresponding unit region. The light is incident on the pixel group P3 and the light that has passed through the fourth optical region D4 is incident on the pixel group of the plurality of pixels P4 in the corresponding unit region.

  Accordingly, the light transmitted through each of the first optical region D1, the second optical region D2, the third optical region D3, and the fourth optical region D4 is detected by the pixels P1 to P4.

  The plurality of pixels P2 in this configuration example are configured with four pixels that are not provided with a spectral filter, but are configured with four pixels that are provided with a spectral filter that transmits light in the near-infrared band. Also good.

  With the configuration described above, the first image information having near-infrared intensity information, the second image information having visible light intensity information, and the polarized light that vibrates in the direction of a specific polarization axis (transmission axis). The third image information having the intensity information and the fourth image information whose exposure amount is 12.5% with respect to the second image information can be acquired simultaneously. Here, since the pixel groups P2, P3, and P4 are provided with RGB color filters, the second, third, and fourth image information each include color information. Therefore, it is possible to colorize a composite image using these image information, and not only can recognize information such as signs, but also can be used as a drive recorder for recording color images if a device for storing moving images is added. You can also.

  Further, in the first and second embodiments, if each of the pixels P2, P3, and P4 is provided with a color filter, the same effect as this configuration can be obtained.

  The imaging device disclosed in the present application can be used as an imaging device such as a digital still camera, a digital video camera, an in-vehicle camera, a surveillance camera, a skin diagnostic camera, an endoscope camera, and a capsule endoscope. Further, it can be applied to an imaging system such as a microscope and an electronic mirror.

A Imaging device L Lens optical system L1 Optical element L2 Lens D1 to D4 Optical region S Aperture K Lenticular lens Li Lighting device N Imaging device Ni Imaging surface M1, Md1 Cylindrical lens Md2 Lenticular lens (or microlens) on pixel
Ms Micro lens on pixel P1 to P4 Pixel on image sensor C Signal processing unit C2 Distance calculation circuit Ve Vehicle

Claims (11)

A lens optical system having a first optical region, a second optical region, and a third optical region;
An imaging device having a plurality of first pixels, a plurality of second pixels, and a plurality of third pixels arranged to receive the light that has passed through the lens optical system;
An array-like optical element disposed between the lens optical system and the imaging element;
With
The first to third optical regions are a region that transmits visible light, a region that transmits red light and near infrared light, a region that transmits light that vibrates in the direction of a specific polarization axis, and a predetermined ratio of visible light. Are three regions selected from four regions that are attenuated and transmitted by
The arrayed optical element causes light transmitted through the first optical region to enter the plurality of first pixels, and allows light transmitted through the second optical region to enter the plurality of second pixels. The light transmitted through the third optical region is configured to enter the plurality of third pixels.
Imaging device. The first to third optical regions are arranged in a first direction perpendicular to the optical axis direction of the lens optical system,
The arrayed optical element is a lenticular lens in which a plurality of cylindrical lenses are arranged in the first direction.
The imaging device according to claim 1. The first optical region is configured to transmit red light and near infrared light;
The second optical region is configured to transmit visible light;
The third optical region is configured to transmit light that vibrates in the direction of a specific polarization axis.
The imaging device according to claim 1 or 2. The lens optical system further includes a fourth optical region that attenuates and transmits visible light at a predetermined rate,
The imaging device further includes a plurality of fourth pixels,
The arrayed optical element is further configured to make light transmitted through the fourth optical region incident on the plurality of fourth pixels.
The imaging device according to claim 3. The first to fourth optical regions are arranged in a first direction perpendicular to the optical axis direction of the lens optical system,
The arrayed optical element is a lenticular lens in which a plurality of cylindrical lenses are arranged in the first direction.
The imaging device according to claim 4.   The imaging apparatus according to claim 1, wherein the arrayed optical element and the imaging element are integrated. Two imaging devices each being an imaging device according to any one of claims 1 to 6,
A distance calculation circuit that generates distance information of a subject based on signals output from the two imaging devices;
Ranging device comprising. Each of the two imaging devices generates and outputs a composite image based on signals output from the plurality of first pixels, the plurality of second pixels, and the plurality of third pixels,
The distance calculation circuit generates the distance information by performing pattern matching between two images output from the two imaging devices.
The distance measuring device according to claim 7. Each of the two image pickup devices is the image pickup device according to claim 4 or 5, wherein the plurality of first pixels, the plurality of second pixels, the plurality of third pixels, and the plurality of the plurality of image pickup devices. Generating and outputting a composite image based on the signal output from the fourth pixel;
The distance calculation circuit generates the distance information by performing pattern matching between two images output from the two imaging devices.
The distance measuring device according to claim 8. The imaging apparatus according to any one of claims 1 to 6, wherein the lens optical system includes an optical region configured to transmit red light and near-infrared light,
A lighting device that emits near infrared rays;
An imaging system comprising:   A vehicle comprising at least one of the imaging device according to any one of claims 1 to 6, the distance measuring device according to any one of claims 7 to 9, and the imaging system according to claim 10.

Images