JP6701316B2 - Imaging device and imaging system - Google Patents

Description

  The present invention relates to an imaging device and an imaging system.

  It is known that near-infrared rays of 800 nm to 1000 nm have transparency to a living body. Attention has been focused on a technique for injecting a drug that emits near-infrared fluorescence into the body by being excited by near-infrared light and observing the fluorescence from outside the body to make it visible. A monochrome imaging device having sensitivity in the near-infrared region can capture fluorescence from the inside of the body. In addition, it is required to simultaneously output color information and monitor a visible light image and a near infrared image at the same time. Although it is possible to use a visible light imaging device and a near-infrared imaging device for shooting and superimposing images, there is a problem in downsizing and cost reduction. It is required to acquire a visible light image and a near infrared image with a single image pickup device.

  Patent Document 1 discloses a solid-state imaging device in which a filter that transmits visible light and a filter that transmits near-infrared light are arranged on pixels of an imaging device, and a single imaging device has sensitivity to visible light and near-infrared light. Has been done. Further, in Patent Document 2, a filter having transparency in a specific visible light region and near infrared region is arranged on a pixel of an image pickup device, and the same pixel receives visible light and near infrared light, and for each frame. A technique of switching between visible light output and near infrared light output is disclosed. In addition, a technology is disclosed in which near infrared rays are added with signals for four pixels and output.

JP, 2008-76084, A JP, 2010-35168, A

  Since the conversion efficiency of near-infrared rays with respect to visible rays in silicon is lowered, the near-infrared ray output is generally smaller than the visible ray output. Therefore, when higher sensitivity is required for the near infrared wavelength, sufficient sensitivity may not be obtained in the arrangement of the filter disclosed in Patent Document 1.

  In order to solve the above problems, Patent Document 2 discloses a technique of adding signals for four pixels to a near infrared ray output. However, since visible light and near infrared rays are received by the same pixel, visible light noise is mixed in the near infrared image, and visible light noise is separately removed to obtain a low noise near infrared image. Processing is required.

  An object of the present invention is to provide an imaging device and an imaging system that easily have high sensitivity to visible light and near infrared light without performing a process of removing visible light noise on a near infrared light image.

The imaging device of the present invention includes a plurality of first pixels that convert visible light into an electric signal, a plurality of second pixels that convert near-infrared rays into an electric signal, the plurality of first pixels and the plurality of pixels. A scanning circuit for driving a second pixel, wherein the scanning circuit causes the plurality of first pixels to output the electric signal in the first frame, and the scanning circuit in the second frame The plurality of second pixels are caused to output the electric signal, and the plurality of first pixels and the plurality of second pixels form a matrix including a plurality of columns, and one column A first output line and a second output line are arranged, and a part of the plurality of first pixels and a plurality of the second output lines are provided in each of the first output line and the second output line. Some of the pixels are connected .

  It is possible to easily provide an imaging device and an imaging system that have high sensitivity to visible light and near infrared light without performing a process of removing visible light noise from the near infrared light image.

It is a figure which shows the structural example of a solid-state imaging device. It is a figure which shows the pixel arrangement pattern in 1st Embodiment. It is a figure which shows the equivalent circuit of the pixel part in 1st Embodiment. It is a figure which shows the read sequence in 1st Embodiment. It is a figure which shows the read timing in 1st Embodiment. It is a figure which shows the image of the output image in 1st Embodiment. It is a figure which shows the pixel arrangement pattern in 2nd Embodiment. It is a figure which shows the equivalent circuit of the pixel part in 2nd Embodiment. It is a figure which shows the read timing in 2nd Embodiment. It is a figure which shows the pixel arrangement pattern in 3rd Embodiment. It is a figure which shows the image of the output image in 3rd Embodiment. It is a figure which shows the pixel arrangement pattern in 3rd Embodiment. It is a figure which shows the image of the output image in 3rd Embodiment. It is a figure which shows the pixel arrangement pattern in 4th Embodiment. It is a figure which shows the image of the output image in 4th Embodiment. It is a block diagram which shows the structural example of an imaging system.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of a solid-state imaging device according to the first embodiment of the present invention. The solid-state imaging device includes a pixel portion 101, a vertical scanning circuit 102, a reading circuit 103, an output amplifier 105, and a horizontal scanning circuit 104. The pixel portion 101 has a plurality of pixels arranged in a two-dimensional matrix that accumulates charges generated according to incident light and outputs a signal. The vertical scanning circuit 102 is a pixel control unit for reading a pixel signal from the pixel unit 101 to the reading circuit 103. The readout circuit 103 processes signals from the pixels of the pixel portion 101. The horizontal scanning circuit 104 controls signal transfer from the reading circuit 103 to the output amplifier 105.

  FIG. 2 is a diagram showing an arrangement example of some pixels in the pixel unit 101. The pixel portion 101 has a plurality of pixels arranged in a two-dimensional matrix. In the n-th row, a pixel (hereinafter referred to as an R pixel) in which a color filter (visible light filter) that transmits a wavelength of red (visible light) is arranged and a color filter (a pixel that transmits a wavelength of green (visible light)) ( Pixels (hereinafter referred to as G pixels) in which visible light filters) are arranged are arranged alternately. Further, in the (n+2)th row, a pixel (hereinafter referred to as a B pixel) in which a color filter (visible light filter) that transmits a wavelength of blue (visible light) is arranged and a G pixel are alternately arranged. Further, in the (n+1)th row and the (n+3)th row, pixels (hereinafter referred to as IR pixels) in which a filter that transmits a wavelength of near infrared rays is arranged are arranged. The R pixel, the G pixel, and the B pixel are visible light pixels (first pixels), and convert visible light into an electric signal. The IR pixel is a near infrared ray pixel (second pixel), and converts the near infrared ray into an electric signal. Here, n is an odd number. In that case, the pixels of the visible light transmission filter are arranged in the odd rows, and the pixels of the near infrared transmission filter are arranged in the even rows. The order of arranging the color filters in the odd rows is arbitrary, and the reading start row is defined as the first row here.

  FIG. 3 is an equivalent circuit diagram of four pixels arranged in the same column of the pixel unit 101 of FIG. The pixel in the nth row has a light receiving element 301a and a transfer transistor 302a. The pixel in the (n+1)th row has a light receiving element 301b and a transfer transistor 302b. The pixel in the (n+2)th row has a light receiving element 301c and a transfer transistor 302c. The pixel in the (n+3)th row has a light receiving element 301d and a transfer transistor 302d. The light receiving elements 301a to 301d are, for example, photodiodes, and generate and accumulate charges by photoelectric conversion. The transfer transistors 302a and 302b transfer the charges accumulated in the light receiving elements 301a and 301b to the floating diffusion (hereinafter, referred to as FD) 306a, respectively. Pixels in the nth row and the n+1th row that are vertically adjacent to each other share one FD 306a. The transfer transistors 302c and 302d respectively transfer the charges accumulated in the light receiving elements 301c and 301d to the FD 306b. Pixels on the n+2th row and the n+3th row that are vertically adjacent to each other share one FD 306b. The amplification transistors 304a and 304b amplify the voltage of the FDs 306a and 306b, respectively. The row selection transistors 305a and 305b select the output voltages of the amplification transistors 304a and 304b, and output them to the vertical output line 307. The reset transistors 303a and 303b are provided between the FDs 306a and 306b and the power supply voltage node, respectively, and reset the signals of the FDs 306a and 306b and the light receiving elements 301a to 301d. The transfer transistors 302a to 302d, the reset transistors 303a and 303b, and the row selection transistors 305a and 305b are controlled by the vertical scanning circuit 102 in FIG. The vertical scanning circuit 102 has a reading scanning circuit and a reset scanning circuit, and controls the reading operation and the reset operation.

  FIG. 4 is a diagram showing a read sequence in this embodiment. After time t2, electric signals of a plurality of visible ray pixels (first pixels) are read and output in the first frame. Next, after time t3, the electric signal of the IR pixel (second pixel) is read and output in the second frame. Further, the resetting of the light receiving elements 301a to 301d is performed after time t4 after the signal of the IR pixel is read out, whereby the charge accumulation time of the IR pixel can be made longer than that of the visible ray pixel, and the IR pixel can be reset. The sensitivity of can be increased.

  FIG. 5 is a diagram showing a signal read timing of pixels of four rows in the same column shown in FIG. The reset signal RES1 indicates the reset signal of the gate of the reset transistor 303a in the nth row and the n+1th row. The transfer signal TX1 indicates the transfer signal of the gate of the transfer transistor 302a in the nth row. The transfer signal TX2 indicates the transfer signal of the gate of the transfer transistor 302b in the (n+1)th row. The row selection signal SEL1 indicates the row selection signal of the gate of the row selection transistor 305a of the nth row and the (n+1)th row. The reset signal RES2 indicates a reset signal of the gates of the reset transistors 303b in the n+2th row and the n+3th row. The transfer signal TX3 indicates the transfer signal of the gate of the transfer transistor 302c in the (n+2)th row. The transfer signal TX4 indicates the transfer signal of the gate of the transfer transistor 302d in the (n+3)th row. The row selection signal SEL2 indicates the row selection signal of the gate of the row selection transistor 305b in the (n+2)th row and the (n+3)th row.

  In FIG. 4, times t1 and t4 indicate the start of the reset operation of the pixels in the first row. Time t2 indicates the start of reading the first row of visible light pixels. Time t3 indicates the start of reading the top row of the near infrared ray pixels. 4 and 5, when the reset signal RES1 goes high, the reset transistors 303a in the nth and n+1th rows are turned on, and when the reset signal RES2 goes high, the reset transistors in the n+2th and n+3th rows are reset. 303b turns on. At time t11, the transfer signals TX1 and TX2 become high level, the transfer transistors 302a and 302b in the nth and n+1th rows are turned on, and the FD 306a and the light receiving elements 301a and 301b in the nth and n+1th rows are reset. To be done. Subsequently, the transfer signals TX3 and TX4 become high level, the transfer transistors 302c and 302d of the n+2th and n+3th rows are turned on, and the FD 306b and the light receiving elements 301c and 301d of the n+2th and n+3th rows are reset.

  Next, at time t21, the reset signal RES1 becomes low level, and the reset transistors 303a in the nth row and the n+1th row are turned off. At the same time, the row selection signal SEL1 goes high and the row selection transistors 305a in the nth and n+1th rows are turned on, so that the pixels in the nth and n+1th rows are activated. Next, when the transfer signal TX1 goes high, the transfer transistor 302a in the n-th row is turned on, and the signal from the light-receiving element 301a in the n-th row is read out to the reading circuit 103 via the vertical output line 307. During this period, the transfer signal TX2 is fixed at a low level, the transfer transistor 302b in the (n+1)th row is turned off, and the signal from the light receiving element 301b in the (n+1)th row is not read. After that, the reset signal RES1 becomes high level, and the reset transistors 303a in the nth row and the n+1th row are turned on. Then, the reset signal RES2 becomes low level, and the reset transistors 303b of the n+2th row and the n+3th row are turned off. At the same time, the row selection signal SEL2 becomes high level, and the row selection transistors 305b of the n+2th row and the n+3th row are turned on, so that the pixels of the n+2th row and the n+3th row are activated. Next, when the transfer signal TX3 becomes high level, the transfer transistor 302c in the (n+2)th row is turned on, and the signal of the light receiving element 301c in the (n+2)th row is read out to the reading circuit 103 through the vertical output line 307. During this period, the transfer signal TX4 is fixed to the low level, the transfer transistor 302d in the n+3th row is turned off, and the signal from the light receiving element 301d in the n+3th row is not read. As described above, the transfer signals TX1 and TX3 cause the signals of the visible ray pixels in the nth row and the n+2th row of the odd rows to be read.

  Next, at time t31, the reset signal RES1 becomes low level, and the reset transistors 303a in the nth row and the n+1th row are turned off. At the same time, the row selection signal SEL1 goes high and the row selection transistors 305a in the nth and n+1th rows are turned on, so that the pixels in the nth and n+1th rows are activated. Next, when the transfer signal TX2 becomes high level, the transfer transistor 302b in the (n+1)th row is turned on, and the signal of the light receiving element 301b in the (n+1)th row is read out to the reading circuit 103 through the vertical output line 307. During this period, the transfer signal TX1 is fixed at a low level, the transfer transistor 302a in the nth row is turned off, and the signal from the light receiving element 301a in the nth row is not read. Then, the reset signal RES2 becomes low level, and the reset transistors 303b of the n+2th row and the n+3th row are turned off. At the same time, the row selection signal SEL2 goes high and the row selection transistors 305b of the n+2th row and the n+3th row are turned on, so that the pixels of the n+2th row and the n+3th row are activated. Next, when the transfer signal TX4 becomes high level, the transfer transistor 302d in the n+3th row is turned on, and the signal of the light receiving element 301d in the n+3th row is read out to the reading circuit 103 via the vertical output line 307. During this period, the transfer signal TX3 is fixed at a low level, the transfer transistor 302c in the (n+2)th row is turned off, and the signal from the light receiving element 301c in the (n+2)th row is not read. As described above, the signals of the near-infrared pixels on the n+1th and n+3th rows of the even-numbered rows are read by the transfer signals TX2 and TX4. After time t41, as in the case of time t11 and after, the reset operation of the pixels in all rows is performed again, and the above operation is repeated.

  6A and 6B are diagrams showing the visible light image frame and the near-infrared image frame read by the above operation. The solid-state imaging device outputs the signal of the visible ray pixel in the first frame of FIG. 6A and the signal of the near infrared ray pixel in the second frame of FIG. 6B. With the arrangement pattern of visible light pixels and near infrared pixels shown in FIG. 2, the visible light image of FIG. 6A and the near infrared image of FIG. 6B can be separately output by a simple driving sequence. The imaging device outputs the visible light image in FIG. 6A and the near infrared image in FIG. 6B in separate frames. Therefore, for example, by alternately outputting the first frame and the second frame, a visible light image and a near infrared image are simultaneously displayed as a moving image by a relatively simple image processing without requiring an external memory or the like. It becomes possible. Further, when it is desired to display an image with reduced resolution and priority on sensitivity, a range of 4 pixels surrounded by broken lines in FIGS. 6A and 6B is configured as 1 pixel. In that case, in the visible ray pixel of FIG. 6A, the pixel output is configured by the color information of each pixel within the broken line. In the near-infrared pixel of FIG. 6B, the sensitivity can be improved by adding signals for four pixels indicated by broken lines. The addition here may be performed inside the imaging device or outside the imaging device. That is, in the second frame of FIG. 6B, the vertical scanning circuit 102 and the horizontal scanning circuit 104 add and output the electrical signals of the plurality of IR pixels within the broken line.

  Further, when it is desired to improve the sensitivity of the near infrared ray pixel with respect to the visible ray pixel, the signal of the near infrared ray pixel may be amplified by the gain in the reading circuit 103 and output. In FIG. 2, the visible light pixels and the near-infrared pixels are arranged on a row-by-row basis, and the read gain is set in each of the first frame and the second frame. It is possible to adjust the output of visible light pixels and the output of near infrared pixels.

(Second embodiment)
The configuration of the image pickup apparatus according to the second embodiment of the present invention is the same as that of the first embodiment, and therefore the points different from the first embodiment will be described. FIG. 7 is a diagram showing a pixel arrangement pattern according to the second embodiment of the present invention. In the nth row, R pixels and G pixels are alternately arranged. In the (n+1)th row, G pixels and B pixels are alternately arranged. Further, IR pixels are arranged in the n+2th row and the n+3th row. In subsequent rows, these are repeatedly arranged. FIG. 8 shows an equivalent circuit of some pixels of the pixel unit 101 according to the present embodiment. FIG. 8 shows an equivalent circuit diagram of four pixels arranged in the same column in the pixel arrangement of FIG. The pixel on the n-th row has a light receiving element 801a, a transfer transistor 802a, a reset transistor 803a, an amplification transistor 804a, a row selection transistor 805a, and an FD 806a. The pixel in the (n+1)th row has a light receiving element 801b, a transfer transistor 802b, a reset transistor 803b, an amplification transistor 804b, a row selection transistor 805b, and an FD 806b. The pixel in the (n+2)th row has a light receiving element 801c, a transfer transistor 802c, a reset transistor 803c, an amplification transistor 804c, a row selection transistor 805c, and an FD 806c. The pixel in the (n+3)th row has a light receiving element 801d, a transfer transistor 802d, a reset transistor 803d, an amplification transistor 804d, a row selection transistor 805d, and an FD 806d. Row selection transistors 805a and 805c are connected to the vertical output line 807a. Row selection transistors 805b and 805d are connected to the vertical output line 807b. This embodiment has two vertical output lines 807a and 807b for the same column. As a result, in the present embodiment, the signals for two rows can be simultaneously read out to the vertical output lines 807a and 807b, so that the reading speed is improved as compared with the first embodiment.

  The transfer transistors 802a to 802d transfer the charges generated in the light receiving elements 801a to 801d to the FDs 806a to 806d, respectively. The amplification transistors 804a to 804d amplify the voltages of the FDs 806a to 806d, respectively. The row selection transistors 805a and 805c respectively select the outputs of the amplification transistors 804a and 804c and output them to the vertical output line 807a. The row selection transistors 805b and 805d respectively select the outputs of the amplification transistors 804b and 804d and output them to the vertical output line 807b. The reset transistors 803a to 803d are connected between the FDs 806a to 806d and the power supply voltage node, respectively, and control resetting of signals of the FDs 806a to 806d and the light receiving elements 801a to 801d. The vertical scanning circuit 102 controls the transfer transistors 802a to 802d, the reset transistors 803a to 803d, and the row selection transistors 805a to 805d.

  The read sequence of this embodiment is the same as that of the first embodiment. FIG. 9 is a read timing chart of the present embodiment corresponding to the read sequence of FIG. 4, and shows the read timing of signals for four rows in the same column. The reset signal RES1 indicates a reset signal for the gates of the reset transistors 803a and 803b in the nth and n+1th rows. The transfer signal TX1 indicates a transfer signal of the gates of the transfer transistors 802a and 802b in the nth row and the n+1th row. The row selection signal SEL1 indicates a row selection signal of the gates of the row selection transistors 805a and 805b of the nth row and the n+1th row. The reset signal RES2 indicates a reset signal of the gates of the reset transistors 803c and 803d in the n+2th row and the n+3th row. The transfer signal TX2 indicates a transfer signal of the gates of the transfer transistors 802c and 802d in the n+2th row and the n+3th row. The row selection signal SEL2 indicates a selection signal for the gates of the row selection transistors 805c and 805d of the n+2th row and the n+3th row.

  When the reset signal RES1 goes high, the reset transistors 803a and 803b in the nth and n+1th rows turn on, and when the reset signal RES2 goes high, the reset transistors 803c and 803d in the n+2th and n+3th rows turn on. To do. At time t11, the transfer signal TX1 becomes high level, the transfer transistors 802a and 802b of the nth and n+1th rows are turned on, and the FDs 806a and 806b and the light receiving elements 801a and 801b of the nth and n+1th rows are reset. To be done. Subsequently, when the transfer signal TX2 becomes high level, the transfer transistors 802c and 802d of the n+2th row and the n+3th row are turned on, and the FDs 806c and 806d and the light receiving elements 801c and 801d of the n+2th row and the n+3th row are reset. To be done.

  Next, at time t21, the reset signal RES1 becomes low level, and the reset transistors 803a and 803b are turned off. At the same time, the row selection signal SEL1 goes high and the row selection transistors 805a and 805b are turned on, so that the pixels in the nth row and the n+1th row are activated. Next, the transfer signal TX1 goes high, turning on the transfer transistors 802a and 802b in the nth and n+1th rows. Then, the signals of the light receiving elements 801a and 801b in the nth row and the n+1th row are read out to the reading circuit 103 via the vertical output lines 807a and 807b, respectively.

  Then, at time t31, the reset signal RES2 becomes low level, and the reset transistors 803c and 803d are turned off. At the same time, the row selection signal SEL2 becomes high level and the row selection transistors 805c and 805d are turned on, so that the pixels on the n+2th row and the n+3th row are activated. Next, when the transfer signal TX2 becomes high level, the transfer transistors 802c and 802d of the n+2th row and the n+3th row are turned on. Then, the signals of the light receiving elements 801c and 801d on the n+2th row and the n+3th row are read out to the reading circuit 103 via the vertical signal lines 807a and 807b, respectively. After time t41, as in the case of time t11 and after, the reset operation for all rows is performed again, and the above operation is repeated.

  By the above operation, the read visible light image frame and the near-infrared image frame are as shown in FIGS. 6A and 6B, as in the first embodiment. The solid-state imaging device outputs the signal of the visible ray pixel in the first frame of FIG. 6A and the signal of the near infrared ray pixel in the second frame of FIG. 6B. With the arrangement pattern of the visible light pixels and the near infrared pixels shown in FIG. 7, the visible light image and the near infrared image can be separately output by a simple driving sequence. The imaging device outputs the visible light image in FIG. 6A and the near infrared image in FIG. 6B in separate frames. Therefore, for example, by outputting the first frame and the second frame alternately, a visible light image and a near-infrared image are simultaneously displayed as a moving image by a relatively simple image processing without requiring an external memory or the like. It becomes possible to do. Also. When it is desired to display an image with reduced resolution and priority on sensitivity, one pixel is formed in the range of 4 pixels surrounded by the dotted line in FIGS. 6A and 6B. In that case, in the visible ray pixel, the pixel output is configured from the color information of each pixel within the range of the broken line, and in the near-infrared pixel, the signals for four pixels within the range of the broken line are added to obtain the sensitivity. Can be improved.

  By operating the pixel arrangement shown in FIG. 7 with the drive pattern shown in FIG. 9, the present embodiment has an improvement in read speed and a high frame rate as compared with the first embodiment. On the other hand, the configuration is more suitable.

(Third Embodiment)
The same points as those of the first embodiment of the third embodiment of the present invention will be omitted, and different points will be described. FIG. 10 is a diagram showing a pixel arrangement pattern according to the third embodiment of the present invention, which is an arrangement in which G pixels in a row including R pixels and G pixels are replaced with IR pixels in FIG. .. That is, in the nth row, R pixels and IR pixels are arranged alternately. Each row of the two-dimensional matrix of pixels is either a row containing multiple visible ray pixels or a row containing only multiple IR pixels. Images output in the first frame and the second frame by the driving sequence and the driving pattern shown in FIGS. 4 and 5 are shown in FIGS. 11A and 11B. The solid-state imaging device outputs signals of visible ray pixels and IR pixels in the first frame of FIG. 11A, and signals of IR pixels in the second frame of FIG. 11B. The vertical scanning circuit 102 outputs an image including electric signals of a plurality of visible ray pixels in the first frame of FIG. 11A, and outputs an electric signal of a plurality of IR pixels in the second frame of FIG. 11B. Output an image containing only.

  The signal of the IR pixel output in the first frame of FIG. 11A can be used as information for correcting the IR component included in the aperture pixel, and compared with the first embodiment. , A higher quality visible light image can be obtained. Further, the IR pixel output in the second frame of FIG. 11B has the same storage time as the IR pixel output in the first frame for the purpose of obtaining the information of the incident near infrared rays. It doesn't have to be. In the second frame, the gain of the pixel signal does not need to be the same as that in the first frame. Therefore, by increasing the accumulation time or applying a gain, a higher sensitivity IR image can be obtained. be able to. As described above, the present embodiment can provide an image with lower noise than the image output in the first frame, as compared with the first embodiment.

  Further, the present embodiment may have the pixel arrangement pattern shown in FIG. FIG. 12 shows an example in which when G pixels are replaced with IR pixels, replacement is performed so that G pixels remain in all columns. The nth row to the n+3th row in FIG. 12 have the same pixel arrangement pattern as the nth row to the n+3th row in FIG. 10. In the n+4th row, R pixels and G pixels are arranged alternately. In the n+5th row and the n+7th row, IR pixels are arranged. In the (n+6)th row, IR pixels and B pixels are arranged alternately. In this case, the output image has the first frame in FIG. 13A and the second frame in FIG. 13B. In the first frame of FIG. 11A, since G pixels are arranged in all columns, this G pixel signal can be used as information when performing offset correction between columns, and a more preferable arrangement pattern can be obtained. Has become. Further, the arrangement is not limited to the pattern of FIG. 12 as long as the purpose of including G pixels in all columns is satisfied.

(Fourth Embodiment)
The differences between the fourth embodiment of the present invention and the second embodiment will be omitted, and different points will be described. FIG. 14 shows a pixel arrangement pattern according to an embodiment of the present invention. 14 is an arrangement in which G pixels in a row including R pixels and G pixels are replaced with IR pixels in contrast to FIG. 7. That is, in the nth row, R pixels and IR pixels are arranged alternately. The images output in the first frame and the second frame by the driving sequence and the driving pattern shown in FIGS. 4 and 9 are the same as those in the third embodiment, and the first frame in FIG. And the second frame in FIG. 11B. The signal of the IR pixel output in the first frame of FIG. 11A can be used as information for correcting the IR component included in the aperture pixel, and compared with the first embodiment. , A higher quality visible light image can be obtained. Further, the IR pixel output in the second frame of FIG. 11B has the same storage time as the IR pixel output in the first frame for the purpose of obtaining the information of the incident near infrared rays. It doesn't have to be. Further, the second frame does not need to have the same gain for the pixel signal as the first frame. Therefore, by increasing the accumulation time or applying a gain, a higher sensitivity IR can be obtained. Images can be obtained. As described above, the present embodiment can provide an image with lower noise than the image output in the first frame, as compared with the second embodiment.

  Further, the present embodiment may have the pixel arrangement pattern shown in FIG. FIG. 15 shows an example in which when G pixels are replaced with IR pixels, replacement is performed so that G pixels remain in all columns. The nth row to the n+3th row in FIG. 15 are the same as the nth row to the n+3th row in FIG. In the n+4th row, R pixels and G pixels are arranged alternately. In the (n+5)th row, IR pixels and B pixels are arranged alternately. The n+6th row and the n+7th row are the same as the n+2nd row and the n+3th row. In this case, the output image has the first frame in FIG. 13A and the second frame in FIG. 13B. The images of FIGS. 13A and 13B are different from the images of FIGS. 11A and 11B in that G pixels are arranged in all columns in the first frame. The pixel signal can be used as information when performing offset correction between columns, and has a more suitable arrangement pattern. Further, the present embodiment only needs to satisfy the purpose of including G pixels in all columns, and the arrangement is not limited to the pattern of FIG.

(Fifth Embodiment)
FIG. 16: is a figure which shows an example of the imaging system to which the imaging device which concerns on the 5th Embodiment of this invention is applied. The imaging system 501 has an optical system 502, an imaging device 100, a signal processing unit 503, and an external device 506. The signal processing unit 503 includes an imaging signal processing circuit 504 and an image signal processing unit 505. The imaging system 501 may also include a memory unit, a timing generator, and the like. The image pickup apparatus 100 is the image pickup apparatus according to the first to fourth embodiments. The light incident on the optical system 502 forms an image of a subject on the imaging surface (pixel unit 101) of the imaging device 100. The imaging device 100 converts an image of a subject formed on the imaging surface into an image signal, reads the image signal from the pixel unit 101, and outputs the image signal. The imaging signal processing circuit 504 is connected to the imaging device 100 and processes the image signal output from the imaging device 100. The image signal processing unit 505 performs arithmetic processing such as various corrections on the image signal processed by the imaging signal processing circuit 504 to generate image data. This image data is output to an external device (for example, personal computer) 506. When the signal supplied from the image pickup apparatus 100 to the image pickup signal processing circuit 504 is an analog signal, it is a digital signal by an A/D converter or the like in the image pickup signal processing circuit 504 before being supplied to the image signal processing unit 505. It is desirable to convert to.

  The signal processing unit 503 forms a visible light image based on the first frame output by the image pickup apparatus 100, and forms a near infrared image based on the second frame. Further, the signal processing unit 503 may alternately display the visible light image and the near-infrared image for each frame on the external device (display device) 506, or may combine the visible light image and the near-infrared image to the external device ( It may be displayed on the display device) 506.

  As described in the first to fourth embodiments, the image pickup apparatus 100 outputs the visible light image data and the near infrared ray image data for each frame. Therefore, the image signal processing unit 505 outputs the visible light ray component from the near infrared ray image, for example. It eliminates the need for a correction system such as removing the noise of. Therefore, it is possible to output an image with high sensitivity to visible rays and near infrared rays without requiring special correction.

  It should be noted that each of the above-described embodiments is merely an example of an embodiment for carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101 pixel unit, 102 vertical scanning circuit, 103 readout circuit, 104 horizontal scanning circuit, 105 output amplifier

Claims (14)

A plurality of first pixels that convert visible light into electrical signals;
A plurality of second pixels for converting near infrared rays into electric signals,
A scanning circuit that drives the plurality of first pixels and the plurality of second pixels,
The scanning circuit causes the plurality of first pixels to output the electric signal in a first frame,
The scanning circuit causes the plurality of second pixels to output the electric signal in a second frame,
The plurality of first pixels and the plurality of second pixels form a matrix including a plurality of columns,
A first output line and a second output line are arranged for one column,
An imaging device, wherein a part of the plurality of first pixels and a part of the plurality of second pixels are connected to each of the first output line and the second output line . The first pixel converts visible light incident during the first period into electric charge and stores the electric charge.
The second pixel converts near-infrared rays incident during a second period at least partially overlapping with the first period into electric charge and stores the electric charge.
The image pickup apparatus according to claim 1, wherein The first pixel converts visible light incident during the first period into electric charge and stores the electric charge.
The second pixel converts near-infrared rays incident during a second period, which is longer than the first period, into electric charges and stores the electric charges.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is provided. Each row of the matrix, according to any one of claims 1-3, characterized in that either line containing only the rows or a plurality of said second pixel comprising a plurality of said first pixel Imaging device. The image pickup apparatus according to claim 1, wherein the electrical signals of the plurality of first pixels are output independently of each other in the first frame. In the second frame, the imaging apparatus according to any one of claims 1-5, characterized in that the electrical signals of a plurality of the second pixel is output is added. Gains are independently set for the electric signal output from the first pixel and the electric signal output from the second pixel,
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The plurality of first pixels and the plurality of second pixels respectively include a light receiving element that converts incident light into an electric charge, a transfer transistor that transfers the electric charge of the light receiving element, and an electric signal based on the electric charge. Including an amplifying transistor for outputting
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. In the operation of outputting the electric signal of the second frame, from turning on the transfer transistor of the second pixel in the first row to turning on the transfer transistor of the second pixel in the second row , Keeping the transfer transistors of the first pixels included in a row between the first row and the second row off.
The imaging device according to any one of claims 1 to 8, characterized in that. In the operation of outputting the electric signal of the second frame, from outputting the electric signal of the second pixel of the first row to outputting the electric signal of the second pixel of the second row , The electric signal is not output from the first pixels included in a row between the first row and the second row,
The imaging device according to any one of claims 1 to 8, characterized in that. An imaging device according to any one of claims 1-10,
An optical system for forming an image on the imaging device,
An image pickup system, comprising: a signal processing unit that processes an image output from the image pickup apparatus to generate image data. The imaging system according to claim 11, wherein the signal processing unit forms a visible light image based on the first frame and a near infrared image based on the second frame. The image pickup system according to claim 12, wherein the signal processing unit alternately displays the visible light image and the near-infrared image for each frame. The image pickup system according to claim 12, wherein the signal processing unit synthesizes the visible light image and the near-infrared image and displays them.

Images