KR20230082544A - Wide dynamic range image sensor and metod performed by the same - Google Patents

Abstract

An image sensor may include: a pixel array including a plurality of pixels; a driver configured to drive the pixel array; a readout circuit configured to generate a digital signal from a pixel signal received from the pixel array; a controller configured to control the driver to allow a first pixel group and a second pixel group among the plurality of pixels to have a first exposure time and a second exposure time, respectively; and a signal processor configured to generate image data by calculating a first value of the digital signal corresponding to the first pixel group and a second value of the digital signal corresponding to the second pixel group.

Description

WIDE DYNAMIC RANGE IMAGE SENSOR AND METOD PERFORMED BY THE SAME}

The technical idea of the present disclosure relates to an image sensor, and more particularly to a wide dynamic range image sensor and a method performed by the same.

An image sensor may refer to a device that captures a 2D or 3D image of a subject. The image sensor may generate an image of the subject using a photoelectric conversion element that reacts according to the intensity of light reflected from the subject. An image sensor may be required to have a wide dynamic range and good noise characteristics for high-quality images.

The technical spirit of the present disclosure provides an image sensor having both a wide dynamic range and good noise characteristics and a method performed by the same.

An image sensor according to one aspect of the technical idea of the present disclosure includes a pixel array including a plurality of pixels, a driver configured to drive the pixel array, a readout circuit configured to generate a digital signal from a pixel signal received from the pixel array, and a plurality of A controller configured to control a driver so that a first pixel group and a second pixel group among the pixels have a first exposure time and a second exposure time, respectively, and a first value and a first value of a digital signal corresponding to the first pixel group; and a signal processor configured to generate image data by calculating a second value of a digital signal corresponding to two pixel groups, and the first pixel group may be independent of illuminance.

An image sensor according to one aspect of the technical idea of the present disclosure includes a pixel array including a plurality of pixels, a driver configured to drive the pixel array, a readout circuit configured to generate a digital signal from a pixel signal received from the pixel array, and a plurality of A controller configured to control a driver so that a first pixel group and a second pixel group among the pixels have a first exposure time and a second exposure time, respectively, and identify illuminance based on the value of the digital signal, and based on the illuminance and a signal processor configured to identify the second pixel group and the second exposure time, and set the controller based on the second exposure time, wherein the first exposure time may be independent of illuminance.

According to one aspect of the technical idea of the present disclosure, a method performed by an image sensor includes generating a digital signal from a pixel signal received from a pixel array including a plurality of pixels, a first exposure time and a second exposure time. driving a first pixel group and a second pixel group among a plurality of pixels to have a first value of a digital signal corresponding to the first pixel group and a second value of a digital signal corresponding to the second pixel group; The method may include generating image data based on a weighted sum of values, and the first pixel group may be independent of illuminance.

According to an image sensor according to an exemplary embodiment of the present disclosure and a method performed by the same, both a wide dynamic range and good noise characteristics can be achieved, and thus the image sensor can be effectively used in various applications.

In addition, according to the image sensor and the method performed by the image sensor according to exemplary embodiments of the present disclosure, the quality of an image generated by the image sensor can be improved, and thus the performance of applications including the image sensor can be improved. can

Effects obtainable in the exemplary embodiments of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are common knowledge in the art to which exemplary embodiments of the present disclosure belong from the following description. can be clearly derived and understood by those who have That is, unintended effects according to the implementation of the exemplary embodiments of the present disclosure may also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

1 is a block diagram illustrating an image sensor according to an exemplary embodiment of the present disclosure.
2A and 2B are circuit diagrams illustrating examples of pixels according to exemplary embodiments of the present disclosure.
3 shows an SNR-HDR graph of an image sensor according to an exemplary embodiment of the present disclosure.
4 is a flowchart illustrating a method performed by an image sensor according to an exemplary embodiment of the present disclosure.
5A-5E show examples of pixel arrays according to exemplary embodiments of the present disclosure.
6A and 6B are diagrams illustrating examples of operations of an image sensor according to exemplary embodiments of the present disclosure.
7A and 7B show SNR-HDR graphs of an image sensor according to exemplary embodiments of the present disclosure.
8A and 8B show SNR-HDR graphs of an image sensor according to exemplary embodiments of the present disclosure.
9 is a flowchart illustrating a method performed by an image sensor according to an exemplary embodiment of the present disclosure.
10A and 10B are flow charts illustrating examples of a method performed by an image sensor according to exemplary embodiments of the present disclosure.
11A and 11B show SNR-HDR graphs of an image sensor according to exemplary embodiments of the present disclosure.
12 is a flowchart illustrating a method performed by an image sensor according to an exemplary embodiment of the present disclosure.
13A is an exploded perspective view of an image sensor according to an exemplary embodiment of the present disclosure, FIG. 13B is a plan view of the image sensor according to an exemplary embodiment of the present disclosure, and FIG. 13C is an image according to an exemplary embodiment of the present disclosure. This is an exploded perspective view of the sensor.
14 is a block diagram illustrating an electronic device including a multi-camera module according to an exemplary embodiment of the present disclosure.
15 is a block diagram illustrating the camera module of FIG. 14 according to an exemplary embodiment of the present disclosure.

1 is a block diagram illustrating an image sensor 10 according to an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the image sensor 10 may include a pixel array 11 , a driver 12 , a readout circuit 13 , a controller 14 and a signal processor 15 .

The image sensor 100 may be included in a system having an image or light sensing function. For example, the image sensor 100 may be included in electronic devices such as cameras, smart phones, wearable devices, internet of things (IoT), tablet PCs, drones, and advanced drivers assistance systems (ADAS). In addition, the image sensor 10 may be included in parts included in vehicles, furniture, manufacturing facilities, doors, various measuring devices, and the like.

The pixel array 11 may include a plurality of pixels, and the plurality of pixels may be connected to a plurality of driving lines DLs and a plurality of sensing lines SLs. In some embodiments, the image sensor 10 may be an active pixel sensor (APS), and each of a plurality of pixels of the pixel array 11 may include at least one photoelectric conversion element and at least one transistor. . The photoelectric conversion element may sense light and generate an electrical signal corresponding to the light. The photoelectric conversion element may be an organic material, such as, but not limited to, an inorganic photo diode, an organic photo diode, a perovskite photo diode, a photo transistor, a photo gate, or a pinned photo diode. Alternatively, a photo-sensing element made of an inorganic material may be included. In some embodiments, as described below with reference to FIGS. 2A and 2B , a pixel may provide two or more conversion gains. In some embodiments, as described below with reference to FIG. 2B , a pixel may include two or more photoelectric conversion elements.

A microlens may be disposed on each of a plurality of pixels or on two or more pixels adjacent to each other. The pixel may detect light in a specific spectral region from light received through the microlens. For example, the pixel array 11 includes a red pixel that converts light in the red spectrum region into an electrical signal, a green pixel that converts light in the green spectrum region into an electrical signal, and a pixel array that converts light in the blue spectrum region into an electrical signal. It may contain blue pixels. A color filter array may be disposed on the plurality of pixels, and a color that the pixel can detect may be determined according to the color filter disposed on the pixel. One pixel may be formed in an image by combining pixel signals generated from each of the red, green, and blue pixels, and each of the red, green, and blue pixels may be referred to as a sub-pixel in this specification.

In some embodiments, a pixel may have a multi-layer structure. A pixel having a multi-layer structure may include stacked photoelectric conversion elements, and the stacked photoelectric conversion elements may respectively convert different spectral regions into electrical signals. Accordingly, electrical signals respectively corresponding to different colors may be output from one pixel. Also, in some embodiments, a pixel may include a photoelectric conversion element that converts light in a specific spectral region into an electrical signal according to an applied electrical signal.

The driver 12 may drive the pixel array 11 based on the first control signal CTR1 provided from the controller 14 . For example, the driver 12 may generate a driving signal based on the first control signal CTR1 and output the driving signal through a plurality of driving lines DLs. In some embodiments, driver 12 may drive a plurality of pixels row by row of pixel array 11 and may be referred to as a row driver. For example, the driver 12 may select a row based on the first control signal CTR1, and outputs pixel signals from the selected row through a plurality of sense lines SLs. can drive

The readout circuit 13 may receive pixel signals from the pixel array 11 through the plurality of sense lines SLs. The readout circuit 13 may convert the pixel signal into a digital signal DSIG based on the second control signal CTR2 provided from the controller 14 . In some embodiments, the readout circuit 13 may include a ramp generator that generates a ramp signal that increases or decreases with a constant slope, and converts the pixel signal into a digital signal DSIG based on the ramp signal. It may include an analog-to-digital converter (ADC). For example, the read circuit 13 may include a plurality of analog-to-digital converters, each corresponding to a plurality of sense lines SLs and commonly receiving a ramp signal. Also, in some embodiments, the readout circuit 13 may generate the digital signal DSIG based on correlation double sampling.

The controller 14 can control the driver 12 through the first control signal CTR1 and the read circuit 13 through the second control signal CTR2. The controller 14 may generate the first control signal CTR1 and the second control signal CTR2 based on the setting of the signal processor 15 . For example, as shown in FIG. 1 , the controller 14 may receive a set signal SET from the signal processor 15, and a first control signal based on a setting corresponding to the set signal SET. (CTR1) and the second control signal (CTR2) may be generated. In some embodiments, the controller 14 may identify a pixel group including at least some of the plurality of pixels of the pixel array 11 and an exposure integration time (EIT) based on the setting signal SET. can The controller 14 may generate a first control signal CTR1 based on the identified pixel group and the exposure time, and the driver 12 may identify the identified pixel group based on the first control signal CTR1. A driving signal may be output through a plurality of driving lines DLs to have a predetermined exposure time. Operations of the driver 12 and the readout circuit 13 may be switched according to the timings of the first control signal CTR1 and the second control signal CTR2, and thus the controller 14 may be referred to as a timing controller. there is.

The signal processor 15 may receive the digital signal DSIG from the readout circuit 13 and generate image data IMG representing an image of a subject. In some embodiments, signal processor 15 may perform various compensation operations. For example, the signal processor 15 may perform noise reduction, gain adjustment, waveform shaping, interpolation, white balance adjustment, gamma processing, edge enhancement, binning, and the like.

In some embodiments, signal processor 15 may identify illuminance of an image based on digital signal DSIG. The signal processor 15 may divide the pixel array 11 into two or more regions based on the identified illuminance. For example, the signal processor 15 may compare the identified illuminance with at least one predefined reference value, and may include at least one low-illuminance area having an illuminance less than the reference value and at least one high-illuminance area having an illuminance greater than or equal to the reference value. The pixel array 11 can be divided into In the following, examples in which the pixel array 11 includes at least one low-intensity area and at least one high-intensity area will be described, but it is noted that exemplary embodiments of the present disclosure are not limited thereto. For example, the pixel array 11 may include regions respectively corresponding to three or more illuminance ranges.

In some embodiments, signal processor 15 may identify illuminance of an image based on digital signal DSIG. The signal processor 15 may identify a pixel group including at least one pixel among a plurality of pixels of the pixel array 11 based on the identified illuminance. As will be described later with reference to FIGS. 5A to 5E , the signal processor 15 determines that a pixel group included in a low luminance area has a relatively long exposure time and a pixel group included in a high luminance area has a relatively short exposure time. The controller 14 can be set to have The setting signal SET may include information about the pixel group and the exposure time, and the controller 14 may generate the first control signal CTR1 to drive the pixel group based on the information included in the setting signal SET. Through this, the driver 12 can be controlled.

In some embodiments, signal processor 15 may set controller 14 such that a group of pixels has a position and exposure time independent of illuminance. In this specification, a pixel group having a location and exposure time independent of illuminance may be referred to as a reference pixel group or a reference group, and a pixel included in the reference pixel group may be referred to as a reference pixel. Also, a pixel group having a location and an exposure time dependent on illuminance may be referred to as an adaptive pixel group or an adaptive group, and pixels included in the adaptive pixel group may be referred to as adaptive pixels.

The signal processor 15 may generate the image data IMG by calculating values of the digital signal DSIG corresponding to pixel groups having different exposure times. Accordingly, the image data IMG may have good quality, and the performance of the image sensor 10 may be improved. In addition, the performance of applications including the image sensor 10 may be improved due to the good quality of the image data IMG. For example, the image sensor 10 may have a high low-light signal-to-noise ratio (SNR), an extended high dynamic range (HDR), and an improved SNR dip at high temperatures, and thus in harsh use environments. It can be effectively used for automotive applications that require support.

Signal processor 15 may have any structure that performs the operations described above. For example, the signal processor 15 may include a programmable component such as a processing core, a reconfigurable component such as a field programmable gate array (FPGA), and an intellectual property (IP) core. It may include at least one of components providing fixed functions.

2A and 2B are circuit diagrams illustrating examples of pixels according to exemplary embodiments of the present disclosure. In some embodiments, each of the pixels 20a and 20b of FIGS. 2A and 2B may be included in the pixel array 11 of FIG. 1 . It is noted that the pixels included in the pixel array 11 of FIG. 1 are not limited to the pixels 20a and 20b of FIGS. 2A and 2B. Hereinafter, overlapping descriptions of FIGS. 2A and 2B will be omitted.

Referring to FIG. 2A , a pixel 20a may include a photodiode PD, a first capacitor C1, a second capacitor C2, and a plurality of transistors. The plurality of transistors may include a transfer transistor TG, a reset transistor RG, a gain control transistor CG, a driving transistor DX, and a selection transistor SX. In some embodiments, the first capacitor C1 may correspond to a parasitic capacitor of the floating diffusion node FD. In some embodiments, the second capacitor C2 may be a passive element structured to have a fixed or variable capacitance. In some embodiments, the second capacitor C2 may correspond to a parasitic capacitor of a node connected to the source of the reset transistor RG and the drain of the gain control transistor CG.

The photodiode (PD) is a photoelectric conversion element and can convert light incident from the outside into an electrical signal. The photodiode PD may accumulate charges according to the intensity of light. The pixel 20a may receive driving signals provided from the driver 12 of FIG. 1 , that is, a reset signal RS, a gain signal GS, a transmission signal TS, and a selection signal SEL, respectively.

The reset transistor RG may be turned on in response to an activated reset signal RS, and the gain control transistor CG may be turned on in response to an activated gain signal GS. Accordingly, the reset voltage VRD may be applied to the floating diffusion node FD, and the floating diffusion node FD may be reset. The transfer transistor TG may be turned on in response to the activated transfer signal TS, and thus the photodiode PD may also be reset.

The transfer transistor TG may be turned off in response to the inactive transfer signal TS, and while the transfer transistor TG is turned off, that is, during the exposure time, the photodiode PD responds to the incident light. charge can accumulate. When the transfer transistor TG is turned on in response to the activated transfer signal TS, charges accumulated in the photodiode PD may be transferred to the floating diffusion node FD. Charges may be accumulated in the first capacitor C1 when the gain signal GS is inactivated, while charges may be accumulated in the first capacitor C1 and the second capacitor C2 when the gain signal GS is activated. there is.

The voltage of the floating diffusion node FD may depend on the charge accumulated in the photodiode PD and may be expressed as a product of the charge accumulated in the photodiode PD and a conversion gain. As described above, the voltage of the floating diffusion node FD corresponding to the same amount of charge may be different depending on whether the gain signal GS is activated or not, and the conversion gain may vary according to the gain signal GS. That is, the pixel 20a may have a high conversion gain (HCG) when the gain signal GS is inactive, while the pixel 20a may have a relatively low conversion gain when the gain signal GS is activated. It may have low conversion gain (LCG). As such, a pixel 20a that provides two different conversion gains may be referred to as a dual conversion gain (DCG) pixel. In this specification, a state in which the gain signal GS is inactivated may be referred to as an HCG mode, and a state in which the gain signal GS is activated may be referred to as an LCG mode. As described later with reference to FIG. 3 , the DCG pixels may extend the dynamic range of the image sensor 10 .

The driving transistor DX may function as a source follower by means of the current source CS connected to the pixel voltage VPIX and the sense line SL, and transfer the voltage of the floating diffusion node FD to the selection transistor SX. there is. In some embodiments, the current source CS may be shared by pixels connected to the sense line SL and may be included in the readout circuit 13 of FIG. 1 . The selection transistor SX may provide the output voltage of the driving transistor DX to the sensing line SL in response to the activated selection signal SEL. The voltage of the sensing line SL may be provided as a pixel signal to the readout circuit 13 of FIG. 1 , and the readout circuit 13 may generate a digital signal DSIG corresponding to the voltage of the sensing line SL. there is.

Referring to FIG. 2B , a pixel 20b may include a large photodiode (LPD), a small photodiode (SPD), a capacitor (SC), and a plurality of transistors. The plurality of transistors include a first transfer transistor TG1, a second transfer transistor TG2, a switch transistor SG, a reset transistor RG, a gain control transistor CG, a driving transistor DX, and a selection transistor SX. ) may be included. Each of the first to third floating diffusion nodes FD1 to FD3 may have a parasitic capacitance. In some embodiments, the pixel 20b may further include capacitors as passive elements respectively connected to the first to third floating diffusion nodes FD1 to FD3.

Large photodiodes (LPDs) and small photodiodes (SPDs) can accumulate charges according to incident light. The large photodiode (LPD) may have a larger size than the small photodiode (SPD), and the large photodiode (LPD) may be suitable for low light subjects, while the small photodiode (SPD) may be suitable for high light subjects. can A pixel structure including photodiodes of different sizes, such as the pixel 20b, may be referred to as a split photodiode (split PD), in particular, a small photodiode (SPD) is disposed at one corner of a large photodiode (LPD). A structure that becomes may be referred to as a corner pixel.

The pixel 20b receives driving signals provided from the driver 12 of FIG. 1 , that is, a reset signal RS, a gain signal GS, a first transmission signal TS1, a second transmission signal TS2, and a switch signal. (SS) and the selection signal (SEL) may be received respectively. The pixel 20b may support the HCG mode and the LCG mode of the large photodiode (LPD) and the HCG mode and LCG mode of the small photodiode (SPD). Accordingly, the pixel 20b may provide a wider dynamic range than the pixel 20a of FIG. 2A.

The reset transistor RG may be turned on in response to the activated reset signal RS, and thus the second floating diffusion node FD2 may be reset. The gain control transistor CG may be turned on in response to the activated gain signal GS, and accordingly, the first floating diffusion node FD1 may be reset. Also, the switch transistor SG may be turned on in response to the activated switch signal SS, and thus the third floating diffusion node FD3 may be reset.

In the HCG mode of the large photodiode LPD, the gain signal GS may be inactivated, and thus the gain control transistor CG may be turned off. When the first transfer transistor TG1 is turned on in response to the activated first transfer signal TS1, charges accumulated in the large photodiode LPD may be transferred to the first floating diffusion node FD1. In the LCG mode of the large photodiode LPD, the gain signal GS can be activated, and thus the gain control transistor CG can be turned on. When the first transfer transistor TG1 is turned on in response to the activated first transfer signal TS1, charges accumulated in the large photodiode LPD are transferred to the first floating diffusion node FD1 and the second floating diffusion node. (FD2).

In the HCG mode of the small photodiode SPD, the switch signal SW may be inactivated, and thus the switch transistor SG may be turned off. When the second transfer transistor TG is turned on in response to the activated second transfer signal TS2, the charge accumulated in the small photodiode SPD is transferred to the third floating diffusion node FD3 to which the capacitor SC is connected. can be forwarded to As shown in FIG. 2B , the capacitor SC may be connected between a node to which the voltage VMIN is applied and the third floating diffusion node FD3. In the LCG mode of the small photodiode SPD, the switch signal SW can be activated, and thus the switch transistor SG can be turned on. When the second transfer transistor TG is turned on in response to the activated second transfer signal TS2, charges accumulated in the small photodiode SPD are transferred to the third floating diffusion node FD3 and the second floating diffusion node. (FD2).

3 shows an SNR-HDR graph of the image sensor 10 according to an exemplary embodiment of the present disclosure. In the graph of FIG. 3, the horizontal axis represents the intensity of light incident on the image sensor 10, that is, brightness, and the vertical axis represents the signal-to-noise ratio (SNR). In the following, FIG. 3 will be described with reference to FIG. 1 .

In some embodiments, curve 30 in the graph of FIG. 3 is an image sensor that includes pixels that include photodiodes of different sizes and support double conversion gain (DCG), such as pixel 20b of FIG. 2B . The characteristics of (10) are shown. Accordingly, the image sensor 10 may provide a wide dynamic range, that is, high dynamic range (HDR). For example, as shown in FIG. 3, HDR may be defined as a section in which the curve 30 has an SNR higher than zero, and the LH section, the LL section, the SH section, and the SL section according to the illuminance can include In the LH period, the pixel may output a pixel signal corresponding to the charge accumulated in the LPD in the HCG mode, and in the LL period, the pixel may output a pixel signal corresponding to the charge accumulated in the LPD in the LCG mode. Also, in the SH section, the pixel may output a pixel signal corresponding to the charge accumulated in the SPD in the HCG mode, and in the SL section, the pixel may output a pixel signal corresponding to the charge accumulated in the SPD in the LCG mode. The signal processor 15 may combine regions corresponding to different intervals in the image.

The image sensor 10 may be required to simultaneously satisfy a variety of capabilities. For example, image sensor 10 may be required to provide good low-light SNR 31, good SNR at image combining points 32, 33, 34 and extended HDR, as shown in FIG. there is. As will be described later with reference to the drawings, pixel groups included in a plurality of pixels may have different exposure times, and image data may be generated by calculating digital signal values corresponding to the different pixel groups. can Accordingly, the above-described performances of the image sensor 10, namely low-illuminance SNR, SNR at the image combining point, and HDR may all be improved.

4 is a flowchart illustrating a method performed by image sensor 10 according to an exemplary embodiment of the present disclosure. In this specification, the method of FIG. 4 may be referred to as a method for image sensing. As shown in FIG. 4 , the method performed by the image sensor 10 may include a plurality of steps S20, S40, S60, and S80. In the following, FIG. 4 will be described with reference to FIG. 1 .

Referring to FIG. 4 , in step S20 , the driver 12 may be controlled so that the first and second pixel groups have a first exposure time and a second exposure time, respectively. For example, the controller 14 may identify the first pixel group and/or the second pixel group based on the setting signal SET provided by the signal processor 15, and may determine the first exposure time and/or A second exposure time can be identified. The controller 14 may control the driver 12 through the first control signal CTR1 so that the first pixel group has a first exposure time, and the driver 12 is driven to have a first exposure time. The first pixel group may be driven through the signals. Also, the controller 14 may control the driver 12 through the first control signal CTR1 so that the second pixel group has the second exposure time, and the driver 12 has the second exposure time. The second pixel group may be driven through the driving signals.

In some embodiments, the setting signal SET may include addresses corresponding to the first pixel group and/or the second pixel group in the pixel array 11, the first exposure time and/or the second exposure time. It may include values corresponding to . In some embodiments, the first pixel group may be a reference group, and the first pixel group and first exposure time may be fixed. Accordingly, information on the first pixel group and the first exposure time may be omitted from the setting signal SET, and the controller 14 determines that the first pixel group sets the first exposure time regardless of the setting signal SET. It is possible to control the driver 12 through the first control signal CTR1 so as to have.

In step S40, a pixel signal may be generated. For example, the pixel array 11 may sense light based on driving signals provided by the driver 12 through a plurality of driving lines DLs, and may sense light through a plurality of sensing lines SLs. A pixel signal can be output. The first pixel group may generate a pixel signal corresponding to the intensity of incident light during the first exposure time, and the second pixel group may generate a pixel signal corresponding to the intensity of incident light during the second exposure time. there is.

In step S60, a digital signal DSIG may be generated. For example, the read circuit 13 may generate the digital signal DSIG from a pixel signal received from the pixel array 11 through a plurality of sense lines SLs. In some embodiments, the readout circuit 13 may generate the digital signal DSIG from the pixel signal based on correlated double sampling (CDS).

In step S80 , the image data IMG may be generated by calculating the first and second values of the digital signal DSIG. A first value of the digital signal DSIG may correspond to a pixel signal generated by a first pixel group, and a second value of the digital signal DSIG may correspond to a pixel signal generated by a second pixel group. The signal processor 15 may calculate the first value and the second value, and generate image data IMG based on the calculation result. In some embodiments, the signal processor 15 may calculate a weighted sum of the first value and the second value and generate the image data IMG based on the weighted sum. An example of step S80 will be described with reference to FIG. 12 .

5A-5E show examples of a pixel array 11 according to exemplary embodiments of the present disclosure. Specifically, FIGS. 5A to 5E show pixel arrays 11a to 11d including differently grouped pixels. As described above with reference to the drawings, a plurality of pixels included in the pixel array 11 of FIG. 1 may be grouped into two or more pixel groups, and each of the pixel groups may have its own exposure time. It should be noted that the pixels included in the pixel array 11 are not limited to those shown in FIGS. 5A to 5E. Hereinafter, overlapping descriptions of FIGS. 5A to 5E will be omitted, and FIGS. 5A to 5E will be described with reference to FIG. 1 .

As described above with reference to FIG. 1 , the pixel array 11 may be divided into two or more regions according to illuminance. For example, each of the pixel arrays 11a to 11d of FIGS. 5A to 5D may be divided into a first region R1 and a second region R2. The first region R1 may be a high illuminance region corresponding to a relatively high illuminance, and the second region R2 may be a low illuminance region corresponding to a relatively low illuminance. The low luminance area may include pixels having a relatively long exposure time, and the high luminance area may include pixels having a relatively short exposure time. Unlike those shown in FIGS. 5A to 5D , the pixel array 11 may be divided into three or more regions. In some embodiments, an area divided from the pixel array 11 may be classified into one of three or more areas including a low-intensity area and a high-intensity area.

In some embodiments, a pixel array may include a reference pixel group (or reference group) that includes pixels at a fixed location and has a fixed exposure time. For example, in the pixel arrays 11a to 1d of FIGS. 5A to 5D , the first pixel group G1 may be a reference pixel group, and has a fixed position and a fixed position in each of the pixel arrays 11a to 11d. exposure time can be obtained. In particular, as in the first pixel group G1 of FIGS. 5A and 5B , when the reference pixel group includes some rows of a pixel array, the reference pixel group may be referred to as a reference row. 5A to 5D each show a first pixel group G1 as an example of a reference pixel group, but it should be noted that the reference pixel group is not limited to the first pixel group G1 of FIGS. 5A to 5D.

In some embodiments, each of the pixels included in a pixel group may include a plurality of sub-pixels. For example, as shown in FIGS. 5A, 5C, and 5D , the first pixel PX1 included in the first pixel group G1 includes two red pixels R, a green pixel G, and A blue pixel (B) may be included. Also, the second pixel PX2 of the second pixel group G2 and the third pixel PX3 of the third pixel group G3 also include two red pixels R, a green pixel G, and a blue pixel ( B) can be included. As shown in FIG. 5A , two red pixels R, a green pixel G, and a blue pixel B corresponding to one pixel in an image may be referred to as a Bayer pattern.

Referring to FIG. 5A , the pixel array 11a may be divided into a first region R1 and a second region R2. For example, the signal processor 15 may divide the pixel array 11a into a first region R1 corresponding to high luminance and a second region R2 corresponding to low luminance based on the digital signal DSIG. . The signal processor 15 may generate the setting signal SET so that the first region R1 includes pixels having a short exposure time and the second region R2 includes pixels having a long exposure time. . For example, as shown in FIG. 5A , the signal processor 15 has a short exposure time for the second pixel group G2 in the first region R1 and a third pixel group in the second region R2. The setting signal SET can be generated so that (G3) has a long exposure time.

The first pixel group G1 may include pixels regularly distributed in the pixel array 11a independently of illuminance. For example, as shown in FIG. 5A , the first pixel group G1 may include odd rows of the pixel array 11a. The first pixel group G1 may have an exposure time independent of illuminance, that is, independent of the first region R1 and the second region R2. In some embodiments, the first pixel group G1 may have an exposure time of 11 ms due to light emitting diode (LED) flicker, and the second pixel group G2 of the first region R1 may have an exposure time of less than 11 ms. It may have a shorter or equal exposure time (eg, 5.5 ms, 9 ms, etc.), and the third pixel group G3 of the second region R2 may have an exposure time longer than or equal to 11 ms (eg, 22 ms).

Referring to FIG. 5B , the pixel array 11b may be divided into a first region R1 and a second region R2. As shown in FIG. 5B , in the signal processor 15, the second pixel group G2 has a short exposure time in the first region R1 corresponding to high luminance and the second region R2 corresponding to low luminance. In , the setting signal SET may be generated so that the third pixel group G3 has a long exposure time. The first pixel group G1 may include pixels regularly distributed in the pixel array 11b independently of illuminance. For example, as shown in FIG. 5B , the first pixel group G1 may include rows having indices corresponding to multiples of 3 in the pixel array 11b.

Referring to FIG. 5C , the pixel array 11c may be divided into a first region R1 and a second region R2. As shown in FIG. 5C , the signal processor 15 has a short exposure time for the second pixel group G2 in the first region R1 corresponding to high luminance and the second region R2 corresponding to low luminance. In , the setting signal SET may be generated so that the third pixel group G3 has a long exposure time. The first pixel group G1 may include pixels regularly distributed in the pixel array 11b independently of illuminance. For example, as shown in FIG. 5C , the first pixel group G1 may include pixels disposed at positions corresponding to a grid of equal intervals. Referring to FIG. 5D , the pixel array 11d may be divided into a first region R1 and a second region R2. As shown in FIG. 5D , the signal processor 15 has a short exposure time for the second pixel group G2 in the first region R1 corresponding to high luminance and the second region R2 corresponding to low luminance. In , the setting signal SET may be generated so that the third pixel group G3 has a long exposure time. The first pixel group G1 may include pixels regularly distributed in the pixel array 11b independently of illuminance. For example, as shown in FIG. 5D , the first pixel group G1 may include pixels disposed at positions corresponding to a lattice at regular intervals, similar to the first pixel group G1 of FIG. 5C. It can correspond to different patterns.

Referring to FIG. 5E , the pixel array 11e may be divided into a first region R1 and a second region R2. Unlike the pixel arrays 11a to 11d of FIGS. 5A to 5D that are divided into the first region R1 and the second region R2 in row units, the pixel array 11e of FIG. 5E is arbitrary according to the illuminance. It can be divided into regions corresponding to the shapes of . For example, as shown in FIG. 5E, the pixel array 11e is disposed in the center and has a second region R2 corresponding to low luminance and a first region surrounding the second region R2 and corresponding to high luminance ( R1) can be divided into The signal processor 15 may generate the setting signal SET so that the first region R1 includes pixels having a short exposure time and the second region R2 includes pixels having a long exposure time. . For example, as shown in FIG. 5E , the signal processor 15 has a short exposure time for the second pixel group G2 in the first region R1 and a third pixel group in the second region R2. The setting signal SET can be generated so that (G3) has a long exposure time. As shown in FIG. 5C , the first pixel group G1 corresponding to the reference pixel group may include pixels arranged at positions corresponding to the grid at regular intervals.

6A and 6B are diagrams illustrating examples of operations of the image sensor 10 according to exemplary embodiments of the present disclosure. Specifically, FIGS. 6A and 6B show examples of operations performed by the signal processor 15 of FIG. 1 . As described above with reference to the drawings, the signal processor 15 may calculate values of the digital signal DSIG corresponding to the pixel groups and generate image data IMG based on the calculation result. In the following, FIGS. 6A and 6B will be described with reference to FIG. 1 .

Referring to the left side of FIG. 6A , the pixel array 11 may include a first region R1 corresponding to high luminance and a second region R2 corresponding to low luminance. The first region R1 may include a first pixel PX1 as a reference pixel and a second pixel PX2 as an adaptive pixel. The second region R2 may include a third pixel PX3 as a reference pixel and a fourth pixel PX4 as an adaptive pixel. As described above with reference to FIGS. 5A to 5D , the first pixel PX1 and the third pixel PX3 may be included in the reference pixel group and may have a fixed exposure time (eg, 11 ms). Also, the second pixel PX2 of the first region R1 may have an exposure time shorter than or equal to that of the reference pixel, and the fourth pixel PX4 of the second region R2 may have an exposure time longer than or equal to the reference pixel. can have

In some embodiments, signal processor 15 may calculate a weighted sum of values of digital signal DSIG. For example, as shown in FIG. 6A , the value of the digital signal DSIG corresponding to the first pixel PX1 of the pixel array 11 and the weight W11 are multiplied by a value corresponding to the second pixel PX2. The product of the value of the digital signal DSIG and the weight W21 may be added, and the value of the first pixel PX1 ′ of the image data IMG may correspond to the calculated weighted sum. The product of the value of the digital signal DSIG corresponding to the first pixel PX1 of the pixel array 11 and the weight W12 and the product of the value of the digital signal DSIG corresponding to the second pixel PX2 and the weight W22 are The value of the second pixel PX2 ′ of the image data IMG may correspond to the calculated weighted sum. The product of the value of the digital signal DSIG corresponding to the third pixel PX3 of the pixel array 11 and the weight W31 is the product of the value of the digital signal DSIG corresponding to the fourth pixel PX4 and the weight W41. The value of the third pixel PX3 ′ of the image data IMG may correspond to the calculated weighted sum. The product of the value of the digital signal DSIG corresponding to the third pixel PX3 of the pixel array 11 and the weight W32 and the product of the value of the digital signal DSIG corresponding to the fourth pixel PX4 and the weight W42 are The value of the fourth pixel PX4 ′ of the image data IMG may correspond to the calculated weighted sum. In some embodiments, differently from that shown in FIG. 6A , a weighted sum of values corresponding to three or more adjacent pixels in the pixel array 11 may be calculated for the image data IMG.

In some embodiments, weights may be defined based on exposure time. For example, the exposure time of the first pixel PX1 may be longer than or equal to the exposure time of the second pixel PX2 , and accordingly, the weight (eg, W11) of the first pixel PX1 is the second pixel PX2. ) may be smaller than the weight (eg, W21). Also, the exposure time of the third pixel PX3 may be shorter than or equal to the exposure time of the fourth pixel PX4 , and accordingly, the weight (eg, W31) of the third pixel PX3 is the weight of the fourth pixel. (eg W41).

In some embodiments, weights may be defined based on resolution or sharpness of the image data IMG. For example, the weights W11 and W12 of the first pixel PX1 may be different, and/or the weights W21 and W22 of the second pixel PX2 may be different. Accordingly, the first pixel PX1' and the second pixel PX2' of the image data IMG may have different values, respectively. Also, the weights W31 and W32 of the third pixel PX3 may be different, and/or the weights W31 and W42 of the fourth pixel PX4 may be different. Accordingly, the third pixel PX3 ′ and the fourth pixel PX4 ′ of the image data IMG may have different values.

As described above, the signal processor 15 may perform binning on pixels each having different exposure times, and accordingly, as described below with reference to FIGS. 7A to 8B , the image sensor 10 ) may have improved performance.

Referring to FIG. 6B , the resolution of image data IMG may vary by binning. For example, as shown in FIG. 6B , first to third image data IMG1 to IMG3 having different resolutions are generated from first to fourth pixels PX1 to PX4 adjacent to each other in the pixel array 11. can Like the image data IMG of FIG. 6A , the first image data IMG1 may have a resolution corresponding to the pixel array 11 . In some embodiments, the values of the first pixel PX1' and the third pixel PX3' of the first image data IMG1 are the values of the first pixel PX1 and the third pixel PX3 of the pixel array 11. ), and the values of the first pixel PX1' and the third pixel PX3' of the first image data IMG1 are the values of the second pixels PX21 and PX21 of the pixel array 11. It may be generated by binning the fourth pixel PX4 . In some embodiments, the values of the first to fourth pixels PX1' to PX4' of the first image data IMG1 are binned with respect to the first to fourth pixels PX1 to PX4 of the pixel array 11. can be created by

The value of the first pixel PX1' of the second image data IMG2 may be generated by binning the first pixel PX1 and the third pixel PX3 of the pixel array 11, and The value of the second pixel PX2 ′ of the data IMG2 may be generated by binning the second pixel PX2 and the fourth pixel PX4 of the pixel array 11 . Accordingly, the second image data IMG2 may correspond to an image having a lower resolution than the first image data IMG1. Also, the value of the first pixel PX1 ′ of the third image data IMG3 may be generated by binning the first to fourth pixels PX1 to PX4 of the pixel array 11 . Accordingly, the third image data IMG3 may correspond to an image having a lower resolution than the first image data IMG1 and the second image data IMG2.

7A and 7B show SNR-HDR graphs of the image sensor 10 according to exemplary embodiments of the present disclosure. Specifically, the graph of FIG. 7A represents the SNR-HDR graph of the image sensor 10 in a low-illuminance area, and the graph of FIG. 7B represents the SNR-HDR graph of the image sensor 10 in a high-illuminance area. Hereinafter, FIGS. 7A and 7B will be described with reference to FIG. 6A.

Referring to FIG. 7A, a first curve 71a may correspond to a case where the same exposure time is applied to all pixels of the pixel array 11, and a second curve 72a, as described above with reference to the drawings, This may correspond to a case in which different exposure times are applied to pixel groups in a low-illuminance area. As shown in A of FIG. 7A , the second curve 72a may have a higher SNR than the first curve 71a in low light, and the image sensor 10 may provide improved low light SNR. For example, when the exposure time of the third pixel PX3 is 11 ms and the exposure time of the fourth pixel PX4 is 22 ms in the second region R2 of FIG. 6A corresponding to low illumination, the exposure time is doubled. Due to time, the SNR of about 6 dB may increase in the case of the fourth pixel PX4, and when the weights W31, W32, W41, and W42 are all the same, by binning, as shown in [Equation 1] below, about 5 dB of SNR may increase.

In [Equation 1], normalization may be performed due to different exposure times, and normalization may be reflected in weights in some embodiments.

Referring to FIG. 7B , a first curve 71b in FIG. 7B represents a case in which the same exposure time is applied to all pixels of the pixel array 11, and a second curve 72b is as described above with reference to the drawings. , may correspond to a case in which different exposure times are applied to pixel groups in the high luminance area. As shown in B of FIG. 7B , the second curve 72b may have extended HDR in the high luminance region than the first curve 71b, and the image sensor 10 may provide improved HDR. For example, when the exposure time of the first pixel PX1 is 11 ms and the exposure time of the second pixel PX2 is 5.5 ms in the first region R1 of FIG. 6A corresponding to high luminance, the second pixel At (PX2), the maximum acceptable illuminance can be increased, and HDR can be extended by 6dB as shown in [Equation 2] below.

8A and 8B show SNR-HDR graphs of the image sensor 10 according to exemplary embodiments of the present disclosure. Specifically, the graph of FIG. 8A shows the SNR-HDR graph of the image sensor 10 in a high-temperature state in a low-illuminance area, and the graph of FIG. 8B shows the SNR-HDR graph of the image sensor 10 in a high-temperature state in a high-illuminance area. indicate In the following, FIGS. 8A and 8B will be described with reference to FIG. 1 .

The image sensor 10 may be required to have a good SNR at the image joint point, as described above with reference to FIG. 3 , and in particular the SNR at the image joint point may degrade when the image sensor 10 is at a high temperature. there is. As described above with reference to the drawings, pixels each having different exposure times may be binned, and when a difference in noise between pixels is large, an SNR of about 5 dB may be increased based on a pixel with high noise. Also, pixels each having different exposure times can be binned, and if the noise in the pixels is similar, the SNR of about 3dB can be increased. Accordingly, SNR dips occurring between different sections, such as dark signal non-uniformity (DSNU), can be greatly improved by binning pixels each having different exposure times. That is, an effect of disposing an additional section in a section having a weak SNR may occur, and as a result, an effect of improving the overall SNR may occur.

Referring to FIG. 8A, a first curve 81a in FIG. 8A may correspond to a case where the same exposure time is applied to all pixels of the pixel array 11, and a second curve 82a may correspond to the case described above with reference to the drawings. As such, it may correspond to a case in which different exposure times are applied to pixel groups in a low-light area. As shown in A, B, and C of FIG. 8A, the second curve 82a may have a higher SNR than the first curve 81a at the image combining points, and the image sensor 10 may have an improved SNR at the image combining points. SNRs can be provided.

Referring to FIG. 8B, a first curve 81b in FIG. 8B may correspond to a case where the same exposure time is applied to all pixels of the pixel array 11, and a second curve 82b may correspond to the case described above with reference to the drawings. As such, it may correspond to a case in which different exposure times are applied to pixel groups in a high luminance area. As shown in D, E and F of FIG. 8B, the second curve 82b may have a higher SNR than the first curve 81b at the image combining points, and the image sensor 10 may have an improved SNR at the image combining points. SNRs can be provided.

9 is a flowchart illustrating a method performed by image sensor 10 according to an exemplary embodiment of the present disclosure. As shown in FIG. 9 , the method performed by the image sensor 10 may include a plurality of steps S12, S14, and S16. In some embodiments, the method of FIG. 9 may be performed by signal processor 15 of FIG. 1 before step S20 of FIG. 4 is performed. In the following, FIG. 9 will be described with reference to FIG. 1 .

Referring to FIG. 9 , illuminance may be identified in step S12. For example, the signal processor 15 may identify the illuminance of a subject based on the digital signal DSIG. In some embodiments, the signal processor 15 may identify illuminance based on values of the digital signal DSIG corresponding to all of the plurality of pixels included in the pixel array 11 . In some embodiments, the signal processor 15 is based on values of the digital signal DSIG corresponding to some of the plurality of pixels included in the pixel array 11 (eg, pixels corresponding to an equally spaced grid). Thus, the intensity can be identified.

At step S14, pixel groups and exposure times can be identified. For example, the signal processor 15 may compare the illuminance identified in step S12 with at least one threshold value. The signal processor 15 may define pixels included in the same illuminance range among two or more illuminance ranges defined by at least one threshold value as one area. The signal processor 15 may divide the pixel array 11 into two or more regions and identify a pixel group in each of the divided regions. In some embodiments, as described above with reference to FIGS. 5A to 5D , the pixel array 11 may include a reference pixel group, and the signal processor 15 may exclude the reference pixel group from the divided area. A pixel group containing pixels may be identified. In some embodiments, signal processor 15 may refer to a lookup table that stores a plurality of pairs of illuminance range and exposure time, and may identify an exposure time corresponding to a pixel group.

In step S16, the controller 14 can be set. For example, the signal processor 15 may set the controller 14 based on the pixel group and exposure time identified in step S14. The signal processor 15 may generate a set signal SET including information on the identified pixel group and exposure time, and the controller 14 may generate the driver 12 according to the information included in the set signal SET. can control. Accordingly, the driver 12 may drive the pixel group identified in step S14 to have the exposure time identified in step S14. Examples of step S16 will be described later with reference to FIGS. 10A and 10B.

10A and 10B are flow charts illustrating examples of a method performed by image sensor 10 according to exemplary embodiments of the present disclosure. Specifically, the flowcharts of FIGS. 10A and 10B show examples of step S16 of FIG. 9 . As described above with reference to FIG. 9 , the signal processor 15 of FIG. 1 may set the controller 13 in step S16a of FIG. 10a and step S16b of FIG. 10b. In some embodiments, step S16 of FIG. 9 may include both step S16a of FIG. 10a and step S16b of FIG. 10b. The second pixel group and the third pixel group in FIGS. 10A and 10B are the second pixel group G2 and the third pixel group G3 of FIGS. 5A to 5D , and the first pixel group G1 is a fixed pixel group. It is assumed to have 1 exposure time (eg, 11 ms). Hereinafter, FIGS. 10A and 10B will be described with reference to FIGS. 5A to 5D.

Referring to FIG. 10A , step S16a may include step S16_2 and step S16_4. In step S16_2 , the controller 14 may set the small photodiode SPD of the second pixel group G2 to have a second exposure time. The second pixel group G2 may be included in the first region R1 corresponding to the high luminance, and thus the second exposure time (eg, 5.5 ms) is the first exposure time of the first pixel group G1 ( For example, it may be shorter than or equal to 11ms). As described above with reference to FIG. 2B , the pixel may include a small photodiode (SPD) and a large photodiode (LPD), and the signal processor 15 may include a small photodiode of the pixels included in the second pixel group G2. The controller 14 may be set so that the photodiode SPD has a second exposure time shorter than or equal to the first exposure time. Accordingly, the image sensor 10 may provide extended HDR.

In step S16_4, the controller 14 may set the large photodiode LPD of the second pixel group G2 to have a fourth exposure time. The second pixel group G2 may be included in the first region R1 corresponding to the high luminance, and accordingly, in step S16_2, the small photo diode SPD of the second pixel group G2 has a shorter exposure time than the first exposure time. Meanwhile, in step S16_4, the large photodiode LPD of the second pixel group G2 has a fourth exposure time equal to or longer than the second exposure time (eg, 9 ms or 11 ms). can have Accordingly, as will be described later with reference to FIG. 11A , SNR deterioration that may occur when a dark subject is captured in a high-illuminance area can be prevented.

Referring to FIG. 10B , step S16b may include step S16_6 and step S16_8. In step S16_6 , the controller 14 may set the large photodiode LPD of the third pixel group G3 to have a third exposure time. The third pixel group G3 may be included in the second region R2 corresponding to low luminance, and thus the third exposure time (eg, 22 ms) is the first exposure time (eg, 22 ms) of the first pixel group G1. 11 ms) or longer. As described above with reference to FIG. 2B , the pixel may include a small photodiode (SPD) and a large photodiode (LPD), and the signal processor 15 determines the large size of the pixel included in the third pixel group G3. The controller 14 may be set so that the photodiode LPD has a third exposure time longer than or equal to the first exposure time. Accordingly, the image sensor 10 may provide a high SNR in low light.

In step S16_8, the controller 14 may set the small photodiode SPD of the third pixel group G3 to have a fifth exposure time. The third pixel group G3 may be included in the second region R2 corresponding to the low luminance, and thus, in step S16_6, the large photodiode LPD of the third pixel group G3 is longer than or equal to the first exposure time. While it may be set to the third exposure time, the small photodiodes (SPDs) of the third pixel group G3 in step S16_8 have a fourth exposure time shorter than or equal to the third exposure time (eg, 5.5 ms, 9 ms, or 11 ms). ) can have. Accordingly, as described below with reference to FIG. 11B , extended HDR may be provided even in a low-illuminance area.

11A and 11B show SNR-HDR graphs of the image sensor 10 according to exemplary embodiments of the present disclosure. Specifically, FIG. 11A shows an SNR-HDR graph of the image sensor 10 performing step S16a of FIG. 10A, and FIG. 11B shows an SNR-HDR graph of the image sensor 10 performing step S16b of FIG. 10B. . Hereinafter, FIGS. 11A and 11B will be described with reference to FIGS. 5A to 5D.

Referring to FIG. 10A, a first curve 111a may correspond to a case in which the same exposure time is applied to all pixels of the pixel array 11, and a second curve 112a may correspond to a case as described above with reference to FIG. 11A. , the small photodiode SPD of the second pixel group G2 in the first region R1 corresponding to the high luminance has a second exposure time shorter than or equal to the first exposure time, and the second pixel group G2 The large photodiode LPD of may have a fourth exposure time longer than or equal to the second exposure time. Accordingly, as shown in A of FIG. 11A, the second curve 112a may have a higher SNR than the first curve 111a at low illumination, and the image data IMG may represent a dark subject well even in a high illumination region. .

Referring to FIG. 10B, the first curve 111b may correspond to a case where the same exposure time is applied to all pixels of the pixel array 11, and the second curve 112b may correspond to the case as described above with reference to FIG. 11B. , the large photodiode LOD of the third pixel group G3 has a third exposure time longer than or equal to the first exposure time in the second region R2 corresponding to low illumination, and the small photodiode LOD of the third pixel group G3 The photodiode SPD may have a fifth exposure time shorter than or equal to the third exposure time. Accordingly, as shown in B of FIG. 11B, the second curve 112b may have extended HDR in the high-illumination area than the first curve 111b, and the image data IMG may well represent a bright subject even in the low-illumination area. can

12 is a flowchart illustrating a method performed by image sensor 10 according to an exemplary embodiment of the present disclosure. Specifically, the flowchart of FIG. 12 shows an example of step S80 of FIG. 4 . As described above with reference to FIG. 4 , the image data IMG may be generated by calculating the first value and the second value of the digital signal DSIG in step S80 ′ of FIG. 12 . As shown in FIG. 12 , step S80' may include a plurality of steps S82, S84, S86, and S88. In some embodiments, step S80' may be performed by the signal processor 15 of FIG. 1, and FIG. 12 will be described with reference to FIG. 1 below.

Referring to FIG. 12 , in step S82, the first value of the reference pixel and the second value of the adaptive pixel may be normalized. As described above with reference to the figures, the reference pixel and the adaptive pixel may each have different exposure times. Accordingly, in an environment where motion artifact and/or LED flicker occur, the reference pixel and the adaptive pixel may each generate pixel signals having a large difference. Accordingly, the signal processor 15 may normalize the first value and the second value by compensating for the exposure time difference in order to compare the first value of the reference pixel and the second value of the adaptive pixel, as will be described later. . For example, when the exposure time of the adaptive pixel is 1/2 of the exposure time of the reference pixel, the second value may be normalized by a factor of two.

In step S84, the difference between the normalized first value and the normalized second value may be compared with the threshold value THR. For example, the signal processor 15 may calculate the difference between the normalized first value and the normalized second value in step S82 and determine whether the difference is greater than or equal to the threshold value THR. When the difference between the first value and the second value is large, the signal processor 15 may consider operation noise and/or LED flicker to occur, and increase the dependence on the reference pixel and decrease the dependence on the adaptive pixel. . As shown in FIG. 12 , when the difference between the first value and the second value is equal to or greater than the threshold value THR, step S86 may be subsequently performed. On the other hand, when the difference between the first value and the second value is less than the threshold value THR, the execution of step S86 may be omitted and step S88 may be subsequently performed.

When operation noise and/or LED flicker occurs, the weight of the second value may be decreased in step S86. For example, the signal processor 15 may decrease the weight of the second value in order to increase the dependence on the reference pixel and decrease the dependence on the adaptive pixel. In some embodiments, signal processor 15 may set the weight of the second value to zero, so that the second value may be ignored. In some embodiments, signal processor 15 may increase the weight of the first value while decreasing the weight of the second value.

At step 88, a weighted sum of the first value and the second value may be calculated. For example, when operation noise and/or LED flicker do not occur, that is, when step S88 is performed following step S84, the signal processor 15 determines the first value and the second value based on predefined weights. A weighted sum of values can be calculated. On the other hand, when operation noise and/or LED flicker occurs, that is, when step S88 is performed following step S86, the signal processor 15 determines the first value and the second value based on the weights adjusted in step S86. A weighted sum of values can be calculated. As a result, operation noise due to different exposure times of pixels can be eliminated, and LED flicker mitigation (LFM) can be effectively achieved.

13A is an exploded perspective view of an image sensor 100a according to an exemplary embodiment of the present disclosure, FIG. 13B is a plan view of the image sensor 100a according to an exemplary embodiment of the present disclosure, and FIG. 13C is an example of the present disclosure. It is an exploded perspective view of the image sensor 100a according to the embodiment.

Referring to FIGS. 13A and 13B , the image sensor 100a may have a structure in which a first chip CH1 and a second chip CH2 are stacked. A pixel core (eg, at least one photoelectric conversion element and other elements) of each of the plurality of pixels PX included in the pixel array ( 11 in FIG. 1 ) is formed in the first chip CH1 , and the second chip A logic circuit, for example, a driver 12, a readout circuit 13, a controller 14, and a signal processor 15 of FIG. 1 may be formed in CH2.

As shown in FIG. 13B , the first chip CH1 and the second chip CH1 may include an active area AA and a logic area LA, respectively, which are disposed in the center, and may be disposed on the periphery of the chip. It may include the disposed peripheral areas PERR and PEI. A photoelectric conversion element and other elements may be disposed in a two-dimensional array structure in the active area AA of the first chip CH1. A logic circuit may be disposed in the logic area LA of the second chip CH2 .

Through vias extending in the third direction (Z direction) may be disposed in the peripheral areas PERR and PEI of the first chip CH1 and the second chip CH2 . The first chip CH1 and the second chip CH1 may be electrically coupled to each other through through vias. Wires and vertical contacts extending in a first direction (X direction) or a second direction (Y direction) may be further formed in the peripheral area PERR of the first chip CH1 . A plurality of wiring lines extending in the first direction (X direction) and the second direction (Y direction) may also be disposed in the wiring layer of the second chip CH2 , and these wiring lines may be connected to a logic circuit. Meanwhile, although the structure in which the first chip CH1 and the second chip CH2 are electrically coupled through the through via has been described, the present invention is not limited thereto. For example, the first chip CH1 and the second chip CH2 ) may be implemented in various coupling structures such as Cu-Cu bonding, coupling of through-vias and Cu pads, coupling of through-vias and external connection terminals, or coupling through integral through-vias.

Referring to FIG. 13C , a plurality of unit pixels included in the pixel array 110a may be embedded in an upper chip. For example, the upper chip may correspond to the first chip CH1 of FIG. 13A. The driver 12 of FIG. 1 may be embedded in the underlying chip. For example, the lower chip may correspond to the second chip CH2 of FIG. 13A. According to various embodiments, at least one control signal may be passed through through vias. For example, a control signal for turning on the first switch DSW1 may be electrically transferred from the lower second chip CH2 to the upper first chip CH1 through the through-via.

14 is a block diagram illustrating an electronic device including a multi-camera module according to an exemplary embodiment of the present disclosure, and FIG. 15 is a block diagram illustrating the camera module of FIG. 14 according to an exemplary embodiment of the present disclosure.

Referring to FIG. 14 , the electronic device 1000 may include a camera module group 1100, an application processor 1200, a PMIC 1300, and an external memory 1400.

The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Although the drawing shows an embodiment in which three camera modules 1100a, 1100b, and 1100c are disposed, the embodiments are not limited thereto. In some embodiments, the camera module group 1100 may be modified to include only two camera modules. Also, in some embodiments, the camera module group 1100 may be modified to include k (k is an integer of 4 or more) camera modules. Hereinafter, a detailed configuration of the camera module 1100b will be described in more detail with reference to FIG. 15 , but the following description may be equally applied to other camera modules 1100a and 1100b according to embodiments.

Referring to FIG. 15, the camera module 1100b includes a prism 1105, an optical path folding element (hereinafter referred to as ½OPFE½) 1110, an actuator 1130, an image sensing device 1140, and a storage unit ( 1150) may be included. The prism 1105 may include a reflective surface 1107 of a light reflective material to change a path of light L incident from the outside. In some embodiments, the prism 1105 may change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism 1105 rotates the reflective surface 1107 of the light reflecting material in the direction A around the central axis 1106 or rotates the central axis 1106 in the direction B to move in the first direction X. A path of the incident light L may be changed in a second direction Y, which is perpendicular to the second direction Y. At this time, the OPFE 1110 may also move in a third direction (Z) perpendicular to the first direction (X) and the second direction (Y). In some embodiments, as shown, the maximum angle of rotation of the prism 1105 in the A direction may be less than 15 degrees in the positive A direction and greater than 15 degrees in the negative A direction. However, the embodiments are not limited thereto. In some embodiments, prism 1105 can move around 20 degrees, or between 10 and 20 degrees, or between 15 and 20 degrees in the positive or negative B direction, where the angle of movement is positive. It can move at the same angle in the (+) or minus (-) B direction, or it can move to an almost similar angle within the range of 1 degree. In some embodiments, the prism 1105 can move the reflective surface 1106 of the light reflecting material in a third direction (eg, Z direction) parallel to the extension direction of the central axis 1106 .

The OPFE 1110 may include, for example, optical lenses consisting of m (where m is an integer greater than 0) groups. The m lenses may move in the second direction (Y) to change the optical zoom ratio of the camera module 1100b. For example, when the basic optical zoom magnification of the camera module 1100b is Z, when m optical lenses included in the OPFE 1110 are moved, the optical zoom magnification of the camera module 1100b is 3Z or 5Z or It can be changed to an optical zoom magnification of 5Z or higher.

The actuator 1130 may move the OPFE 1110 or an optical lens (hereinafter referred to as an optical lens) to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is positioned at the focal length of the optical lens for accurate sensing.

The image sensing device 1140 may include an image sensor 1142 , a control logic 1144 and a memory 1146 . The image sensor 1142 may sense an image of a sensing target using light L provided through an optical lens. The image sensor 1142 may merge HCG image data and LCG image data to generate image data having a high dynamic range.

The control logic 1144 may control the overall operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

The memory 1146 may store information required for operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 may include information necessary for the camera module 1100b to generate image data using light L provided from the outside. The calibration data 1147 may include, for example, information about a degree of rotation, information about a focal length, information about an optical axis, and the like, as described above. When the camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 1147 is a focal length value for each position (or state) of the optical lens and It may include information related to auto focusing.

The storage unit 1150 may store image data sensed through the image sensor 1142 . The storage unit 1150 may be disposed outside the image sensing device 1140 and may be implemented in a stacked form with a sensor chip constituting the image sensing device 1140 . In some embodiments, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiments are not limited thereto.

Referring to FIGS. 14 and 15 together, in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

In some embodiments, one of the plurality of camera modules 1100a, 1100b, 1100c (eg, 1100b) is a folded lens including the prism 1105 and the OPFE 1110 described above. camera module, and the remaining camera modules (eg, 1100a, 1100b) may be vertical camera modules that do not include the prism 1105 and the OPFE 1110, but embodiments are limited thereto. it is not going to be

In some embodiments, one camera module (eg, 1100c) among the plurality of camera modules 1100a, 1100b, and 1100c is of a vertical type that extracts depth information using IR (Infrared Ray), for example. It may be a depth camera. In this case, the application processor 1200 merges image data provided from the depth camera and image data provided from other camera modules (eg, 1100a or 1100b) to obtain a 3D depth image. can create

In some embodiments, at least two camera modules (eg, 1100a, 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c may have different fields of view (viewing angles). In this case, for example, optical lenses of at least two camera modules (eg, 1100a, 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other, but the present invention is not limited thereto.

Also, in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may have different viewing angles. In this case, optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, but are not limited thereto.

In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be disposed physically separated from each other. That is, the sensing area of one image sensor 1142 is not divided and used by a plurality of camera modules 1100a, 1100b, and 1100c, but an independent image inside each of the plurality of camera modules 1100a, 1100b, and 1100c. A sensor 1142 may be disposed.

Referring back to FIG. 14 , the application processor 1200 may include an image processing device 1210 , a memory controller 1220 , and an internal memory 1230 . The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be separately implemented as separate semiconductor chips.

The image processing device 1210 may include a plurality of sub image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216. The image processing device 1210 may include a plurality of sub image processors 1212a, 1212b, and 1212c corresponding to the number of the plurality of camera modules 1100a, 1100b, and 1100c.

Image data generated from each of the camera modules 1100a, 1100b, and 1100c may be provided to the corresponding sub image processors 1212a, 1212b, and 1212c through separate image signal lines ISLa, ISLb, and ISLc. . For example, image data generated from the camera module 1100a is provided to the sub image processor 1212a through the image signal line ISLa, and image data generated from the camera module 1100b is provided to the image signal line ISLb. Image data generated from the camera module 1100c may be provided to the sub image processor 1212c through the image signal line ISLc. Such image data transmission may be performed using, for example, a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface), but embodiments are not limited thereto.

Meanwhile, in some embodiments, one sub image processor may be arranged to correspond to a plurality of camera modules. For example, the sub image processor 1212a and the sub image processor 1212c are not separately implemented as shown, but integrated into one sub image processor, and the camera module 1100a and the camera module 1100c Image data provided from may be selected through a selection element (eg, multiplexer) and the like, and then provided to the integrated sub image processor.

Image data provided to each of the sub image processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may generate an output image using image data provided from each of the sub image processors 1212a, 1212b, and 1212c according to image generating information or a mode signal.

Specifically, the image generator 1214 merges at least some of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles according to image generation information or a mode signal to obtain an output image. can create In addition, the image generator 1214 may generate an output image by selecting any one of image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles according to image generation information or a mode signal. .

In some embodiments, the image creation information may include a zoom signal or zoom factor. Also, in some embodiments, the mode signal may be a signal based on a mode selected by a user, for example.

When the image generating information is a zoom signal (zoom factor) and each of the camera modules 1100a, 1100b, and 1100c have different fields of view (viewing angles), the image generator 1214 operates differently according to the type of zoom signal. can be performed. For example, when the zoom signal is the first signal, after merging the image data output from the camera module 1100a and the image data output from the camera module 1100c, the merged image signal and the camera module not used for merging An output image may be generated using the image data output from step 1100b. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not merge the image data and uses any one of the image data output from each of the camera modules 1100a, 1100b, and 1100c. You can choose to generate an output image. However, the embodiments are not limited thereto, and a method of processing image data may be modified and implemented as needed.

In some embodiments, the image generator 1214 receives a plurality of image data having different exposure times from at least one of the plurality of sub image processors 1212a, 1212b, and 1212c, and HDR (high dynamic range) for the plurality of image data. range) processing, it is possible to generate merged image data with an increased dynamic range.

The camera module controller 1216 may provide a control signal to each of the camera modules 1100a, 1100b, and 1100c. Control signals generated from the camera module controller 1216 may be provided to corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

One of the plurality of camera modules 1100a, 1100b, and 1100c is designated as a master camera (eg, 1100b) according to image generation information including a zoom signal or a mode signal, and the remaining camera modules (eg, 1100b) For example, 1100a and 1100c) may be designated as slave cameras. Such information may be included in the control signal and provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

Camera modules operating as a master and a slave may be changed according to a zoom factor or an operation mode signal. For example, when the viewing angle of the camera module 1100a is wider than that of the camera module 1100b and the zoom factor indicates a low zoom magnification, the camera module 1100b operates as a master and the camera module 1100a operates as a slave. can act as Conversely, when the zoom factor indicates a high zoom magnification, the camera module 1100a may operate as a master and the camera module 1100b may operate as a slave.

In some embodiments, a control signal provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a sync enable signal. For example, when the camera module 1100b is a master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may transmit a sync enable signal to the camera module 1100b. The camera module 1100b receiving such a sync enable signal generates a sync signal based on the provided sync enable signal, and transmits the generated sync signal to the camera modules (through the sync signal line SSL). 1100a, 1100c). The camera module 1100b and the camera modules 1100a and 1100c may transmit image data to the application processor 1200 in synchronization with the sync signal.

In some embodiments, a control signal provided from the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c may include mode information according to the mode signal. Based on this mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operation mode and a second operation mode in relation to sensing speed.

The plurality of camera modules 1100a, 1100b, and 1100c generate an image signal at a first rate (eg, generate an image signal having a first frame rate) in a first operation mode, and generate an image signal at a second frame rate higher than the first rate. encoding (eg, encoding an image signal having a second frame rate higher than the first frame rate) and transmitting the encoded image signal to the application processor 1200 . In this case, the second speed may be 30 times or less than the first speed.

The application processor 1200 stores the received image signal, that is, the encoded image signal, in the internal memory 1230 or the external storage 1400 of the application processor 1200, and then the memory 1230 or storage 1200. The encoded image signal from 1400 may be read out and decoded, and image data generated based on the decoded image signal may be displayed. For example, a corresponding sub-processor among the plurality of sub-processors 1212a, 1212b, and 1212c of the image processing device 1210 may perform decoding and may also perform image processing on the decoded image signal.

The plurality of camera modules 1100a, 1100b, and 1100c generate image signals at a third rate lower than the first rate in the second operation mode (eg, image signals having a third frame rate lower than the first frame rate). generation), and transmit the image signal to the application processor 1200. An image signal provided to the application processor 1200 may be an unencoded signal. The application processor 1200 may perform image processing on a received image signal or store the image signal in the memory 1230 or the storage 1400 .

The PMIC 1300 may supply power, eg, a power supply voltage, to each of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the PMIC 1300 supplies first power to the camera module 1100a through the power signal line PSLa under the control of the application processor 1200, and supplies the first power to the camera module 1100a through the power signal line PSLb ( 1100b) and third power may be supplied to the camera module 1100c through the power signal line PSLc.

The PMIC 1300 may generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal (PCON) from the application processor 1200, and may also adjust the level of the power. . The power control signal PCON may include a power control signal for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode may include a low power mode, and in this case, the power control signal PCON may include information about a camera module operating in the low power mode and a set power level. Levels of the powers provided to each of the plurality of camera modules 1100a, 1100b, and 1100c may be the same or different from each other. Also, the level of power can be dynamically changed.

As above, exemplary embodiments have been disclosed in the drawings and specifications. Although the embodiments have been described using specific terms in this specification, they are only used for the purpose of explaining the technical idea of the present disclosure, and are not used to limit the scope of the present disclosure described in the claims. . Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Claims (20)

a pixel array including a plurality of pixels;
a driver configured to drive the pixel array;
readout circuitry configured to generate a digital signal from a pixel signal received from the pixel array;
a controller configured to control the driver so that a first pixel group and a second pixel group among the plurality of pixels have a first exposure time and a second exposure time, respectively; and
a signal processor configured to generate image data by calculating a first value of the digital signal corresponding to the first pixel group and a second value of the digital signal corresponding to the second pixel group;
The image sensor, characterized in that the first pixel group is independent of the illuminance. The method of claim 1,
The signal processor identifies the illuminance based on the value of the digital signal, identifies the second pixel group and the second exposure time based on the illuminance, and determines the second pixel group and the second exposure time. An image sensor, characterized in that configured to set the controller based on. The method of claim 2,
The first pixel group includes a plurality of first rows of the pixel array;
The image sensor of claim 1 , wherein the second pixel group includes at least one second row among the plurality of rows. The method of claim 2,
The signal processor is configured to normalize the first value and the second value, and to reduce a weight of the second value when a difference between the normalized first value and the normalized second value is greater than or equal to a reference value. Characteristic image sensor. The method of claim 1,
Each of the plurality of pixels includes a first optoelectronic element and a second optoelectronic element larger than the first optoelectronic element;
The controller causes the first photoelectric devices of the second pixel group to have the second exposure time and the second photoelectric devices of the second pixel group to have a third exposure time equal to or longer than the second exposure time. Image sensor, characterized in that configured to control the driver. The method of claim 5,
The second exposure time is less than or equal to the first exposure time,
The image sensor, characterized in that the third exposure time is equal to or longer than the second exposure time. The method of claim 5,
The second exposure time is greater than or equal to the first exposure time,
The image sensor, characterized in that the third exposure time is equal to or less than the second exposure time. The method of claim 5,
the controller is configured to control the driver such that first photoelectric elements and second photoelectric elements of the first pixel group have the first exposure time;
The image sensor, characterized in that the first exposure time is longer than the second exposure time and shorter than the third exposure time. The method of claim 1,
wherein the signal processor is configured to set the controller based on a value of the digital signal such that each of the plurality of pixels has a first conversion gain or a second conversion gain. The method of claim 1,
The signal processor normalizes at least one of the first value and the second value based on the first exposure time and the second exposure time, and calculates a weighted sum of the normalized first value and the second value An image sensor configured to compute. a pixel array including a plurality of pixels;
a driver configured to drive the pixel array;
readout circuitry configured to generate a digital signal from a pixel signal received from the pixel array;
a controller configured to control the driver so that a first pixel group and a second pixel group among the plurality of pixels have a first exposure time and a second exposure time, respectively; and
a signal processor configured to identify an illuminance based on a value of the digital signal, identify the second pixel group and the second exposure time based on the illuminance, and set the controller based on the second exposure time; include,
The image sensor, characterized in that the first pixel group is independent of the illuminance. The method of claim 11,
The first pixel group includes a plurality of first rows spaced apart from each other at equal intervals in the pixel array;
The second pixel group includes at least one of a plurality of second rows between the plurality of second rows. The method of claim 11,
The signal processor determines a first value of the digital signal corresponding to the first pixel group and a second value of the digital signal corresponding to the second pixel group based on the first exposure time and the second exposure time. Image sensor, characterized in that configured to normalize. The method of claim 13,
wherein the signal processor is configured to generate image data based on a weighted sum of the normalized first value and the second value. The method of claim 13,
The signal processor is configured to normalize the first value and the second value, and to reduce a weight of the second value when a difference between the normalized first value and the normalized second value is greater than or equal to a reference value. Characteristic image sensor. The method of claim 11,
Each of the plurality of pixels includes a first optoelectronic element and a second optoelectronic element larger than the first optoelectronic element;
wherein the signal processor is configured to set the controller such that the first photoelectric devices of the second group of pixels have the second exposure time and the second photovoltaic devices of the second group of pixels have a third exposure time. image sensor to be. The method of claim 16
The second exposure time is less than or equal to the first exposure time,
The image sensor, characterized in that the third exposure time is equal to or longer than the second exposure time. The method of claim 16
The second exposure time is greater than or equal to the first exposure time,
The image sensor, characterized in that the third exposure time is equal to or less than the second exposure time. The method of claim 11,
wherein the signal processor is configured to set the controller based on a value of the digital signal such that each of the plurality of pixels has a first conversion gain or a second conversion gain. As a method performed by an image sensor,
generating a digital signal from a pixel signal received from a pixel array including a plurality of pixels;
driving a first pixel group and a second pixel group among the plurality of pixels to have a first exposure time and a second exposure time, respectively; and
generating image data based on a weighted sum of a first value of the digital signal corresponding to the first pixel group and a second value of the digital signal corresponding to the second pixel group;
The method of claim 1 , wherein the first group of pixels is independent of the illuminance.

Images