TWI767484B - Dual sensor imaging system and depth map calculation method thereof - Google Patents

Abstract

A dual sensor imaging system and a depth map calculation method thereof are provided. The dual sensor imaging system includes at least one color sensor, at least one infrared ray (IR) sensor, a storage device and a processor. The processor is configured to load and execute a computer program stored in the storage device to control the color sensor and the IR sensor to respectively capture multiple color images and multiple IR images using multiple exposure conditions under an imaging scene, adaptively select a combination of the color image and the IR image that are comparable to each other from the captured color images and IR images, and calculate a depth map of the imaging scene by using the selected color image and IR image.

Description

Dual-sensor camera system and its depth map calculation method

The present invention relates to a camera system and method, and more particularly, to a dual-sensor camera system and a depth map calculation method thereof.

The camera's exposure conditions (including aperture, shutter, and sensitivity) affect the quality of the images captured, so many cameras automatically adjust exposure conditions during image capture to obtain clear and bright images. However, in scenes with high contrast such as low light source or backlight, the result of camera adjustment of exposure conditions may result in excessive noise or overexposure in some areas, and the image quality in all areas cannot be balanced.

In this regard, the current technology adopts a new image sensor architecture, which utilizes the characteristics of high light sensitivity of infrared (IR) sensors to intersperse and configure IR pixels among the color pixels of the image sensor to assist brightness detection. Measurement. For example, FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. Referring to FIG. 1 , in addition to red (R), green (G), blue (B) and other color pixels, the conventional image sensor 10 is also interspersed with infrared (I) pixels. Thereby, the image sensor 10 can combine the color information 12 captured by the R, G, and B color pixels with the luminance information 14 captured by the I pixel to obtain an image 16 with moderate color and brightness.

However, in the above-mentioned single image sensor structure, the exposure conditions of each pixel in the image sensor are the same, so only the exposure conditions that are more suitable for color pixels or infrared pixels can be selected to capture images, and the result is still ineffective. The characteristics of the two pixels are used to improve the image quality of the captured image.

The invention provides a dual-sensor camera system and a depth map calculation method thereof, which can accurately calculate the depth map of a camera scene.

The dual-sensor camera system of the present invention includes at least one color sensor, at least one infrared sensor, a storage device, and a processor coupled to the color sensor, the infrared light sensor, and the storage device. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the infrared sensor to respectively capture a plurality of color images and multiple infrared images; adaptively selecting a combination of color images and infrared images that are comparable to each other from the multiple color images and the multiple infrared images; and calculating a depth map of the camera scene using the selected color images and infrared images.

The depth map calculation method of the dual-sensor camera system of the present invention is suitable for a dual-sensor camera system comprising at least one color sensor, at least one infrared sensor and a processor. The method includes the following steps: controlling a color sensor and an infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; A combination of color images and infrared images that are comparable to each other is adaptively selected in the images; and a depth map of the camera scene is calculated using the selected color images and infrared images.

Based on the above, the dual-sensor camera system and the depth map calculation method of the present invention capture multiple images by adopting different exposure conditions suitable for the current camera scene on the independently configured color sensor and infrared sensor. image, and select the color and infrared images that can be compared with each other to calculate the depth map of the shooting scene, so that the depth map of the shooting scene can be accurately calculated.

The embodiments of the present invention are applicable to a dual-sensor camera system independently configured with a color sensor and an infrared sensor. The color image and the infrared image captured by the color sensor and the infrared sensor can be used to calculate the depth map of the camera scene due to the parallax between the color sensor and the infrared sensor. In view of the situation that the color image captured by the color sensor may be overexposed or underexposed due to the influence of light reflection, shadow, high contrast and other factors in the shooting scene, the embodiment of the present invention uses the infrared image to have better The signal-to-noise ratio (SNR) has the advantage of containing more texture details of the camera scene. The texture information provided by the infrared image is used to assist the calculation of the depth value of the defect area, so as to obtain accurate camera scenes. depth map.

FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. Referring to FIG. 2 , the image sensor 20 of the embodiment of the present invention adopts a dual-sensor structure in which a color sensor 22 and an infrared (IR) sensor 24 are independently configured, and the color sensor 22 and the infrared sensor are used. 24 with their respective characteristics, a plurality of images are respectively captured using a plurality of exposure conditions suitable for the current shooting scene, and a color image 22a and an infrared image 24a with appropriate exposure conditions are selected from among them. In some embodiments, through image fusion, the infrared image 24a can be used to supplement the texture details lacking in the color image 22a, so as to obtain a scene image 26 with good color and texture details. In some embodiments, the color image 22a and the infrared image 24a can be used to calculate the depth map of the camera scene, and the texture details provided by the infrared image 24a can be used to compensate for the lack of texture details in the color image, and assist in the calculation of defect areas depth value.

FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 , the dual-sensor camera system 30 of this embodiment can be configured in electronic devices such as mobile phones, tablet computers, notebook computers, navigation devices, driving recorders, digital cameras, digital cameras, etc., to provide a camera function . The dual-sensor camera system 30 includes at least one color sensor 32, at least one infrared sensor 34, a storage device 36 and a processor 38, and its functions are described as follows:

The color sensor 32 includes, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) device, or other types of photosensitive devices, and can sense light intensity to generate a camera scene image. The color sensor 32 is, for example, a red-green-blue (RGB) image sensor, which includes red (R), green (G), and blue (B) color pixels for capturing red light and green light in the camera scene , blue light and other color information, and combine these color information to generate a color image of the camera scene.

The infrared sensor 34 includes, for example, a CCD, a CMOS element or other types of photosensitive elements, which can sense infrared light by adjusting the wavelength sensing range of the photosensitive element. The infrared sensor 34, for example, uses the above-mentioned photosensitive elements as pixels to capture infrared light information in the imaging scene, and synthesizes the infrared light information to generate an infrared image of the imaging scene.

The storage device 36 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard drive A disk or similar element, or a combination of the foregoing, for storing computer programs executable by the processor 38 . In some embodiments, the storage device 36 may also store, for example, the color image captured by the color sensor 32 and the infrared image captured by the infrared sensor 34 .

The processor 38 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessors), microcontrollers (Microcontrollers), and digital signal processors (Digital Signal Processors). Processor, DSP), programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, the present invention does not this limit. In this embodiment, the processor 38 can load a computer program from the storage device 36 to execute the depth map calculation method of the dual-sensor camera system according to the embodiment of the present invention.

FIG. 4 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 4 at the same time, the method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and the following describes the depth map calculation method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

In step S402 , the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for the identified shooting scene.

In some embodiments, the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture color images with shorter or longer exposure times based on the exposure time in standard exposure conditions, for example. The difference between the exposure times of each other is, for example, any value in the exposure value (Exposure Value, EV) between -3 and 3, which is not limited here. For example, if the A image is twice as bright as the B image, the EV of the B image can be increased by 1, and so on, the exposure value can have a decimal (eg +0.3EV), which is not limited here.

In some embodiments, the processor 38 controls at least one of the color sensor 32 and the infrared sensor 34 to capture at least one standard image of the camera scene using standard exposure conditions, and uses these standard images to Identify the camera scene. The standard exposure conditions include, for example, parameters such as aperture, shutter, and sensitivity determined by using the existing light metering technology. , Chroma, white balance and other image parameters to identify the camera scene, including the location of the camera scene (indoor or outdoor), light source (high light source or low light source), contrast (high contrast or low contrast), The type (object or portrait) or state (dynamic or static) of the photographed object, etc. In other embodiments, the processor 38 may also use a positioning method to identify the camera scene or directly receive user operations to set the camera scene, which is not limited herein.

In step S404, the processor 38 adaptively selects a combination of color images and infrared images that are comparable to each other from the plurality of color images and the plurality of infrared images. In some embodiments, the processor 38 selects a combination of color images and infrared images that are comparable to each other, for example, according to color details of each color image and texture details of each infrared image. In some embodiments, the processor 38 uses the color image or the infrared image as a reference to compare the image histograms of each color image and the infrared image, thereby determining a combination of color images and infrared images that are comparable to each other.

In step S406, the processor 38 calculates the depth map of the camera scene using the selected color image and infrared image. In some embodiments, the processor 38 captures, for example, a plurality of feature points with robust features in the selected color image and the infrared image, and calculates the camera scene according to the positions of the feature points corresponding to each other in the color image and the infrared image. depth map.

Through the above method, the dual-sensor camera system 30 can select a color image with better color detail and an infrared image with better texture detail to calculate the depth map of the camera scene, and use the infrared image to compensate or replace all of the color images. The lack of texture details to calculate the depth value, which can accurately calculate the depth map of the camera scene.

In some embodiments, the processor 38 first selects one of the color images as the reference image according to the color details of each color image, and then identifies at least one defective area lacking texture details in the reference image, and then selects one of the color images as the reference image according to the color details of each The texture details of the images corresponding to these defect areas in the infrared images are selected, and one of the infrared images is selected as an image that can be compared with the reference image, and is used for the calculation of the depth map together.

To be more specific, because the color sensor 32 can only use a single exposure condition to capture color images at a time, when the shooting scene is low light source or high contrast, each color image may have high noise and excessive noise. Areas that are exposed or underexposed (i.e. the defective areas described above). At this time, the processor 38 can use the characteristics of the high light sensitivity of the infrared sensor 34 to select the infrared image with the texture details of the defect area from the multiple infrared images captured previously for the above-mentioned defective area, and Can be used to complement texture details in defective areas in color images.

FIG. 5 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 5 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following is a description of the depth map calculation method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

In step S502, the processor 38 selects a color image that can reveal the color details of the shooting scene from the plurality of color images as a reference image.

In some embodiments, the processor 38 selects the color image with the most color details from the plurality of color images as the reference image, for example, according to the color details of each color image. The amount of color detail can be determined, for example, by the size of the overexposed or underexposed areas in the color image.

In detail, the color of the pixels in the overexposed areas tends to be white, and the color of the pixels in the underexposed areas tends to be black, so the color details of these areas will be less. Therefore, if the color image includes more such regions, it means that the color details are less, and the processor 38 can determine which color image has the most color details accordingly, and use it as the reference image. In other embodiments, the processor 38 can also distinguish the color details of each color image according to the contrast, saturation or other image parameters, which is not limited herein.

In step S504, the processor 38 identifies at least one defect region in the reference image that lacks texture details. The defect area is, for example, the above-mentioned overexposed area or underexposed area, or an area with higher noise captured under low light source, which is not limited herein.

In step S506 , the processor 38 selects one of the infrared images according to the texture details of the images corresponding to the defective area in each of the infrared images, and uses it as a combination to be compared with the reference image.

In some embodiments, the processor 38 selects, for example, the infrared image corresponding to the image of the defective area with the most texture detail as the combination to be compared with the reference image. Wherein, the processor 38, for example, determines the amount of texture details of each infrared image according to the contrast ratio or other image parameters, which is not limited herein.

In step S508, the processor 38 executes a feature extraction algorithm to extract multiple feature points with robust features from the reference image and the selected infrared image.

In some embodiments, the feature extraction algorithm is, for example, Harris corner detector, Hessian-affine region detector, Maximally stable extremum region Stable Extremal Regions, MSER), Scale invariant feature transform (SIFT) or Speeded up robust features (SURF), the feature points are, for example, edge or corner pixels in the image, not here set limits. In some embodiments, the processor 38 may also align the color image and the infrared image according to the correspondence between the captured features.

In step S510, the processor 38 calculates the depth map of the imaging scene according to the positions of the feature points corresponding to each other in the reference image and the infrared image.

In some embodiments, the processor 38 directly calculates the aberration of each pixel corresponding to the reference image and the infrared image, and captures the image according to the color sensor and the infrared sensor 34 of the dual-sensor camera system 30 . The focal length, the distance between the two sensors, and the aberration of each pixel are used to estimate the depth of each pixel. The processor 38, for example, calculates the displacement of each pixel between the reference image and the infrared image according to the position of each pixel in the reference image and the infrared image, as the aberration.

In detail, the aberration of the corresponding pixels in the reference image and the infrared image captured by the dual-sensor camera system 30 is determined by the focal length (determining the size of the image), the sensor distance (determining the overlapping range of the images), and the object corresponding to the pixel The distance between the sensor and the sensor (ie the depth value, which determines the size of the object in the image) is determined, there is a certain proportional relationship, and the relationship table recording this proportional relationship can be used in the dual-sensor camera system 30. pre-tested. Therefore, when the user uses the dual-sensor camera system 30 to capture an image, and the processor 38 calculates the aberration of each pixel in the image, the depth value of each pixel can be obtained by querying the pre-established relationship table.

Through the above method, the dual-sensor camera system 30 can calculate the depth value of each pixel using the positional relationship of the corresponding pixels in the color image and the infrared image, thereby obtaining an accurate depth map of the camera scene.

For example, FIG. 6 is an example of a depth map calculation method of a dual-sensor camera system according to an embodiment of the present invention. Referring to FIG. 6 , in this embodiment, the color image 62 with the most color details is selected as the reference image by the depth map calculation method in the above-mentioned FIG. ), and selects the infrared image 64 with the most texture detail in the defect area from the multiple infrared images captured under different exposure conditions, and compares it with the color image 62 to calculate an accurate depth map 66 of the camera scene.

In some embodiments, the processor 38 controls the color sensor 32 to capture a plurality of color images, for example, when the user activates the live view mode, so as to perform auto focus, thereby obtaining the captured images. The focal length of the object, and based on this focal length, the color image that can reveal the most color details of the object is determined.

In the real-time display mode, the processor 38 uses, for example, the exposure time corresponding to the color image that can reveal the most color details of the object as a reference, and controls the color sensor 32 to use a plurality of exposure times that are longer or shorter than the exposure time. Capture multiple color images to monitor environmental changes in the camera scene. Similarly, the processor 38 can also control the infrared sensor 34 to capture multiple infrared images with multiple exposure times longer or shorter than the exposure time corresponding to the infrared image that can reveal the most texture details of the object as a reference. . Finally, the processor 38 can select a combination of the color image and the infrared image that is most comparable to each other from the images captured by the color sensor 32 and the infrared sensor 34 to calculate the depth map of the camera scene.

For example, in some embodiments, the processor 38 calculates an image histogram for each of the color images and the infrared image, and selects the color image or the infrared image as a benchmark to compare the color and infrared images. The image histogram is used to determine the combination of color image and infrared image that can best compare with each other, and used to calculate the depth map of the camera scene.

Specifically, in some embodiments, the processor 38 selects one of the color images (eg, selects the color image that can reveal the most color details of the object) as the reference image, and selects one of the infrared images (eg, is Select the color image that can reveal the most texture details of the object) to compare with the reference image, and judge whether the brightness of the selected infrared image is higher than the brightness of the reference image according to the image histogram of these images. Wherein, if the determination result is yes, the processor 38 selects an infrared image with a shorter exposure time than the selected infrared image from a plurality of infrared images pre-captured by the infrared sensor 34, or controls the infrared sensor The device 34 uses an exposure time shorter than the exposure time of the selected infrared image to capture the infrared image, which is used as a combination to be compared with the reference image. On the other hand, if the determination result is no, the processor 38 selects an infrared image with a longer exposure time than the selected infrared image from a plurality of infrared images pre-captured by the infrared sensor 34, or controls the infrared sensing The device 34 uses an exposure time longer than the exposure time of the selected infrared image to capture the infrared image, which is used as a combination to be compared with the reference image.

On the other hand, in some embodiments, the processor 38 selects one of the infrared images (eg, selects a color image that can reveal the most texture details of the object) as the reference image, and selects one of the color images (eg, is Select the color image that can reveal the most color details of the object) to compare with the reference image, and determine whether the brightness of the selected color image is higher than the brightness of the reference image according to the image histogram of these images. Wherein, if the determination result is yes, the processor 38 selects a color image with a shorter exposure time than the exposure time of the selected color image from the plurality of color images pre-captured by the color sensor 32, or controls the color sensing The controller 32 uses an exposure time shorter than the exposure time of the selected color image to capture the color image, which is used as a combination to be compared with the reference image. Conversely, if the determination result is no, the processor 38 selects a color image with a longer exposure time than the selected color image from the plurality of color images pre-captured by the color sensor 32, or controls the color sensing The controller 32 uses an exposure time longer than the exposure time of the selected color image to capture the color image, which is used as a combination to be compared with the reference image.

Through the above method, the dual-sensor camera system 30 can adaptively select a combination of color images and infrared images that are most comparable to each other from a plurality of color images and infrared images, and use it to calculate an accurate depth map of the camera scene. .

In some embodiments, even if the combination of the most comparable color image and infrared image is selected to calculate the depth map of the camera scene, the selected color image may still be due to reflection or insufficient dynamic range of the color sensor 32 and other factors, there are many defective areas lacking color and/or texture details, which is called occlusion. In this case, the texture details provided by the infrared image can be used as a reference to estimate the depth value of the occlusion from the depth values of the surrounding pixels of the occlusion.

In detail, FIG. 7 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 7 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and additionally, an infrared light emitting diode (LED) is configured in the dual-sensor camera system 30 . ) and other IR projectors (not shown) to enhance the texture details of the captured infrared images. The following describes the detailed steps of the depth map calculation method of the present embodiment in conjunction with various elements of the dual-sensor camera system 30 .

In step S702, the processor 38 detects at least one occlusion lacking color details or texture details in the selected color image, and in step S704, it is determined whether an occlusion is detected.

If blocking is detected in step S704, in step S706, the processor 38 controls the infrared projector to project infrared rays to the camera scene, and controls the infrared sensor 34 to capture the infrared image of the camera scene. Among them, by projecting infrared rays to the camera scene, the texture details of the dark area in the camera scene captured by the infrared sensor 34 can be enhanced, so as to assist the calculation of the subsequent depth map.

In step S708 , the processor 38 determines the depth value of the occlusion from the depth values of the pixels around each occlusion according to the texture details around each occlusion provided by the infrared image captured by the infrared sensor 34 . In detail, since the infrared image can provide accurate texture details of the occluded surrounding pixels, the depth values of the surrounding pixels with homogeneity with the occlusion can be used to fill in the depth values of the holes in the depth map, so that the depth map The holes are filled with the correct depth values with the aid of infrared imagery.

On the other hand, if no occlusion is detected in step S704, then in step S710, the processor 38 calculates the depth map of the camera scene according to the positions of the corresponding feature points in the reference image and the infrared image. This step is the same as or similar to step S510 in the foregoing embodiment, so the detailed content thereof will not be repeated here.

Through the above method, the dual-sensor camera system 30 can effectively fill the holes in the calculated depth map, thereby obtaining a complete and accurate depth map of the camera scene.

To sum up, the dual-sensor camera system and the depth map calculation method of the present invention configure the color sensor and the infrared sensor independently, and use multiple exposure conditions suitable for the current shooting scene to capture multiple A color image and an infrared image that are comparable to each other are selected to calculate the depth map, so that the depth map of various camera scenes can be accurately calculated. By using the texture details provided by the infrared image to assist in calculating the depth value of the holes in the depth map, a complete depth map of the camera scene can be generated.

10, 20: Image sensor 12: Color Information 14: Brightness information 16: Video 22: Color sensor 22a, 62: Color image 24: Infrared sensor 24a, 64: Infrared imagery 26: Scene image 30: Dual-sensor camera system 32: Color Sensor 34: Infrared sensor 36: Storage Device 38: Processor 62a: face area 66: Depth Map R, G, B, I: pixels S402~S406, S502~S510, S702~S710: Steps

FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. FIG. 4 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. FIG. 5 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. FIG. 6 is an example of a depth map calculation method of a dual-sensor camera system according to an embodiment of the present invention. 7 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention.

S402~S406: Steps

Claims (20)

A dual-sensor camera system, comprising: at least one color sensor; at least one infrared sensor; a storage device for storing computer programs; and a processor coupled to the at least one color sensor, the at least one The infrared light sensor and the storage device are configured to load and execute the computer program to: control the at least one color sensor and the at least one infrared sensor to use multiple capturing a plurality of color images and a plurality of infrared images respectively for each exposure condition; selecting a combination of the color images and the infrared images that are compared with each other from the plurality of color images and the plurality of infrared images; and using The selected color image and the infrared image calculate a depth map of the camera scene. The dual-sensor camera system of claim 1, wherein the processor comprises: selecting a color image that can reveal color details of the camera scene from the plurality of color images as a reference image; identifying the color image at least one defect area lacking texture details in the reference image; and selecting an infrared image from the plurality of infrared images that can reveal the texture details of the defect area as a combination for comparing with the reference image . The dual-sensor camera system of claim 2, wherein the processor further comprises: using the color image to perform auto-focusing to obtain the focal length of the object captured in the camera scene, and determine the focal length according to the focal length The color image that can reveal the most color details of the object is used as the reference image. The dual-sensor camera system of claim 1, wherein the processor comprises: calculating an image histogram of each of the color images and each of the infrared images; and selecting one of the color images or the One of the infrared images is used as a reference, and each of the color images and the image histogram of each of the infrared images are compared to determine a combination of the color images and the infrared images that are compared with each other. The dual-sensor camera system of claim 4, wherein the processor comprises: selecting one of the color images as a reference image, and selecting one of the infrared images to compare with the reference image, and Determine whether the brightness of the selected infrared image is higher than the brightness of the reference image according to the image histogram; if so, select an exposure time from the captured infrared images that is longer than the selected infrared image An infrared image with a short exposure time of the image, or controlling the at least one infrared sensor to capture an infrared image with an exposure time shorter than the exposure time of the selected infrared image, for use as a reference image with the reference image and if not, selecting an infrared image with a longer exposure time than the exposure time of the selected infrared image from the captured infrared images, or controlling the at least one infrared sensor to use An infrared image is captured with an exposure time longer than the exposure time of the selected infrared image, and is used as a combination with the reference image to be compared with each other. The dual-sensor camera system of claim 4, wherein the processor comprises: selecting one of the infrared images as a reference image, and selecting one of the color images to compare with the reference image, and According to the image histogram, determine whether the brightness of the selected color image is higher than the brightness of the reference image; A color image with a shorter exposure time of the color image, or controlling the at least one color sensor to capture a color image with an exposure time shorter than the exposure time of the selected color image, for use as a reference image with the reference image and if not, selecting a color image with a longer exposure time than the exposure time of the selected color image from the plurality of captured color images, or controlling the at least one color sensor to use A color image is captured with an exposure time longer than the exposure time of the selected color image, as a combination with the reference image to be compared with each other. The dual-sensor camera system of claim 1, wherein the processor comprises: detecting at least one occlusion lacking color detail or texture detail in the selected color image; and performing hole filling ( hole filling) algorithm, according to the texture details around each of the occlusions provided by the selected infrared image, the depth values of the occlusions are determined from the depth values of a plurality of pixels around the occlusions. The dual-sensor camera system according to claim 7, wherein the dual-sensor camera system further comprises an IR projector, and the processor further comprises: when detecting the occlusion, controlling The infrared projector projects infrared rays to the imaging scene, and controls the at least one infrared sensor to capture an infrared image of the imaging scene; and each of the shielding provided according to the captured infrared image For the surrounding texture details, the depth value of the occlusion is determined by the depth values of a plurality of pixels around each occlusion. The dual-sensor camera system of claim 1, wherein the processor comprises: capturing a plurality of feature points with robust features in the selected color image and the infrared image; and according to the color image and the positions of the feature points corresponding to each other in the infrared image to calculate the depth map of the imaging scene. The dual-sensor camera system of claim 1, wherein the processor comprises: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture using standard exposure conditions at least one standard image of the camera scene, and identifying the camera scene using the at least one standard image. A depth map calculation method for a dual-sensor camera system, the dual-sensor camera system includes at least one color sensor, at least one infrared sensor, and a processor, and the method includes the following steps: by the processing The processor controls the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; selecting a combination of the color images and the infrared images that are compared with each other from the plurality of color images and the plurality of infrared images; and calculating by the processor using the selected color images and the infrared images The depth map of the camera scene. The method of claim 11, wherein the step of selecting a combination of the color images and the infrared images that are compared with each other from the plurality of color images and the plurality of infrared images comprises: selecting from the plurality of color images and the infrared images. selecting a color image from the color images that can reveal the color details of the shooting scene as a reference image; identifying at least one defect area lacking texture details in the reference image; and selecting from the plurality of infrared images to reveal the color details of the defective area Infrared image of texture detail. The method of claim 12, wherein the step of selecting a color image from the plurality of color images that can reveal the color details of the shooting scene as a reference image comprises: using the color image to perform auto-focusing to obtain The focal length of the photographed object in the shooting scene is determined, and a color image that can reveal the most color details of the object is determined as the reference image according to the focal length. The method of claim 11, wherein the step of selecting a combination of the color images and the infrared images that are compared with each other from the plurality of color images and the plurality of infrared images comprises: calculating each of the colors an image histogram of an image and each of the infrared images; and selecting one of the color images or one of the infrared images as a reference, and comparing the image histograms of each of the color images and each of the infrared images , to determine the combination of the color image and the infrared image that are compared with each other. The method of claim 14, wherein the step of comparing the image histograms of each of the color images and each of the infrared images to determine combinations of the color images and the infrared images that are compared to each other comprises: selecting One of the color images is used as a reference image, and one of the infrared images is selected to be compared with the reference image, and whether the brightness of the selected infrared image is higher than the reference image is determined according to the image histogram the brightness of the image; If so, select an infrared image with a shorter exposure time than the selected infrared image from the captured infrared images, or control the at least one infrared sensor to use the selected infrared image. capturing an infrared image with an exposure time with a short exposure time of the infrared image for use as a combination to be compared with the reference image; and if not, selecting an exposure time shorter than the selected one from the plurality of captured infrared images The infrared image with a longer exposure time of the infrared image, or the at least one infrared sensor is controlled to capture an infrared image with an exposure time longer than the exposure time of the selected infrared image, which is used as a comparison with the reference A combination of images that contrast with each other. The method of claim 14, wherein the step of comparing the image histograms of each of the color images and each of the infrared images to determine combinations of the color images and the infrared images that are compared to each other comprises: selecting One of the infrared images is used as a reference image, and one of the color images is selected to be compared with the reference image, and according to the image histogram, it is determined whether the brightness of the selected color image is higher than the brightness of the selected color image. the brightness of the reference image; if so, select a color image with a shorter exposure time than the selected color image from the multiple color images captured, or control the at least one color sensor to use a color image with a shorter exposure time than the selected color image. The selected exposure time of the color image is short to capture the color image as a combination to be compared with the reference image; and If not, select a color image with a longer exposure time than the exposure time of the selected color image from the plurality of captured color images, or control the at least one color sensor to use a color image with a longer exposure time than the selected color image. The color image is captured with a long exposure time of the color image and used as a combination with the reference image to be compared with each other. The method of claim 11, further comprising: detecting at least one occlusion lacking color details or texture details in the selected color image; and executing a hole filling algorithm to For the provided texture details around each of the occlusions, the depth values of the occlusions are determined by the depth values of a plurality of pixels around each of the occlusions. The method of claim 17, wherein the dual-sensor camera system further comprises an infrared projector, after the step of detecting at least one occlusion lacking color detail or texture detail in the selected color image, the The method further includes: when the blocking is detected, controlling the infrared projector to project infrared rays to the camera scene, and controlling the at least one infrared sensor to capture the infrared image of the camera scene; and according to For the texture details around each of the occlusions provided by the captured infrared image, the depth values of the occlusions are determined by the depth values of a plurality of pixels around each of the occlusions. The method of claim 11, wherein the step of calculating a depth map of the camera scene using the selected color image and the infrared image comprises: capturing a plurality of feature points with robust features in the selected color image and the infrared image; and calculating the camera according to the positions of the feature points corresponding to each other in the color image and the infrared image the depth map of the scene. The method of claim 11, wherein the at least one color sensor and the at least one infrared sensor are controlled to capture a plurality of color images and a plurality of color images respectively using a plurality of exposure conditions suitable for a shooting scene Before the step of infrared image, the method further includes: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture at least one standard image of the camera scene using standard exposure conditions , and use the at least one standard image to identify the camera scene.

Images