JP6456466B2 - Image sensor and image sensor control method - Google Patents

Description

  The present invention relates to an image sensor and a method for controlling the image sensor.

  A technique for detecting a partial region or a specific subject satisfying a specific condition from an image signal and controlling image processing on the image signal based on the detection result is known. In Patent Document 1, regarding a method of detecting a human face and its direction from an image signal, a principal component analysis is used after roughly classifying the face direction using a face detection template for face detection. A technique for calculating the face orientation in detail based on the calculated eigenvector is disclosed. Further, in Patent Document 2, a method of extracting a specular reflection component from an image signal based on a dichroic reflection model and estimating a color temperature of a light source in a shooting environment, using a difference in pixel values, the specular reflection component. A technique for extracting the above is disclosed.

JP 2003-141551 A JP 2007-13415 A

  According to the technique of the above-mentioned patent document 1, information relating to a face in an image can be acquired with high accuracy by combining detection with a small amount of calculation and low accuracy and a detection method with a large amount of calculation and high accuracy. . However, when image processing such as image signal color and brightness correction processing is performed based on the face detection result, in the conventional technology, it is necessary to wait for the start of image processing until the face detection processing is completed. Time lag occurs. In particular, when a two-stage detection method such as that described in Patent Document 1 is used, there is a problem that the processing time becomes long.

  The same applies to the case of the above-mentioned Patent Document 2. In order to perform white balance correction on the image signal, it is necessary to wait for the process of extracting the specular reflection component from the image signal to be completed, and the processing time lag is reduced. May occur. Furthermore, as in the case of face detection processing, when extracting specular reflection components in combination with rough and high-speed processing and processing with high accuracy and a large amount of processing, the processing time lag may increase.

  An object of the present invention is to provide an image sensor and an image sensor control method capable of reducing the waiting time of the processing result of the first signal processing unit when performing signal processing based on the processing result of the first signal processing unit. Is to provide.

The image sensor of the present invention is an image sensor in which an image sensor layer and a signal processing layer are stacked, and is included in the image sensor layer, and includes a plurality of pixels that convert an optical image into an image signal, and the signal processing layer. A first signal processing unit that identifies a position of an image region based on the image signal; a first output terminal that outputs the image signal to a second signal processing unit included in an external device; A second output terminal for outputting, to the second signal processing unit, output region information relating to the position of the image region specified by the first signal processing unit for processing in the second signal processing unit ; to have a first input terminal for inputting a setting data for said first signal processing unit.

  Since the processing of the second signal processing unit and the processing of the first signal processing unit can be performed in parallel, the overall processing time can be shortened.

It is a block diagram which shows the structure of the imaging device which concerns on 1st Embodiment. It is a block diagram which shows the structure of the lamination | stacking sensor which concerns on 1st Embodiment. It is a figure which shows the structure of the pixel part of the lamination | stacking sensor which concerns on 1st Embodiment. It is a figure which shows the flow of the image processing which concerns on 1st Embodiment. It is a figure explaining the extraction method of the specular reflection component which concerns on 1st Embodiment. It is a block diagram which shows the structure of the imaging device which concerns on 2nd Embodiment. It is a figure which shows the flow of the image processing which concerns on 2nd Embodiment. It is a figure which shows the flow of the image processing which concerns on 2nd Embodiment.

(First embodiment)
The imaging apparatus according to the first embodiment of the present invention will be described below with reference to FIGS. As the first embodiment, an imaging apparatus that performs white balance correction by performing light source estimation based on a result of extracting a specular reflection component from an image signal will be described as an example.

  FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to the first embodiment of the present invention. The imaging apparatus 100 includes an imaging optical system 101, a stacked sensor 110, an image processing logic unit 120, a display unit 132, an operation unit 133, and an external memory 134. Reference numeral 110 denotes a stacked sensor in which an image sensor 111 that converts an optical image into an image signal and an image processing unit that will be described later are configured in the same package. An image processing logic unit 120 performs image processing on an image signal output from the stacked sensor 110. 131 is a recording medium for recording image data, 132 is a display unit for presenting an image to the user, 133 is an operation unit for receiving an operation from the user, and 134 is an external memory. S101 is a captured image signal, S102 is specular reflection candidate area information, S103 is camera signal processing setting data, S104 is a recording image signal, S105 is a display image signal, S106 is a user instruction input signal, and S107 is external memory R / W information. is there.

  The laminated sensor 110 has the following components. Reference numeral 111 denotes an image sensor that converts an optical image into an image signal, and reference numeral 112 denotes a captured image output unit. Reference numeral 113 denotes a specular reflection candidate region extraction unit (first image processing unit) that extracts a region that may contain a specular reflection component from the image signal. 114 is a specular reflection candidate area information output unit, and 115 is a control signal input unit. S111 is a sensor readout signal a, S112 is a sensor readout signal b, S113 is specular reflection candidate area information, S114 is a sensor drive setting signal, and S115 is a specular reflection extraction setting signal.

  The image processing logic unit 120 includes the following components. 121 is a captured image input unit, 122 is a development processing unit (second image processing unit), 123 is a specular reflection candidate area information input unit, 124 is a specular reflection extraction unit, 125 is a control signal output unit, and 126 is a system control unit. It is. S121 is a development processing input image signal, S122 is specular reflection candidate area information, S123 is camera signal processing setting information, and S124 is white balance gain information.

  FIG. 2 is a block diagram illustrating a configuration example of the stacked sensor 110. 202a and 202b are a row scanning circuit (control unit) a and a row scanning circuit (control unit) b for selecting image signals to be output to the captured image output unit 112 and the specular reflection candidate area extraction unit 113, respectively. 203a and 203b are column ADCs (analog-digital converters) that convert analog signals into digital signals. A column operation circuit 204 scans each column.

  The image sensor 111 has a plurality of pixels 211 arranged in a two-dimensional matrix. Reference numeral 212a denotes a transfer signal line a, and reference numeral 213a denotes a reset signal line a. A row selection signal line a 214a reads an image signal from the pixel 211 under the control of the row scanning circuit 202a. The read image signal is output to the column signal line 215a as described later. 212b is a transfer signal line b, and 213b is a reset signal line b. A row selection signal line b 214b reads an image signal from the pixel 211 in accordance with control by the row scanning circuit 202b. The read image signal is output to the column signal line 215b as will be described later.

  A column signal line a 215a outputs a sensor read signal S111 to the captured image output unit 112 via the column ADC 203a. A column signal line b 215b outputs a sensor read signal S112 to the specular reflection candidate area extraction unit 113 via the column ADC 203b. As illustrated, the transfer signal line 212a, the reset signal line 213a, the row selection signal line 214a, and the column signal line 215a are connected to all the pixels 211. On the other hand, the transfer signal line 212b, the reset signal line 213b, the row selection signal line 214b, and the column signal line 215b are connected to only some of the pixels 211. That is, the sensor readout signal S112 output to the specular reflection candidate area extraction unit 113 is thinned out with respect to the sensor readout signal S111 output to the captured image output unit 112. For example, the sensor readout signal S111 is horizontal 4096 pixels and vertical 2160 pixels. The sensor readout signal b (S112) is thinned out to horizontal 400 pixels and vertical 216 pixels.

  The row scanning circuits 202a and 202b, the image sensor 111, the column ADCs 203a and 203b, the specular reflection candidate area extraction unit 113, the column operation circuit 204, the specular reflection candidate area information output unit 114, and the captured image output unit 112 are included in the same semiconductor chip. Formed. The image sensor 111 and the specular reflection candidate area extraction unit 113 are arranged in a stacked manner. That is, the image sensor 111 is disposed on the light incident side, and the specular reflection candidate area extraction unit 113 is disposed on the back side thereof, and is electrically connected to each other. Thereby, the speed at which the image sensor 111 outputs the image signal to the specular reflection candidate area extraction unit 113 can be higher than the output speed of the image sensor 111 to the development processing unit 122.

  FIG. 3 is a circuit diagram illustrating a configuration example of the pixel 211. Reference numeral 301 denotes a photodiode. Reference numeral 302 denotes a transfer transistor. Reference numeral 303 denotes an FD (floating diffusion). Reference numeral 304 denotes a reset transistor. Reference numeral 305 denotes an amplification transistor. Reference numerals 306a and 306b denote selection transistors. The photodiode 301 is a photoelectric conversion unit in which an anode is grounded and a cathode is connected to the source of the transfer transistor 302 to convert light into electric charges. Transfer signal lines 212a and 212b are connected to the gate of the transfer transistor. Therefore, when the transfer transistor 302 is turned on by the transfer signal line 212a or 212b, the charge stored in the photodiode 301 is transferred to the FD 303 and stored in the FD 303. The amplification transistor 305 has a drain to which the power supply voltage Vdd is applied and a gate connected to the FD 303. The amplification transistor 305 amplifies the charge accumulated in the FD 303 and converts it into a voltage signal.

  The selection transistor 306a is disposed between the source of the amplification transistor 305 and the column signal line 215a, and its gate is connected to the row selection signal line 214a. Accordingly, when the transfer transistor 306a is turned on by the row selection signal line 214a, a voltage signal corresponding to the voltage of the FD 303 is output to the column signal line 215a.

  On the other hand, the selection transistor 306b is disposed between the source of the amplification transistor 305 and the column signal line 215b, and the gate thereof is connected to the row selection signal line 214b. Therefore, when the transfer transistor 306b is turned on by the row selection signal line 214b, a voltage signal corresponding to the voltage of the FD 303 is output to the column signal line 215b.

  The reset transistor 304 has a gate connected to the reset signal lines 214a and 214b, a drain connected to the node of the power supply voltage Vdd, and a source connected to the FD 303. Accordingly, when the reset transistor 304 is turned on by the reset signal line 214a or 214b, the voltage of the FD 303 is reset to the power supply voltage Vdd. The sensor readout signal S111 is read out by the scanning control by the row scanning circuit 202a, and the sensor readout signal S112 is read out by the scanning control by the row scanning circuit 202b.

  Next, with reference to FIG. 4, the reading of an image signal in the imaging apparatus 100 and the flow of image processing for the read image signal will be described. FIG. 4 is a diagram for explaining the operation timing of the imaging apparatus 100 when a moving image signal is read out with a driving cycle of 1/30 seconds and image processing is performed.

  The sensor read signal a (S111) read by the image sensor 111 is output to the outside of the stacked sensor 110 by the captured image output unit 112 as a captured image signal S101. The captured image signal S101 is captured by the captured image input unit 121 and supplied to the development processing unit 122 as the development processing input image signal S121.

  As shown in FIG. 4, reading of the first frame image signal (frame 1) of the sensor reading signal a (S111) starts at time t10 and ends at time t20. Thereafter, from time t20 to t30, the next frame image signal (frame 2) is read, and thereafter periodically read.

  The development processing unit 122 is a second image processing unit, and executes various correction processes (second image processing) such as gamma correction and color correction on the development process input image signal S121. Further, the development processing unit 122 performs white balance correction (second image processing) according to the white balance gain calculated by a method described later. Image processing in the development processing unit 122 is performed using the external memory 134 via the external memory R / W information S107, and is performed in a period from time t30 to t31 in FIG.

  That is, the image signal corresponding to the first frame of the moving image read during the period from time t10 to time t20 is output from the development processing unit 122 at time t31 delayed by two frames, and the image data on the recording medium 131 is output. It is used for recording and displaying an image on the display unit 132.

  On the other hand, the sensor readout signal b (S112) is output to the specular reflection candidate region extraction unit 113 at times t10 to t11. As illustrated, reading of the sensor read signal b (S112) is completed in a shorter time than reading of the sensor read signal a (S111). This is because the sensor readout signal b (S112) has a thinned number of pixels, and the image sensor 111 and the specular reflection candidate area extraction unit 113 are connected by a stacked structure.

  Next, the specular reflection candidate area extraction unit 113 is a first image processing unit, and extracts a specular reflection candidate area that may cause specular reflection on the subject surface from the sensor readout signal b (S112). The first image processing is performed. A specific method will be described with reference to FIGS. FIGS. 5A and 5B are diagrams illustrating a state in which the image signal is divided into blocks of an appropriate size. In FIGS. 5A and 5B, the block is divided into 8 × 8 = 64 blocks. Further, it is assumed that the shaded area on the subject surface represents an area where specular reflection occurs. The specular reflection candidate area extraction unit 113 extracts a block that may include an area where specular reflection occurs from the divided blocks, and outputs the position information as specular reflection candidate area information S113. It is considered that the luminance of the image signal is higher in the region where the specular reflection occurs than in the region where only the surrounding diffuse reflection occurs. For this reason, the specular reflection candidate area extraction unit 113 calculates the average luminance in the block for each block, and searches for and extracts blocks having a higher average luminance than the surrounding blocks. There may be a plurality of blocks that are determined to have a high possibility of occurrence of specular reflection under this condition. In this case, a plurality of pieces of information are output as the specular reflection candidate area information S113. In FIG. 5 (b), the extracted blocks are represented by diagonal lines. The extraction process of the specular reflection candidate area is performed in a period from time t11 to t12.

  The specular reflection candidate area information S113 is output by the specular reflection candidate area information output unit 114 to the specular reflection candidate area information input unit 123 of the image processing block unit 120.

  Next, processing of the specular reflection extraction unit 124 will be described. The specular reflection extracting unit 124 estimates the light source color by extracting only the reflection component corresponding to the specular reflection from the development processing input image signal S121 input via the captured image input unit 121. However, the processing time is shortened by using only the region corresponding to the specular reflection candidate region information S122 in the development processing input image signal S121.

  The processing content will be described with reference to FIGS. Similar to the specular reflection candidate area extraction unit 113, the specular reflection extraction unit 124 divides the image signal into a plurality of blocks. Here, the number of divisions and the division position are set to be the same as those of the specular reflection candidate area extraction unit 113, and in this embodiment, the division is made into 8 × 8 blocks. The specular reflection extraction unit 124 generates a luminance histogram for the pixels in the block included in the specular reflection candidate area information S122, and extracts specular reflection components. FIG. 5C shows an example of a brightness histogram generated by the specular reflection extraction unit 124. 5C represents the luminance signal Y of each pixel, and the horizontal axis of FIG. 5C represents the frequency representing the number of pixels included in each luminance signal Y section. As described above, since the region where the specular reflection occurs is considered to have a higher luminance than the region where the reflection does not occur, the pixels included in the interval where the luminance signal Y is high in the luminance histogram of FIG. Is considered to contain a lot of specular reflection components. Therefore, in the luminance histogram, the specular reflection signal is calculated by calculating the difference in pixel value between the pixel included in the interval corresponding to the high luminance signal and the pixel included in the interval corresponding to the lower luminance signal. It can be extracted. Since the specular reflection component extracted here represents the color of the light source in the shooting environment, the specular reflection extraction unit 124 may calculate a white balance gain suitable for the shooting environment based on the extracted specular reflection component. it can. The specular reflection extraction unit 124 outputs the calculated white balance gain to the development processing unit 122. The above-described processing is executed in the period from time t20 to t21 in the sequence of FIG. At times t21 to t22, the system control unit 126 updates the white balance gain information S124 for the development processing unit 122. As described above, at time t30 to t31 in FIG. 4, the development processing unit 122 can perform image processing including white balance correction.

  As described above, the imaging apparatus 100 according to the present embodiment includes the laminated sensor 110 in which the image sensor 111 and the image processing unit are integrated. The specular reflection candidate region extraction unit (image processing unit) 113 inside the multilayer sensor 110 extracts a region that is highly likely to cause specular reflection, and then the specular reflection extraction unit (image processing unit) outside the multilayer sensor 110. ) 124, the specular reflection component is extracted with higher accuracy.

  According to the present embodiment, the sensor readout signal is from the start to the completion of the readout of the sensor readout signal b in FIG. 4 and the subsequent extraction processing of the region where the specular reflection based on the luminance signal for each block is highly likely to occur. It is completed between the start of reading of a and the completion thereof. When not according to the present embodiment, first, a thinned image corresponding to the sensor readout signal b must be generated based on the sensor readout signal a, and then an extraction process using the luminance value for each block. This requires a lot of processing time. On the other hand, according to the present embodiment, the above time can be shortened.

  In this embodiment, an area having a high possibility of including a specular reflection component is detected using the luminance information of the block unit, and the specular reflection component is extracted using the luminance histogram of the pixel unit. The detection and extraction processing method is not limited to this. For example, a partial area whose luminance value is higher than the luminance value of the surrounding area may be detected from the read image signal as an area having a high possibility of including a specular reflection component. Alternatively, the specular reflection component may be extracted by calculating a pixel value difference in units of pixels and extracting only a difference value corresponding to a representative light source color stored in advance from the calculated difference value. . That is, any method may be used as long as it is processing for detecting a region having a high possibility of including specular reflection and processing for extracting a specular reflection component based on an image signal.

  Further, in the present embodiment, a case has been described in which position information of an area that is likely to include a specular reflection component is output from the laminated sensor 110, but the output content from the laminated sensor 110 is not limited to this. For example, you may output the information showing whether the area | region with high possibility of containing a specular reflection component exists. In this case, when there is a region that is highly likely to include a specular reflection component, the development processing unit 122 performs white balance processing based on detection of specular reflection. When the area does not exist, the development processing unit 122 performs white balance processing based on the detection of the achromatic area of the image signal. As a result, when the white balance processing based on detection of specular reflection cannot perform appropriate white balance correction, unnecessary processing is not performed, and the entire processing time can be shortened.

  Further, as an output result from the laminated sensor 110, an image signal of an area having a high possibility of including a specular reflection component may be output. In this case, the specular reflection extracting unit 124 may generate a luminance histogram for the output partial region image signal and extract the specular reflection component. Accordingly, the specular reflection extraction unit 124 can calculate the white balance gain without waiting for the completion of reading of the high-resolution image signal to the development processing unit 122, thereby reducing the overall processing time. Is possible.

  That is, the output content from the laminated sensor 110 may be in any format as long as it represents information indicating the result of detecting a region that is likely to include a specular reflection component. In this embodiment, the case where the laminated sensor 110 in which the image sensor 111 and the specular reflection candidate region extraction unit (image processing unit) 113 are connected by a laminated structure is used has been described. It is not limited. That is, any sensor may be used as long as the pixel 211, the row scanning circuits 202a and 202b, and the specular reflection candidate region extraction unit 113 are configured as an integrated package.

  In the present embodiment, the case of the imaging apparatus 100 including the stacked sensor 110 and the image processing logic unit 120 has been described. However, the present invention is not limited to the imaging apparatus. For example, an image processing system including an imaging device including a stacked sensor 110, a specular reflection candidate region extraction unit 113, and a recording unit, and an image processing device including a recorded signal reading unit and an image processing logic unit 120 is used. It doesn't matter. In this case, the recording unit of the imaging apparatus may record the information related to the image signal output from the image sensor 111 and the image region that is highly likely to include a specular reflection component. By doing in this way, the image processing logic unit 120 can shorten the entire processing time even when image processing is performed outside the imaging apparatus.

(Second Embodiment)
Next, as a second embodiment of the present invention, an imaging apparatus that detects a face from an image signal and controls image processing based on the detection result will be described as an example. FIG. 6 is a block diagram illustrating a configuration example of an imaging apparatus according to the second embodiment of the present invention. The same components as those in the first embodiment shown in FIG. Hereinafter, the points of the present embodiment different from the first embodiment will be described.

  S601 is face candidate coordinate information. S602 is face candidate image information. 610 is a face candidate detection unit, 611 is a face candidate normalization unit, 612 is a face candidate image memory, 613 is a face candidate image output unit, 614 is a face candidate coordinate memory, 615 is a face candidate coordinate output unit, and 616 is a face template image. It is memory. S610 is individual face candidate image information, S611 is normalized face candidate image information, S612 is face candidate image information, S613 is face template image information, S614 is face template setting information, S615 is individual face candidate coordinate information, and S616 is a face candidate. Coordinate information.

  The image processing logic unit 120 includes the following components. 621 is a face candidate image input unit, 622 is a face determination unit, 623 is a face candidate coordinate input unit, and 624 is an eigenvector memory. S621 is face candidate image information, S622 is face candidate coordinate information, S623 is eigenvector information, and S624 is face determination result information.

  Since the image sensor 111 has the configuration described in the first embodiment, detailed description thereof is omitted. Note that, as in the case of the first embodiment, the sensor readout signal a (S111) is an image signal of all the pixels of the image sensor 111. The sensor readout signal b (S112) is an image signal that is thinned out for all pixels of the image sensor 111. For example, the sensor readout signal a (S111) is 4096 horizontal pixels and 2160 vertical pixels. The sensor readout signal b (S112) is thinned out to horizontal 400 pixels and vertical 216 pixels.

  Next, the operation timing of the imaging apparatus according to the second embodiment of the present invention will be described with reference to FIGS. Note that FIG. 7 is a diagram showing the operation timing at the time of moving image shooting operation at a 1/30 second cycle, and FIG.

  The sensor readout signal a (S111) is output as a captured image signal S101 by the captured image output circuit 112 to the outside of the stacked sensor 110. The captured image signal S101 is taken into the image processing logic unit 120 by the captured image input circuit 121 and supplied to the development processing unit 122 as the development processing input image signal S121. As shown in FIG. 7, the sensor read signal a (S111) starts reading at time t70, ends reading at time t80, and then periodically outputs moving images at times t90 and t100.

  Based on the face determination result information S624, the development processing unit 122 executes image processing including face area image processing such as skin color correction processing for the face area of the subject and noise suppression processing specialized for the face area of the subject. . The image processing is executed using the external memory 134 via the external memory R / W information S107. The display image signal S105 and the recording image signal S104 are output as the output of the development processing unit 122 at times t90 to t91 in FIG. That is, the output of the development processing unit 122 for the image of the first frame of the sensor readout signal a (S111) read out at times t70 to t80 is delayed by two frames from time t90 to t91. Pointer information indicating the face area of the subject is superimposed on the display image signal S105 and displayed on the display unit 132. The recorded image signal S104 is recorded on the recording medium 131.

  On the other hand, the sensor readout signal b (S112) is input to the face candidate detection unit 610. Since the sensor readout signal b (S112) is thinned out to horizontal 400 pixels and vertical 216 pixels, readout is completed at times t70 to t71 in FIG.

  In the face template image memory 616, face template images are recorded in advance in a plurality of face sizes. Since the size of a face as a subject varies depending on the situation at the time of shooting, it is desirable to record face template images in a plurality of sizes. For example, four types of horizontal, vertical pixels of 160 pixels, 80 pixels, 40 pixels, and 20 pixels are recorded, and a face template image of any face size is converted into a face template image according to face template setting information S614. It is possible to specify whether to read as information S613.

  The face candidate detection unit 610 uses the template image information S613 of the face with the specified face size, and the template image information S613 of the face with the specified face size for all the angles of view of the sensor read signal b (S112). Search for partial areas highly correlated with. That is, the face candidate detection unit 610 extracts a partial area image having the same size as the template image information S613 of the face having the designated face size from the sensor read signal b (S112) while shifting the partial area. Then, the face candidate detection unit 610 determines whether or not there is a high possibility that a face exists in the partial area by comparing the correlation coefficient or the difference between the partial area and the template image information S613 of the face of the designated face size. , Rough correlation evaluation. As described above, since the size of a face as a subject varies depending on the situation at the time of shooting, it is desirable to perform a search process for all the template images of a plurality of sizes recorded in advance.

  Therefore, as shown in FIG. 7, at times t71 to t72, the face candidate detection unit 610 performs a search process on the face template image information S613 having 160 horizontal and vertical pixels. At times t72 to t73, the face candidate detection unit 610 performs a search process on the face template image information S613 in which the horizontal and vertical pixels are 80 pixels each. From time t73 to t74, the face candidate detection unit 610 performs a search process on the face template image information S613 in which the horizontal and vertical pixels are 40 pixels each. From time t74 to t75, the face candidate detection unit 610 performs search processing on the face template image information S613 in which the horizontal and vertical pixels are 20 pixels each.

  If it is determined by the above search processing that there is a high possibility that a face exists, the face candidate detection unit 610 determines the face from the partial region information compared to the face template image information S613 of the designated face size. The individual face candidate coordinate information S615 including the position and size information is generated. Then, the face candidate detection unit 610 stores the individual face candidate coordinate information S615 in the face candidate coordinate memory 614. Since there may be a plurality of partial regions that are determined to have a high possibility that a face exists, there may be a plurality of individual face candidate coordinate information S615.

  Further, the face candidate detection unit 610 cuts out the partial region image compared with the face template image information S613 of the designated face size, generates individual face candidate image information S610, and outputs it to the face candidate normalization unit 611. Since the individual face candidate image information S610 has a plurality of image sizes in conjunction with the face template image information S613 of the designated face size, the face candidate normalization unit 611 normalizes the images to a common image size. Process. For example, the face candidate normalization unit 611 performs a reduction process so that the horizontal and vertical pixels are each 20 pixels in size. In this way, normalized face candidate image information S611 is generated and stored in the face candidate image memory 612 as a face candidate image in which horizontal and vertical pixels are unified to a size of 20 pixels.

  The normalization processing of the face candidate normalization unit 611 is included in times t71 to t72, t72 to t73, t73 to t74, and t74 to t75 in FIG. As shown in FIG. 7, from the start of reading of the sensor readout signal a (S111), from the start of reading out of the sensor readout signal a (S111) to the completion of the template matching processing using the template images of the four face sizes. Complete before completion.

  Next, the face candidate coordinate memory 614 and the face candidate image memory 612 output the recorded face candidate coordinate information S616 and normalized face candidate image information S612. The face candidate coordinate information S616 and the normalized face candidate image information S612 are transmitted from the face candidate coordinate output unit 615 and the face candidate image output unit 613 to the outside of the layer sensor 110 at time t80 in FIG. Output as face candidate image information S601. The face candidate coordinate information S602 and the face candidate image information S601 are input to the image processing logic unit 120 by the face candidate coordinate input unit 623 and the face candidate image input unit 621, respectively.

  The face candidate coordinate input unit 623 outputs the face candidate coordinate information S602 to the face determination unit 622 as face candidate coordinate information S622. Further, the face candidate image input unit 621 outputs the face candidate image information S601 as face candidate image information S621 to the face determination unit 622. As described above, the face candidate image information S621 is only roughly evaluated as having a high possibility of being a face based on the correlation coefficient or difference of the partial area images having the same size as the face template image information S613. Therefore, the accuracy rate is not so high.

  In order to further increase the accuracy rate, the eigenvector memory 624 records face eigenvector information S623 previously extracted by principal component analysis as data that accurately represents the feature amount of the face image. The face determination unit 622 compares the face candidate image information S621 with face eigenvector information S623 to determine whether or not the face is finally a face. The face eigenvector information S623 is recorded in advance in the eigenvector memory 624, read out as eigenvector information S623, and input to the face determination unit 622. As described above, the face determination unit 622 performs face determination processing by principal component analysis, finally determines whether or not the face is a face, and develops face determination result information S624 as a result of face determination. To the unit 122.

  At times t80 to t81 in FIG. 7, the face determination unit 622 performs final face determination processing on the face candidate image information S621. At times t81 to t82 in FIG. 7, the setting parameters relating to various correction processes controlled based on the face determination result information S624 in the development processing unit 122 are updated. As described above, the development processing unit 122 updates the setting parameter at the time t90 to time t91 in FIG. 7 so that the development processing unit 122 performs skin color correction processing on the face area of the subject or noise specialized for the face area of the subject. Image processing including face area image processing such as suppression processing is performed.

  In the present embodiment, the process from the start of reading of the sensor read signal a (S112) to the completion of the template matching process using the template images of faces of four types of face sizes is completed from the start of reading of the sensor read signal a (S111). Has been completed.

  When not according to the present embodiment, first, a thinned image corresponding to the sensor readout signal b (S112) must be generated based on the sensor readout signal a (S111). Furthermore, since it is necessary to perform template matching processing using template images of faces of four types of face sizes after that, it takes a lot of processing time. Therefore, according to the present embodiment, the above time is greatly shortened.

  Next, a case where the control parameter for the imaging apparatus 100 according to the present embodiment is changed by the user will be described. Here, the user instructs via the operation unit 133 to set the frame rate at the time of moving image shooting from a 1/30 second period (FIG. 7) to a double frame rate of 1/60 second period (FIG. 8). Assume that you have entered it.

  The system control unit 126 gives a 1/60 second period setting to the camera signal processing setting information S123, and outputs the camera signal processing setting information S103 to the outside of the image processing logic unit 120 through the control signal output unit 125. The control signal input unit 115 of the stacked sensor 110 receives the camera signal processing setting information S103, and outputs a face template setting signal S617 for the face candidate detection unit 610 and a sensor drive setting signal S114 for the image sensor 111. The sensor drive setting signal S114 is a setting signal for changing the thinning setting of the sensor readout signal b (S112) generated in the image sensor 111. Further, the face template setting signal S617 is a setting signal for changing which of the above-described four face size face template images is to be subjected to the template matching process with a lower face size.

  In this embodiment, even when the frame rate is set to a 1/60 second cycle, the reading of the sensor readout signal b (S112) is completed from the start of reading, and the faces of four types of face sizes that follow. The period until the template matching process using the template image is shortened. During this period, the following changes are made to complete the period from the start to the end of reading of the sensor read signal a (S111).

  That is, first, by the sensor drive setting signal S114, the thinning setting of the sensor readout signal b (S112) is set such that the horizontal 400 pixels and the vertical 216 pixels are further thinned to the horizontal 320 pixels and the vertical 180 pixels. To do. As a result, the time from the start to the end of reading of the sensor read signal b (S112) from time t120 to time t121 in FIG. 8 is further shortened.

  Further, by setting the face template setting signal S617, the image size of the face template used for the template matching processing is a face template image with 160 pixels for horizontal and vertical pixels and a face template image with 80 pixels for horizontal and vertical pixels. Limit with only. At times t121 to t122 in FIG. 8, the face candidate detection unit 610 performs a search process on a face template image having 160 horizontal and vertical pixels. At times t122 to t123, the face candidate detection unit 610 performs a search process on the face template image whose horizontal and vertical are each 80 pixels, and ends the template matching process.

  Thereby, the template matching processing time of the face candidate detection unit 610 is shortened to ½. Furthermore, time t120 to t121 in FIG. 8 of the time from the start to the end of reading of the sensor read signal b (S112) is shortened. Completion from the start of reading of the sensor read signal a (S111) is doubled to 1/60 second. Even in that case, the completion of the sensor readout signal b (S112) from the start to the completion and the subsequent template matching process can be completed from the start of the readout of the sensor readout signal a (S111) to the completion.

  As described above, according to the present embodiment, a search process for a partial region having a high correlation with a face template under a predetermined condition is performed on all pixel regions of the image sensor 111. Can be completed in parallel during the period. Furthermore, since the face determination process, which is a detailed determination process in the image processing logic unit 120, can be performed immediately thereafter, the face detection process in the subject can be executed more efficiently.

  Further, according to the present embodiment, even when the readout time of all the pixels of the image sensor 111 is changed, the search process of the partial region having a high correlation with the face template is completed within the readout time of all the pixels of the image sensor 111. It is possible to easily perform the control.

  In the first and second embodiments, the row scanning circuit (control unit) 202a sends a signal based on the charge of the photodiode 301 to the column signal line (first signal line) by the transfer signal line (first control line) 212a. ) Output to 215a. The row scanning circuit (control unit) 202b also outputs a signal based on the charge of the photodiode 301 to the column signal line (second signal line) 215b through the transfer signal line (second control line) 212b. The specular reflection candidate area extraction unit 113 and the face candidate detection unit 610 (and the face candidate normalization unit 611) are first image processing units, and perform first image processing on the signal output to the column signal line 215b. I do. The development processing unit 122 is a second image processing unit, and based on the result of the first image processing of the specular reflection candidate region extraction unit 113 or the face candidate detection unit 610 (and the face candidate normalization unit 611), Second image processing is performed on the signal output to the signal line 215a.

  The row scanning circuit 202a causes the development processing unit 122 to output a signal based on the charges of the photodiodes 301 of all the pixels 211 of the plurality of pixels 211 through the transfer signal line 212a. The row scanning circuit 202b causes the specular reflection candidate area extraction unit 113 or the face candidate detection unit 610 to output a signal based on the charges of the photodiodes 301 of some of the plurality of pixels 211 through the transfer signal line 212b. The specular reflection candidate region extraction unit 113 or the face candidate detection unit 610 (and the face candidate normalization unit 611) performs first image processing on a signal based on the charges of the photodiodes 301 of some pixels 211. The development processing unit 122 performs second image processing on a signal based on the charges of the photodiodes 301 of all the pixels 211. The number of pixels 211 that the row scanning circuit 202b outputs to the column signal line 215b through the transfer signal line 212b is smaller than the number of pixels 211 that the row scanning circuit 202a outputs to the column signal line 215a through the transfer signal line 212a.

  The specular reflection candidate area extraction unit 113 or the face candidate detection unit 610 (and the face candidate normalization unit 611) does not have the first time until the development processing unit 122 inputs all the signals of the pixels 211 output to the column signal line 215a. 1 image processing is completed.

  A first image processing unit such as the specular reflection candidate region extraction unit 113 or the face candidate detection unit 610 (and the face candidate normalization unit 611) detects a predetermined image region. Specifically, the first image processing unit detects an image region including a specular reflection component on the surface of the subject, an image region including the face of the subject person, or an image region including a specific subject. The first image processing unit outputs presence / absence of a predetermined image area, a predetermined number of image areas, and position information of the predetermined image area, or outputs an image signal of the predetermined image area. The development processing unit 122 performs second image processing on the image area detected by the first image processing unit.

  The image processing system includes an imaging device and an image processing device. The imaging apparatus includes an image sensor 111, row scanning circuits 202a and 202b, a first image processing unit, and a recording unit. The first image processing unit is the specular reflection candidate region extraction unit 113 or the face candidate detection unit 610 (and the face candidate normalization unit 611). The image processing apparatus includes a development processing unit 122. The recording unit records the signal of the column signal line 215a and the result of the first image processing of the first image processing unit. The development processing unit 122 performs second image processing on the signal of the column signal line 215a recorded in the recording unit, based on the result of the first image processing of the first image processing unit recorded in the recording unit. I do.

  According to the first and second embodiments, the development processing unit 122 performs the second image processing on the signal of the column signal line 215a based on the result of the first image processing of the first image processing unit. Do. In particular, the detection processing is performed in two stages, that is, a rough and high-speed detection process of the specular reflection candidate region extraction unit 113 or the face candidate detection unit 610 and a high-accuracy and high-volume detection process of the mirror reflection extraction unit 124 or the face determination unit 622 I do. The development processing unit 122 can shorten the time lag caused by waiting for the completion of the detection process, and can shorten the entire processing time.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 Imaging device, 110 Stack sensor, 111 Image sensor, 113 Specular reflection candidate area extraction part, 120 Image processing logic part, 122 Development processing part

Claims (28)

An image sensor in which an image sensor layer and a signal processing layer are stacked,
A plurality of pixels included in the image sensor layer for converting a light image into an image signal;
A first signal processing unit that is included in the signal processing layer and identifies a position of an image region based on the image signal;
A first output terminal for outputting the image signal to a second signal processing unit included in an external device;
A second output terminal for outputting, to the second signal processing unit, output region information relating to the position of the image region specified by the first signal processing unit for processing in the second signal processing unit ; ,
Image sensor characterized by have a first input terminal for inputting a setting data for said first signal processing unit. The image sensor according to claim 1, wherein the signal processing layer includes the first output terminal, the second output terminal, and the first input terminal. A first signal line connected to the plurality of pixels to output the image signal to the first output terminal;
The image sensor according to claim 1, further comprising a second signal line connected to the plurality of pixels in order to output the image signal to the first signal processing unit. The first signal processing unit completes a specifying process for specifying the position of the image area before the first output terminal outputs all of the image signals of the plurality of pixels. The image sensor according to any one of claims 1 to 3. The first signal processing unit, and characterized by identifying the position of the image region including the position of the image area including the specular reflection component, the position of the image area including a face of a subject person, or a particular subject in the subject surface The image sensor according to any one of claims 1 to 4.   The image sensor according to claim 1, wherein the first signal processing unit performs template matching processing of the image signal. The image sensor according to claim 1, wherein the first signal processing unit specifies a position of the image region based on luminance of the image signal. The image sensor according to claim 1, wherein the first signal processing unit specifies positions of the plurality of image regions based on the image signal. A first control unit and a second control unit for selecting image signals output from the plurality of pixels;
The image sensor according to claim 1, wherein the second control unit is different from the first control unit.   The image sensor according to claim 3, further comprising a selection switch that selects whether to output the image signal to the first signal line or the second signal line.   The data amount of the image signal processed by the first signal processing unit is smaller than the data amount of the image signal processed by the second signal processing unit included in an external device. The image sensor according to any one of 1 to 10. A first memory in which a plurality of setting data is recorded in advance;
12. The first signal processing unit according to claim 1, wherein the first signal processing unit sequentially acquires a plurality of parameters from the first memory in order to specify a position of the image region. Image sensor. The image sensor according to claim 12, further comprising a second memory in which the identification result of the first signal processing unit is recorded.   The image sensor according to claim 12, wherein the amount of the plurality of parameters acquired from the first memory is limited based on a frame rate. An image sensor in which an image sensor layer and a signal processing layer are stacked,
A plurality of pixels included in the image sensor layer for converting a light image into an image signal;
A first signal processing unit that is included in the signal processing layer and detects an image area based on a luminance level of the image signal;
A first output terminal for outputting the image signal to a second signal processing unit included in an external device;
A second output terminal that outputs, to the second signal processing unit, output region information relating to the image region detected by the first signal processing unit for processing in the second signal processing unit ;
Image sensor characterized by have a first input terminal for inputting a setting data for detecting the image region. The image sensor according to claim 15 , wherein the signal processing layer includes the first output terminal, the second output terminal, and the first input terminal. The first signal processing unit, wherein said first output terminal before outputting all of said image signals of said plurality of pixels, thereby completing a detection process for detecting the front Kiga image area The image sensor according to claim 15 or 16 . The first signal processing unit, an image area including a specular reflection component in the surface of the object, claim 15 to 17 and detecting an image region including an image region including the face of a subject person, or a particular subject The image sensor according to any one of 1. The image sensor according to any one of claims 15 to 18 , wherein the first signal processing unit performs a template matching process on the image signal. The first signal processing unit includes an image sensor according to any one of claims 15 to 19, wherein the detecting a plurality of pre-outs image region based on the height of the luminance of the image signal . A first control unit and a second control unit for selecting image signals output from the plurality of pixels;
Said second control unit, an image sensor according to any one of claims 15 to 20, wherein different from said first control unit. A first signal line connected to the plurality of pixels to output the image signal to the first output terminal;
A second signal line connected to the plurality of pixels for outputting the image signal to the first signal processing unit;
The selection switch according to any one of claims 15 to 21, further comprising a selection switch for selecting whether to output the image signal to the first signal line or the second signal line. Image sensor. The data amount of the image signal processed by the first signal processing unit is smaller than the data amount of the image signal processed by the second signal processing unit included in an external device. The image sensor according to any one of 15 to 22 . A first memory in which a plurality of setting data is recorded in advance;
The first signal processing unit, pre-outs in order to detect the image area, according to any one of claims 15 to 23, characterized in that sequentially acquires a plurality of parameters from the first memory Image sensor. The image sensor according to claim 24 , further comprising a second memory in which a detection result of the first signal processing unit is recorded. The image sensor according to claim 24 , wherein the amount of the plurality of parameters acquired from the first memory is limited based on a frame rate. An image sensor control method in which an image sensor layer and a signal processing layer are stacked,
Converting a light image into an image signal by a plurality of pixels included in the image sensor layer ;
Identifying a position of an image region based on the image signal by a first signal processing unit included in the signal processing layer ;
Outputting the image signal to a second signal processing unit included in the external device by a first output terminal;
Output region information related to the position of the image region specified by the first signal processing unit is output to the second signal processing unit by the second output terminal for processing in the second signal processing unit. And steps to
The first input terminal, the image sensor control method, characterized by chromatic and a step of inputting setting data for said first signal processing unit. An image sensor control method in which an image sensor layer and a signal processing layer are stacked,
Converting a light image into an image signal by a plurality of pixels included in the image sensor layer ;
A step of detecting an image region based on a luminance level of the image signal by a first signal processing unit included in the signal processing layer ;
Outputting the image signal to a second signal processing unit included in the external device by a first output terminal;
A step of outputting output region information related to the image region detected by the first signal processing unit to the second signal processing unit for processing in the second signal processing unit by a second output terminal; When,
The first input terminal, a control method of the image sensor, characterized by chromatic and a step of inputting setting data for detecting the image region.

Images