JP2000152263A - Solid-state image pickup device and image data generating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a solid-state image pickup device where requirements of enhancement of image quality and high processing speed that are trade-off requirements can be satisfied and to provide a pixel data generating method. SOLUTION: A light receiving element of an image pickup section 18 placed corresponding to a color filter of a color separation filter CF of a digital still camera 10 receives a light, an A/D converter section 20 converts an obtained image pickup output into image digital data and a buffer memory 22 stores the image data. Then an image selection section 28 selects an image read from the buffer memory 22, The buffer memory 22 outputs the selected image data to a signal processing section 24. The signal processing section 24 applies signal processing in response to the mode with priority among settings of resolution of an image or a read speed of the image under the control of a system control section 16. The processed signal is especially fed to a matrix section 24b via the buffer memory 22 and an output destination changeover section 240a by the processing of a display signal processing section 242a. In the case of placing priority of the image read speed, image data that are grouped as they are outputted without decreasing the number of pixels.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION The present invention relates to a light-receiving element which is arranged such that the center of the geometrical shape of each light-receiving element is half the pitch of the light-receiving elements in the row direction and / or the column direction with respect to each other. Image data is created by performing signal processing based on signals from a plurality of light receiving elements that receive the color-separated incident light that is arranged and shifted by a corresponding distance and generate signal charges corresponding to this incident light. Solid-state imaging device and an image data generation method, and in particular, a solid-state imaging device that outputs an image obtained from a square lattice-like arrangement in which the arrangement of pixels used for image formation differs from a square lattice-like arrangement from a signal of a primary color filter array The apparatus is suitably applied to, for example, a digital still camera or an image input apparatus.

[0002]

2. Description of the Related Art There is a demand for high quality images. As a method for improving the vertical resolution of an image, an all-pixel reading method is generally applied as one method. Furthermore, in colorization, a method of improving image quality in consideration of the arrangement of color filters has been evaluated and examined, and proposed. One of these proposals is disclosed in Japanese Patent Application Laid-Open No. 8-340455.

[0003] This method assumes a case where a single-plate color separation filter in which color filters are arranged in an RGB stripe arrangement is used. Therefore, each pixel of RGB occupies 1/3 of the total pixels. When each color of the pattern repeated in the column direction is viewed in the row direction, the light receiving elements, that is, the pixels are arranged in a positional relationship shifted from each other by half the pixel interval (pitch). Colors that are close to each other in this arrangement
A Δ (delta) pattern including all RGB is treated as one pixel, and the RGB values of each color in this pattern are used as the values of the three primary colors of the pixel. By handling pixels in this way, the number of pixels contributing to imaging is reduced to 1/3 of all the arranged pixels.

[0004]

As described above, there is a demand that the higher the image quality of a captured image is, the better the image quality is. There is. The former requirement is that, when all pixels are effectively read due to an increase in the number of pixels accompanying higher image quality, there is a possibility that the read time may be longer than before. The latter requirement means that the reading time and the processing speed of the image pickup signal are improved as the image display speed is increased, and the reading time is shortened. Therefore.
These requirements are in conflict with each other.

Further, when the latter requirement is to be satisfied, a false signal may be generated in the generated image.
The generation of the false signal degrades the image quality even if the number of pixels is increased.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a solid-state imaging device and a pixel data generating method capable of solving the above-mentioned drawbacks of the prior art and satisfying the conflicting demands of improved image quality and faster image display. And

[0007]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides a light-collecting means for condensing incident light from an object scene, and a device for converging incident light incident through the light-condensing means into three primary colors. RG
B Color filters for color separation are arranged in a square grid.
Color separation means arranged in a complete checkerboard pattern in which the same color R or B is arranged diagonally across G, and arranged in the row and column directions to receive incident light color-separated by this color separation means And a plurality of light receiving elements for generating signal charges according to the incident light. Digital centers which are arranged such that the centers of the geometric shapes are shifted from each other in the row direction and / or the column direction by a distance corresponding to a half of the pitch of the light receiving elements, and convert the imaging outputs from the plurality of light receiving elements into digital signal data. Conversion means; storage means for storing data from the digital conversion means for a plurality of images; and image resolution or image reading speed among a plurality of modes set for images taken from the storage means. Mode setting means for setting a mode in accordance with the setting, image selection means for selecting an image when reading an image from the storage means, and signal processing according to the mode set for the image selected by the image selection means. It is characterized by including signal processing means to be applied, and control means for controlling the image processing means and the digital conversion means and for controlling the signal processing means in accordance with information supplied from the mode setting means and the image selection means, respectively.

Here, the mode setting means instantaneously displays the output from the storage means to the signal processing means.
And a second mode in which the output from the storage means is used for either high-resolution display or recording on a recording medium for recording an image as compared with the first mode. . In this way, it is possible to respond to a user request separately as a mode.

The signal processing means uses the first signal processing means for performing signal processing by sampling when reading an image taken from the storage means in the first mode, and the output from the storage means in the second mode. A second signal processing means for generating high-frequency luminance data and performing signal processing by interpolating data between light receiving elements using the luminance data and data supplied via the storage means; And an output destination selecting means for switching and outputting the output of the first signal processing means to the second signal processing means under the control of the control means. Matrix means for generating, an aperture adjusting means for performing an outline emphasizing process on the luminance data among outputs from the matrix means, and an output from the aperture adjusting means. Preferably includes a filtering means for applying respective band-limited color difference data from the force and matrix means. As a result, the first signal processing means does not perform the interpolation processing as in the second signal processing means, and is a signal processing only for sampling. Therefore, the result of the signal processing performed by the second signal processing means is represented by a matrix. Output to the means.

The first signal processing means is disposed obliquely with respect to the color G stored in the storage means, and converts the colors R, G disposed in the same row direction at the position of this color G. A first extraction unit that performs sampling as combination data of three primary colors, wherein the first extraction unit includes a color R,
It is preferable that sampling is performed by overlapping combination data in which at least one of Gs is adjacent. This allows
The signal processing is almost the same as the case where the data of the light receiving elements (pixels) corresponding to the three primary colors RGB are combined into one data, but the number of output pixels is 1.5 times that of the conventional sampling. Can be increased. For example, three data are created from light receiving elements that create two data at a time. Therefore, the first signal processing means outputs data of half the total number of pixels while satisfying the speed requirement. In other words, the color G originally in the imaging means
Is obtained by the number of pixels corresponding to

The first signal processing means samples data of the same color R or B, which is obliquely arranged with respect to the color G stored in the storage means, as data at the position of the color G. 2 extracting means, and RB calculating means for calculating the colors R and B by the averaging of the data using the data extracted by the second extracting means,
It is desirable that the value calculated by the RB calculation means be combined data representing the three primary colors at the position of the color G. Thereby, although the speed is slightly lower than that of the above-described first signal processing means, data in which false colors are suppressed is provided by a simple arithmetic operation of averaging. Therefore, the image quality can be improved as compared with the output of the above-mentioned first signal processing means.

A second signal processing means for interpolating pixel data corresponding to a gap position of the light receiving element based on an imaging signal from the light receiving element in sampling the signal from the light receiving element in the second mode; And generating luminance data at a position where a light receiving element is present based on an output from the storage means when generating luminance data and color data from pixel data including the interpolated pixel data. Calculating brightness data in the virtual light receiving element when an empty region of the light receiving element is used as a virtual light receiving element based on the calculating means and the brightness data located in the horizontal and / or vertical direction from the first brightness calculating means; Second luminance calculating means, and each color using the luminance data created by the second luminance calculating means and each color data of RGB obtained from the light receiving element. And plane calculation means for generating plane data of the entire screen. Accordingly, it is possible to cope with a case where the user's request gives priority to image quality.

[0013] In addition to the above-described configuration, the solid-state imaging device includes:
Further, the optical system includes a spectral unit that divides the optical path of the incident light into a plurality of light paths, and a plurality of image capturing units that color-separate and receive the spectral light from the spectral unit, respectively. Are used as a color filter for each color or a color filter combining a plurality of colors, and a spatially superimposed color pattern is arranged in a G square grid, and the same color R or B is positioned diagonally across the color G. Are preferably arranged in a complete checkered pattern. Thus, even when the image data is constituted by a multi-plate color filter, it is possible to correspond to the high-speed reading priority of image data and the image quality priority.

Further, in addition to the above-described configuration, the solid-state imaging device further includes a color filter switching means for switching a color filter means inserted between the optical system and the two-dimensionally arranged light receiving elements; Moving means for moving the light-receiving elements of the light-receiving elements arranged two-dimensionally in a two-dimensional plane parallel to the imaging surface to be formed; Recording means for recording the captured image in a frame-sequential manner, and each time the image is moved, the center of the geometric imaging surface shape is shifted by half pitch in the row direction and the column direction with respect to the image captured immediately before. A color pattern that is shifted each time and is spatially superimposed is arranged in a G square grid pattern, and the same color R or B is arranged diagonally across the color G to form a complete checkered pattern Preferably This makes it possible to cope with the high-speed reading priority of image data and the priority of image quality, respectively, even in the imaging of the frame sequential method.

The solid-state imaging device according to the present invention receives light by a light-receiving element of the imaging means provided corresponding to the color separation means,
The obtained imaging output is converted into digital signal data by digital conversion means, and the image data is stored in the storage means. Thereafter, an image to be read is selected in the storage unit by the image selection unit. The storage unit outputs the selected signal to the signal processing unit. In the signal processing means, of the setting of the image resolution or the image reading speed, the signal processing according to the priority mode is performed by the control means, so that the user performs the signal processing according to the request of the user who has higher priority. In the case of speed priority, a decrease in the number of grouped image data can be suppressed.

Further, the present invention provides a plurality of prepared light sources for condensing incident light from an object scene, passing through a color filter for separating the incident light into three primary colors RGB, and further performing photoelectric conversion of the incident light. Each of the light receiving elements is shifted relative to the adjacent light receiving elements by a distance corresponding to a half of the pitch of the light receiving elements in the row direction and / or the column direction with respect to each other. The image data generating method for generating the image data of the object scene based on the signal charges obtained by the light receiving elements arranged in the image data generating method includes: A mode setting step of setting one mode in consideration of the resolution of the image or the reading speed of the image;
G are arranged in a square lattice, and the same color R or B is arranged diagonally across the color G of this color filter. An imaging step of converting and reading out to a signal represented by a voltage, a digital conversion step of converting an output obtained by the imaging step into digital signal data, and a process of converting the data obtained in the digital conversion step into a plurality of image data. An image storing step of storing, an image selecting step of selecting a captured image stored in the image storing step, and performing an image processing by performing signal processing according to a set mode on the image selected in the image selecting step. And a signal processing step of generating data.

Here, in the mode setting step, a first mode in which a captured image is instantaneously displayed and an output from the storage means having a higher resolution than the first mode are recorded or an image is recorded. It is preferable to set the second mode used for any of recording on a recording medium. As a result, although it is not possible to satisfy a plurality of requests, it is possible to perform a process for accurately satisfying each of the prioritized requests.

The signal processing step includes an output destination switching step of switching a reading destination of data stored in the image storage step corresponding to the first mode and the second mode, and an output destination switching step in the first mode. A first signal processing step of performing sampling when reading a captured image, and a high-frequency luminance signal using the output obtained by the digital conversion step in the second mode. A second signal processing for interpolating the data between the light receiving elements using the data supplied from the storage step;
A signal processing step, a matrix step of generating luminance data and chrominance data using the output obtained in the first signal processing step or the second signal processing step, It is preferable to include an aperture adjustment step of performing contour enhancement processing on the data, and a filter processing step of performing band limitation on the output from the aperture adjustment step and the color difference data from the matrix unit. As a result, signal processing of read priority and image quality priority is performed according to the set mode.

In the first signal processing step, the colors R and G arranged in the same row direction are arranged obliquely with respect to the color G stored in the image storing step, and the positions of the colors G are determined. It is preferable that the first extraction step includes sampling the combination data in which at least one of the colors R and G of the combination data is adjacent to the combination data. Thus, although the signal processing is almost the same as the case where the data of the light receiving elements (pixels) corresponding to the three primary colors RGB are simply combined into one data, the number of output pixels is smaller than that of the conventional sampling. Can be increased 1.5 times. For example, three data are created from light receiving elements that create two data at a time. Therefore, the first signal processing means outputs data of half the total number of pixels while satisfying the speed requirement.

In the first signal processing step, data of the same color R or B, which is obliquely arranged with respect to the color G stored in the image storing step, is sampled as data at the position of the color G. A second extraction step, and an RB operation step of calculating the colors R and B by averaging the data using the data extracted in the second extraction step, and calculating the calculated value in the RB operation step. It is preferable to use combination data representing the three primary colors at the position of the color G. As a result, although a little longer processing time is required than in the case of the above-described first signal processing step, it is possible to create and output image data in which image quality, particularly, false color is suppressed.

The second signal processing step is a signal processing for interpolating pixel data corresponding to a gap position of the light receiving element based on an image pickup signal from the light receiving element in sampling a signal from the light receiving element in the second mode. And when generating luminance data and color data from pixel data including the interpolated pixel data, generating first luminance data at a position where the light receiving element exists based on an output from the image storage step. When the empty region of the light receiving element is set as a virtual light receiving element based on the calculating step and the luminance data located in the horizontal and / or vertical directions from the first luminance calculating step, the luminance data in the virtual light receiving element is created. A second luminance calculation step, and using the luminance data created in the second luminance calculation step and each of the RGB color data obtained from the light receiving elements, for each color. And a plane operation step of creating plane data of the entire screen. Accordingly, it is possible to cope with a case where the user's request gives priority to image quality.

In the image data creating method, in addition to the above-described steps, a spectral step of dividing an optical path of incident light from the object field into a plurality of parts, and an incident light supplied by the spectral separation in the spectral step are color-separated. A plurality of imaging steps for receiving light, and using a color filter for each color or a color filter combining a plurality of colors corresponding to the color separation processing performed in the imaging step, a color pattern superimposed spatially is a square G It is preferable to form a complete checkered pattern in which the same color R or B is arranged diagonally with the color G interposed therebetween in a lattice shape. Thus, for example, high-speed reading priority and image quality priority can be made to correspond to the image data obtained through the multi-plate color filters.

In addition, in addition to the above-described steps, the image data creating method uses an optical system for condensing incident light from the object scene and inserts the optical system between the optical system and the two-dimensionally arranged light receiving elements. A color filter switching step of switching a color filter to be subjected to color separation, a moving step of moving a two-dimensionally arranged light receiving section of the light receiving elements in a two-dimensional plane parallel to an imaging surface formed by the light receiving elements, and a moving step And moving the light-receiving unit a plurality of times, each time the movement is performed, and recording a captured image in a frame-sequential manner. Are obtained by shifting the center of the geometric imaging plane shape by half a pitch in the row direction and the column direction, and the spatially superimposed color patterns are arranged in a G square lattice, and the diagonal across the color G Same color R or B at position It is desirable to be completely checkered pattern. This makes it possible to cope with the high-speed reading priority of image data and the priority of image quality, respectively, even in the imaging of the frame sequential method.

The plane calculation step includes, in the digital conversion step, a pixel of each color obtained corresponding to an actual light receiving element in an RB complete checkerboard pattern shifted by a distance of half a pitch with respect to the G square lattice. Using the data and the luminance data created in the second step, the G plane interpolation of the pixel data is performed in the horizontal and / or vertical directions adjacent to the pixel to be interpolated. A step obtained by adding the luminance data corresponding to the position of the interpolation target pixel to the difference between the average of the pixel data G and the average of the luminance data adjacent to the interpolation target pixel in the horizontal direction and / or the vertical direction. In the plane interpolation of pixel data R, the average of the actually obtained pixel data R adjacent to the pixel to be interpolated in the oblique direction and the A first R step obtained by adding the luminance data corresponding to the position of the pixel to be interpolated to the difference between the average of the luminance data adjacent in the same direction as the direction and the pixel for the remaining color R of the pixel to be interpolated Data R
Is obtained by adding the luminance data for the remaining color R of the pixel to be interpolated to the difference between the averaging of the pixel data located at the same distance and the averaging of the luminance data located at the same distance obtained in the R step. A second R step, an average of the first and second R steps that locate the pixel data R with respect to the remaining color R of the pixel to be interpolated, and the average of the actually obtained pixel data R 2 A third R step obtained by adding the luminance data corresponding to the position of the pixel to be interpolated to the difference between the average of the luminance data corresponding to the pixels used for the pixel data and the pixel data. Is the average of the pixel data B actually obtained adjacent to the interpolation target pixel in the oblique direction and the average of the luminance data adjacent to the interpolation target pixel in the same direction as the oblique direction. Pixel to be interpolated to the difference from The first B step obtained by adding the luminance data corresponding to the position and the pixel data B for the remaining color B of the pixel to be interpolated are obtained by dividing the pixel data of the equidistant pixel data obtained in the first B step. A second B step obtained by adding the luminance data for the remaining color B of the pixel to be interpolated to the difference between the averaging and the averaging of the luminance data located at the same distance; The pixel data B is interpolated by the difference between the first and second B steps located closest to each other and the average of the actually obtained pixel data B and the luminance data corresponding to the pixels used for the average. It is preferable to include a third B step obtained by adding luminance data corresponding to the position of the target pixel.

According to the image data generating method of the present invention, a mode is set in consideration of the resolution of the final image or the reading speed of the image, color separation is performed, and the obtained image pickup signal is converted into digital data. Storing a plurality of captured images,
When reading out image data, a stored captured image is selected. Since the selected image is subjected to signal processing according to the set mode, the image data is subjected to signal processing according to the user's priority request, and in particular, speed reduction suppresses a decrease in the number of grouped image data. be able to.

[0026]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of a solid-state imaging device and a method for creating image data according to the present invention;

The solid-state imaging device according to the present invention is provided with a mode corresponding to an item which the user gives priority to, one of the modes is set, and a honeycomb type light receiving element is two-dimensionally arranged.
The signal processing unit performs signal processing in accordance with a mode in which an imaging signal obtained from the imaging unit is set, so that a reduction in resolution is suppressed while the required time is approximately the same as that of a conventional high-speed display. . When importance is placed on the image quality, for example, only the image selected by the user can be recorded.

In this embodiment, a case where the solid-state imaging device of the present invention is applied to, for example, a digital still camera 10 will be described with reference to FIGS. The digital still camera 10 includes an optical system 12, a release shutter 14, a system control unit 16, an imaging unit 18 including a color separation filter CF,
An A / D conversion unit 20, a buffer memory 22, a signal processing unit 24, a mode setting unit 26, and an image selection unit 28 are provided.

The optical system 12 has a plurality of optical lenses. The optical system 12 is, for example, a first shutter of the release shutter 14.
The focal length of the subject and the amount of incident light are measured based on the transmitted light of these optical lenses by the pressing operation of the step. Although not shown, the system controller 16 automatically adjusts the focal length and performs exposure control based on the photometric value.

The release shutter 14 is a selection key for taking in information necessary for taking an image as described above. When a deep pressing operation is performed by the first-stage pressing operation, the system control unit controls the imaging timing desired by the user. It has the role of a timing switch to supply to 16.

The system control unit 16 is a central processing unit that controls the optical system 12 (including distance measurement and exposure control), the imaging unit 18, the A / D conversion unit 20, the buffer memory 22, and the signal processing unit 24. Equipment included. The system control unit 16 includes a release shutter 14, a mode setting unit 26, and an image selection unit.
From 28, the timing and the setting / selection result are supplied as information necessary for performing operation control.

In the imaging section 18, a color separation filter CF is arranged on the side of the incident light, and a light receiving element for photoelectrically converting the light transmitted through the color separation filter CF is two-dimensionally arranged. The color separation filter CF is a single plate. Therefore, there is a one-to-one correspondence between the color filters of the color separation filter CF and the light receiving elements.
As the color separation filter CF, for example, a color filter of three primary colors RGB as shown in FIG. 2 is arranged. This color pattern is a pattern in which the colors G are arranged in a square lattice, and the same color R or B is arranged diagonally across the color G 1 on a complete checkered pattern. The square lattice shape of the color G described above does not indicate the shape of the pixel but indicates the arrangement shape of the pixel.
In addition, the color filters (and light receiving elements) are arranged such that the centers of the geometrical shapes of the color filters (and light receiving elements) are aligned with each other in the row direction and the column direction with respect to each other. Are shifted by a distance corresponding to half the pitch of the elements. This color pattern is hereinafter referred to as a honeycomb type G square lattice RB perfect checkered pattern. The number of pixels shown in FIG.
G is six, and colors R and B are four each. In this case, the pixels formed collectively by the subsequent signal processing are the color G
(6) corresponding to the number of.

The light receiving element is, for example, a charge-coupled device (CC
D) or a metal oxide film type semiconductor (MOS). The imaging unit 18 forms the light receiving elements two-dimensionally so as to be arranged at the above-mentioned interval (pitch) to constitute an image sensor. The imaging unit 18 converts the signal charge obtained by the light receiving element into an analog signal of a voltage level by I / V conversion and outputs the analog signal to the A / D conversion unit 20. This imaging unit 18
A noise removal may be performed by providing a correlated double sampling unit between the A / D conversion unit 20 and the A / D conversion unit 20. Here, gamma conversion may be performed.

The A / D conversion section 20 converts the image signal (analog signal) obtained by the image section 18 into a digital signal. A /
The D conversion unit 20 converts the signal level of the image pickup signal into the number of bits obtained by slicing at the quantization level, for example, in order to represent the signal level of the image signal by bits, which are information units. Thereby, the imaging signal is converted into a digital signal represented by bits.

The buffer memory 22 has a function of temporarily storing digital data. The buffer memory 22 has a capacity to store a plurality of images. The image data converted into a digital signal by the A / D converter 20 is stored in the buffer memory 22 for each image under the control of the system controller 16. Since the image data is stored in this manner, the buffer memory 22 can also read out the image data in image data units. Reading of image data from the buffer memory 22 is also performed by the system control unit.
Controlled by 24. The buffer memory 22 is stored in the address control unit 2420 described later by the system control unit 22.
To control the write / read position of the memory. In particular, when reading a captured image, the system control unit 22 performs control according to the designation of the image selection unit 28.

The signal processing unit 24 includes a mode corresponding processing unit 24a
, A matrix unit 24b, an aperture adjusting unit 24c, and a filter unit 24d. Mode support processing unit 24a
Performs the processing of the image data supplied from the buffer memory 22 in the designated mode. In performing this processing, the mode corresponding processing unit 24a includes an output destination switching unit 240a, a display signal processing unit 242a, and a recording signal processing unit 244a.

The output destination switching section 240a is supplied with a control signal PC1 corresponding to the setting mode received by the system control section 16. In response to the control signal PC1, the output destination switching unit 240a supplies the output from the buffer memory 22 to the matrix unit 24b or the recording signal processing unit 244a.

The display signal processing unit 242a includes an address control unit.
2420 is provided. The address control unit 2420 performs high-speed display processing based on a control signal PC2 from the system control unit 16. In order to emphasize that the address control unit 2420 performs the high-speed display processing, the address control unit 2420 is shown in a block diagram as in FIG. Because the address control unit 2420
This is because the operation of reading the image data from the storage 22 is not limited to the display signal processing unit 242a, and the operation of reading the image data from the recording signal processing unit 244a is also controlled by the control signal PC2. The display signal processing will be described later in detail.

As shown in FIG. 3, the recording signal processor 244a includes a luminance data generator 2440 and a high-frequency luminance data generator 2442.
And a plane interpolation developing unit 2444. The image data (RGB) from the buffer memory 22 is output destination selection unit
The luminance data is supplied to a luminance data creation section 320b and a plane interpolation development section 320d via a 240a. Since the colors R and B are arranged in a checkered pattern, the brightness data creation unit 220b creates the brightness data Y at each target position of the checkered pattern by arithmetic processing. In the luminance data creation section 220b, the luminance data Y is created corresponding to the position where the light receiving element actually exists.

The high-frequency luminance data generator 2442 is a digital filter that performs a filtering process using the luminance data Y generated by the luminance data generator 2440. This digital filter is a low-pass filter that performs interpolation on a position to which the light-receiving element does not actually correspond, that is, on a virtual pixel, and makes the frequency band of the obtained luminance data high.
As a result, high-frequency luminance data generating unit 2442 outputs the high-frequency luminance data Y _h the plane interpolation expansion unit 2444.

The plane interpolation section 2444 includes an R interpolation section 44a, a G interpolation section 44b, and a B interpolation section 44c as shown in FIG. These units are operation units. R interpolation expansion section 44a, G interpolation expansion section 44b and B
The interpolation developing unit 44c has high-frequency luminance data at one end.
Y _h are supplied, the pixel data corresponding to a color to be interpolated from another terminal side, i.e. R data, G data and B data are supplied. When performing this plane interpolation development, a virtual pixel for a target color located further around is obtained using pixel data obtained by the arithmetic processing. The procedure of this calculation will be described in detail later.

The recording signal processing section 244a has such a configuration mainly considering recording of high-resolution image data, but displays the high-resolution image data on a high-resolution monitor or the like. In this case, this signal processing is used. Therefore, in the recording signal processing unit 244a of FIG. 1, such a case is assumed (display).

The matrix portion 24b are luminance data Y and color difference data C _r on the basis of the image data supplied, and generates a C _b. As shown in FIG. 1, the matrix unit 24b stores the image data from the output destination switching unit 240a and the RG signals by the recording signal processing unit 244a.
Either one of the B and the interpolated image data is supplied according to the mode. The matrix unit 322 includes an arithmetic unit (not shown) so as to generate these data. Matrix unit 24b, the luminance data Y and color difference data C _r = RY _h by the arithmetic processor, to generate a C _b = BY _h.

The aperture adjustment unit 24c has the luminance data Y
A conventional configuration is used for an aperture effect, for example, contour enhancement. That is, the aperture adjuster 24c adjusts the signal level in the high frequency region so as to increase the signal level in the vicinity of the band-limited frequency in terms of frequency.

The filter section 24d is a low-pass filter 240d, 242d, 244d for anti-aliasing adjusted to include the high frequency component of the luminance data Y. These low-pass filters 240d, 242d, 244d are all constituted by digital filters. The mode setting unit 26 is a selection switch for selecting one mode from a plurality of modes. Although not shown, the digital still camera 10 is provided with a display unit. A mode selection item may be displayed on this display unit, and the display area of this item may be clicked and selected. In this case, the screen of the display unit also has a sensor function as a selection key.

The image selection section 28 is a selection key for selecting, for example, an image to be recorded, displayed, and further printed on the display section while viewing the thumbnail-displayed image. The image selection unit 28 is configured to move the cursor in four directions and to allow selection.

Note that the digital still camera 10 is the same as that shown in FIG.
A compression / decompression processing unit (not shown) performs compression processing on the image data obtained in this way under the control of the system control 16. The compressed image data is transferred to the system control 16
Is recorded on a recording medium via a built-in or integrally mounted recording / reproducing device under the control of. Conversely, when reproducing the compressed image data recorded on the recording medium, the image data is reproduced from the recording / reproducing apparatus under the same control as during recording. Since the reproduced image data has been subjected to compression processing, it is subjected to expansion processing by a compression / expansion processing unit, supplied to a display unit, and displayed.

Next, the operation of the digital still camera 10 will be described. Reference is made to the drawings used in the above-described configuration as necessary. The digital still camera 10 operates according to the main flowchart in the photographing shown in FIG. In step S10, the user himself / herself as an operator sets what mode the operation of the digital still camera 10 should be. There are a plurality of modes, for example, a still image shooting mode, a movie mode, an index display mode, a high image quality display mode, and the like. The setting mode selected by the operator and selected is supplied from the mode setting unit 26 to the system control unit 16. System control unit 16
Sets the digital still camera 10 in a photographable state based on the setting mode and other conditions. Thereafter, the process proceeds to step S12.

In step S12, the photographing of the scene is performed.
Immediately before this stage, the release shutter 14 is pressed halfway, exposure and distance measurement are performed, and proper exposure and focus adjustment are performed. Thereafter, the operator completely presses the release shutter 14 at a desired timing to take an image. Imaging unit
In 18, the incident light supplied via the optical system 12 is color-separated by a single-plate color separation filter CF. The color-separated light is supplied to the light receiving elements of the imaging unit 18 which are two-dimensionally arranged in a honeycomb shape. The color separation filter CF is the honeycomb type G square lattice RB perfect checkerboard pattern shown in FIG. The light receiving elements respectively receive the light transmitted through the color separation filter CF,
Photoelectric conversion. The photoelectrically converted signal charges are transferred to the imaging unit.
The light is output as pixel data detected by each light receiving element via 18 vertical transfer paths and horizontal transfer paths (both not shown). The imaging unit 18 performs I / V conversion so that the signal charge detected in the current form can be represented by a voltage, and outputs the signal charge to the A / D conversion unit 20.

Next, in step S14, the signal level supplied from the imaging unit 18 is quantized and converted into a digital signal that can be handled as bit information, that is, image data. Immediately after this conversion process, the image data is stored in the buffer memory.
Supplied to 22.

In step S16, the image data is temporarily stored in the buffer memory 22. In supplying this image data, the control of the system control unit 16 and the address control unit 24
Under the control of 20, image data is written to a predetermined address, for example, in image units. The address control unit 2120
It is controlled by the system control unit 16. The buffer memory 22 can store a plurality of captured image data. An image that has been shot before this shooting may be stored.

Next, the process proceeds to step S18, and the process proceeds to step S18.
In, it is determined whether or not the preset mode is a mode corresponding to high-speed display of image data. When the high-speed display mode is set (Yes), the subroutine SU
Proceed to B1. If the high-speed display mode is not set (No), the process proceeds to the subroutine SUB2. The setting of this mode is performed by the system controller 16 detecting and determining the mode performed in step S10. The image data is read from the buffer memory 16 in accordance with the determination result of the system control unit 16. Further, the read image data is supplied to the output destination selection unit 240a.
At this time, the output destination selection unit 240a switches the image data supply destination under the control of the system control unit 16.

In the subroutine SUB1, priority processing of the display speed is performed on the image data. In this case, the display signal processing unit
At 2420, the image data is read from the buffer memory 22 and output to the matrix unit 24b without performing a time-consuming process such as an arithmetic process. This processing will be described later in detail.

In the subroutine SUB2, image data read from the buffer memory 22 is given priority over image quality,
Processing to increase the effective resolution is performed on the honeycomb type G square lattice RB perfect checkerboard pattern. In performing this processing, the image data is supplied to the recording signal processing unit 244a via the output destination selection unit 240a. This processing will be described later in detail. In the subroutines SUB1 and SUB2, RGB data subjected to completely different processes by the respective signal processes is output to the matrix unit 24b. After this, the subroutine SUB3
Proceed to.

In the subroutine SUB3, the luminance data Y and the color difference data (BY), (RY) are generated, the aperture adjustment is performed on the luminance data Y, and the three signals are subjected to the filter processing to perform the signal processing. Will be Although not shown, the image data obtained in this manner is supplied to the display unit as it is when the setting mode is the display speed priority (high-speed display).
Proceed to. When priority is given to image quality, the image data is supplied to a compression processing unit (not shown), and the process proceeds to step S20.

Next, in step S20, subroutine SUB3
Compression processing is performed when the image data subjected to the signal processing is recorded on the recording medium. Examples of compression processing include, for example,
JPEG (Joint Photographic coding Experts Group) or MPEG (Moving Picturecoding Experts Group)-
There is signal processing used for 1 etc. The image data that has been subjected to the compression processing is recorded and stored on a recording medium mounted on a recording and reproducing apparatus.

In step S22, it is determined whether or not to select an image. In this case, the operator determines whether to display or record another image data. Until this determination is completed, the digital still camera 10 keeps its operation in a standby state. When selecting an image (Yes
), And proceed to step S18. When not selecting an image (N
o), proceed to step S24.

In step S24, it is determined whether the photographing is to be ended. If shooting is still to be continued (No), the process returns to step S12 to continue a series of processes. When the photographing is completed (Yes), the power supply used to operate the digital still camera 10 is turned off. The digital still camera 10, which has an imaging unit in which the light-receiving elements are arranged in a honeycomb shape, can perform high-quality signal processing so far, so that good image quality can be provided, but it takes much time to display. Have been. However, such an operation enables high-speed display and satisfies the user's request.

Further, a subroutine SUB1 will be described as a method for performing high-speed display. In three colors RGB, color G
Based on the direction of the vertices of the triangle formed by the colors R and B used as the color of the position, the form of the data combination in the sampling is selected. In the form of this combination, there are four types of triangular shapes to be regarded collectively. As for the setting of the form of this combination, for example, one of four combinations is set by default, and the other three combinations can be set at the mode setting stage, for example. In the subroutine SUB1, it is first determined which combination is set.

First, in sub-step SS10, it is determined whether or not the color G located at the vertex of the triangle is on the lower side. If it is on the lower side (Yes), proceed to sub-step SS12. When it is determined that the state is not in this state (No), the process proceeds to sub-step SS14.

In the sub-step SS12, the light receiving elements (hereinafter referred to as pixels) of the colors R and B are arranged side by side above the pixels of the color G. Although the pixel corresponding to the color R, B at the pixel position of this color G does not actually exist, the address control unit 2420 operates the buffer memory 22 so as to enable sampling using the values of the color R, B at the above-mentioned position instead. Control the reading of the data. For example, when this reading is performed in the honeycomb type G square lattice RB complete checkerboard pattern of FIG.
As shown in (b), a combined pattern is formed. That is, it is regarded as pixel data in which colors R ₀₀ and B ₀₂ are grouped with respect to color G ₁₁ . The next adjacent group is the color
A pixel data color B _02, R ₀₄ and group for G _13. As described above, the colors R and B are grouped so as to include at least one color.

In sub-step SS14, it is determined whether the color G located at the vertex of the triangle is on the upper side. When it is on the upper side (Yes), the process proceeds to sub-step SS16. When it is determined that the state is not in this state (No), the process proceeds to sub-step SS18.

In sub-step SS16, pixels of colors R and B are
It lies side by side below the G pixel. In this case as well, the pixel corresponding to the color R, B at the pixel position of this color G does not actually exist, but the address control unit 24 enables sampling using the value of the color R, B at the above-mentioned position instead.
20 controls the reading of the buffer memory 22. For example,
In the honeycomb-type G square lattice RB completely checkered pattern of FIG. 7 (a), when the reading is performed, the color for the color G ₁₁ B
₂₀ and R ₂₂ are regarded as pixel data. The next adjacent group, a pixel data color R _22, B ₂₄ and groups the color G _13. As described above, the colors R and B are grouped so as to include at least one color.

In sub-step SS18, it is determined whether the color G located at the vertex of the triangle is on the right side. When it is on the right side (Yes), proceed to sub-step SS20. When it is determined that the state is not in this state (No), the process proceeds to sub-step SS22.

In sub-step SS20, pixels of colors R and B are
It is arranged vertically on the left side of the G pixel. In this case as well, the pixel corresponding to the color R, B at the pixel position of this color G does not actually exist, but the address control unit 24 enables sampling using the value of the color R, B at the above-mentioned position instead.
20 controls the reading of the buffer memory 22. For example,
In the honeycomb-type G square lattice RB completely checkered pattern of FIG. 7 (a), when the reading is performed, the color relative to the color G ₁₁ R
_00, regarded as the pixel data for the B ₂₀ and group. The next adjacent group is pixel data in which colors B ₀₂ and R ₂₂ are grouped with respect to color G ₁₃ . As described above, the colors R and B are grouped so as to include at least one color.

Further, in sub-step SS22, it is determined whether or not the color G located at the vertex of the triangle is on the left side. If it is on the left (Yes), proceed to sub-step SS24. When it is determined that the state is not in this state (No), it is determined that the corresponding setting is not made, and the process returns to the return.

In sub-step SS24, pixels of colors R and B are
It is arranged vertically to the right of the G pixel. In this case as well, the pixel corresponding to the color R, B at the pixel position of this color G does not actually exist, but the address control unit 24 enables sampling using the value of the color R, B at the above-mentioned position instead.
20 controls the reading of the buffer memory 22. For example,
In the honeycomb-type G square lattice RB completely checkered pattern of FIG. 7 (a), when the reading is performed, the color for the color G ₁₁ B
_02, regarded as pixel data to the R ₂₂ group. The next adjacent group is pixel data in which colors R ₀₄ and B ₂₄ are grouped with respect to color G ₁₃ . As described above, the colors R and B are grouped so as to include at least one color.

When the relationship between the patterns for reading the image data from the buffer memory 22 is thus determined, it is determined whether or not the image reading has been completed in each setting (sub-steps SS26, SS28, SS30, SS32). . If the reading has not been completed yet (No), the process returns to the previous sub-steps SS12, SS16, SS20 and SS24, respectively, to continue reading the image data. On the other hand, if all the readings have been completed (Yes), the process proceeds to return. By this processing, the image data is read out as pixel data of half the total number of pixels. In this pattern, the number of pixels read out is the same as the number of pixels provided with the color G. In this embodiment, the image data is read out without performing any operation on the originally obtained image data. As a result, the reading speed can be read while keeping the reading speed at about half the total number of pixels while maintaining the same reading speed as the case of reading the three pixels (RGB) at a high speed. . As a result, in terms of resolution,
This can be improved compared to the case where only 1/3 could be read.

Incidentally, a honeycomb type color separation filter CF is used.
When a signal captured by a G square lattice RB perfect checkerboard pattern is represented by a spatial frequency, each color is distributed as shown in FIG.
That is, the frequency distribution of the colors R and B is a distribution obtained by rotating the square by 45 °. This distribution indicates that the sample frequency in the oblique direction is low. For this reason, in an image obtained using this image data, a false signal due to aliasing in an oblique manner is likely to occur. Since the spatial frequency distribution of the color G is arranged in a square lattice, the colors R, B
Becomes a square distribution including the frequency distribution of

While coping with the occurrence of such a false signal,
A configuration for reading image data at high speed will be described as a modification of the above-described embodiment. In this case, the display signal processing unit 242a includes not only the address control unit 2420 but also an RB calculation unit 2422 (see FIG. 9). The RB operation unit 2422 is an operation unit that uses pixel data of the colors R and B located around the color G and performs an operation on the pixel data. Therefore, as shown in FIG. 9, the RB calculation unit 2422 passes the pixel data of the color G from the output destination switching unit 240a to the matrix unit 24b as it is,
Only the R and B pixel data are input.

In this operation, so-called averaging processing is performed in which pixel data (in this case, same colors) positioned diagonally across the color G is added and the average is calculated. At this time,
The pixels obtained by the averaging are the pixel data of the colors R and B to be treated as a group together with the color G. As a result, it means the same as the color R and B pixels in the color G pixel. Specifically, the relationship is as shown in FIG. That is, the color located across the color G ₁₁ R
₀₀ , R ₂₂ and the color B located between the colors B ₀₂ , B ₂₀ and the color G _13
02, B ₂₄ and the color R _04, is calculated using the R _22. Finally, the color G
In order to identify the pixel data of the color R and B at the position of, the obtained pixel data is represented by small letters r and b, and the suffix is made the same as the numerical value of the color G. Therefore, the operation is

[0072]

[Number 1] _{r 11 = (R 00 + R} 22) / 2 b 11 = (B 02 + B 20) / 2 ··· (1) r 13 = (R 04 + R 22) / 2 b 13 = ( B ₀₂ + B ₂₄ ) / 2... (2) In other words, the pixel data r ₁₁ can be said to have been newly calculated color R _00. Similarly, pixel data b ₁₁ ,
b ₁₃ and r ₁₃ can be said to be the calculated colors B ₀₂ , B ₀₂ and R ₀₄ , respectively. The characteristic point is that the calculation is performed for the same position as the colors B ₀₂ and B ₀₂ , but the group handled is different. As described above, one group includes at least one or two adjacent pixels. One overlap corresponds to a value near the boundary, and two overlaps correspond to a case that is located inside a pixel to be handled as the boundary. The pixel data thus calculated is output to the matrix unit 24b.

When the calculated pixel data of the colors R and B is used, a low-pass effect is exerted on the original signal,
The S / N of the R and B signals will also be improved. As a result, generation of a false signal in the honeycomb type G square lattice RB perfect checkerboard pattern is suppressed. Also, since the calculation here is only a simple averaging, it can be easily realized by a hardware configuration (not shown) that performs bit shifting with an adder. Since the calculation is performed by hardware, high-speed processing can be performed as compared with the calculation performed by software. Further, as described in the above configuration, since the most important color G used for generating the luminance signal is passed through without any change, the fidelity in the generation of the luminance data Y is maintained and the final The number of pixels that can be handled as many as the number of colors G can be secured. By this. Since the false color can be improved and the luminance signal can be improved, the image quality can be improved as compared with the previous embodiment.

Next, the operation of the recording signal processing section 244a will be described with reference to a subroutine SUB2. That is, subroutine SUB
4. Operate in the order of SUB5 and SUB6. Subroutine SUB4
Then, checkerboard-like luminance data Y is created in consideration of a color obtained corresponding to the position of an existing light receiving element. subroutine
In SUB4, luminance data Y is calculated according to a preset mode as described later. In the subroutine SUB5, based on the obtained luminance data Y, luminance data at a virtual pixel without a corresponding light receiving element is generated, and when the luminance data is viewed in a frequency band, the luminance data is raised. The high Ikika luminance signal represented by Y _h. Further, the subroutine SUB6 the luminance data Y _h and the supplied G,
RGB plane development is performed by interpolating using R and B, respectively.

The operation of each subroutine SUB4, SUB5 and SUB6 will be described. In sub-step SS400 of subroutine SUB4 shown in FIG. 12, first, it is determined whether or not the mode is the adaptive processing mode. If the mode is not the adaptive processing mode (No), the sub-step SS40 of FIG.
Proceed to 2. In the case of the adaptive processing mode (Yes), the process proceeds to sub-step SS401. In sub-step SS401, a selection is made as to whether to perform diagonal correlation processing. When performing the diagonal correlation processing (Yes), the process proceeds to sub-step SS404. When the diagonal correlation processing is not performed (No), sub-step SS410
Proceed to.

In the above-described sub-step SS402, the calculation of the luminance data is performed irrespective of the adaptive processing mode. In performing this processing, the CCD image sensors of the imaging unit 31 are originally two-dimensionally arranged as shown in FIG. Here, the suffix is a position when the position of each light receiving element as a pixel is represented by a matrix expression. The pixels of the actual light receiving element are indicated by solid lines, and the pixels corresponding to the virtual light receiving elements are indicated by broken lines. Basically, it is known that the luminance data Y can be calculated as (0.5 * R + 0.5B) using the pixel data G and the pixel data R and B. Also in this case, the pixel data G is treated as it is as luminance data (pixel data G = luminance data). The luminance data by the pixel data R, B, if the color corresponding to the position of the real light receiving elements of the R / B rather than G, for example, luminance data Y with respect to the position of the pixel data R ₂₂ shown in FIG. 14 (a) _Reference numeral ₂₂ denotes four pixels of pixel data R ₂₂ and pixel data B located around the pixel data R ₂₂ , that is, pixel data B ₀₂ ,
Using B ₂₀ , B ₂₄ , B ₄₂

[0077]

[Number 2] obtained from _{Y 22 = R 22/2 +} (B 02 + B 20 + B 24 + B 42) / 8 ··· (3). The luminance data Y ₂₄ corresponding to the position of the pixel data B ₂₄ is four pixels of the pixel data R which is located around the pixel data B _24, i.e. pixel data R _04,
Using R ₂₂ , R ₂₆ , R ₄₄

[0078]

Equation 3] obtained from the _{Y 24 = B 24/2 +} (R 04 + R 22 + R 26 + R 44) / 8 ··· (4). This calculation is performed for each pixel to obtain the luminance data Y. The result obtained in this way is
A checkerboard pattern of the luminance data shown in FIG. 14 (b) is obtained. Note that such calculation is also performed when there is no correlation between the oblique direction, the vertical direction, and the horizontal direction. After this processing, the flow advances to sub-step SS430.

In the sub-step SS404, comparison data is calculated. The comparison data is used to determine, for example, in which direction the pixel data around the pixel data to be subjected to the adaptive processing are correlated. For example, if the object pixel data of R _22, comparison data
AG uses the surrounding pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃

[0080]

AG = | G ₁₁ + G _33- (G ₁₃ + G ₃₁ ) | (5) Also when the pixel data is B, it is calculated from the surrounding pixel data G. By this calculation, a larger value having a slope on one of the left and right sides is obtained as the comparison data AG. After this operation, sub-step SS406
Proceed to.

In sub-step SS406, it is determined whether or not there is a correlation (ie, a diagonal correlation) between the pixel data positioned diagonally across the target pixel data. In this determination, J1 is set as a determination reference value. comparison data
When AG is larger than the judgment reference value J1 (Yes), the process proceeds to sub-step SS408. Further, the comparison data AG is the judgment reference value J1.
If smaller than (No), proceed to sub-step SS410.

In sub-step SS408, the four pixel data G used to calculate the comparison data AG are averaged to obtain the luminance data
Calculate Y. Although not least shown by oblique correlation so that six patterns is determined for the pixel data R = R _22. By the way, even in this case, there is a possibility that a false color is generated. Therefore, when the luminance data Y in the pixel data R located near the boundary of the pixel having the possibility of being calculated by the above-described calculation, it is possible to satisfactorily suppress the occurrence of the false color at the color boundary when the entire image is viewed. Can be. It is possible to create an adaptive luminance data Y on the basis of whether the calculated oblique correlation similar comparison data sub-steps SS406, SS408 even for specific explanation pixel data B = B ₂₄ is omitted . After this process, the flow advances to sub-step SS430 in FIG. 13 via connector C.

As described above, in sub-step SS410,
The vertical direction comparison data ABR _V and the horizontal direction comparison data ABR _H for the pixel data R = R ₂₂ are obtained by using pixel data of the other color arranged around, that is, pixel data B.

[0084]

ABR _V = | B ₀₂ -B ₄₂ | (6) ABR _H = | B ₂₀ -B ₂₄ | (7) is calculated. After this processing, the flow advances to sub-step SS411.

At sub-step SS411, the calculated comparison data ABR _V and ABR _H are used to further calculate the correlation value (ABR _H -ABR
_V ) and (ABR _V -ABR _H ), when each of these correlation values is smaller than the newly provided predetermined criterion value J2 (Yes), the correlation in the horizontal and vertical directions is Substep SS40 via connector B
Proceed to 2. When the above conditions are not satisfied (N
o) If there is some correlation, proceed to sub-step SS412. When the process is terminated at this stage, the process may proceed to sub-step SS430 via a connector C (not shown).

In sub-step SS412, it is determined whether or not there is a correlation (that is, a vertical correlation) between pixel data positioned vertically across the target pixel data. In this judgment, J2a is set as a judgment reference value. When the difference between the comparison data ABR _H and the comparison data ABR _V is larger than the determination reference value J2a (Yes), it is determined that there is a vertical correlation, and the flow advances to sub-step SS414. Also, the difference of the comparison data (ABR _H -ABR _V )
Is smaller than the determination reference value J2a (No), it is considered that there is no vertical correlation, and the flow advances to sub-step SS416.

In the sub-step SS414, since there is a correlation between the pixel data, it means that the luminance data Y is calculated using the pixel data B ₀₂ and B ₄₂ . In this case, the luminance data Y ₂₂ is

[0088]

[6] _{Y 22 = R 22/2 +} (B 02 + B 42) / 4 is obtained by (8). Thereafter, it is considered that the calculation of the luminance data Y in the pixel data has been completed, and the process proceeds to the sub-step SS430 via the connector C.

Next, in sub-step SS416, it is determined whether or not there is a correlation (that is, a horizontal correlation) between pixel data positioned horizontally across the target pixel data. In this determination, J2b described above is used as a determination reference value. When the difference between the comparison data ABR _V and the comparison data ABR _H is larger than the determination reference value J2b (Yes), it is determined that there is a horizontal correlation, and the flow advances to sub-step SS418. In addition, the difference (ABR _V -AB
When R _H ) is smaller than the determination reference value J2b (No), it is determined that there is no horizontal correlation, and the sub-step SS41 is performed via the connector D.
Go to 9.

In the sub-step SS418, the luminance data Y is calculated by using the pixel data B ₂₀ and B ₂₄ as having a correlation. In this case, the luminance data Y ₂₂ is

[0091]

Y ₂₂ = R _22/2 + (B ₂₀ + B ₂₄ ) / 4 (9) Thereafter, it is considered that the calculation of the luminance data Y in the pixel data has been completed, and the process proceeds to the sub-step SS430 via the connector C.

Next, through the connector D, the sub-step SS419
Then, for example, it is selected whether or not to perform the correlation judgment of the surrounding color B pixel with the color R pixel. Since the pixel of the color R is arranged at the center position of the surrounding pixels of the color B, the distance between the pixels in the sub-steps SS412 and SS416 is short. Therefore, it is determined in a subsequent process whether or not there is a correlation in a narrower range. If this correlation is to be determined (Yes), the flow proceeds to sub-step SS420. When the correlation determination is not performed (No), the process proceeds to sub-step SS402. In this case, a criterion value J2a different from the criterion value J2,
It is determined that none of the criteria of J2b was satisfied. Note that a processing procedure that does not perform the subsequent processing may be used. In this case, simply via connector D
Proceed to 2.

In sub-step SS420, comparison data is calculated again. In this case, the comparison data is calculated in the vertical and horizontal directions by calculating each correlation between the target pixel data and the surrounding pixel data, and adding the obtained correlation values. Calculation of the luminance data Y for the case as well as the pixel data R ₂₂ described above, the vertical direction of the comparison data AC
_RV and the comparison data ACR _H in the horizontal direction are obtained by using the pixel data of the other color arranged around, that is, the pixel data B.

[0094]

ACR _V = │B ₀₂ -R │ + │B ₄₂ -R │ ・ ・ ・ (10) ACR _H = │B ₂₀ -R │ + │B ₂₄ -R │ ・ ・ ・ (11) I do. After this processing, the flow advances to sub-step SS412. By using the comparison data, the correlation value can be obtained by further reducing the distance of the pixel data to the pixel data, so that the range defined in the calculation of the correlation value in the procedure of the previous sub-steps SS412 to SS418 can be obtained. Can also be checked for a narrow range. After this calculation,
Proceed to sub-step SS422.

In sub-step SS422, it is determined whether or not there is a correlation (ie, a vertical correlation) between pixel data positioned vertically across the target pixel data. In this determination, J3 is set as a determination reference value (here, the determination reference value J3 may be divided into J3a and J3b for horizontal and vertical use). When the difference between the comparison data ACR _H and the comparison data ACR _V is larger than the determination reference value J3 (Yes), it is determined that there is a vertical correlation, and the flow advances to substep SS424. In addition, the difference
When (ACR _H -ACR _V ) is smaller than the judgment reference value J3 (No),
It is determined that there is no vertical correlation, and the flow advances to substep SS426.

In sub-step SS424, the same processing as in the above-described sub-step SS414 is performed. Therefore,
Equation (9) is used for the calculation. Also, sub-step SS42
In step 6, it is determined whether or not there is a correlation (that is, a horizontal correlation) between pixel data positioned horizontally across the target pixel data. The determination reference value J3 is also used for this determination.

In sub-step SS426, when the difference (ACR _V -ACR _H ) of the comparison data is equal to or larger than the judgment reference value J3 (Yes),
It is determined that there is a horizontal correlation, and the flow advances to substep SS428.
In this case, the luminance data Y in sub-step SS428 uses pixel data as described above in sub-step SS418,
It is calculated based on (10). After this, sub-step SS43
Go to 0. In sub-step SS426, the difference
When (ACR _V -ACR _H ) is smaller than the determination reference value J3 (No), it is determined that there is no horizontal correlation, and the flow advances to substep SS402. In sub-step SS402, the target pixel data and the surrounding pixel data (in this case, pixel data B) are averaged and multiplied by a coefficient of 0.5 by equation (3) to obtain luminance data.
Y is calculated. After this calculation, the process proceeds to sub-step SS430.

In sub-step SS430, it is determined whether the creation of the checkered luminance data has been completed for one frame. This determination can be easily made, for example, by counting the calculated number of pieces of luminance data Y and determining whether or not this count value matches the number of light receiving elements. When the count value is smaller than the number of light receiving elements (N
o), it is determined that the processing has not been completed yet. As a result, the process of calculating the luminance data Y is returned to the sub-step SS400 in FIG. When the count value matches the number of light receiving elements (Yes), the process shifts to return. After this return, the process is shifted to subroutine SUB5.
By calculating the luminance data Y in this way,
As shown in FIG. 14 (b), data is created at a position where a checkered light receiving element actually exists.

Next, the operation of the subroutine SUB5 will be described.
The operation of the subroutine SUB5 is performed based on the configuration of the digital filter of the high-frequency luminance data creation unit 2442 (see FIG. 3) as described above.

In sub-step SS50, as shown in FIG. 15, low-pass filter processing which is a characteristic of the digital filter is performed, and pixel data at the position of the virtual light receiving element is generated to perform data interpolation. This relationship is briefly shown in FIG. In FIG. 16 as well, pixels d _(-3) , d _(-1) , d ₍₁₎ , and d ₍₃₎ corresponding to a real light receiving element are indicated by solid lines, and pixels corresponding to virtual light receiving elements are indicated by broken lines. Are arranged between the four light receiving elements (existing pixels). Pixels d _{n (-4)} , d _{n (-2)} , d _{n (0)} , d corresponding to the virtual light receiving element
_{n (2)} and dn ₍₄₎ are treated as having the same relationship as the state in which no data is contained, in consideration of the correspondence with the existing light receiving element. That is, zero is set in advance for these pixels. For example, when horizontally interpolating the pixel dn ₍₀₎ as shown in FIG. 16 (a), the tap coefficients of the digital filter are k ₀ , k ₁ , k ₂ , k ₃ , k ₄ ,..., K _When arranged as _n ,
The luminance data Yh ₍₀₎ including the high-frequency component is expressed by equation (12).

[0101]

Equation 9] _{Y h (0) = k 0} * d n (0) + k 1 * (d (1) + d (-1)) + k 2 * (d n (-2) + d n (2 _{)) + k 3 * (d} (-3) + d (3)) + k 4 * (d n (-4) + d n (4)) + ··· k n * (d n (-n) + d _{n (n)} )... (12) However, in this case, FIG.
As can be seen from (a), the coefficient is doubled because zero data is alternately entered. This relationship is based on the other pixels dn _(-4) , dn _(-2) , dn ₍₂₎ , dn _{(4) to be} interpolated in FIG.
Also apply to By performing these interpolation processes, the luminance data Yh _(-4) , Yh _(-2) ,
Yh ₍₂₎ and Yh ₍₄₎ are obtained (see FIG. 16 (b)).

In the vertical direction, low-pass filter processing is also performed by the high-frequency luminance data creating section 2442 using a digital filter. In this case, the pixel data corresponding to the virtual light receiving element has already been interpolated by the horizontal interpolation process, so that the pixel data is dense. Therefore,
The coefficients of the low-pass filter can be made the same as usual. When the luminance data including the high frequency component obtained in this way is represented by a matrix expression as shown in FIG.
Luminance data Y _h including high-frequency component is generated as shown in FIG. 17. Luminance data Y _h including high-frequency component is referred to as the high-frequency luminance data in the following description.

Next, the operation of the subroutine SUB6 will be described. The subroutine SUB6 is performed by the plane interpolation developing unit 2444 as shown in FIG. The plane interpolation expansion unit 2444, the high frequency luminance data Y _h Toko of high-frequency luminance data Y color interpolated corresponding to _h of the pixel data generated by the subroutine SUB5 is supplied to each of the arithmetic processing unit. High frequency luminance data Y _h is, R interpolation expansion unit 44, as is apparent from FIG. 4
a, the G interpolation developing section 44b, and the B interpolation developing section 44c are commonly supplied. Using these supplied pixel data, the pixel data of the pixel of each virtual light receiving element is interpolated for each color according to the flowchart shown in FIG. In this case, the interpolation processing of the pixel data G is first performed in sub-step SS60. At this time, as shown in FIG.
Since the square lattice RB perfect checkerboard pattern is used,
A pixel having existing pixel data G is represented by a solid square grid. Pixels having no pixel data G, that is, pixels having a color different from the color G while having corresponding pixels of the virtual light receiving element and existing pixel data, are represented by a square grid indicated by broken lines. A pixel having no pixel data G is called a virtual pixel. For the interpolation processing, existing pixel data is used four by four.

FIG. 19 shows this relationship specifically. As shown in the pattern of FIG. 19, the virtual pixels G ₁₂ , G ₁₄ , G ₁₆ , G ₂₁ to G _21
In the case of interpolating one line of ₂₆ , G ₃₂ , G ₃₄ , G ₃₆ , the interpolation processing is performed by four adjacent pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃ and pixel data G ₁₃ , G ₁₅ , G ₃₃ , G _35, etc. are used. The calculation is also performed using the high-frequency luminance data in FIG. 17 corresponding to the pixel data G used for the interpolation. For example, interpolated pixel data G ₂₁ virtual pixel is interpolated using the high-frequency luminance data of existing data and high-luminance data and the position to be interpolated corresponding to two pixels of the same column

[0105]

G ₂₁ = (G ₁₁ + G ₃₁ ) / 2− (Y _h11 + Y _h31 ) / 2 + Y _h21 (13) Using the equation of Equation (26), the virtual pixel G
₂₃ can be interpolated. Also, the interpolation of the virtual pixel G _12, using the high-frequency luminance data of existing data and high-luminance data and the position to be interpolated corresponding to two pixels of the same row

[0106]

G ₁₂ = (G ₁₁ + G ₁₃ ) / 2− (Y _h11 + Y _h13 ) / 2 + Y _h12 (14) Using the calculation formula of Expression (27), the virtual pixel G
₃₂ can be interpolated. And pixel data located at the center of each of the four pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃
G _22, using the pixel data and the high luminance data of these four positions

[0107]

Obtained from Equation 12] _{G 22 = (G 11 + G} 13 + G 31 + G 33) / 4- (Y h11 + Y h13 + Y h31 + Y h33) / 4 + Y h22 ··· (15) . Using the calculation formula of Expression (28), the virtual pixel G
₂₃ can be interpolated. Pixel data G ₁₃ , G ₁₅ ,
G _33, when the G ₃₅ the four regarded as a set of data interpolation, already because the pixel data G ₂₃ are calculated, by calculating the pixel data _{G 14, G 34, G 25} , G 25 remaining Good. By repeating this process, a plane image of the pixel data G is created. However, the outermost edge of the plain image is
Since such a relationship does not occur, it is preferable to set a boundary value when performing strict interpolation. In consideration of the effective screen, the data in the peripheral portion is out of the range of the effective screen, and thus need not be calculated.

Next, the calculation of the pixel data R is performed in the sub-step SS.
Perform at 62. Also in this case, the pixels corresponding to the existing data and the pixel data calculated by the calculation are represented by a solid square grid, and the virtual pixels are represented by a broken square grid. Pixel data
Existing pixel data in R 1 is represented by R ₀₀ ,
_{R 04, R 22, R 26} , R 40, R 44 only. In this case, in the sub-step SS62, the pixel data obliquely adjacent to the virtual pixel to be interpolated and the high-frequency luminance data of FIG. 17 corresponding to this position are used. For example, the pixel data R _11, using the pixel data R _00, R ₂₂ and high-luminance data Y _h00, Y _h22,

[0109]

R ₁₁ = (R ₀₀ + R ₂₂ ) / 2− (Y _h00 + Y _h22 ) / 2 + Y _h11 (16) Similarly, virtual pixels R ₁₁ , R ₃₁ , R _{33
Are the} same as the pixel data R ₀₄ , R ₂₂ ,
It is calculated by applying the pixel data R ₄₀ and R ₂₂ and the pixel data R ₄₄ and R ₂₂ . If an existing pixel data R ₂₆ is also calculated in consideration, it is possible to also create the virtual pixel R _15, R ₃₅ by the adjacent diagonal interpolation process. The result is shown in FIG.

Next, in sub-step SS64, the pixel surrounded by the pixel calculated in the immediately preceding sub-step SS62 is set as the pixel to be interpolated, and the four pixels calculated in the interpolation are calculated.
Interpolation processing is performed using the two pixel data and the high-frequency luminance data at that position. For example, as can be seen from FIG. 21 around the pixel data R ₂₄ , the surrounding pixel data R ₁₃ , R ₁₅ ,
Using the data of the positions of R ₃₃ and R ₃₅ ,

[0111]

Equation 14] _{R 24 = (R 13 + R} 15 + R 33 + R 35) / 4- calculated by _{(Y h13 + Y h15 + Y} h33 + Y h35) / 4 + Y h24 ··· (17) You. When a relationship equivalent to Expression (29) is obtained from peripheral pixels, pixel data R _02, R ₂₀ , and R ₄₂ are obtained by performing interpolation, as shown in FIG. In other words, from the viewpoint of the pixel to be interpolated, all the pixel data used for the interpolation are positioned obliquely.

Next, in sub-step SS66, pixel data obtained so far is used, and among these pixels, interpolation is performed from pixel data located at the top, bottom, left, and right with respect to the interpolation target pixel. For example, using a high-frequency luminance data of four pixel data and its vertical and horizontal positions around the pixel data R ₁₂

[0113]

Equation 15] _{R 12 = (R 02 + R} 11 + R 13 + R 22) / 4- calculated by _{(Y h02 + Y h11 + Y} h13 + Y h22) / 4 + Y h12 ··· (18) You. Pixel data with similar positional relationship
R ₁₄ , R ₃₂ , R ₁₄ , and R ₃₄ can be calculated by substituting the data corresponding to Expression (17). Further, if pixels continue on the right side of FIG. 22, pixel data R ₁₆ and R ₃₆ can be calculated.

Note that, as shown in FIG. 23, uninterpolated virtual pixels remain in the peripheral portion, so that the virtual pixels may be surrounded by, for example, three pixels. Also in the case of this interpolation, if the above-described interpolation method is used, the pixel data R ₀₁ of the virtual pixel becomes

[0115]

Equation 16] _{R 01 = (R 00 + R} 02 + R 11) / calculated by _{3- (Y h00 + Y h02 +} Y h11) / 3 + Y h02 ··· (19). In this way, the pixel data R ₀₃ , R ₀₅ , R ₁₀ , R ₃₀ , R ₄₁ , R ₄₃ , and R ₄₅ are interpolated.
Finally, the entire plane screen for the pixel data R is interpolated.

Next, an interpolation process for the pixel data B is performed in sub-steps SS68, SS70 and SS72. Sub-step SS
68, SS70, and SS72 are adjacent oblique interpolation processing on pixel data B, central interpolation processing using four pieces of interpolation data, and central interpolation processing using four pixels (up, down, left, and right). These interpolation processes are based on the above-described interpolation process of the pixel data R (that is, sub-steps SS62, SS64, and SS66). This can be seen from the relationship between the pixel arrangement of the pixel data R in FIG. 20 and the pixel data B in FIG. That is, the pixel arrangement of the pixel data B in FIG. 24 is such that the pixel data R in FIG. 20 is shifted by two columns in the horizontal (that is, row) direction as a whole from the matrix display represented by the subscript of each color. I have. From this, when the virtual pixels are interpolated by applying the expressions (16) to (19) so far, the number of the subscript column of each pixel data on the right side of two or more columns in the matrix display It is good to calculate by the relation which added +2 to. For example, the pixel data B ₁₃ and the pixel data B ₃₃ are obtained by replacing the color R in the equation (29) with the color B and setting the positional relationship between the pixel data R ₀₀ and R ₃₁ to pixel data B ₀₂ and B _33.

[0117]

## EQU17 ## B _{11 + 2} = (B _{00 + 2} + B _{22 + 2} ) / 2- (Y _{h00 + 2} + Y _{h22 + 2} ) / 2 + Y _{h11 + 2} B ₁₃ = (B ₀₂ + B ₂₄ ) / 2- (Y _h02 + Y _h24 ) / 4 + Y _h13・ ・ ・ (20) B _{31 + 2} = (B _{22 + 2} + B _{40 + 2} ) / 2- (Y _{h22 + 2} + Y _{h40 + 2} ) / 4 + Y _{h31 + 2} B ₃₃ = (B ₂₄ + B ₄₂ ) / 2- (Y _h24 + Y _h42 ) / 4 + Y _h33・ ・ ・ (21) You. Further, when the number of columns in the matrix display of the pixel data to perform interpolation processing of each pixel data in less than 2 left, using the relation for calculating the pixel data R ₁₃ by using the pixel data R _04, R _22, It is better to subtract -2 from the subscript number.
For example, pixel data B ₁₁ is

[0118]

( _Equation 18) B _13-2 = (B _04-2 + B _22-2 ) / 2- (Y _h04-2 + Y _h22-2 ) / 2 + Y _h13-2 B ₁₁ = (B ₀₂ + B ₂₀ ) / 2- (Y _h02 + Y _h20 ) / 4 + Y _h11 (22) A similar relationship holds in other equations (17) to (19). Be aware of this relationship
When the interpolation processing is performed in 70 and SS72, plane interpolation development for the pixel data B can be performed. After this processing, the flow advances to sub-step SS74.

In sub-step SS74, it is determined whether or not all the plane interpolation development has been completed for each color. When a series of processing is not completed yet (No), sub-step
Return to SS60 and repeat the process. Note that this confirmation processing may be performed for each color. When a series of processing is completed (Yes), the process returns to the return. After this transition, the processing of the subroutine SUB6 ends, and the process proceeds to the subroutine SUB3.

Here, the data and the like obtained by the processing of the subroutine SUB6 will be described with reference to FIG. 8 showing the frequency band of the signal, and the signal will be described again. The horizontal axis is the horizontal frequency axis (f _h ), and the vertical axis is the vertical frequency axis (f _V ). From the spatial frequency display in the honeycomb arrangement of FIG. 8, it is represented by a solid line RB distribution according to a relationship arranged in a two-pixel pattern in which R / Bs are completely alternately switched in a checkered pattern. On the other hand, the frequency of the pixel G is high because the pixel G is arranged in a stripe pattern with four pixels including pixel shift, and the frequency of the pixel R / B is included in the range. The spatial frequency of pixel G passes through the horizontal and vertical frequency axes at approximately f _s / 4. High frequency luminance signal Y _h obtained by the interpolation process, pixel G, include R / B, and extend the bandwidth to f _s / 2. As described above, the recording signal processing unit 244a raises the frequency to a higher frequency and uses this signal to generate RGB plane data whose image quality is further improved as compared with the image displayed by the signal of the display signal processing unit 242a. Matrix part
Output to 24b.

Next, the operation of the subroutine SUB3 will be described with reference to FIG. In sub-step SS30, the mode corresponding processing unit
Matrix processing is performed in the matrix unit 24b using the RGB data output from 24a3. By this matrix processing, luminance data Y and color difference data (RY) and (BY) are generated.
After this processing, the flow advances to sub-step SS32. Sub-step SS
At 32, the aperture is adjusted. This processing is performed in sub-step SS72. The aperture adjustment is performed by the aperture adjustment unit 24c in FIG. In this way, the luminance data Y 2 with the aperture adjustment is obtained. After this processing, the flow advances to sub-step SS34.

In the sub-step SS34, the supplied luminance data Y and color difference data (RY) and (BY) are subjected to LPF processing (anti-aliasing filter processing) over a wide band. By this processing, generation of aliasing distortion is suppressed. This process is performed by the LPFs 240d, 242d, and 244d. Through this processing, the luminance data Y and the color difference data signal (RY) = C _r ,
(BY) = C _b is obtained. After this processing, the process proceeds to the return, ends the subroutine SUB3, and returns to the main routine of the solid-state imaging device 10.

By performing the above-described processing as described above, the obtained image pickup signal can be converted into a signal (data) having a higher resolution than the original resolution. Can be satisfactorily suppressed. Thereby, the image quality of the obtained image can be improved.

The solid-state imaging device 10 can be applied to a multi-plate system. For example, a description will be given of a case of a two-plate system in which an object image is projected onto at least two imaging surfaces by an optical system. Here, the light receiving elements forming the imaging surface are two-dimensionally arrayed in the row direction and the column direction, and one two-dimensional array is formed when the same object image projected on the imaging surface is spatially superimposed. The center of the corresponding geometric imaging surface shape of the arranged light receiving element and the other two-dimensionally arranged light receiving element is arranged so as to be shifted by a half pitch in the row direction and the column direction.

Here, a description will be given of a specific example of a multi-plate color filter in which a plurality of combinations using a honeycomb arrangement are used. First, a plurality of prisms (not shown) are joined to the optical system as spectral means, and light transmitted from the lens is transmitted to the imaging unit 18.
The light is sent to a light receiving unit on each optical path provided in the optical path. A color filter CF is provided in front of each light receiving unit. Here, when the light receiving section is a two-plate type, the color filter CF uses a color filter CF1 and a color filter CF2.

In the case of the first color filter CF, the spatial arrangement of the colors of the color filter CF1 shown in FIG. 26A is a honeycomb arrangement in which the colors of G ₁ and R are alternately arranged for each row. Spatial arrangement of colors of color filters CF2 shown in FIG. 26 (b) using alternately the color of G ₂ and B in each row, and we arranged color B to color line in G ₁ of the color filter CF1 Use honeycomb arrangement. For example, color filter CF
When the color filter CF2 is attached to the color filter CF1 by shifting it by one pixel pitch in the row direction, the color filter CF shown in FIG. 26 (c) can be easily formed into a honeycomb type G square lattice RB complete checkerboard pattern. In FIG. 26 (c), for convenience, the G stripe RB
Expressed in a complete checkered pattern.

In the case of the second color filter CF, FIG.
As shown in (d), the honeycomb-arranged color filter CF1 is composed of only G colors, and the spatial arrangement of the color
As shown in FIG. 26 (e), a honeycomb arrangement is adopted in which the colors R and B are alternately arranged for each row. Again, for example, color filters
The color filter CF2 is attached to CF1 with a shift of one pixel pitch in the row direction. By this bonding, the second color filter CF can be easily formed into a Bayer pattern (see FIG. 26 (f)).

As the third color filter CF, the above-described color filters CF1 and CF2 of the first color filter are used (FIG. 27).
(See (a), (b)). The third color filter CF is a color filter CF
By bonding the pixel positions of 1, CF2 so as to be completely coincident with each other, a pattern of a honeycomb arrangement shown in FIG. 27 (c) is formed. The color filter CF used in this manner is arranged on the front surface of the light receiving element corresponding to the light receiving element, and has a color arrangement that becomes a color filter of three primary colors RGB when spatially superimposed. When the above-described signal processing is performed on the imaging signal having such a relationship, a similar effect, that is, high image quality,
In addition, an image in which false colors are suppressed is obtained.

Further, the solid-state imaging device 10 may be used in a frame sequential system. This type of solid-state imaging device 10 has two light receiving elements.
An imaging unit 18 arranged in a three-dimensional manner, a moving mechanism 31 for moving, for example, in a horizontal (X) direction and a vertical (Y) direction in a two-dimensional plane parallel to the imaging surface of the imaging unit 18, a subject and an imaging unit A plurality of color filters CF arranged in a honeycomb arrangement disposed between the optical paths with, for example, immediately before the imaging unit 18, and a color filter for selecting one color filter from the plurality of color filters CF and switching the insertion between the optical paths. A switching mechanism 32, a recording / reproducing unit 33 for recording the output from the imaging unit 18, and a color signal processing unit 32 for performing signal processing on the outputs of the imaging unit 18 and the recording / reproducing unit 33 are provided.

Here, the imaging unit 18, the plurality of color filters CF, and the color signal processing unit 32 are configured in the same manner as the configuration described in the above embodiment. In particular, the color filter CF
Since the image capturing unit 18 is moved by the moving mechanism, the image capturing area is formed larger than the image capturing area so that the image capturing surface is covered even when the image capturing unit 18 is moved.

The moving mechanism 31 performs the minute movement of the image pickup section 18 according to the type of the color filter CF to be used. That is, the movement is performed in accordance with the pixel shift of the color filter CF. When minute control is difficult, the relationship between the pixel pitches to be shifted has periodicity, so that the adjustment may be performed by setting an appropriate movement amount. The moving mechanism 31 includes, for example, a driving unit 31a such as a stepping motor, a gear 31b transmitting a driving force from the driving unit 31a, and a gear 31b in order to move the imaging unit 11 with high accuracy in the XY directions. A rack 31c that meshes and converts the rotational motion of the gear 31b into a translational motion
And a translation control unit 31d for controlling the operation of the driving unit 31a of the stepping motor, and the imaging unit 18 is mounted on the rack 31c.
Is placed. The imaging unit 18 is connected to a translation control unit 31d, a recording / reproducing unit 33, and a color signal processing unit 32 via a flexible substrate or the like so that signals can be input and output even when the imaging unit 18 is moved. With this configuration, the color filter
The imaging section 18 can be moved according to the type of CF.

The color filter switching mechanism 32 includes a color filter installation section in which a plurality of color filters CF are provided at regular intervals on a disk at regular angles in consideration of the number of filters and at the same radial position so as to allow transmission. 32a, a motor 32b as a rotation drive unit for rotating the disk around the center of rotation, and a motor
A rotation control unit 32c for controlling the operation of 32b is used. The filter opening surface of the color filter installation section 32a may be disposed so as to be kept parallel to the imaging surface. The rotating shaft 32d attached to the motor is inserted through a through hole formed at the center of the disk of the color filter installation portion 32a. The rotating shaft 32d is fixed to the disk by, for example, applying an adhesive or the like to the through hole. The rotation control unit 32c controls the rotation start and the rotation end so as to perform the rotation drive at a certain angle. By performing such control, the color filter CF can be switched as required.

The pixel data is supplied to the recording / reproducing section 33 via, for example, a SCSI interface or the like. The recording / reproducing unit 33 records a signal obtained from the imaging unit 18 as pixel data each time according to a combination of operations of the color filter switching mechanism and the moving mechanism. Therefore, imaging is performed a plurality of times to create one image. The pixel data thus obtained is recorded in the recording / reproducing unit 23, and when the pixel data reproduced from the recording / reproducing unit 33 is supplied to the color signal processing unit 12, color reproduction is performed by performing any of the above-described embodiments. -A high-quality image with an emphasis on resolution can be obtained. As described above, when the solid-state imaging device 10 is compatible with color imaging, since the color filter CF and the imaging unit 18 in the honeycomb arrangement are used,
The complicated arrangement and adjustment in the manufacturing process can be avoided, and the necessity of providing an on-chip color filter provided on the imaging device (light receiving device) can be eliminated.

The color imaging apparatus is not limited to the single-panel type, but may be a two-panel type using two sets of color filters and image pickup units arranged in a honeycomb configuration. It can be regarded as a color image pickup apparatus of a frame sequential type as an image pickup unit. Each time the image is moved, a relationship is obtained in which the center of the geometric imaging surface shape is shifted by a half pitch in the row direction and the column direction with respect to the image captured immediately before. In this case as well, the above-described signal processing is performed on the supplied imaging signal to generate an image with high image quality and suppressed false colors (generation of false signals).

With such a configuration, in the case of a honeycomb type G square lattice RB complete checkerboard pattern, a reduction in the number of pixels due to this readout processing is suppressed as compared with the conventional high-speed readout, so that the arrangement is originally performed. The image data is generated so that the number of pixels of the color G is about the same, and the reading speed is not reduced.
The image quality of the provided image can be improved as compared with the related art. In addition, while trying to read at high speed and improve the image quality,
Image data can be read without reducing the reading speed in consideration of the generation of a false signal derived from the pattern of the color separation filter. As a result, both high-speed reading of image data and improvement of image quality can be satisfied.

Further, by performing the operation in accordance with which one of the high-speed readout and the high-quality readout is prioritized, it is possible to perform a process corresponding to the user's request.

In addition, the color separation filter can be constituted not only by a single plate but also by a plurality of plates, and even in a solid-state image pickup device of a plane sequential system, processing for high-speed reading and high image quality can be performed in the same manner as described above. it can.

[0138]

As described above, according to the solid-state imaging device of the present invention, light is received by the light receiving element of the imaging means provided corresponding to the color separation means, and the obtained imaging output is converted by the digital conversion means. The image data is converted into digital signal data and stored in the storage means. Thereafter, an image to be read is selected in the storage unit by the image selection unit. The storage unit outputs the selected signal to the signal processing unit. In the signal processing means, of the setting of the image resolution or the image reading speed, the signal processing according to the priority mode is performed by the control means, so that the user performs the signal processing according to the request of the user who has higher priority. Output.
In particular, image data is generated such that the number of pixels due to this readout process is suppressed from being reduced as compared with the conventional high-speed readout so that the number of pixels is originally about the number of pixels of the color G, and image generation is performed. Since the reading speed does not decrease, the image quality of the provided image can be improved as compared with the related art while the image display speed is high.

Further, according to the image data generating method of the present invention, a mode is set in consideration of the resolution of the final image or the image reading speed, the color separation is performed, and the obtained signal charges are represented by the level voltage. And read it out. After converting this signal into digital data, a plurality of images are stored, and the stored captured image is selected for reading. Since the selected image is subjected to the signal processing according to the set mode, the user can perform the signal processing according to the request that the user has higher priority and output the processed image. In particular, since the primary color RGB is read by reading control while generating an image while suppressing the decrease in the number of pixels due to this reading process as compared with the conventional high-speed reading, the reading speed is not reduced and the image display is performed at high speed. However, the image quality of the provided image can be improved as compared with the related art.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration when a solid-state imaging device according to the present invention is applied to a digital still camera.

FIG. 2 shows a honeycomb type G used for the color separation filter of FIG.
FIG. 3 is a schematic diagram showing an arrangement of a square lattice RB perfect checkerboard pattern.

FIG. 3 is a block diagram illustrating a schematic configuration of a recording signal processing unit in FIG. 1;

FIG. 4 is a block diagram illustrating a schematic configuration of a plane interpolation developing unit in FIG. 3;

FIG. 5 is a main flowchart for explaining the operation of the digital still camera in FIG. 1;

FIG. 6 is a flowchart illustrating an operation of a subroutine SUB1 in the main flowchart of FIG.

7A and 7B are schematic diagrams illustrating pixel data reading processing of a subroutine SUB1 in FIG. 6; FIG. 7A illustrates a pixel arrangement including a color attribute of an imaging unit; and FIG. It is a schematic diagram which shows the relationship of.

FIG. 8 is a graph showing a relationship between spatial frequencies of image data for each color generated by the digital still camera of FIG. 1;

FIG. 9 is a block diagram illustrating a schematic configuration of a modification of the display signal processing unit in FIG. 1;

10A and 10B are schematic diagrams showing a relationship between pixel data used in calculation of colors R and B performed in the display signal processing unit in FIG. 9; FIG. 4 is a schematic diagram illustrating the relationship between

FIG. 11 is a flowchart illustrating an operation of a subroutine SUB2 in the main flowchart of FIG.

FIG. 12 is a flowchart illustrating an operation of a subroutine SUB4 used in the subroutine SUB2 of FIG.

FIG. 13 is a flowchart illustrating an operation after each connector in a subroutine SUB4 of FIG. 12;

14A is a schematic diagram illustrating the relationship between a real pixel and a virtual pixel, more specifically, the pixel arrangement of FIG. 7A; FIG.
(b) is a schematic diagram showing pixels for which luminance data Y is created.

FIG. 15 is a flowchart illustrating an operation of a subroutine SUB5 used in the subroutine SUB2 of FIG. 11;

FIG. 16 shows an LP performed in subroutine SUB5 of FIG.
It is a schematic diagram explaining F process.

17 is a schematic diagram illustrating the relationship between the high-frequency luminance data Y _h generated in the subroutine SUB5 in Figure 15.

FIG. 18 is a flowchart illustrating an operation of a subroutine SUB6 used in the subroutine SUB2 of FIG.

19 is a schematic diagram showing a positional relationship between a pixel to be interpolated with respect to pixel data G in a subroutine SUB6 of FIG. 11 and an existing pixel.

20 is a schematic diagram showing a positional relationship between a pixel to be interpolated with respect to pixel data R in a subroutine SUB6 of FIG. 11 and an existing pixel.

21 is a schematic diagram showing a positional relationship when the result of the adjacent oblique interpolation processing is added to the positional relationship of FIG. 20;

22 is a schematic diagram illustrating a positional relationship when the result of the interpolation process using four pixel data obtained by the adjacent oblique interpolation process is added to the positional relationship of FIG. 21;

FIG. 23 is a schematic diagram showing a positional relationship when a result of performing interpolation processing using pixel data positioned vertically, horizontally, and horizontally with respect to a pixel to be interpolated is added to the positional relationship of FIG. 21;

24 is a schematic diagram showing a positional relationship between a pixel to be interpolated and an existing pixel regarding pixel data B in a subroutine SUB6 of FIG. 11;

FIG. 25 is a flowchart illustrating an operation of a subroutine SUB3 in the main flowchart of FIG. 5;

FIG. 26 is a schematic diagram showing a relationship between color pattern arrangements used when a color filter applied to the solid-state imaging device of the present invention has a two-plate configuration and spatial arrangements obtained by combining these pattern arrangements.

FIG. 27 is a schematic diagram showing a relationship between color pattern arrangements used when a color filter applied to a solid-state imaging device has a two-plate configuration and spatial arrangements obtained by combining these pattern arrangements, similarly to FIG. 26; .

[Explanation of symbols]

 10 Digital still camera 12 Optical system 14 Release shutter 16 System control unit 18 Imaging unit 20 A / D conversion unit 22 Buffer memory 24 Signal processing unit 26 Mode setting unit 28 Image selection unit 24a Mode-compatible processing unit 24b Matrix unit 24c Aperture adjustment unit 24d filter unit 240a output destination switching unit 242a display signal processing unit 244a recording signal processing unit 2420 address control unit 2440 luminance data generation unit 2442 high-frequency luminance data generation unit 2444 plane interpolation development unit

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C051 AA01 BA03 DA06 DB01 DB23 DC02 DC04 DE11 EA01 5C065 AA03 BB10 BB48 CC01 CC02 CC03 DD02 DD15 EE06 GG03 GG13 GG18 GG21

Claims (17)

[Claims] 1. A light condensing means for condensing incident light from an object scene, and a color filter for separating RGB light into three primary colors, which are incident through the light condensing means, are arranged in a color G square lattice. Color separation means arranged in a complete checkered pattern in which the same color R or B is arranged at diagonal positions across the color G; and the incident light which is arranged in the row and column directions and color separated by the color separation means. A plurality of light receiving elements that receive light and generate signal charges corresponding to the incident light, wherein each of the plurality of light receiving elements is a light receiving element with respect to an adjacent light receiving element. The centers of the geometric shapes of the light receiving elements are arranged in the row direction and / or the column direction so as to be shifted from each other by a distance corresponding to a half of the pitch of the light receiving elements. Digital conversion means for converting, and the digital conversion means And a storage unit for storing the data for a plurality of images from among the plurality of modes set for the captured images from the storage unit, according to the setting of the resolution of the image or the reading speed of the image. Mode setting means for performing setting, image selecting means for selecting an image when reading the image from the storage means, signal processing means for performing signal processing according to the mode set for the image selected by the image selecting means And a control means for controlling the image processing means and the digital conversion means, and controlling the signal processing means in accordance with information supplied from the mode setting means and the image selection means, respectively. Imaging device. 2. The apparatus according to claim 1, wherein said mode setting means includes: a first mode for displaying an output from said storage means to said signal processing means instantaneously; A solid-state imaging device, wherein a second mode is set in which an output from the storage unit is used for either display with a high resolution or recording on a recording medium for recording the image as compared with the mode. 3. The apparatus according to claim 1, wherein the signal processing means performs signal processing by sampling when reading an image captured from the storage means in the first mode. Generating high-frequency luminance data using the output from the storage means in the second mode, and using the luminance data and data supplied via the digital conversion means to generate a luminance signal between light receiving elements. A second signal processing unit that performs signal processing that also interpolates data; an output destination selection unit that switches an output of the storage unit under the control of the control unit and supplies the output to the second signal processing unit; Matrix means for generating luminance data and chrominance data using the output from the signal processing means or the second signal processing means; and among the outputs from the matrix means, A solid-state imaging device comprising: aperture adjustment means for performing contour enhancement processing on luminance data; and filter processing means for band-limiting each of the output from the aperture adjustment means and the color difference data from the matrix means. . 4. The apparatus according to claim 3, wherein said first signal processing means includes a color signal stored in said storage means.
A first extracting unit that is arranged obliquely with respect to G and samples the colors R and G arranged in the same row direction as combination data of three primary colors at the position of the color G; The solid-state imaging device, wherein the extraction unit performs sampling by overlapping the combination data in which at least one of the colors R and G of the combination data is adjacent. 5. The apparatus according to claim 3, wherein said first signal processing means includes a color storage means for storing a color stored in said storage means.
Second extracting means for sampling data of the same color R or B arranged obliquely with respect to G as data at the position of the color G, and using the data extracted by the second extracting means, RB calculation means for calculating the colors R and B respectively by the addition and averaging of the solid-state imaging device, wherein the value calculated by the RB calculation means is combined data representing the three primary colors at the position of the color G. apparatus. 6. The apparatus according to claim 3, wherein the second signal processing means is configured to perform the sampling of a signal from the light receiving element in the second mode based on an image signal from the light receiving element. While performing signal processing to interpolate pixel data corresponding to the gap position of the light receiving element, when generating luminance data and color data from pixel data including the interpolated pixel data, based on the output from the storage means First brightness calculating means for creating brightness data at a position where the light receiving element is present; and empty space of the light receiving element based on the horizontal and / or vertical brightness data from the first brightness calculating means. A second brightness calculating means for creating brightness data in the virtual light receiving element when the virtual light receiving element is used; and a brightness data created by the second brightness calculating means. A solid-state imaging device, comprising: plane calculation means for creating plane data of the entire screen in each color using each color data of RGB obtained from the light receiving element. 7. The apparatus according to claim 1, wherein the optical system splits the optical path of the incident light into a plurality of light paths, and separates the spectral light from the spectral means into colors. A plurality of image pickup means for receiving light; a color separation means corresponding to the image pickup means is used as a color filter for each single color or a color filter combining a plurality of colors; G A solid-state imaging device which is arranged in a square lattice pattern and has a complete checkerboard pattern in which the same color R or B is arranged diagonally across the color G 1. 8. The apparatus according to claim 1, wherein a color filter switching means for switching a color filter means inserted between the optical system and the two-dimensionally arranged light receiving elements; A moving means for moving a light receiving unit two-dimensionally arrayed in the light receiving element within a two-dimensional plane parallel to an imaging surface formed by the light receiving element; and Recording means for recording a photographed image in a frame-sequential manner each time the image is moved; and And a relationship shifted by a half pitch in the column direction is obtained, and the spatially superimposed color patterns are arranged in a G square
A solid-state image pickup device characterized by a complete checkerboard pattern in which colors RB are arranged diagonally between G 1. 9. Each of a plurality of light-receiving elements prepared for converging incident light from an object scene, passing through a color filter that separates the incident light into three primary colors RGB, and further photoelectrically converting the incident light. However, with respect to adjacent light receiving elements, the center of the geometrical shape of each light receiving element is shifted in the row direction and / or
Or an image data generating method for generating image data of the object scene based on signal charges obtained by a light receiving element, which is arranged by being shifted by a distance corresponding to a half of a pitch of the light receiving element in a column direction. The method includes: a mode setting step of setting one mode in consideration of a final image resolution or an image reading speed among a plurality of modes to be set for the captured image; and The color G is arranged in a square lattice, and the color G is separated by a complete checkerboard pattern in which the same color R or B is arranged diagonally across the color G of the color filter, and the signal charge obtained by the light receiving element is An imaging step of converting and reading out a signal represented by a level voltage; a digital conversion step of converting an output obtained in the imaging step into digital signal data; An image storage step of storing data for a plurality of images; an image selection step of selecting a captured image stored in the image storage step; and an image corresponding to a mode set in the image selected in the image selection step. A signal processing step of performing signal processing to generate image data. 10. The method according to claim 9, wherein the mode setting step includes: a first mode for displaying the captured image instantaneously; and a mode from the storage means as compared to the first mode. An image data creation method, comprising setting a second mode in which output is used for either display with high resolution or recording on a recording medium for recording the image. 11. The method according to claim 9, wherein said signal processing step switches an output destination of data stored in said image storage step in accordance with said first mode and said second mode. A first switching step; a first signal processing step of performing sampling when reading an image captured in the image storage step in the first mode; and an output obtained by the image storage step in the second mode. A second signal processing step of generating a high-frequency luminance signal by using the luminance signal and data supplied from the image storage step, and performing signal processing in which data between light-receiving elements is also interpolated, A matrix step of generating luminance data and color difference data using an output obtained in the first signal processing step or the second signal processing step; An aperture adjusting step of performing an outline emphasizing process on the luminance data among the outputs from the stages; and a filtering step of performing band limiting on the output from the aperture adjusting step and the color difference data from the matrix means. A method for creating image data, comprising: 12. The method according to claim 11, wherein the first signal processing step is arranged obliquely with respect to the color G stored in the image storing step and in the same row direction. A first extraction step of sampling the arranged colors R and G as combination data of the three primary colors at the position of the color G, wherein the first extraction step is such that at least one of the colors R and G of the combination data is adjacent An image data creation method characterized by performing sampling by duplicating combination data to be processed. 13. The method according to claim 11, wherein the first signal processing step includes the processing of the same color R or B data obliquely arranged with respect to the color G stored in the image storing step. A second extraction step of performing sampling as data at the position of the color G; and an RB calculation step of calculating colors R and B by an averaging of the data using the data extracted in the second extraction step. An image data creation method, wherein the value calculated in the RB calculation step is combination data representing three primary colors at the position of the color G. 14. The method according to claim 11, wherein the second signal processing step includes the step of sampling the signal from the light receiving element in the second mode based on an image signal from the light receiving element. While performing signal processing to interpolate pixel data corresponding to the gap position of the light receiving element, when generating luminance data and color data from pixel data including the interpolated pixel data, based on the output from the digital conversion step A first luminance calculating step of creating luminance data at a position where the light receiving element is present; and a vacant area of the light receiving element based on the horizontal and / or vertical luminance data from the first luminance calculating step. Is a virtual light receiving element, a second luminance calculating step of creating luminance data in the virtual light receiving element; Image data generating method characterized by comprising a plane calculating step of creating a plain data of the entire screen in each color by using respective color data of RGB obtained from the light receiving element and the luminance data. 15. The method according to claim 9, wherein the light path of the incident light from the object scene is divided into a plurality of light paths, and the incident light is supplied by the light separation of the light separating step. Including a plurality of imaging steps for receiving color-separated light, respectively, using a color filter for each single color or a color filter combining a plurality of colors corresponding to the color separation processing performed in the imaging step, spatially A method of producing image data, characterized in that a superimposed color pattern is arranged in a G square lattice pattern, and a complete checkered pattern is formed in which the same color R or B is arranged diagonally across the color G. 16. The method according to claim 9, wherein an optical system for condensing incident light from the object scene is used, and a light receiving element arranged two-dimensionally with the optical system. A color filter switching step of switching a color filter to be subjected to color separation by inserting between the two, and a movement of moving a light receiving unit in which the light receiving elements are two-dimensionally arranged in a two-dimensional plane parallel to an imaging surface formed by the light receiving elements. And a recording step of recording a captured image in a frame-sequential manner each time the movement is performed while moving the light receiving unit a plurality of times in the moving step, and each time the movement is performed, the captured image is moved forward by one. The relationship obtained by shifting the center of the geometrical imaging surface shape by half a pitch in the row direction and the column direction with respect to the image captured at a time is obtained, and spatially superimposed color patterns are arranged in a G square lattice shape. ,color
A method of creating image data, wherein a complete checkered pattern in which the same color R or B is arranged diagonally across G. 17. The method according to claim 14, wherein the plane operation step includes the step of converting the G square grid and the RB perfect checker pattern shifted by half the pitch with respect to the G square grid by the digital conversion step. Using the pixel data of each color obtained corresponding to the actual light receiving element and the luminance data created in the second step, the plane interpolation of G of the pixel data is performed horizontally with respect to the interpolation target pixel. The difference between the average of the actually obtained pixel data G adjacent in the direction and / or the vertical direction and the average of the luminance data adjacent in the horizontal and / or the vertical direction to the interpolation target pixel And a step obtained by adding luminance data corresponding to the position of the pixel to be interpolated. The difference between the average of the actually obtained pixel data R and the average of the luminance data adjacent to the interpolation target pixel in the same direction as the oblique direction corresponds to the position of the interpolation target pixel. A first R step obtained by adding the luminance data to be processed, and an averaging of pixel data R 1 for the remaining color R of the pixel to be interpolated obtained by the first R step, A second R step obtained by adding the luminance data for the remaining color R of the interpolation target pixel to the difference between the average of the luminance data located at the same distance and the pixel data for the remaining color R of the interpolation target pixel. The difference between the first and second R steps located closest to the data R and the averaging of the actually obtained pixel data R and the averaging of the luminance data corresponding to the pixel used for the averaging is calculated as The position of the pixel to be interpolated is A third R step obtained by adding luminance data to the pixel data. The method further comprises: Add the luminance data corresponding to the position of the interpolation target pixel to the difference between the obtained average of the pixel data B and the average of the luminance data adjacent to the interpolation target pixel in the same direction as the oblique direction. And the pixel data B for the remaining color B of the pixel to be interpolated is obtained by adding the pixel data B at the same distance obtained by the first A second B step obtained by adding the luminance data for the remaining color B of the pixel to be interpolated to the difference between the average of the luminance data located and the pixel data B for the remaining color B of the pixel to be interpolated; The first, located at In the second step B, the difference between the average of the pixel data B actually obtained and the average of the luminance data corresponding to the pixels used for the average is calculated by adding the luminance data corresponding to the position of the pixel to be interpolated. A third B step obtained by the addition.