JPH07283999A - Image synthesizer and image photographing device - Google Patents

Abstract

PURPOSE:To obtain the image signal of an image with high resolution by circuit configuration with low cost and low power consumption. CONSTITUTION:The motion vector of the image photographed by moving an active prism 1 is detected by a motion vector detection circuit 6, and it is sent to a microcomputer 8. The microcomputer 8 finds moving quantity based on a detected motion vector, and the image information of the image photographed by moving by found moving quantity is moved by a memory controller 9. Such image information is stored in memory 7 for high resolution still picture. The information is added on the image information of an original image photographed by moving.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for synthesizing a still image using a video signal of a photographed image and an image photographing apparatus having a high resolution photographing mode.

[0002]

2. Description of the Related Art Conventionally, as a method for obtaining an image signal of a higher resolution image by using a video signal of an image of a normal resolution taken by a handy type image pickup device such as an 8 mm video, that is, a video camera, There is a method of interpolating between pixels to synthesize a high resolution image. In this case,
It requires multiple image sensors, so-called imagers. As this image sensor, specifically, a CCD (Charge Coupled Device) which is a solid-state image sensor is used.

At the time of photographing an image, one image is divided, each divided area is imaged using a plurality of CCDs, and the range of the image of the plurality of imaged divided areas is adjusted to obtain a high resolution image. Combining two images.

[0004]

By the way, since the range of the image picked up by the plurality of CCDs is equal to the range of the divided area, when the high-resolution images are combined by the above-mentioned method, the image is picked up. In order to match the range of a plurality of images, a range matching technique that accurately matches the range of images is required. It is also necessary to process the video signal of the captured image at high speed. Therefore, a circuit for synthesizing a high-resolution image using a video signal of a normal-resolution image is expensive and consumes a large amount of power.

In view of the above situation, the present invention is directed to an image synthesizing apparatus capable of synthesizing a high-resolution image with an inexpensive and low power consumption constituent circuit, and a video signal used in the image synthesizing apparatus. The present invention provides an image capturing device for capturing.

[0006]

SUMMARY OF THE INVENTION An image synthesizing apparatus according to the present invention provides image information of an image captured by moving a captured image in units smaller than one pixel of an image sensor.
The problem described above is solved by moving the image by the amount of the movement and adding the image information of the moved image to the image information of the image captured at the original imaging position.

Here, storage means for storing image information of an image taken at the original image pickup position, motion vector detection means for detecting a motion vector between fields or frames using the image information, and Discrimination for determining whether or not the moving amount is obtained using the motion vector from the motion vector detecting means, and based on this moving amount, the image information of the image photographed by moving the above is added to the image information of the original image. Means and the discrimination result from the discriminating means, when the image information of the image which is moved and photographed is added to the image information of the original image, the movement is performed by the movement amount from the discriminating means. Move the image information of the captured image,
Image control means for adding to the image information of the original image.

The first storage means for storing a representative point of the image signal of the previous field or the previous frame in the video signal, and the image signal of the previous field or the previous frame stored in the first storage means. Calculating means for calculating the difference between the representative point and the image signal of the current field or current frame including a shift smaller than one pixel; second storing means for storing the difference value from the calculating means; Calculating means for obtaining a parameter using the difference value from the storage means of
A motion vector detecting means for detecting a motion vector based on a parameter from the calculating means, and detecting a motion vector between the previous field and the current field or between the previous frame and the current frame. To do.

Further, the image photographing apparatus according to the present invention has a high resolution photographing mode in which the position of the image is moved by a unit smaller than one pixel of the image pickup device and the image is photographed in this high resolution photographing mode. The image information of the image is moved by the above-described movement amount and is sent to an image synthesizing device for synthesizing a high-resolution image in addition to the image information of the image photographed at the original image pickup position. To solve.

Here, the above-mentioned image photographing device is provided with the above-mentioned image synthesizing device, and in the high resolution photographing mode, the photographed image is subjected to signal processing by the image synthesizing device to obtain high resolution image information. It is desirable to output.

Further, in the high resolution photographing mode, the frequency range obtained by the optical low-pass filter for limiting the frequency component of the image pickup light of the photographed image is widened to the high frequency component.

Further, in the high resolution photographing mode, the image pickup device is driven to control movement of the moved and photographed image by a movement amount of a unit smaller than one pixel.

[0013]

In the present invention, the amount of movement of an image taken by moving in a unit smaller than one pixel of the image sensor is obtained,
The image information of the moved and photographed image is moved by the amount of movement and added to the image information of the original image to synthesize the image.

Further, the difference between the representative point of the image signal of the previous field in the video signal and the image signal of the current field which is the next field of this previous field, or the representative point of the image signal of the previous frame and the next point of this previous frame. The difference from the image signal of the current frame, which is a frame, is taken, a parameter is obtained from this difference value, a motion vector between two fields or between two frames is detected using this parameter, and the above movement amount is calculated from this motion vector. To do.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A preferred embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows a schematic configuration of an image synthesizing apparatus and an image capturing apparatus according to the present invention. The image synthesizing apparatus and the image capturing apparatus shown in this embodiment capture an image by moving an image sensor that receives image capturing light by one pixel, that is, a sub pixel unit that is a unit smaller than the pixel unit. The amount of motion is detected to synthesize a high-resolution still image.

Here, the principle of image composition in the image composition apparatus and the image photographing apparatus of the present invention will be briefly described below. An image is captured by utilizing blurring caused by camera shake of the image capturing apparatus or by forcibly moving the image sensor in the image capturing apparatus. When capturing an image, the image captured without movement is used as the image in the initial state, and the image information of the image subsequently moved and captured is added to the image information of the image in the initial state to obtain a high resolution image. Combine images.

Specifically, when synthesizing an image having a resolution of X times the normal resolution, the field image in the initial state is photographed by moving by 1 / X pixel of the image pickup device. 1 using the image information of one field image
Make up a pixel. At this time, if a field image after the two field images is photographed by moving the image sensor to the left, for example, the image is photographed while being shifted to the right in the screen. For example, as shown in a of FIG. 2, a field image A indicated by a circle is taken as a previous field image in an initial state which is an original image, and the field image A is shifted to the right to The field image is photographed as a field image B indicated by Δ.

For example, when the value X is 2 and the value n is 1,
When combining one pixel of an image having twice the resolution of a normal image with two pieces of image information, the field image B shown in a of FIG. 2 is photographed with a shift of 1/2 pixel from the field image A. The field image A is composed of pixel information as shown in FIG. 2B, and the field image B is shown in FIG.
It is composed of pixel information as shown in c. By constructing one pixel by using the image information of each one pixel of the field image A and the field image B, as shown in FIG.
An image having a resolution twice as high as that of the normal resolution image is synthesized as shown in d. That is, an image signal that has been moved by 1 / X pixel is obtained, and the operation of adding this image signal to the field image in the initial state is performed n times for each 1 / X to 1 pixel to obtain the horizontal resolution of the original image. It is possible to synthesize an image having a resolution of X times higher than.

Next, the structure of an embodiment of the present invention will be described. Imaging light from an image enters through an active prism 1 described later in FIG. 1 and is received by a solid-state imaging device, so-called CCD 2, and converted into an electric signal. This electric signal is sent to the camera signal processing circuit 3 to be subjected to signal processing and converted into a video signal of a luminance signal Y and a color difference signal C. This video signal is converted into a digital image signal by an A / D (analog / digital) converter 4.

The image signal is stored in the frame memory 5 for each frame. In addition, a high-resolution still image memory 7
Among the image signals sent to the frame memory 5, only the image signal of the frame image in the initial state is sent and stored in the. Further, the image signal of the frame image sent to the frame memory 5 is sent to the motion vector detection circuit 6. The motion vector detection circuit 6 detects a motion vector between frames of the sent frame image. The detected motion vector is sent to the microcomputer 8. The microcomputer 8 uses the sent motion vector to determine the amount of motion and sends the amount of motion to the memory controller 9. By the control of the memory controller 9, the image signal of the frame image for performing image combination is read from the frame memory 5. The memory controller 9 moves the read frame image by the amount of movement and adds it to the frame image in the initial state stored in the high resolution still image memory 7.

The active prism 1 will be described below. The active prism 1 is formed by enclosing a liquid having a high refractive index between two pieces of glass and connecting them with a bellows. Rotation axes are attached to the two glasses in the vertical and horizontal directions so that they can move freely. Normally, this active prism 1 is used in front of a camera lens to bend the optical axis of image-pickup light incident vertically and horizontally to correct blurring of a captured image due to camera shake or the like. However, in this embodiment, it is used to capture an image obtained by intentionally moving the captured image.
That is, at the time of capturing the image, by moving the active prism 1 after capturing the initial state of the image, the image pickup light received by the CCD 2 changes, and the image in which the position of the image in the initial state is moved is captured. Will be.

As a method of moving the original image to be photographed, in addition to using the active prism 1,
The camera body is moved by a gimbal mechanism method that mechanically moves the lens unit itself, an ABS method that moves a combination of convex and concave afocal lenses, and a piezo element that expands and contracts. It is conceivable to use a CCD drive control system, a system in which a ferroelectric is pasted to the CCD to change the optical path, or a system in which the position of the CCD is moved by liquid crystal.

In the method of moving the position of the image described above, when the position of the image is moved by a method other than the method of utilizing the blur due to the camera shake or the like, the amount of movement of the image with a certain degree of accuracy is obtained. Can be predicted in advance. Specifically, for example, when a motion indicated by an arrow in FIG. 3B is given to an image, a motion vector as indicated by an arrow in FIG. 3A is detected in each divided region obtained by dividing the image into a plurality of regions. In this case, in the plurality of divided areas, b in FIG.
It is possible to easily discriminate the movement indicated by the arrow indicating the same direction as the movement indicated by the arrow. Therefore, this predicted motion amount can be used when the microcomputer 8 detects the image motion amount. Also,
In the CCD drive control method and the method of moving the position of the CCD by using the liquid crystal, the CCD 2 can be controlled with high accuracy, so that the microcomputer 8 does not need to detect the movement amount.

Next, the operation of synthesizing high resolution images will be described in detail below. When synthesizing high-resolution images, the high-resolution shooting mode is turned on by a switch (not shown) such as a high-resolution still image selection switch.
Information is input to the mode information input terminal 14 and sent to the microcomputer 8. The microcomputer 8 sends a control signal for high-resolution image composition to the memory controller 9 based on the ON information. Further, the microcomputer 8 sends a changeover control signal to the signal changeover switch 13 in order to send the image signal output from the A / D converter 4 to the high resolution still image memory 7. As a result, the signal changeover switch 13
Is switched to the terminal a side.

On the basis of the control signal from the microcomputer 8, the memory controller 9 causes the frame memory 5 and the high-resolution still image memory 7 to display an initial image of the images to be combined, That is, a control signal for reading the image signal of the original frame image is sent. Therefore, the image signal of the original frame image via the A / D converter 4 is stored in the frame memory 5 and the high-resolution still image memory 7.

The frame memory 5 is a memory for storing the image signal of the frame image for each frame.
Therefore, under the control of the memory controller 9, the frame memory 5 sequentially stores the image signals of the frame images moved in subpixel units after the original frame image. The high-resolution still image memory 7 is a memory used when performing high-resolution image synthesis. For example, in order to synthesize an image having a resolution of X times the resolution of the original frame image, FIG. As shown, X _v subpixels can be written in the vertical direction by the number of effective pixels in the vertical direction, and X _v subpixels can be written in the horizontal direction by the number of effective pixels in the horizontal direction. A memory with a possible memory capacity. When the image signal of the original frame image is written in the high-resolution still image memory 7, each image information of the original image is written as shown by the diagonal lines in FIG.

The image signal passed through the A / D converter 4 is also sent to the motion vector detection circuit 6. The motion vector detection circuit 6 performs frame matching using the original frame image and the moved frame image to calculate a predetermined parameter. Further, a motion vector is detected from this parameter, and this motion vector is sent to the microcomputer 8.

The microcomputer 8 uses the sent motion vector to calculate the amount of pixel motion in subpixel units. Further, based on the detected motion vector, it is determined whether or not the moved frame image is combined with the original frame image stored in the high resolution still image memory 7. When performing image synthesis, the control signal and the amount of movement for performing image synthesis are output from the microcomputer 8 to the memory controller 9.
Further, the microcomputer 8 switches the signal changeover switch 13 to the terminal b side in order to send the moved frame image stored in the frame memory 5 to the high resolution still image memory 7.

The memory controller 9 sends a control signal for sending the image signal of the moved frame image to the high resolution still image memory 7 in the frame memory 5.
Send to. As a result, the image signal of the frame image for which the motion amount has been obtained is read from the frame memory 5, and this image signal is sent to the high resolution still image memory 7 via the signal changeover switch 13. In the high resolution still image memory 7, each pixel of the moved frame image is moved by the memory controller 9 by the amount of movement and is combined with the original frame image. For example, each piece of image information of the moved frame image is written next to the image information of the original frame image indicated by diagonal lines, as indicated by the vertical line in FIG. As described above, the operation of obtaining the motion amount of the moved frame image and moving and writing each pixel of the moved frame image by the amount of this motion is performed by sub-pixel information forming each pixel of the composite image. Sequentially until all are written. As a result, an image having a resolution that is X times the resolution of the original frame image is combined.

Next, the method of detecting the motion vector will be described in detail with reference to FIG. The digital image signal for each frame via the A / D converter 4 has a representative point of the image signal in the frame in the motion vector detection circuit 6, and the representative point of this image signal is written in the representative point memory 21. Be done.

Thereafter, the image signal at the representative point is read out and sent to the adder 22 shown in FIG. The image signal of the next frame of the frame in which the representative point is taken is sent to the adder 22 and the difference from the image signal of the representative point is taken. This difference value is stored in the residual memory 23. The stored difference value is sent to the parameter calculation circuit 24. The parameter calculation circuit 24 calculates parameters such as minimum value, minimum address, and average value, and these calculated parameters are sent to the microcomputer 25. The microcomputer 25 detects a motion vector using the sent parameters.

When the microcomputer 8 determines that the high resolution image is not to be combined, the signal changeover switch 12 is changed over to the terminal a side. Therefore, the image signal stored in the frame memory 5 is D through the terminal a of the signal changeover switch 12.
/ A (digital / analog) converter 10.
The image signal converted into an analog signal by the D / A converter 10 is subjected to signal processing by the video signal processing circuit 11 and output as a predetermined video signal to an external image display device or the like.

In the above embodiment, the CCD 2
An optical low-pass filter (OLF) is provided in the preceding stage to suppress in advance a frequency component higher than a frequency component that can be reproduced as an image in the imaging light from the active prism 1. Therefore, the active prism 1
There is a limit to increase the resolution of the high-frequency component of the imaging light obtained when the is moved. Therefore, the frequency characteristics of the optical low-pass filter are switched between the case of outputting the image signal of a normal image and the case of outputting the image signal of a high-resolution synthesized image, and the optical low-pass filter is switched during the high-resolution image synthesis. It is possible to extend the frequency characteristic of to the high frequency component. Specifically, the frequency characteristic of the ordinary optical low-pass filter is as shown by the line A in FIG. 6, but at the time of high-resolution image synthesis, the frequency characteristic is extended to a high frequency component as shown by the line B in FIG. 6, for example. .

Further, in the above-described embodiment, the case where the inter-frame motion vector is detected by using the frame image when detecting the motion vector is explained. However, the field image is used in place of the frame image instead of the field image. You may make it detect the motion vector between them.

[0035]

As is apparent from the above description, in the image synthesizing apparatus according to the present invention, the position of the captured image is moved in units smaller than one pixel of the image sensor, and the image of the image is captured. By moving the information by the moving amount and adding the image information of the moved image to the image information of the image taken at the original imaging position, the cost is high and the circuit configuration with low power consumption is high. An image signal of an image with resolution can be obtained. That is, a high-resolution still image can be obtained from the image captured by the single-plate color imaging device.

Further, storage means for storing image information of the image taken at the original image pickup position, motion vector detection means for detecting a motion vector between fields or frames using the image information, and the motion Discriminating means for determining the amount of movement using the motion vector from the vector detecting means, and discriminating whether or not to add the image information of the image photographed by the movement based on the amount of movement to the image information of the original image. When the image information of the moved and photographed image is added to the image information of the original image according to the determination result from the determination means, the image is moved and moved by the movement amount from the determination means. By moving the image information of the image, and having the image control means for adding to the image information of the original image, when a video signal including blur due to camera shake is input, In the image display device or a printer device comprising a line input terminal for inputting an image signal, the image becomes possible to output it to the high resolution.

Further, the first storage means for storing a representative point of the image signal of the previous field or the previous frame of the video signal, and the image signal of the previous field or the previous frame stored in the first storage means. Calculating means for calculating the difference between the representative point and the image signal of the current field or current frame including a shift smaller than one pixel; second storing means for storing the difference value from the calculating means; Between the previous field and the current field by the calculation means for obtaining the parameter using the difference value from the storage means and the motion vector detection means for detecting the motion vector based on the parameter from the calculation means, or the previous field. By detecting the motion vector between the frame and the current frame, it is possible to easily detect the motion vector between fields or frames. Kill.

Further, the image photographing apparatus according to the present invention has a high resolution photographing mode in which the position of the image is moved in a unit smaller than one pixel of the image pickup device, and the image is photographed in this high resolution photographing mode. The image information of the image is moved by the moving amount and is sent to the image synthesizing device for synthesizing the high-resolution image in addition to the image information of the image taken at the original image pickup position. Can be easily obtained.

In the high resolution photographing mode, the frequency range obtained by the optical low pass filter for limiting the frequency component of the picked-up light of the photographed image is expanded to the high frequency component, so that the high frequency component has higher accuracy. High resolution images can be combined.

In the high resolution photographing mode, the image pickup device is driven to control the movement of the moved and photographed image by a movement amount of a unit smaller than one pixel, thereby causing a shift such as a hand shake. High-resolution images can be synthesized with higher accuracy than in the case of utilizing high-resolution images to be synthesized.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of an image capturing device and an image synthesizing device according to the present invention.

FIG. 2 is a diagram for explaining the principle of combining high-resolution images.

FIG. 3 is a diagram for explaining a motion vector.

FIG. 4 is a diagram showing a state of pixel information of a high resolution image.

FIG. 5 is a diagram for explaining motion vector detection.

FIG. 6 is a frequency characteristic diagram of an optical low pass filter.

[Explanation of symbols]

 1 Active Prism 2 CCD 3 Camera Signal Processing Circuit 4 A / D Converter 5 Frame Memory 6 Motion Vector Detection Circuit 7 High Resolution Still Image Memory 8 Microcomputer 9 Memory Controller 10 D / A Converter 11 Video Signal Processing Circuit 12, 13 signal changeover switch 14 mode information input terminal

Front page continued (72) Inventor Naohisa Arai 6-735 Kitashinagawa, Shinagawa-ku, Tokyo Sony Corporation

Claims (6)

[Claims] 1. An image synthesizing apparatus for synthesizing a still image from a video signal of a photographed image, wherein an image of the image photographed by moving a position of the photographed image by a unit smaller than one pixel of an image sensor. An image synthesizing apparatus, wherein information is moved by an amount corresponding to the moving amount, and the image information of the moved image is added to the image information of the image taken at the original imaging position. 2. Storage means for storing image information of an image captured at the original image capturing position; motion vector detection means for detecting a motion vector between fields or frames using the image information; Discriminating means for determining the amount of movement using the motion vector from the vector detecting means, and discriminating whether or not to add the image information of the image photographed by the movement based on the amount of movement to the image information of the original image. According to the discrimination result from the discriminating means, when the image information of the moved and photographed image is added to the image information of the original image, the image is moved and moved by the movement amount from the discriminating means. The image synthesizing apparatus according to claim 1, further comprising image control means for moving the image information of the selected image and adding it to the image information of the original image. 3. A first storage means for storing a representative point of the image signal of the previous field or the previous frame in the video signal, and the image signal of the previous field or the previous frame stored in the first storage means. Calculating means for calculating the difference between the representative point and the image signal of the current field or current frame including a shift smaller than one pixel; second storing means for storing the difference value from the calculating means; Between the previous field and the current field by the calculation means for obtaining the parameter using the difference value from the storage means and the motion vector detection means for detecting the motion vector based on the parameter from the calculation means, or The image synthesizing apparatus according to claim 1, wherein a motion vector between the frame and the current frame is detected. 4. A high-resolution shooting mode in which the position of the image is moved in units smaller than one pixel of the image pickup device to shoot, and the image information of the image shot in this high-resolution shooting mode corresponds to the amount of movement. The image capturing apparatus is characterized in that the image capturing apparatus is moved only by the image capturing apparatus and sends the image information of the image captured at the original image capturing position to an image combining apparatus that combines a high resolution image. 5. The image according to claim 4, wherein, in the high-resolution shooting mode, a frequency range obtained by an optical low-pass filter that limits a frequency component of imaged light of a shot image is expanded to a high-frequency component. Imaging device. 6. In the high-resolution shooting mode, the image pickup device is driven so that the image taken by the movement is set to 1.
The image capturing apparatus according to claim 4 or 5, wherein movement control is performed with a movement amount of a unit smaller than a pixel.