JPH08317429A - Stereoscopic electronic zoom device and stereoscopic picture quality controller - Google Patents

Abstract

PURPOSE: To perform the segmentation and zoom processing on right and left images by using the size perceptive characteristic and depth perceptive characteristic of an observer and without changing the size feeling and depth perception of the observer so as to put an object within the both eye merging range of the observer and to display a stereoscopic image within the both eye merging range without impairing the depth feeling and size feeling of the observer. CONSTITUTION: This device is a stereoscopic electronic zoom device which comprises a segmentation area decision part 15 which decides the segmenting areas of the right and left images so as to put a display stereoscopic image within the both eye merging range of the observer, a stereoscopic image segmenting part 13 which segments the stereoscopic image in a decided segmenting area, and a zoom part 14 which enlarges and contracts the segmenting area designated by the stereoscopic image segmenting part 13 by image processing and obtains the stereoscopic image to which the zoom processing is applied.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic electronic zoom device and a stereoscopic image quality control device that allow an observer to see a natural stereoscopic image.

[0002]

2. Description of the Related Art A conventional stereoscopic image pickup device is shown in FIG.
There is something like that shown in This is one in which two cameras are installed in parallel and the left and right images of a twin-lens stereoscopic image can be taken at the same time. Further, FIG. 10A is a configuration diagram of a conventional electronic zoom device, and FIG. 10B is a diagram showing an operation of the electronic zoom. As shown in FIG. 10A, the captured image is subjected to an electronic zoom process using the image memory 8 to obtain the image shown in FIG.
Part of the input image is enlarged or reduced to the size of the output image by interpolation processing of each pixel value. Normal Figure 10
The shaded area in (b) is selected at the center of the screen and is set so that the optical axis does not shift during electronic zoom.

In such a conventional example, as shown in FIG.
In the stereoscopic image capturing including two cameras, the captured images of the subject are displaced from each other in horizontal position. This amount of shift is called binocular parallax, but the presence of binocular parallax allows a stereoscopic image observer to see a subject stereoscopically. Here, in order for an observer to stereoscopically observe a stereoscopic image, the magnitude of this binocular parallax must be smaller than a certain value. When an observer sees a binocular parallax larger than this, the observer no longer sees a stereoscopic image, but merely sees a double image.

A conventional stereoscopic image quality control circuit is disclosed in, for example, Japanese Patent Application Laid-Open No. 4-35491, and has a configuration shown in FIG. With this configuration,
The depth difference adjusting unit 9 changes the phase difference between the left and right images, and sets the subject at the viewer's favorite position before and after the display surface of the stereoscopic display unit 11. At this time, the discomfort that the size of the subject does not change, while the distance between the displayed subject image and the observer changes, is adjusted by using the zooming adjusting means 10 linked with the depth feeling adjusting means 9 as shown in the figure. The size of the subject is reduced so that the size of the subject is increased when approaching the observer and decreased when approaching the observer.

[0005]

However, as shown in FIG.
As shown in, when two stereoscopic images are enlarged by the conventional electronic zoom method, the binocular parallax is also enlarged, and if the binocular parallax is set to a zoom magnification exceeding the allowable limit of the observer, the stereoscopic image is viewed. It will be impossible. Not only that, but the problem is that a stereoscopic image that looks double becomes a very unpleasant image. Further, in the conventional stereoscopic image quality control circuit, as shown in FIG.
The binocular parallax changes when the image is scaled up and down by the zooming adjusting means 10, which causes a change in the distance between the subject and the observer, which makes it impossible to completely eliminate the discomfort of the change in the size of the subject. Further, the size of the sense of discomfort felt by the observer is different from the size geometrically inferred from the control amount of the depth sensation adjusting unit 9, and is determined from the control amount of the depth sensation adjusting unit 9. When the zoom magnification was directly obtained, it was not possible to completely eliminate the discomfort of the viewer's size change.

The present invention solves the above-mentioned problems, and utilizes the observer's size perception and depth perception characteristics, and within the binocular fusion range without changing the observer's size perception and depth perception. The purpose is to clip and zoom the left and right images so that the subject fits within the viewer's binocular fusion range, and to display a stereoscopic image within the binocular fusion range without impairing the viewer's sense of depth and size. To do.

[0007]

In order to achieve the above-mentioned object, a stereoscopic electronic zoom device of the present invention has a cutout area for determining a cutout area of left and right images so that a display stereoscopic image fits within a fusional range of both eyes of an observer. A determination unit, a stereoscopic image cutout unit that cuts out a stereoscopic image using the output of the cutout region determination unit, and a stereoscopic image that has been zoomed by enlarging or reducing the clipping region specified by the stereoscopic image cutting unit by image processing. It is composed of a zoom unit.

Further, the stereoscopic image quality control apparatus of the present invention comprises a horizontal phase difference control unit for changing the horizontal phase difference between the images and a change in the size of the display object which an observer feels when changing the horizontal phase difference between the images. And a zoom magnification for compensating the size of the subject using the output of the size perceptual change amount holding unit and the horizontal phase difference, and the reproduction position of the stereoscopic image does not change. It is composed of a fusion range confirmation unit that calculates a clipping region for performing such zoom processing, and a zoom unit that performs image clipping / zoom processing according to the output of the fusion range confirmation unit.

[0009]

With the above-described structure, the present invention obtains a stereoscopic image that has been subjected to electronic zoom processing so that the observer can always perform stereoscopic viewing with both eyes. In addition, when the horizontal phase of each image is changed and the subject is set within the range of binocular fusion of the observer, the three-dimensional position of the displayed subject is used by using the characteristic of the sense of discomfort of the size change of the observer. The optimum region of each image is cut out and zoomed so that the image does not change as much as possible, thereby obtaining a stereoscopic image that does not impair the viewer's sense of depth and size.

[0010]

1 shows the structure of a stereoscopic electronic zoom device according to a first embodiment of the present invention. In FIG. 1, 1 and 2 are lenses, 5 and 6 are cameras, 13 is a stereoscopic image cutout unit, 14 is a zoom unit, and 15 is a cutout region determination unit. The operation of the stereoscopic electronic zoom device of the present embodiment configured as above will be described.

First, the right and left image signals picked up by the left and right cameras 5 and 6 are input to the stereoscopic image clipping section 13. The three-dimensional image cutout unit 13 is a cutout area determination unit 1
5 cuts out the cut-out areas of the right and left images determined by the distance (near point data) to the closest subject and the zoom magnification. Then, the cutout area is enlarged by the zoom unit 14 to the size of the screen of a predetermined TV standard, and the left and right images that have been electronically zoomed and enlarged are obtained. When this zoom process is performed, the cutout region determining unit 15 performs an operation of compensating for an increase in binocular parallax due to image enlargement by controlling the horizontal position of the cutout region.

Hereinafter, with reference to FIG.
The enlarging operation will be described in more detail. In FIG. 2, the subject is the letter “A”, the right image is shown by a solid line, and the left image is shown by a broken line. Here, the binocular parallax of the English character "A" is Δ1. In this case, “A” in the right image is captured on the left side of the screen, and “A” in the left image is captured on the right side of the screen,
When an observer sees this stereoscopic image, it should appear as if it jumps out to the front side of the image display surface. Of these images, the right image cuts out a solid line region, the left image cuts out a broken line region, and zoom processing is performed to obtain a zoomed image. Also in the image after the zoom processing, the solid line shows the right image and the broken line shows the left image. In this image, Δ2 is the binocular parallax of the subject “A” after zoom processing. Here, if the zoom magnification is m, the relationship between Δ1 and Δ2 is

[0013]

[Equation 1]

[0014] Here, the maximum value of binocular parallax that allows an observer to favorably perform stereoscopic binocular viewing is fixed, and the viewing angle is approximately several degrees. If this is Δmax, the binocular parallax after zooming is

[0015]

[Equation 2]

Must be satisfied. This condition
Considering the cutout area before zoom processing (FIG. 2), the center position of the cutout area in FIG.

[0017]

(Equation 3)

[0018]

[Equation 4]

If the points ΔSL and ΔSR satisfying the above conditions are moved horizontally in opposite directions to each other, the binocular parallax Δ2 after electronic zoom can be made equal to or less than the observer's binocular parallax allowable range Δmax. Also, ΔSL, ΔSR

[0020]

(Equation 5)

[0021]

(Equation 6)

By setting so as to satisfy, it is possible to keep the binocular parallax of the subject constant before and after the electronic zoom. The cutout area determination unit 15 calculates ΔSL and ΔSR that satisfy the above formula from the near point data Δ1 (binocular parallax of the closest subject) and the zoom magnification m, and outputs the result to the stereoscopic image cutout unit 13. The stereoscopic image cutout unit 13 horizontally moves the center of the cutout region by ΔSL for the left image and ΔSR for the right image in opposite directions, and cuts out each image, and the zoom unit 14 enlarges it.

The electronic zoom by image processing can be easily realized by using the technique introduced in the existing video camera.

As described above, according to this embodiment, the zoom process is performed so that the binocular parallax falls within the binocular fusion range of the observer by using the parallax (near point data) and the zoom magnification of the closest subject. It is possible to obtain a stereoscopic image, and to generate a stereoscopic image that has been subjected to electronic zoom processing that does not hinder the binocular stereoscopic vision.

FIG. 3 shows the configuration of a stereoscopic electronic zoom device according to the second embodiment of the present invention. In FIG. 3, reference numerals 1 and 2 are lenses, 5 and 6 are cameras, 13 is a stereoscopic image cutout unit, 14 is a zoom unit, and 15 is a cutout region determination unit. The above is the same as in the first embodiment. The difference from the first embodiment is that the near point data is automatically obtained by the parallax detection unit 16. The operation of the stereoscopic electronic zoom device of the present embodiment configured as above will be described.

The basic operation is similar to that of the first embodiment of the present invention. That is, the right and left image signals picked up by the left and right cameras are input to the stereoscopic image cutout unit 13, and the stereoscopic image cutout unit 13 zooms out the distance (near point data) to the closest subject to the cutout region determination unit 15 and zooms. The cutout areas of the right and left images determined by the magnification are cut out. Then, the zoom unit 14 enlarges the cutout area to the size of the screen of a predetermined TV standard.

Here, in the first embodiment, the binocular parallax Δ1 of the closest subject was manually input, but in the present embodiment, this is automatically detected by the parallax detector 16. Various methods for detecting parallax have been proposed, but here, a method using correlation matching processing will be described.

In FIG. 4, consider left and right images of size N × M. Consider a block window of n × n pixels (3 × 3 pixels in the figure) in the left image. The same image as this block window is searched for in the right image using a window of the same size, and the horizontal component Δx of the vector (Δx, Δy) indicating the displacement of the left and right block positions at this time is the center coordinate of the block window. It becomes the binocular parallax of the left and right images. If the position of the block window of the reference left image is translated over the entire screen and the position of the corresponding block (binocular parallax) of the right image is obtained in all cases, the parallax map of the entire screen (each screen What shows the depth distance at the place) is required. Here, the shift between the left and right images at the image coordinates (x, y), that is, binocular parallax (Δx, Δy)
Is

[0029]

(Equation 7)

Here,

[0031]

(Equation 8)

It is However, Σ in (Equation 8) indicates that the coordinates xk and yk are changed in the n × n block window to obtain the sum within the absolute value. Of binocular parallax Δx, Δy,
The depth position is directly indicated by Δx. When the binocular disparity value is positive, the right image is located on the right side of the reference image, the left image is located on the left side, and the depth position of binocular parallax is 0. When the binocular parallax value is negative, it indicates that the subject is present in front of the depth position where the binocular parallax is 0. As the near point data Δ1, the binocular parallax Δ1 of the closest subject can be obtained by using the negative value having the largest absolute value among the calculated Δx.

The binocular parallax Δ1 obtained as described above,
The cutout area determination unit 15 calculates ΔSL and ΔSR that satisfy (Equation 3) and (Equation 4) using the zoom magnification m set by the user, and the stereoscopic image cutout unit 13 and the zoom unit 14 determine predetermined positions according to the calculated ΔSL and ΔSR. The image is subjected to zoom processing to obtain a stereoscopic image with binocular parallax setting that allows the observer to favorably perform stereoscopic viewing.

As described above, according to this embodiment, the zoom process is performed so that the binocular parallax falls within the binocular fusion range of the observer by using the parallax (near point data) and the zoom magnification of the closest subject. It is possible to obtain a stereoscopic image, and to generate a stereoscopic image that has been subjected to electronic zoom processing that does not hinder the binocular stereoscopic vision.

In the present embodiment, the cameras are installed in parallel, but in the case of the convergence photographing in which the optical axis of each camera is directed to a subject, or when the convergence angle is changed in the optical zoom. This method is effective.

In the first and second embodiments of the present invention, the cutout area is determined using the parallax (near point data) of the closest object and the zoom magnification, but the parallax of the farthest object (far point data). Similarly, it is possible to obtain a stereoscopic image in which the binocular parallax after zooming is within the fusion range of the observer. In this case, Δmax is the maximum binocular parallax that allows binocular fusion at the far point of the observer, Δ1 is the binocular parallax of the subject at the far point, and the parallax detection unit 16 detects the binocular parallax Δ1 of the farthest subject. Will be done.

FIG. 5 shows a block diagram of a three-dimensional image quality control device in the third embodiment of the present invention. In FIG. 5, 13 is a three-dimensional image cutout unit, 14 is a zoom unit, and 16 is a zoom unit.
Is a parallax detection unit, 17 is a horizontal phase difference control unit, 18 is a gazing point calculation unit, 19 is a size perceptual change amount holding unit, and 20 is a fusion range confirmation unit.

The operation of the third embodiment having the above configuration will be described. First, the left and right images are input to the parallax detection unit 16, and the binocular parallax of the left and right images is calculated. The calculation method can be performed with the same configuration as in the second embodiment (FIG. 4). Thereby, the binocular parallax at an arbitrary position in the image is calculated. Of the detected binocular parallax, the binocular parallax indicating the closest position is calculated by the gazing point calculation unit 18, and the horizontal phase difference control unit 17 separates the left and right images from each other so that the binocular parallax falls within the binocular fusion range of the observer. Translate in the horizontal direction in the opposite phase direction. For example, as shown in FIG. 6A, the right image display surface 22,
Point images AR and AL are displayed on the left image display surface 23, and point P
From the state where the observer 21 is stereoscopically viewing at the position 1, the left and right images are moved in parallel by D and shifted, and as shown in FIG. 6B, the subject can be viewed stereoscopically at the position P2. . If the position of P1 is too far from the image display surfaces 22 and 23, the observer 21 becomes very difficult to stereoscopically view.
By controlling the horizontal phase difference between the left and right images as shown in FIG. 6B, an image that is easy to stereoscopically view can be obtained.

In this case, since the left and right images are only moved in parallel, the size of the displayed subject does not change. Normal 3
In the three-dimensional world, when the subject approaches, the viewing angle of the subject should increase, but if such control is performed, the size does not change, which is very uncomfortable. In order to remove this discomfort, the stereoscopic image cutout unit 13 and the zoom unit 1
According to 4, the sense of discomfort is compensated by performing the zooming in the case where the position of the subject becomes closer and the zooming out in the case of the position far away. However, as described with reference to FIG. 2, when the zoom process is performed, the binocular parallax of the subject changes. Further, the perception of the change in the size of the subject by the observer has a characteristic different from the change in the geometric size calculated from the display position of the subject. Considering these two points, the size perceptual change amount holding unit 19 and the fusion range confirmation unit 20 determine the cutout positions and zoom magnifications of the left and right images. Less than,
This operation will be described in more detail.

First, from the shift amount D of the left and right images output by the gazing point calculation unit 18, the size perceptual change amount holding unit 19 outputs the amount of change in size felt by the observer when this control is performed. . From these two data, the fusion range confirmation unit 20 calculates the zoom magnification necessary to correct the size change and the cutout regions of the left and right images that can be zoomed so that the depth position of the displayed subject does not change. Is output to the zoom unit 14 and the stereoscopic image cutout unit 13. As a result, without changing the depth position of the subject,
It is possible to control the image quality of a stereoscopic image while compensating for changes in the size of the object that the observer feels.

The clipping / zooming process will be described more specifically with reference to FIG. First, the clipping method will be described. FIG. 7A shows a state in which the left and right images 22 and 23 are horizontally phase-controlled by D and the observer 21 recognizes the subject at the position P3. In this case, the subject P3
Is moving toward the front side compared to before horizontal phase control, so the subject should be large. Therefore, the left and right images are magnified by zooming and corrected, but if the center of the screen is magnified m times as it is, the binocular parallax DLR of P3 is obtained.
Is also enlarged m times, and the depth position of P3 also changes. For example, if the horizontal coordinates of AL and AR are respectively x1 and x2, the binocular parallax x1-x2 (= DLR) becomes m (x1-x2) after zoom processing (Fig. 7 (b)). . Therefore, when performing zoom processing, the left image

[0042]

[Equation 9]

A region which is horizontally translated from the center to the right side, a region of the right image which is horizontally translated from the center, and a region which is horizontally translated from the center to the left side are clipped by the stereoscopic image clipping unit 13, and the zoom unit 14 is used. It will be enlarged m times. As a result, as shown in FIG. 7C, the subject A after zoom processing is performed.
The binocular parallax of R "and AL" can be kept at DLR. These operations are calculated by the fusion range confirmation unit 20 from the zoom magnification m and the parallax DLR of the subject.

The zoom magnification m is calculated by the fusion range confirmation unit 20 from the phase difference D (FIGS. 6 and 7) applied to the left and right images obtained by the gazing point calculation unit 18. This calculation is not the characteristic obtained by the geometrical calculation from the phase difference D, but the amount of change in the size perceived by humans is measured in advance, and this characteristic is stored in the size perceptual change amount holding unit 19 and stored. Determines the zoom magnification m. For example, as shown in FIG. 8, the zoom magnification m has a smaller value than the geometric calculation (broken line).

As described above, according to the present embodiment, the horizontal phase of the left and right images is changed without changing the size of the observer or the three-dimensional position of the displayed subject is changed. Can be set within the binocular fusion range of the observer.

Further, in the third embodiment, an example in which the subject is on the front side of the display surface is shown, but the same control can be performed when the subject is on the back side of the display surface. In this case, the horizontal movement of the cutout area of the image is set in the direction opposite to that of the third embodiment.

Further, in the third embodiment, an example in which the enlarging zoom process is performed when the horizontal phase of the left and right images is controlled so that the subject moves toward the front side of the display surface has been shown. It is also conceivable to control the phase difference between the left and right images so as to move to the back side. In this case, the zoom process is a process of reducing the image (the characteristic of the second quadrant in FIG. 8 is used). Further, the sign of the calculated value of (Equation 9) is changed, and the horizontal movement of the cutout regions of the left and right images is in the opposite direction.

In the first to third embodiments of the present invention, the case of a natural image using a camera has been described, but the same operation can be easily realized in CG (computer graphics) using the same method. it can.

[0049]

As described above, according to the present invention, it is possible to obtain a stereoscopic image which has been subjected to an electronic zoom process in which an observer can always perform stereoscopic viewing with both eyes. In addition, the horizontal phase of each image is changed by zooming and clipping the image according to the characteristics of the observer, which causes a sense of discomfort in the change in size of the observer and a change in the three-dimensional position of the displayed subject. It is possible to display a stereoscopic image within the range of binocular fusion of the observer without doing so, and its practical effect is great.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a stereoscopic electronic zoom device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing the operation of stereoscopic electronic zoom processing in the first embodiment of the present invention.

FIG. 3 is a block diagram showing a configuration of a stereoscopic electronic zoom device according to a second embodiment of the present invention.

FIG. 4 is a block diagram showing the operation of the parallax detection unit according to the second embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration of a stereoscopic image quality control device according to a third embodiment of the present invention.

6 (a) and 6 (b) are diagrams showing the operation of horizontal phase difference control according to the third embodiment of the present invention.

FIG. 7 is a diagram showing the operation of horizontal phase difference control / zoom processing according to the third embodiment.

FIG. 8 is a characteristic diagram showing a correction amount at a zoom magnification according to the present embodiment.

FIG. 9 is a diagram showing a configuration of a conventional stereoscopic image pickup device.

10A is a block diagram showing a configuration of a conventional electronic zoom process, and FIG. 10B is a diagram showing the same operation.

FIG. 11 is a diagram showing an operation of a conventional stereoscopic electronic zoom process.

FIG. 12 is a block diagram showing a configuration of a conventional stereoscopic image quality control device.

[Explanation of symbols]

 1 lens 1 2 lens 2 3 image sensor 4 image sensor 5 left image camera 6 right image camera 13 stereoscopic image cutout section 14 zoom section 15 cutout area determination section 16 parallax detection section 17 horizontal phase difference control section 18 gazing point calculation section 19 Size Perception Change Amount Holding Unit 20 Fusion Area Confirmation Unit 21 Observer 22 Right Screen 23 Left Screen

Claims (2)

[Claims] 1. In a stereoscopic image characterized by obtaining a stereoscopic effect by utilizing binocular parallax, the cutout regions of the left and right images are determined so that the displayed stereoscopic image fits within the binocular fusion range of the observer. A cutout area determination unit, a stereoscopic image cutout unit that cuts out a stereoscopic image using the output of the cutout area determination unit, and a stereoscopic image obtained by enlarging or reducing the cutout area specified by the stereoscopic image cutout unit and performing zoom processing. A stereoscopic electronic zoom device comprising a zoom unit for obtaining an image. 2. In a stereoscopic image characterized by obtaining a stereoscopic effect by utilizing binocular parallax, a horizontal phase difference control unit for changing the horizontal phase difference of each image and a horizontal phase difference for each image. In order to compensate for the size of the subject by using the size perceptual change amount holding unit that stores the size change of the displayed subject that the observer feels when performing, and the output from the size perceptual change amount holding unit and the horizontal phase difference. And a zoom range for performing a zoom process so that the reproduction position of the stereoscopic image does not change, and a zoom unit for performing an image crop / zoom process according to the output of the fusion range check unit. A three-dimensional image quality control device characterized by the following.