JP5296218B2 - 3D image display method and 3D image display apparatus - Google Patents

Abstract

In a stereoscopic image imaging method which captures an image of an object to be displayed as a stereoscopic image, a three-dimensional image is generated in real time from multi-viewpoint images captured by a multi-camera, and is displayed on a viewer provided to the multi-camera, thus allowing a photographer to adjust an imaging condition. The photographer is informed of parameters, which implements a display state adjusted by an observer of a three-dimensional display while observing the three-dimensional image, via the viewer, and the photographer can capture appropriate multi-viewpoint images.

Description

    The present invention relates to a 3D image display method and a 3D image display apparatus for photographing a subject and displaying a 3D image.

  Various types of display devices that can display a three-dimensional image (stereoscopic image) are known. In recent years, there has been a growing demand for a method of displaying a stereoscopic image without using special glasses or the like, particularly in a flat panel type. A light beam from the display panel is displayed on the front surface of a display panel (display device) in which pixels are fixed on a plane such as a direct-view or projection-type stereoscopic moving image display device such as a liquid crystal display device or a plasma display device. A method of installing a parallax barrier (also referred to as a light beam controller) that controls and directs a light beam to an observer has been developed. This method is a method that can display stereoscopic images relatively easily and can be put into practical use.

  The parallax barrier is generally referred to as a parallax barrier, and controls light rays so that different images can be seen depending on the viewing angle even when the parallax barrier at the same position is observed. Specifically, when left and right parallax (horizontal parallax) is given, a slit or a lenticular sheet (cylindrical lens array) is used as a parallax barrier. When left and right parallax and vertical parallax (vertical parallax) are given, a pinhole array or a lens array is used as a parallax barrier. In this specification, one slit or one lens which is a structural unit of the parallax barrier is referred to as an exit pupil.

  The system using the parallax barrier further includes a binocular system, a multi-view system, a super multi-view system (a system in which the multi-view system has super multi-view conditions), and an integral imaging system (hereinafter simply referred to as the II system). )are categorized. These basic principles were invented about 100 years ago and are substantially the same as the stereoscopic photography system. However, since the number of pixels of the display device is finite, the number of pixels assigned to one of the exit pupils is also finite. In this specification, the number of pixels assigned to the exit pupil is referred to as the number of parallaxes, and a planar image composed of pixels assigned to the exit pupils is referred to as an element image.

  The II system is a term for stereoscopic photography and may be called integral photography (hereinafter also referred to as IP).

  In order to display a stereoscopic image by these II methods, an image (multi-viewpoint image) taken from a plurality of directions is required. That is, in the binocular stereoscopic video display method, two multi-viewpoint images are prepared, and in the multi-view or II stereoscopic video display method, the number of pixels corresponding to the number of parallaxes assigned to each exit pupil is large. A viewpoint image is prepared. In this specification, a pixel means a minimum display unit. Basically, a multi-viewpoint image is taken on the assumption of the relationship between pixels and exit pupils. The multi-viewpoint image generation method includes a plurality of generation methods such as live-action shooting or CG rendering. However, a multi-viewpoint image is usually prepared by a live-action shooting in which a subject is shot using a camera.

  In actual shooting with a camera, specifically, cameras for multi-viewpoint image shooting are laid out by the number of parallaxes so as to be similar to the relationship between the exit pupil and the corresponding pixel position. A camera laid out for taking multi-viewpoint images in this way is called a multi-camera. Since the pixels of the display device are arranged on a plane, the multi-camera is arranged on the plane as well. In the stereoscopic display device, when the pixel interval is given by pp and the interval between the exit pupil and the pixel surface of the display device is given by g, in the stereoscopic display device, the imaging reference distance Lc and the interval x_c of the multi-camera 1 are (1 ).

g: pp = Lc: x_c (1)
In order to satisfy the above-described shooting conditions, the size and resolution of the shooting reference plane of the multi-camera are the same as those of the flat display unit of the display device. Matching the size and resolution means the most efficient. Here, the imaging reference plane is referred to as a projection plane on the assumption that it matches the display plane, the imaging reference distance is the observation reference viewing distance of the 3D display, and the imaging position is the viewpoint on the observation reference plane of the 3D display. The shooting and playback rays are the same, and the shooting target is displayed in actual size.

  However, the conditions for this live-action shooting need not be strictly satisfied. In recent years, it has been said that it is possible to devise so that a stereoscopic image can be seen even if it is deviated from the viewing distance by designing the information of adjacent pixels to be mixed to some extent (Non-Patent Document 1). Furthermore, the parallax barrier type three-dimensional display has a limited display range in the z direction (a direction orthogonal to the display surface). (Non-Patent Document 2) Accordingly, a multi-camera for capturing multi-view images larger than the number of viewpoints is prepared, and a multi-view image having an interval x_c narrower than a design value is selected from the multi-camera, and the image is crushed in the z direction. May be displayed. (Patent Document 1) Here, the z direction means a depth direction corresponding to the back surface of the display surface orthogonal to the horizontal direction x and the vertical direction y of the three-dimensional display surface. Also, by adjusting the clipping range to be used as a parallax image, that is, the clipping method, for existing multi-viewpoint images, the z-coordinate when displaying a stereoscopic image is shifted, or the image is enlarged or reduced in the xy direction. A method of displaying a stereoscopic image within a display range is also known (Patent Document 2). In any case, these non-patent documents and patent documents only disclose a method of displaying a stereoscopic image displayed by selecting a multi-viewpoint image that has already been shot or adjusting a clipping range.

JP-A-2005-331844 JP 2004-343290 A

R. Fukushima et al., Proceedings of SPIE-IS & T Electronic Imaging, 7237,72370W-1 (2009) J. Opt. Soc. Am. A vol. 15, p. 2059 (1998)

  In order to change the z-coordinate when displaying the stereoscopic video, specifically, the shift value for each viewpoint image in the range used as the parallax image may be changed. However, in live-action shooting, since the multi-viewpoint image is a perspective projection, when the shift value is changed and the projection plane is shifted forward or backward along the z axis, the shooting reference distance is three-dimensional. Strictly speaking, there is a problem that distortion occurs because it is different from the viewing standard viewing distance of the display. In order to display 3D images without distortion, instead of the acquired multi-viewpoint image being post-processed and reconstructed for 3D image display, the imaging reference distance to the object to be displayed is mainly the 3D display. It must be set equal to the observation reference viewing distance. However, there is no method for the photographer to correctly grasp the photographing reference distance, and there is a problem that the photographing reference distance cannot be set correctly. In addition, there is no way for the photographer to know which object the viewer of the three-dimensional display at a remote place intends to display mainly three-dimensionally. It is said that the photographing reference distance cannot be set because the stereoscopic display target is not found.

  The present invention has been made in consideration of the above circumstances, and an object of the present invention is to obtain an appropriate photographing reference distance in a method for acquiring and displaying a multi-viewpoint image for a three-dimensional display in a parallax barrier method by a real photograph. It is intended to provide a stereoscopic video display device capable of appropriately displaying a desired display target within a display range of a three-dimensional display.

According to one aspect of the invention,
A display unit in which pixels having a predetermined width are arranged in a matrix form in a display surface having a vertical direction and a horizontal direction, and an optical aperture that controls the emission direction of light rays from the pixels. The parallax barriers are arranged in the horizontal direction at intervals corresponding to substantially integral multiples of the pixels, the width is determined depending on the observation reference viewing distance, and the elements correspond to the optical apertures, respectively. A pixel that is composed of a parallax barrier that divides the display surface into element image regions in which an image is displayed and is observed through the optical aperture, and a parallax image that is determined depending on an observation direction is used as the pixel. A first three-dimensional display device that distributes and displays the elemental image in the elemental image area to generate a stereoscopic video of a continuous viewpoint;
In order to generate the parallax image, a multi-camera that captures multi-viewpoint images of real objects from a plurality of directions at a multi-camera angle of view, and a plurality of imaging units arranged at a certain interval and having a certain angle of view are arranged A multi-camera having a projection plane as a photographing reference plane,
An image processing unit that processes the multi-viewpoint image from the multi-camera,
A multi-viewpoint image storage unit for storing a multi-viewpoint image shot by the shooting unit;
A clipping condition storage unit for storing a range to be used as a parallax image necessary for generating a parallax image and a shift value determined corresponding to a clipping region selectable within the range to be used;
A parallax image generation unit that clips a range to be used as a parallax image from the multi-viewpoint image stored in the multi-viewpoint image storage unit based on the information of the clipping condition storage unit and stores the range as parallax image data;
A rearrangement processing unit that rearranges parallax image data read from the parallax image generation unit into image data to be displayed on the display unit;
A shooting condition storage unit for storing shooting conditions of a multi-viewpoint image for displaying a three-dimensional image without distortion on the display unit;
A display condition adjusting unit that adjusts parameters for displaying a photographing object in a desired size near the projection plane with reference to the three-dimensional image displayed on the display unit;
From the parameters and the shooting conditions, a corrected shooting reference distance and a corrected camera interval for displaying without distortion are derived, and information regarding the corrected shooting reference distance is clearly displayed on the display unit and shot to the photographer. An image processing unit that prompts the user to change the distance;
A stereoscopic video display device is provided.

  According to the stereoscopic image display apparatus for capturing a stereoscopic image of the present invention, in a method for acquiring and displaying a multi-viewpoint image for a three-dimensional display in a parallax barrier method by a live action, an appropriate shooting reference distance is notified to a photographer and desired. Can be appropriately displayed within the display range of the three-dimensional display.

FIG. 1 is a block diagram showing a multi-camera photographing system for photographing a stereoscopic image according to an embodiment of the present invention, and schematically showing an arrangement relationship between the multi-camera, a real object to be photographed, and a projection plane. FIG. FIG. 2 is a plan layout view showing a relationship among a multi-camera, a real object to be photographed, and a projection plane in the system for photographing a stereoscopic video shown in FIG. FIG. 3 is a plan layout diagram showing the relationship between the display device, the visible region, and the observation reference plane in the apparatus for displaying the stereoscopic video shown in FIG. 4A is a plan view schematically showing an image captured by the left imaging element of the multi-camera shown in FIG. 4B is a plan view schematically illustrating an image captured by the right image sensor of the multi-camera illustrated in FIG. FIG. 5 is a block diagram schematically showing a stereoscopic video imaging display apparatus according to an embodiment for capturing an image of a subject and displaying it as a stereoscopic video in the multi-camera imaging system shown in FIG. FIG. 6 is a flowchart showing a method of displaying a stereoscopic video from an image acquired by a multi-camera in the stereoscopic video shooting and display apparatus shown in FIG. FIG. 7 is a layout diagram for explaining a method of adjusting the photographing reference distance in the stereoscopic image capturing method shown in FIG. FIG. 8 is a layout diagram for explaining a method of adjusting the photographing reference distance in the stereoscopic image capturing method shown in FIG. FIG. 9 is a layout diagram for explaining a method of adjusting the photographing reference distance in the stereoscopic image capturing method shown in FIG. FIG. 10 is a block diagram schematically showing a stereoscopic video shooting and display apparatus according to another embodiment for capturing a subject and displaying it as a stereoscopic video in the multi-camera shooting system shown in FIG. FIG. 11 is a flowchart showing a method of displaying a stereoscopic video from an image acquired by a multi-camera in the stereoscopic video shooting and display apparatus shown in FIG. FIG. 12 is a schematic diagram schematically showing an example of a display surface displayed on the display unit for displaying the stereoscopic image shown in FIG.

  Hereinafter, a method for acquiring a captured image for displaying a stereoscopic video and a method and apparatus for displaying a stereoscopic video from the acquired image according to an embodiment of the present invention will be described in detail with reference to the drawings.

  In the following description of the embodiment of the present invention, the presentation direction of parallax on a three-dimensional display is limited to one dimension (horizontal direction: X direction). However, the direction (vertical direction: Y direction) orthogonal to the one-dimensional direction (horizontal direction: X direction) can also be applied to a display method and apparatus for displaying parallax information. That is, if the present invention is applied to the vertical direction (Y direction) as well as the horizontal direction (X direction), disparity information can be given to the two-dimensional direction (horizontal and vertical directions: X and Y directions) as well. . Therefore, the stereoscopic image display method and apparatus of the present invention are not limited to an example in which parallax is presented only in a one-dimensional direction, but includes an embodiment in which parallax is substantially presented in a two-dimensional direction.

  FIG. 1 shows a multi-camera photographing system for photographing a stereoscopic video according to an embodiment of the present invention. FIG. 1 schematically shows an arrangement relationship between the real objects 3-1, 3-2 and 3-3 to be photographed to be photographed by the multi-camera 1 and the projection plane 2.

  As shown in FIG. 1, the real object 3-1 indicated by ◯ is arranged in front of the projection plane 2 along the Z direction with respect to the multi-camera 1 and the projection plane 2, or the real object indicated by Δ. 3-2 is arranged on the projection plane 2, or the real object 3-3 indicated by □ can be arranged behind the projection plane 2 along the Z direction. Here, the multi-camera 1 side is defined as the front with respect to the projection plane 2, and the opposite side of the multi-camera 1 is defined as the back of the projection plane 2. FIG. 1 shows coordinate axes (X, Y, Z) in order to clarify the positional relationship among the multi-camera 1, the real objects 3-1, 3-2 and 3-3, and the projection plane 2. Referring to this coordinate axis, the projection plane 2 is defined as a plane extending in the X and Y axes orthogonal to the Z axis, and in the example of FIG. 1, real objects 3-1 to 3-3 are arranged along the Z axis. Has been.

  Data of a photographed image photographed at a certain angle of view by the multi-camera 1 shown in FIG. 1 is processed and converted into elemental image data by the image processing unit 40 for displaying a stereoscopic video. The element image data is supplied as a viewer device to the drive circuit unit 44 of the photographer display unit 42 provided in the housing of the multi-camera 1. A stereoscopic image is displayed on the display panel 46 in real time and is observed as a captured image by the photographer through the parallax barrier 47. As will be described in detail later, the photographer gives an instruction to the image processing unit 40 from the input unit 45 while referring to the photographed image. As a result, it is possible to specify a photographing target that the photographer is interested in, for example, any one of the real objects 3-1 to 3-3. It is possible to set an optimum shooting condition for displaying the specified shooting target. The photographer can photograph the subject according to the photographing conditions.

  In the imaging system shown in FIG. 1, element image data from the image processing unit 40 is transferred to the transmission / reception unit 58 of the observation display device 52 for the observer via the transmission / reception unit 48. Here, the photographing display device 42 is visually recognized by a photographer and used for stereoscopic photographing in order to photograph a subject in a studio. On the other hand, the observation display device 52 corresponds to a display device installed in a monitor room away from the studio or a display device visually recognized by a general viewer who views a stereoscopic video program. The observer display device 52 includes a drive circuit unit 54 to which the element image data transferred to the transmission / reception unit 58 is supplied in the same manner as the photographer display device 42. The drive circuit unit 54 displays the element image on the display panel 56 in real time, and the stereoscopic image is observed as a captured image by the observer through the parallax barrier 57. As will be described in detail later, the observer gives an instruction to the image processing unit 40 from the input unit 55 while referring to the captured image. As a result, it is possible to specify a photographing target that the observer is interested in, for example, any of the real objects 3-1 to 3-3. It is possible to set an optimum shooting condition for displaying the specified shooting target. The photographer can photograph the subject according to the photographing conditions.

  As illustrated in FIG. 2, the multi-camera 1 includes a plurality of photographing units 30-1 to 30-according to combinations of lenses 4, 4-1 to 4-n and imaging elements 5, 5-1 to 5-n. n is arranged linearly on a plane along the X direction. Here, n corresponds to the number of parallaxes, and the arrangement interval (camera pitch) of the lenses 4-1 to 4-n is referred to as a camera interval Ls. The camera interval Ls may be given by the arrangement interval (camera pitch) of the image sensors 5-1 to 5-n.

  In the planar arrangement shown in FIG. 2, only two photographing units 30-1 and 30-n corresponding to two cameras are shown to simplify the description. The photographing units 30-1 and 30-n are configured by lenses 4-1 and 4-n and image sensors 5-1 and 5-n, respectively, and the lenses 4-1 and 4-n and image sensors 5-1 and 5-1. The interval from 5-n is set to the imaging distance f. In order to efficiently photograph the finite distance Lc (referred to as the photographing distance Lc) and the surrounding space from the lenses 4-1, 4-n, the distance between the lenses 4-1, 4-n and the image sensor 5 is not limited. The imaging position f (distance along the Z direction) between the two may be adjusted by shifting the relative position. That is, the lenses 4-1 and 4-n are shifted with respect to the image pickup device 5, and the imaging relationship of the lenses 4-1 and 4-n with respect to the image pickup device 5 is changed to change the photographing range. By adjusting the imaging relationship of the lenses 4-1 and 4-n, the shooting range of the multi-camera 1 can be superimposed at the shooting reference distance Lc as shown by the oblique lines. The lenses 4-1 and 4-n have a certain angle of view, and the photographing range of the multi-camera 1 is determined by the angle of view and the photographing reference distance Lc.

  The range of the XY plane of the shooting reference distance L in the space where the shooting range is superimposed is defined as the projection plane 2 under certain conditions described later. Before and after the projection plane 2, a photographing range 6 is widened as shown in FIG. If it is within this imaging range 6, the real object 3-1 is positioned in front (projecting direction) by a distance z_n from the projection plane 2. Alternatively, even if the real object 3-2 is positioned on the projection plane 2 or the real object 3-3 is positioned a distance z_f away from the projection plane 2 (in the depth direction) and the subject is photographed, a 3D image is displayed on the display device. It can be displayed as a video. The distances z_n and z_f that can be displayed as the stereoscopic video correspond to the pop-up display area and the depth display area on the display device side. That is, the distances z_n and z_f also define the area of the stereoscopic photographing limit area in the multicamera.

  As shown in FIG. 3, each display device 42, 52 is configured by providing parallax barriers 47, 57 in front of the flat display panels 46, 56 on the front surface. A gap g is provided between the flat display panels 46 and 56 and the parallax barriers 47 and 57. The flat display panels 46 and 56 include a display surface having a vertical direction and a horizontal direction. On the display surface, pixels each having a predetermined width are arranged in a matrix at a constant pixel pitch pp. In the one-dimensional II, the parallax barriers 47 and 57 are constituted by a lenticular sheet or a slit sheet. In the parallax barriers 47 and 57, a large number of cylindrical lenses or slits are arranged at the lens pitch Pe along the horizontal direction (x direction). In the two-dimensional II, the parallax barriers 47 and 57 are configured by fly-eye lens sheets or pinhole sheets, and a large number of microlenses or horizontal lenses are arranged along the horizontal direction (x direction) and the vertical direction (y direction). Pinholes are arranged at lens pitches Pe (H) and Pe (V). These cylindrical lenses, slits, microlenses, or pinholes are called optical apertures or optical pupils. In the II-type stereoscopic image display system, the optical aperture or optical pupil array pitch Pe (horizontal pitch Pe (H) or vertical pitch Pe (V)) is a pixel in which pixels are arrayed on the display surface. It is determined to be an integral multiple of the pitch pp.

  The display surfaces of the flat display panels 46 and 56 are defined as segmented element image regions 60 divided into optical apertures or regions optically opposed to the pupil. That is, in the one-dimensional II type stereoscopic image display system, an area 60 where an element image is displayed is determined corresponding to each of the cylindrical lens or the slit, and the area of the element image 60 extends along the x direction. It is arranged continuously. Even in the two-dimensional II system stereoscopic image display system, an area 60 where an element image is displayed is determined corresponding to each of the microlens and the pinhole, and the element image area 60 is arranged in the x direction and the y direction. It is arranged in a matrix form continuously. The element image area 60 is determined depending on the observation reference viewing distance Lo that is a reference of the range in which the stereoscopic images can be normally viewed and defined on the display devices 42 and 56, and the observation reference plane 62 on the observation reference viewing distance Lo. It is done. The parallax images taken in various directions with the multi-camera are distributed to the element image area, and the element image is displayed in the element image area 60. For details of the distribution of the parallax images to the element image areas, refer to Patent Document 1 as an example.

  Regarding the multi-camera 1, as already described, the photographing reference distance Lc and the interval x_c are given by the equation (1).

g: pp = Lc: x_c (1)
Here, pp is a pixel interval (pixel pitch) on the display panels 46 and 56, and g is an interval (gap length) between the parallax barriers 47 and 57 and the display surfaces of the display panels 46 and 56. Yes. When Expression (1) is established and the photographing reference distance Lc is set to the observation reference viewing distance Lo, a stereoscopic image having the same size as the real objects 3-1 to 3-3 is displayed in front of or behind the display devices 42 and 52. It is formed. Here, an imaging reference distance Lc that establishes a relationship in which a stereoscopic image having the same size as the real objects 3-1 to 3-3 is formed in front of or behind the display devices 42 and 52 is the projection plane 2 shown in FIG. Be called. By using the projection plane 2 as an imaging reference plane, it is possible to display a stereoscopic image with high resolution and no distortion on the display devices 42 and 52 as imaging data. The projection plane 2 as the imaging reference plane is different from the imaging plane that is determined by the distance to the subject when the subject is focused on and formed as an image on the imaging elements 30-1 to 30n. The projection plane 2 is defined as an imaging reference plane capable of obtaining imaging data that can display high-resolution and undistorted stereoscopic video on the display devices 42 and 52.

  The projection plane 2 corresponds to a photographing reference plane that matches the display plane of the display devices 42 and 52. The shooting reference distance is set to the observation reference viewing distance of the three-dimensional display, the shooting position is set to the viewpoint on the observation reference plane of the three-dimensional display, and the shooting target is displayed in actual size. The projection plane 2 corresponds to the display plane displayed on the display devices 42 and 52 shown in FIG. 1, and the subject is preferably photographed with reference to the projection plane 2 as a photographing reference plane. Instead of displaying the projection plane 2 in the display devices 42 and 52, numerical values related to the shooting distance to the projection plane or other displays, for example, the multi-camera is moved forward or backward from the current position. A display requesting that may be made.

  In the photographic optical system shown in FIG. 2, the range used as a parallax image within the shooting range 6 of the multi-camera 1 is subjected to clipping editing (cut-out editing), so that any real object is shot and the three-dimensional display Can be enlarged and displayed as a stereoscopic image near the display surface. Control of the range used as the parallax image will be described with reference to FIGS. 4A and 4B.

  4A shows an image 8-1 picked up by the leftmost image sensor 5-1 of the multi-camera 1 shown in FIG. 2, and FIG. 4B shows an image sensor 5-n at the right end of the multi-camera 1 shown in FIG. The image 8-n imaged in FIG. The imaging elements 5-1 to 5-n having the imaging width W_s have the same size and output parallax images of the same size. A real object (shown by a circle) located at a protruding position at a distance z_n from the projection plane 2 is copied to the right end on the image sensor 5-1 as shown in FIG. 4A, and the image sensor 5 is shown in FIG. 4B. -It is copied to the left end on n. Further, an actual object (shown by Δ) on the projection plane 2, that is, on the photographing reference distance Lc, is not shifted to the left or right in any of FIG. 4A and FIG. It is copied in the center of. Furthermore, the real object (shown by □) located at the depth position of the distance z_f is copied to the left end on the image sensor 5-1 in FIG. 4A, and is copied to the right end on the image sensor 5-n in FIG. 4B. The

  The parallax images captured by the image sensors 5-1 to 5-n are processed by the image processing unit 40 and cut out according to the range used for display. A stereoscopic video can be displayed on the display device 42 in accordance with the cut-out parallax image. In FIG. 4A and FIG. 4B, a range used as a parallax image, that is, clipping regions 7-1.7-2 and 7-3 cut out for use as a parallax image are shown surrounded by a thick line. Here, the clipping regions 7-1.7-2 and 7-3 are set to the same size on the imaging surfaces of all the image sensors 5-1 to 5-n. The center reference line 10 of the clipping regions 7-1.1.7-2 and 7-3 is shifted from the center 12 of the image sensors 5-1 and 5-n by the shift value s_n or s_f defined by the equation (2). . If a real object to be clipped (indicated by ○, Δ, or □) is determined, the captured image is displayed on the three-dimensional display regardless of which real object (indicated by ○, Δ, or □) is captured. Thus, a stereoscopic image can be displayed near the display surface. Here, the equation (2) is established for the shift value s_n (or s_f) from the similarity of triangles.

s_n: x_c = z_n: Lc (2)
Here, x_c represents the X coordinate of the lenses 4-1, 4-n with the center of the camera 1 (corresponding to the center reference line 10) as a reference.

  In the case of perspective projection, the real object (◯) at the pop-out position is large and the real object (□) at the depth position is small. Accordingly, the real object (◯) at the pop-out position can be displayed in a constant size by setting the clipping range 7 wide, and the real object (□) at the depth position can be set narrow.

  The image in the clipping range 7 is decomposed to the pixel level and assigned to the display surface of the display panel 46 as a component of the element image. The information of the pixels of the three-dimensional display composed of the exit pupil and the pixel group on the back of the exit pupil changes depending on the viewing angle, and behaves as a pixel on which parallax information is presented, and a stereoscopic image is displayed on the display panel 46. . Therefore, the display panel 46 displays a clip image in which the objects 3-1 to 3-3 that the clipped photographer is interested in are displayed.

  As described above, areas to be used as parallax images, that is, clipping areas 7-1.7-2 and 7-3 are selected from the captured multi-viewpoint images 8-1 and 8-n. As long as the real objects 3-1 to 3-3 are photographed within the photographing range 6, the three-dimensional image can be displayed near the display surface based on the photographed image. More specifically, the size of the clipping range 7-1 to 7-3 indicates the size of the image when it is displayed on the three-dimensional display. For each viewpoint image in the clipping range 7-1 to 7-3, The position shift value s_n or s_f defines the depth or pop-up displayed on the display surface. In other words, the shift value s_n or s_f has a correlation with the distance in the depth direction or the protruding direction from the projection plane 2.

  As a method for controlling the clipping regions 7-1.1.7-2 and 7-3, other real objects (shown by Δ) other than the real objects (shown by Δ) on the projection plane 2 are displayed near the display surface. In the case of displaying on the screen, there is a problem that the image is distorted. The layout of the multi-camera 1 shown in FIG. 2 should be designed to reflect the relationship between the pixels of the three-dimensional display and the exit pupil as described with reference to FIG. However, when a real object in front of or behind the projection plane 2 is displayed in the vicinity of the display plane by adjusting the clipping range, strictly speaking, since the observation reference viewing distance and the imaging reference distance are different, there is a mismatch in transparency. . In the display, this discrepancy in transparency appears as distortion. Although not significant when the distance z_n or z_f is a small value, the display image is preferably displayed without distortion. Furthermore, when the shift positions of the lenses 4-1 and 4-n with respect to the image pickup device 5 are fixed, the shooting range is overlapped at a certain shooting reference distance Lc on the projection plane 2, so that an object to be shot is determined in advance. In this case, it is desirable to shoot from a position where the projection plane 2 substantially coincides with the shooting target, that is, from a position away from the shooting target by the shooting reference distance Lc. This is because when the objects 3-1 to 3-3 to be mainly displayed are arranged too far from the projection surface 2, there are the following problems in addition to the above-described transparency problem. The first problem is that the shooting range may be insufficient when clipping is performed in consideration of the shift values (s_n, s_f). Here, the clipping range 7 based on ◯ in FIG. 2 corresponds to the shooting range. The second problem is that the resolution may be insufficient when clipping is performed in consideration of the shift values (s_n, s_f). Here, since clipping is performed in the clipping range 7 with reference to □ in FIG. 2, there is a possibility that the resolution is insufficient. Also from such a viewpoint, adjustment (correction) of the photographing reference distance is preferable.

  When the shift positions of the lenses 4-1 and 4-n with respect to the image pickup device 5 can be changed, there is no possibility that the shooting range will be insufficient, but distortion due to the difference between the observation reference viewing distance and the shooting reference distance, The possibility of lack of resolution cannot be solved.

  From the above viewpoints, as long as the subject of interest of the photographer or the observer is set as the photographing target on the projection plane, even if the photographing target is clipped, the degree of transparency may be inconsistent. Therefore, the display image can be displayed without distortion.

  In the multi-camera system according to the embodiment of the present invention shown in FIG. 1, when a photographer mainly decides objects 3-1 to 3-3 to be displayed, or an object to be displayed mainly by an observer of a three-dimensional display. In any case, the photographer can grasp the target and optimize it. When the subject 3-1 to 3-3 to be displayed is determined by the photographer, the display target is specified by referring to the display on the display device 42 and specifying the display target via the input unit 45. The screen is displayed and the display target is determined. Further, as an example of the latter in which an observer mainly decides an object to be displayed, a 3D display unit 52 is installed in a remote place, and the content displayed on the 3D display unit 52 is viewed in more detail. It is assumed that an instruction for designating a range to be viewed is sent via the input unit 55. In accordance with an instruction for designating a range to be viewed in more detail, a display target surface for specifying a display target may be displayed and the display target may be determined. The display target plane is subjected to multi-camera shooting in which the shooting distance of the multi-camera with respect to the display target is corrected so that the projection plane matches and clipping is not performed even if clipping is performed.

  FIG. 6 is a flowchart for performing optimum photographing by determining an object that the photographer mainly wants to display using the image processing unit 40 indicated by blocks in FIG. In the image processing unit 40 shown in FIG. 5, the display unit 5 corresponds to the display device 42 shown in FIG. 1, and corresponds to a viewer provided in the multi-camera 1 like a viewer of a digital camera. With reference to FIG. 5 and FIG. 6, the processing procedure of the multi-viewpoint image in the embodiment of the present invention will be described below.

  As described above, the layout of the multi-camera 1 reflects the configuration of the display unit in the stereoscopic image display device. Therefore, preferably, the multi-camera 1 is configured such that the display surface of the viewer as the display unit 5 coincides with the projection surface 2. In addition, designing a multi-camera with such a configuration can improve the usability of the multi-camera 1.

  When shooting is started with the multi-camera 1 (step S10), first, in the vicinity of the projection plane 2 in which the object to be displayed mainly in the state where the shooting is started (initial state) is displayed in the viewer as the display unit 42 Is displayed as a three-dimensional image (step S11). When a display target is not displayed near the projection plane displayed in the viewer, the shooting position is moved back and forth while viewing the viewer, which is a three-dimensional image display device, and the display target is displayed on the projection plane 2. Such a shooting position is searched (step S12). It is determined whether the object is displayed in an appropriate display size in a state where the display object is substantially displayed on the projection plane 2 (step S13). In step S13, if the display size of the display object is to be adjusted, an instruction to change the display range (clipping range) is input to the multi-camera 1 (step S14). Specifically, the shift values (s_n, s_f) of the clipping regions 7-1.7-2, 7-3 are maintained as they are, and the display size of the object is enlarged or reduced.

  In order to maintain the transparency when the range of the clipping regions 7-1.1.7-2 and 7-3 is enlarged or reduced, the imaging reference distance and the camera interval are adjusted (corrected) by providing feedback to the imaging reference distance. Is done. (Step S15) FIG. 7 and FIG. 8 show how the photographing reference distance is changed from the distance L to the distance L ′ and the photographing reference distance is adjusted based on the change of the photographing distance.

  FIGS. 7 and 8 show the shooting positions of the shooting units 30-1 to 30-n for displaying images without distortion when the clipping range 7-1 is set to a range narrower than the projection plane 2. FIG. Yes. When the width of the projection plane 2 is set to W_t and the clipping region 7-1, that is, the width of the clipping range is set to W_c, the photographing reference distance L ′ and the camera position x_c ′ are changed to satisfy the expression (3). (Step S15).

W_c / W_t = x_c ′ / x_c = L ′ / L (3)
When the clipping range 7-1 is changed as shown in FIG. 7 while maintaining the shooting position, the ideal shooting reference distance L is set to a short distance L ′. Accordingly, the projection plane 2 after the change of the clipping range 7-1 is set to the front as shown in FIG. 8 as compared to the conventional projection plane 2 before the change of the clipping range shown in FIG. As a result, the z coordinate of the display object displayed on the viewer is relatively moved to the back. As a result, the photographer is informed that he / she is shooting at a distance greater than the shooting reference distance, and the shooting position is changed in order for the photographer to take a shot at the shooting reference distance (specifically, the camera moves forward). Encourage

  In the processing described above, the camera position x_c of the photographing units 30-1 to 30-n is changed to the camera position x_c ′. This change is performed as effective photographing data selected from the photographing units 30-1 to 30-n. This corresponds to a change of the photographing units 30-k to 30-m used. (K and m are integers having a relationship of 1 <k <m <n) With the change to the imaging units 30-k to 30-m, as shown in step S16, the imaging units 30-k to 30-m As for the images, parallax images having the required number of parallax images are prepared by interpolation processing. The interpolated parallax image is preferably colored in an image different from other parallax images in order to clearly indicate that the parallax image has been interpolated.

  As described above, the photographer confirms that the shooting distance is changed in accordance with the change of the clipping range, and that the projection 2 is changed in the display screen. What is necessary is just to move so that the distance to an object may be shortened. The camera coordinate x_c should ideally be changed, but it is particularly difficult to reduce the camera interval x_c because of the structure of the multi-camera. Therefore, it is preferable to leave the actual camera pitch of the multi-camera as it is. Then, a multi-viewpoint image in which the shooting position (x coordinate) x_c ′ is changed is generated by image interpolation processing by either interpolation or extrapolation depending on the shooting conditions as it is. This multi-viewpoint image may be used (step S16). A screen is displayed with the interpolated multi-viewpoint image, and step S12 is executed according to the display screen.

  As described above, even if the display object can be displayed at the display position by moving the shooting position and image interpolation, the depth of the three-dimensional display is large because the depth of the display object is large. In some cases, the direction display range may not be entered (NO in step S17). In such a case, as shown in FIG. 9, if the photographing units 30-1 to 30-n are shifted so that the camera coordinates x_c are, for example, ½, the displayed space is z. The direction can be reduced to approximately ½ (step S18). In the multi-camera, since the photographing units 30-1 to 30-n cannot actually be shifted, photographing data at ½ camera intervals is prepared by image interpolation processing even for images photographed at the camera coordinates x_c. It should be done.

  In step S17, when the display target is within the display range in the depth direction of the 3D display, or when the display target is within the display range in the depth direction of the 3D display in the process of step S18. The image data from the imaging units 30-1 to 30-n or 30-k to 30-m is rearranged into display images under these imaging conditions to create element image data, and a storage device (not shown) ). (Step S19) In this way, the data of the element image is prepared and the series of processes is completed. (Step S17) If necessary, the processing from step S21 is repeated again for detailed setting.

  As shown in FIG. 5, the video processing unit that executes the above-described flow includes a storage unit 20 that stores a multi-viewpoint image captured by the multi-camera 1 and a shooting condition storage unit that stores shooting conditions of the multi-camera 1. 24. The following clipping conditions C1 to C3 in the multi-viewpoint image captured by the multicamera 1 are stored in the clipping condition storage unit 22.

(C1) Clipping size (initial value is 1 as a standardized value)
(C2) Multicamera 1 interval (initial value, 1 as standardized value)
(C3) Shooting reference distance (pseudo adjustment is possible with shift value (initial value is 0))
In accordance with the position of the multi-camera 1 stored in the clipping condition storage unit 22, the parallax image generation unit 26 performs image interpolation processing and clipping processing when the image acquisition position needs to be changed. The parallax image created by the parallax image generation unit 26 is rearranged in units of pixels by the rearrangement processing unit 28 and is changed to a format to be displayed on the three-dimensional display. Here, the format for a three-dimensional display means an element image array in which element images are arranged in a tile shape.

  The element image array created by the rearrangement processing unit 28 is supplied to the display unit 5 and a three-dimensional image is displayed on the display unit 5. The element image array supplied to the display unit 5 is supplied to the display condition adjustment unit 32. Here, while viewing the video displayed on the display unit 5, the display condition adjustment unit 32 adjusts the clipping size and the position (interval) of the multi-camera 1. Here, as already described, the adjustment of the position (interval) of the multi-camera 1 includes a virtual camera position or camera interval at which shooting data prepared by image interpolation can be acquired. When adjustment is performed by the display condition adjustment unit 32, especially when the clipping size is changed, it is necessary to reflect the shooting conditions and to correct the shooting reference distance and the camera interval while keeping the similarities to the layout of the shooting conditions. The change of the photographing reference distance can be realized in a pseudo manner by adjusting the shift values of the clipping ranges 7-1, 7-2, and 7-3. However, basically, the display condition adjusting unit 32 does not deal with the adjustment of the shift value of the clipping range, and basically prompts the change of the photographing reference distance. The parameters adjusted by the display condition adjustment unit are reflected in the contents of the clipping condition storage unit 24. As a result, the display state of the three-dimensional image on the display unit 5 is updated in real time.

  Here, when the display condition set in the display condition adjustment unit 32 is reflected and displayed on the display unit 5, a case where the shooting range of the multi-viewpoint image is insufficient is assumed. The image in the area where the shooting range is insufficient may be supplemented by processing for replacing the parallax information of the multi-view image that is adjacent and already obtained. In order to display the replacement parallax image to indicate that the image is replaced, the replaced image may be colored to notify the observer.

  The display condition adjustment unit 32 recognizes an object to be displayed in the vicinity of the display surface by image processing, and can track the object even if there is a change in the shooting condition or a movement of the shooting target. It is preferable to include a tracking processing unit (not shown) that can be used. It is preferable that the tracking processing unit automatically change or update the parameter stored in the clipping condition storage unit 22 to always display the object on the display surface.

  FIGS. 10 and 11 show an image processing unit 40 that can determine an object that an observer of a three-dimensional display mainly wants to display, and a processing flow in the image processing unit 40. The apparatus shown in FIG. 10 is different from the processing apparatus shown in FIG. 5 and includes two display units 5-1 and 5-2 as shown in FIG. Here, one display unit 5-1 corresponds to a viewer provided in the multi-camera 1, and the other display unit 5-2 corresponds to a three-dimensional display.

  As with the process in the float shown in FIG. (Step S <b> 10) First, a three-dimensional image in a state where imaging is started (initial state) is observed by an observer, and parameters are adjusted by the observer. The parameter adjustment is performed while viewing the display unit 5-2. The adjustment parameter is displayed on the display unit 5-2, and the parameter is transmitted via the input unit 55 to the image processing unit 40 via the transmission / reception units 48 and 58. This is input to the display condition adjustment unit 32. (Step S21) Accordingly, an image reflecting parameters other than the shift amount as a parameter is displayed on the display unit 5-2. An image of the virtual projection plane 2 converted from the shift amount is generated by CG. This projection plane 2 is colored and displayed in the display section 5-2. (Step S22) Here, unlike the photographer, since the observer cannot change the photographing reference distance L, the projection surface 2 is displayed on the display surface of the three-dimensional display even if the shift amount as a parameter is operated. It will only be done. The photographer observes a screen similar to the screen observed by the observer in the viewer as the display unit 42. When a display target is not displayed near the projection plane displayed in the viewer, the shooting position is moved back and forth while viewing the viewer, which is a three-dimensional image display device, and the display target is displayed on the projection plane 2. Such a shooting position is searched (step S12).

  The observer determines whether the object is displayed with an appropriate display size in a state where the display object is substantially displayed on the projection plane 2 by the movement of the photographer (step S13). In this step S13, when the observer wants to adjust the display size of the display object, an instruction to change the display range (clipping range) is input by the observer to the image processing unit 40 via the input unit (step S14). . Specifically, the shift values (s_n, s_f) of the clipping regions 7-1.7-2, 7-3 are maintained as they are, and the display size of the object is enlarged or reduced.

  When the range of the clipping regions 7-1.1.7-2 and 7-3 is enlarged or reduced, actually, feedback is given to the photographing reference distance as described with reference to FIGS. The photographing reference distance and the camera interval are adjusted. (Step S15) Thereafter, image interpolation processing is performed in accordance with the change of the photographing reference distance, and a multi-viewpoint image in which the photographing position (x coordinate) x_c ′ is changed is generated. (Step S16) A screen is displayed with this interpolated multi-viewpoint image, and step S12 is executed according to this display screen.

  When the depth of the display object is large, i.e., thick, it does not fall within the display range in the depth direction of the three-dimensional display (NO in step S17), the camera coordinates x_c are shifted as shown in FIG. Then, the displayed space can be reduced (compressed) in the z direction (step S18). In the multi-camera, since the photographing units 30-1 to 30-n cannot actually be shifted, photographing data at ½ camera intervals is prepared by image interpolation processing even for images photographed at the camera coordinates x_c. It should be done.

  In step S17, when the display target is within the display range in the depth direction of the 3D display, or when the display target is within the display range in the depth direction of the 3D display in the process of step S18. The image data from the imaging units 30-1 to 30-n or 30-k to 30-m is rearranged into display images under these imaging conditions to create element image data, and a storage device (not shown) ). (Step S19) In this way, the data of the element image is prepared and the series of processes is completed. (Step S17) If necessary, the processing from step S21 is repeated again for detailed setting.

  Note that the input clipping size of the clipping region, the camera interval linked to the clipping size, and the camera interval operation of the squashing display processing of the depth method are taken into consideration and stored in the clipping condition storage unit 22 as a clipping condition. In this process, the shift value is not reflected. This is because it is necessary to display a shift on the display unit 1 to prompt the photographer to move the photographing position. At this time, the shift value desired to be displayed on the display surface, that is, the imaging reference distance, is determined from the shift value operated by the observer using the display condition adjustment unit 32 as to which part of the imaging target is desired to be the display surface. For example, as shown in FIG. 12, a plane 9 to be imaged is displayed in a pseudo manner by CG. The photographer only needs to move the photographing position back and forth so that the photographing target surface 9 coincides with the projection surface.

  By the above-described method, it is possible to inform the photographer of the deviation from the ideal value of the photographing reference distance, which causes distortion in the three-dimensional image display, and to provide a guideline for moving to the ideal photographing reference distance. .

  Although details of the image interpolation processing are not described, an existing method such as a known bilinear method or a bicubic method may be used. The camera is expected to have functions such as zoom in, out, lens shift, and focal length movement, but it is obvious that the present invention can be applied to these operations.

  A method for acquiring a captured image for displaying a stereoscopic image that enables acquisition of a multi-viewpoint image for displaying a display object without distortion within a display range of a three-dimensional display including a parallax barrier, and a three-dimensional image from the acquired image. A method and apparatus for displaying an image can be provided.

  1. . . Multi camera, 2. . . Projection plane, 3-1, 3-2, 3-3. . . Real object, 4-1 to 4-n. . . Lenses, 5-1 to 5-n. . . Imaging device, 7-1 to 7-n. . . 5. clipping region; . . Shooting range, 30-1 to 30-n. . . Photographing section, 40. . . Image processing unit, 42. . . Photographer display section, 44. . . Drive circuit, 45. . . Input unit, 46. . . Display panel, 47. . . Parallax barrier, 52. . . Observer display section, 54. . . Drive circuit section, 55. . . Input unit, 56. . . Display panel, 57. . . Parallax barrier, 60. . . Element image area

Claims (5)

A display unit in which pixels are arranged in a matrix of rows and columns in a display region ; and an optical control unit that is installed in front of the display unit and controls the direction of light emitted from the display unit ;
A first three-dimensional display that generates a stereoscopic image by displaying element image data in which parallax image data is rearranged in an element image area in the display area that is determined according to the direction of the light beam to be controlled Equipment,
A multi-camera for capturing multi-viewpoint images of real objects from a plurality of directions , having a plurality of photographing units arranged at specific intervals, and having a projection surface as a photographing reference plane;
An image processing unit for processing the multi-viewpoint images from the multi-camera, from said multi-viewpoint images, based on the clipping region information about the range utilized for generating the parallax image, said from the multi-viewpoint image A parallax image generation unit that generates parallax image data by clipping a range to be used as a parallax image;
A rearrangement processing unit that rearranges the values of the pixels of the parallax image data into the element image data ;
An imaging condition storage unit that stores the imaging conditions of the multi-viewpoint image,
A display condition adjusting unit for adjusting a parameter for displaying the real object in a desired size near the projection plane;
An image processing unit that generates an image for displaying information on the corrected shooting reference distance for displaying a specific stereoscopic image from the parameter and the shooting condition on the display unit;
A stereoscopic video display device comprising: A second display unit in which pixels are arranged in a matrix of rows and columns in the second display region;
A second optical control unit that is installed in front of the second display unit and controls a direction of a light beam emitted from the second display unit;
And displaying the element image data in which the parallax image data is rearranged in an area within the second display area and determined according to the direction of the light beam controlled by the second optical control unit. A second three-dimensional display device for generating a stereoscopic image with
An input unit for inputting instructions;
The stereoscopic video display apparatus according to claim 1, comprising: The display condition adjustment unit adjusts the parameter based on the input clipping condition when an instruction regarding the clipping condition is input to the input unit ,
The parallax image generation unit generates the parallax image data according to the adjusted parameters,
Wherein the image processing section, based on the input clipping conditions, determine the virtual projection plane, the stereoscopic image display device according to the virtual projection plane in claim 2, characterized in that causes the display on the display unit. The parallax image producing section, when generating the parallax image reflecting the set display condition on the display condition adjusting unit, wherein when the shooting range of the multi-view image is insufficient, adjacent already obtained wherein replacing the parallax information of the multi-view image in the image area to be insufficient the stereoscopic image display apparatus according to claim 3, characterized in that coloring the parallax images this substitutions. The display condition adjusting unit, including a tracking processor to track the real object, the stereoscopic image display apparatus according to claim 1, characterized in that.

Images