JP2011166316A - Imaging method, imaging display device and remote face-to-face communication device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a dead space before a video display device without causing deterioration in image quality or the defect of a display image. <P>SOLUTION: At the upper part of a display device 110, an imaging part 8 comprising a visible light video camera 10, a distance camera 11 and a half mirror 12 is installed through an arm part 6. The imaging part 8 is fixed rotatably in the vertical direction to the arm part 6 through a rotating mechanism 9. The visible light video camera 10 photographs the video image of the face of a user 100. The distance camera 11 photographs a distance video image to the face of the user 100. A front face video image generation part 13 uses the video image of the face of the user 100 photographed by the visible light video camera 10 and the distance video image photographed by the distance camera 11 to generate the face video image of the user 100 photographed from a position (front) different from the video image photographed from the position of the visible light video camera 10 (video image photographed from the upper direction). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging method, an imaging display device, and a remote face-to-face communication device.

  FIG. 5 is a diagram illustrating a schematic configuration of a conventional imaging display device. In FIG. 5, 100 is a user, 110 is a display device, 111 is a half mirror, and 120 is an imaging device (camera). In the conventional imaging display device, the user 100 can observe the display device 110 through the half mirror 111 and simultaneously can capture a face image of the user 100 using the reflection of the half mirror 111.

  However, in the conventional imaging display device, since the half mirror 111 is installed at an angle of 45 degrees between the display device 110 and the user, the space in which the half mirror 111 is installed is a dead space. There was a problem of becoming.

  Therefore, as shown in FIG. 6, a method has been devised in which a half mirror 111 is installed on the back surface of a display device 110 made of a liquid crystal panel (see, for example, Patent Document 1). In FIG. 6, reference numeral 140 denotes a first polarizing plate, 150 denotes a second polarizing plate, and 160 denotes a backlight.

  In addition, as shown in FIG. 7, there is also a technique in which a minute hole 170 is formed on the front screen of the display device 110 and a user (not shown) is photographed with the imaging device 6 installed behind the display device 110 through the hole 170. It has been devised (for example, see Patent Document 2).

JP-A-4-145789 Japanese Patent Laid-Open No. 4-167690

  However, in the prior art disclosed in Patent Document 1, the transmittance of the display device 110 formed of a liquid crystal panel is low, and further, diffraction blur due to the image structure of the liquid crystal panel occurs and the image is blurred. There was a problem that I could not shoot.

  Further, in the related art disclosed in Patent Document 2, when the diameter of the hole 170 is increased so that the diffraction by the hole 170 can be suppressed, a defect in the display image (opened in the display device 110 for photographing the subject with the imaging device 6). There is a problem in that a part of the image displayed on the display device 110 is lost due to the hole 170).

  The present invention has been made in view of such circumstances, and an object of the present invention is to reduce the dead space in front of the video display device without causing deterioration in image quality or defect in the display image. An imaging method, an imaging display device, and a remote face-to-face communication device are provided.

  In order to solve the above-described problems, the present invention provides a first image of a subject viewed from the front of a display surface of a display device that displays an image by a visible light video camera from the display device side. Using a distance image captured by the distance camera from a first image captured by the visible light image camera, a second step of capturing a distance image indicating a distance to the subject by the distance camera And a third step of generating a second video taken from a position different from the position of the visible light video camera.

  In the present invention according to the above invention, the third step generates the 3D video in real time from the first video shot by the visible light video camera and the distance video shot by the distance camera. And a fifth step of generating a second video imaged from a position different from the position of the visible light video camera by rotating the 3D video image in a virtual space. To do.

  In order to solve the above-described problem, the present invention captures from the display device side a display device that displays an image, and a first image of a subject viewing the display surface of the display device from the front. The visible light image camera, a distance camera that captures a distance image indicating a distance to the subject, and a first image captured by the visible light image camera, the distance image captured by the distance camera, and the visible image. An imaging display device comprising: a video generation unit that generates a second video shot from a position different from the position of the optical video camera.

  The present invention further includes a half mirror disposed on an optical axis of the visible light video camera and the distance camera in the above invention, and the visible light video camera and the distance camera are interposed via the half mirror. And shooting the subject from the same viewpoint.

  The present invention further includes a half mirror disposed on an optical axis of the visible light video camera and the distance camera in the above invention, and the visible light video camera and the distance camera are interposed via the half mirror. Thus, both the optical axes are made to coincide with each other, and the subject in front of the display surface of the display device is photographed.

  The present invention is the above invention, further comprising a half mirror disposed on an optical axis of the visible light video camera and the distance camera, wherein the distance camera includes infrared light irradiation means for irradiating infrared light. The half mirror reflects infrared light irradiated from the infrared light irradiation means of the distance camera to irradiate the subject, and transmits visible light from the subject to the visible light video camera. It is a half mirror to be incident, or a half mirror that reflects visible light from the subject to enter the visible light video camera, transmits infrared light emitted from the distance camera, and irradiates the subject It is characterized by being.

  Further, in order to solve the above-described problem, the present invention is a remote face-to-face communication device configured by connecting imaging display devices to each other, the imaging display device including a display device that displays an image, and the display A visible light video camera for photographing a first image of a subject viewing the display surface of the device from the front, a distance camera for photographing a distance image indicating a distance to the subject, and the visible light A video generation unit configured to generate a second video shot from a position different from the position of the visible light video camera from a first video shot by the video camera using a distance video shot by the distance camera; Is a remote face-to-face communication device.

  According to the present invention, the user who is looking at the display surface of the display device can be photographed from an arbitrary position, and an image of the user photographed from a position different from the position of the visible light video camera is generated in real time on the display device. Since it can be displayed, a sufficient work space in front of the user can be secured. In addition, by using a plurality of imaging display devices as remote face-to-face communication devices, natural communication is possible at each point because the face image of the other party at another point is felt as if facing the front. .

It is a figure which shows schematic structure of the imaging display apparatus by embodiment of this invention. It is a figure which shows schematic structure of the imaging part of the imaging display apparatus by embodiment of this invention. It is a flowchart which shows the process sequence of the imaging display method by embodiment of this invention. 1 is a diagram illustrating a schematic configuration of a remote face-to-face communication device using a plurality of imaging display devices according to an embodiment of the present invention. It is a figure which shows schematic structure of the conventional imaging display apparatus. It is a figure which shows schematic structure of the other example of the conventional imaging display apparatus. It is a figure which shows schematic structure of the other example of the conventional imaging display apparatus.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  The present invention relates to a remote face-to-face communication device (one-to-one TV conference system) that combines an imaging device and a display device. The feature of the present invention is that the optical axis of the visible light camera and the distance camera are matched using a half mirror and the optical axis of the visible light camera is adjusted in advance so as to capture a human face with the visible light camera. Is attached to the display device. With this configuration, the dead space in front of the display device can be eliminated, and compared with Patent Documents 1 and 2, it is possible to reduce the image deterioration and display the face of the conversation person in any direction on the screen. With this process, even when the camera is installed at the top of the screen, a face facing the front can be displayed on the screen, and communication without a sense of incongruity becomes possible.

  FIG. 1 is a diagram showing a schematic configuration of an imaging display device according to an embodiment of the present invention. In all the drawings for explaining the present embodiment, the same reference numerals are given to those having the same function, and the repeated explanation thereof is omitted. In FIG. 1, an imaging unit 8 including a visible light video camera 10, a distance camera 11, and a half mirror 12 is installed on the upper part of the display device 110 via an arm unit 6 attached to the back side of the display device 110. ing. The imaging unit 8 is fixed to the arm unit 6 via the rotation mechanism 9 so as to be rotatable in the vertical direction. For this reason, it can be adjusted by the rotation mechanism 9 so that the position of the face of the user 100 who is looking at the display surface of the display device 110 and the optical axis of the imaging unit 8 match.

  The visible light video camera 10 captures an image of the face of the user 100. The distance camera 11 captures a distance image from the distance camera 11 to the face of the user 100. The face image of the user 100 captured by the visible light video camera 10 of the imaging unit 8 and the distance image captured by the distance camera 11 are supplied to the front face image generation unit 13.

  The front face image generation unit 13 uses the image of the face of the user 100 photographed by the visible light image camera 10 and the distance image from the distance camera photographed by the distance camera 11 to the face of the user 100, A face image (directly in front) of the user 100 captured from a position different from the image captured from the position of the visible light image camera 10 (image captured from above) is generated. The display device 110 displays the front face video generated by the front face video generation unit 13. When viewed from the user 100, the image of the front face displayed on the display device 110 feels as if it is facing directly in front.

  In the present embodiment, an image taken from the front is generated. However, not only the front but also an image at an arbitrary position can be generated.

  FIG. 2 is a diagram illustrating a schematic configuration of the imaging unit 8 of the imaging display device according to the present embodiment. In FIG. 2, the viewpoints of the visible light video camera 10 and the distance camera 11 are matched through a half mirror 12. By matching the viewpoints of the cameras, the user 100 can be photographed from the same position. Further, the optical axes of the visible light video camera 10 and the distance camera 11 are matched with each other through a half mirror. In this way, since the viewpoint and the optical axis can be taken to coincide with each other, the dead space is reduced due to the installation locations of a plurality of cameras.

  The half mirror 12 is installed at an angle of 45 degrees with respect to the lens surfaces of the visible light video camera 10 and the distance camera 11. Further, in order to make it as compact as possible, the half mirror 12 is installed at a position closest to the lens of each camera while securing a position where the angles of view of the visible light video camera 10 and the distance camera 11 do not deviate from the half mirror 12.

  The visible light video camera 10 photographs the face of the user 100 looking at the display surface through the half mirror 12. The distance camera 11 irradiates the user 100 with the infrared light irradiated by the infrared light irradiation function of the distance camera 11 and hits the user 100, and the reflected light of the infrared light hitting the user 100. Corresponds to the half mirror 12, and the infrared light is captured by the CCD camera included in the distance camera 11. The arrival time from irradiation of infrared light to reflection by the subject and return to the CCD camera is measured by an image sensor in the distance camera 11 and photographed as a distance image.

  When the position of the visible light video camera 10 and the position of the distance camera 11 are interchanged, the half mirror 12 transmits the infrared light of the distance camera 11 and reflects the visible light of the visible light video camera 10. However, it is also possible to shoot a subject image and a distance image.

  FIG. 3 is a flowchart illustrating a processing procedure of the imaging display method according to the present embodiment. In the imaging display device shown in FIG. 1, the position of the visible light camera 10 is determined from the RGB image of the face taken by the visible light video camera 10 of the imaging unit 8 and the distance video taken by the distance camera 11. 12 is a flowchart illustrating a procedure from generating a face image captured from a different front to outputting a front face image on the display surface of the display device 110;

  First, the RGB image of the face of the user 100 photographed by the visible light video camera 10 and the distance video photographed by the distance camera 11 are acquired from the imaging unit 8 (step 10, S12). Next, gamma processing is performed by adjusting the RGB image of the face and the distance image using an 8-bit table (step S14). Next, face segmentation processing is performed from the gamma-processed face RGB image and face distance image (step S16).

  Next, an alignment process is performed between the face part of the RGB image subjected to the face part cut-out process and the face part of the distance image (step S18). Since there are fewer pixels in the distance image compared to the RGB image, the RGB image is adjusted to the size of the distance image. Next, the face portion of the RGB image subjected to the alignment process and the face portion of the distance image are synthesized to generate a 3D image of the face (step S20).

  Next, a 3-axis rotation process is performed on the 3D image of the face to generate a front image of the face taken from the front different from the position of the visible light video camera 10 (step S22). The three-axis rotation processing can be performed by inputting a tilt angle of the installed visible light video camera 10 and rotating it to make it a front face, or by using a face detection engine to make a front face.

  Next, a process of aligning the position of the center of the front face image and the center of the display surface of the display device 110 is performed (step S24). Next, a process for returning the gamma (adjustment in the 8-bit table) processed in step S14 to the original state for the front face image obtained by aligning the center of the front face image with the center of the display surface of the display device. This is performed (step S26). Next, the front face image obtained by aligning the center of the front face image with the gamma restored to the original state and the center of the display surface is output to the display surface of the display device 110 (step S28).

  FIG. 4 is a diagram showing a schematic configuration of a remote face-to-face communication device using a plurality of imaging display devices according to the present embodiment. As shown in FIG. 4, the remote face-to-face communication device is configured by using two of the above-described imaging display devices and connecting the display device 110 and the front face image generation unit 13 at two points to each other. That is, two imaging display devices are prepared, and the display device 110 of the other imaging display device is connected to the front face image generation unit 13 of one imaging display device. Needless to say, the connection may use not only video distribution but also a communication network. At each point, it feels as if the video of the other party at the other point is directly in front, so natural and highly realistic communication is possible.

  Although the invention made by the present invention has been specifically described based on the above-described examples, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention. Of course there is.

  Further, by recording a program for realizing each step by the front face image generation unit 13 described above on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. The encoding process and the decoding process may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

6 Arm unit 8 Imaging unit 9 Rotating mechanism 10 Visible light image camera 11 Distance camera 12 Half mirror 13 Front face image generation unit 100 User 110 Display device

Claims (7)

A first step of photographing a first image of a subject viewing a display surface of a display device that displays an image from the front by a visible light video camera from the display device side;
A second step of photographing a distance image showing a distance to the subject with a distance camera;
A third step of generating a second video shot from a position different from the position of the visible light video camera from the first video shot by the visible light video camera, using the distance video shot by the distance camera. An imaging method comprising: and. The third step includes
A fourth step of generating a 3D image in real time from the first image captured by the visible light image camera and the distance image captured by the distance camera;
5. A fifth step of generating a second video imaged from a position different from the position of the visible light video camera by rotating the 3D video image in a virtual space. The imaging method described. A display device for displaying images;
A visible light video camera that captures a first image of a subject viewing the display surface of the display device from the front, from the display device side;
A distance camera that captures a distance image showing the distance to the subject;
A video generation unit that generates a second video shot from a position different from the position of the visible light video camera using a distance video shot by the distance camera from a first video shot by the visible light video camera; An imaging display device comprising: A half mirror disposed on the optical axis of the visible light video camera and the distance camera;
The visible light video camera and the distance camera are:
The imaging display device according to claim 3, wherein the subject is photographed from the same viewpoint through the half mirror. A half mirror disposed on the optical axis of the visible light video camera and the distance camera;
The visible light video camera and the distance camera are:
The imaging display device according to claim 3, wherein the object in front of the display surface of the display device is photographed by causing both optical axes to coincide with each other through the half mirror. A half mirror disposed on the optical axis of the visible light video camera and the distance camera;
The distance camera includes infrared light irradiation means for irradiating infrared light,
The half mirror is
A half mirror that reflects infrared light irradiated from the infrared light irradiation means of the distance camera to irradiate the subject, transmits visible light from the subject, and enters the visible light video camera; Alternatively, it is a half mirror that reflects visible light from the subject to enter the visible light video camera, transmits infrared light emitted from the distance camera, and irradiates the subject. The imaging display device according to claim 3. A remote face-to-face communication device configured by connecting imaging display devices to each other,
The imaging display device
A display device for displaying images;
A visible light video camera that captures a first image of a subject viewing the display surface of the display device from the front, from the display device side;
A distance camera that captures a distance image showing the distance to the subject;
A video generation unit that generates a second video shot from a position different from the position of the visible light video camera using a distance video shot by the distance camera from a first video shot by the visible light video camera; A remote face-to-face communication device comprising:

Images