JP2005117285A - Information input device, communication terminal and communication method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information input device which can recognize a position on a screen efficiently and can be constituted inexpensively, as a means for recognizing a position on a screen designated by a user, and to provide a high presence communication terminal provided with a television conference function and an electronic board function and a communication method for obtaining it as a new communication system. <P>SOLUTION: An information input device by image processing is obtained by incorporating a polarization filter. A high presence communication terminal having an electronic board function and a television conference function and capable of communicating between terminals having different display sizes is enabled by employing image processing by the transmission of camera parameters and photography environment parameters and mirror plane transformation and then displaying the video image on an information sharing screen to coincide with the video image of a communication partner. Furthermore, the problem of an existing information input device for recognizing a position on a screen designated by a user that the device is intricate and expensive because a sensor is employed, and a problem that a high presence television conference system having an electronic board function cannot be obtained because there is no technique for synthesizing a shared information screen and the video image of the communication partner accurately are solved. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an information input device for inputting a position on a screen and a locus of a user-designated position, and a communication terminal for communicating a moving image, and in particular, information on a presentation device, a display device, an electronic board system, a video conference system, and the like. The present invention relates to a communication display device.

  An electronic board system has been developed in which various information is displayed on a large display in a conference room or the like, characters and pictures are drawn according to user instructions, and the conference is performed in the form of a whiteboard. In these systems, a means for recognizing the position pointed to by the user on the screen and enabling drawing and computer operation is necessary. There are a method, a resistance film method, a capacitance method, and the like.

In the ultrasonic method and the infrared method, an ultrasonic transmitter and an infrared transmitter are attached around the display device, and a position designated by a user with a pointer such as a pen is recognized by reflection or shielding of ultrasonic waves or infrared rays. The resistance film method and the capacitance method recognize a position designated by the user by attaching a film whose resistance value or capacitance changes by contact of an object on the display. However, these methods require a large transmitter to be placed around the display or a film having a special mechanism on the display, which makes the device complicated and extremely expensive. There was a problem.
On the other hand, a video conference system has been developed as a system for conducting a conference with a person at a remote place. Such a video conference system has a function to operate the camera from a remote location by exchanging information such as the direction the camera is facing and the zoom value, but the positional relationship between the camera and the display Since information about the shooting environment such as the size of the display and the display is not transmitted, it is not possible to adjust the communication partner's video to be displayed in full size, and there is a problem that the sense of reality is poor. In addition, since it does not have the functions of the electronic board system described above, it is possible to hold a meeting while sharing information with a remote person, to grasp the position on the screen pointed to by the communication partner, and to display it accurately. That was difficult.

On the other hand, a video conference system in which a user is displayed in actual size on two communication terminals of the same type using a large display with an emphasis on realism has been proposed (see Non-Patent Document 1, for example).
Further, as a device having a function of drawing on a photographed image, there is a sticker printing device including a camera for photographing a user, a display for displaying an image, and a pen-type tablet-type input device (see, for example, Patent Document 1). ).

IPSJ Technical Report on Audio Visual Complex Information Processing 23-2 pp. 3-8 Published December 4, 1998 Hiroshi Yasuda, Teruyoshi Aoki, Custard Widyo, Nobuki Sakamoto, Kazunori Suzuki, Yuki Sabu, co-authored

JP-A-10-108112

As described above, the conventional sensor method includes an ultrasonic method, an infrared method, a resistance film method, a capacitance method, and the like, but these have a problem that the apparatus is complicated and very expensive.
Until now, electronic board systems and video conferencing systems existed independently, and there was no highly realistic system that fused them. For example, since the technique described in the non-patent document is based on the premise that the display size is fixed, information on the shooting environment is not transmitted, and communication with terminals of different sizes is not possible. In addition, unlike the electronic board system, it does not have a function of drawing on the screen. In addition, the above sticker printing apparatus can draw with a pen or the like while projecting one's own figure, but since it is not a communication terminal, it transmits the image of the communication partner and has the same size as the actual one. It doesn't have the function to adjust and display.

  In view of the above-described conventional technology, the present invention is an information input device that can be efficiently recognized and configured at low cost as means for recognizing a position on a screen pointed to by a user and performing drawing or computer operation. As a new communication system, a highly realistic communication terminal having both a video conference function and an electronic board function, and a communication method for realizing the communication terminal are provided.

In order to solve the above problems, typical ones disclosed in the present application are as follows.
An information input having a display device that displays an image, a pointer that indicates a position on the screen, and a camera device that recognizes the position of the pointer, and a polarization filter that allows each of the camera device and the display device to pass different polarized light Use the device.
In addition, an imaging device that captures a user, an image display device that displays an image, an information input device that indicates a position on the screen and inputs a designated position, a network device that performs communication, and image conversion And an image processing device that superimposes information screens and captured images, a voice input / output device that inputs and outputs audio, and a controller that controls and manages these devices. A communication terminal that converts the captured image so that it looks the same as the actual size by processing such as enlargement / reduction, projection, clipping, etc., using the transmitted shooting environment parameters, and displays it on the display device in a superimposed manner on the information screen Is used.
In addition, as a communication method in the video conference system, video information of the camera that captured the participant, data information such as a computer screen and numerical values, camera parameters such as camera orientation and zoom, positional relationship between the camera and display, display A communication method for transmitting shooting environment parameters indicating the size, resolution, distance to the subject, and the like is used.

  According to the present invention, an information input device capable of recognizing a position on a screen pointed to by a user and performing drawing or computer operation can be configured at low cost. In addition, the information sharing screen on the electronic board and the video of the other party in the video conference are displayed so as to match, and even with terminals with different display sizes, highly realistic communication is possible such as having a meeting while drawing a picture through the glass. It becomes possible to provide a communication terminal and a communication method to be realized.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 shows an embodiment of an information input device according to the present invention, and its features will be described.
First, the display device (101) is a device that displays various image information, and displays information on the display screen (102). The display screen (102) is provided with a polarizing filter, or has a function equivalent to this in the structure of the display device. For example, a liquid crystal display has a structure in which a polarizing filter is incorporated, and in this case, it is not necessary to add a polarizing filter on the display. The polarizing filter may be generally used and has a property of passing only light having a certain polarization and blocking light having other polarization. There are two types of polarizing filters: a circular polarizing filter that passes only polarized light in a fixed rotational direction and a linear polarizing filter that passes only polarized light having a fixed polarization plane, either of which may be used. In addition, since the light displayed on the normal display screen includes polarized light in all directions, it appears to the human eye in the same way as normal even through the polarizing filter, and there is no problem with respect to normal viewing.

The camera device (103) is a camera that captures an image on the display screen (102). The position of the pointer (104) operated by the user (105) is recognized by image processing. The camera device (103) has a polarizing filter that transmits polarized light different from that of the display screen (102) at the lens opening or inside. Due to this mechanism, the captured image (106) of the camera device (103) appears black on the display portion of the display screen due to the action of the polarizing filter, and other portions appear as usual. For this reason, only the pointer (104) operated by the user (105) pointing on the screen is clearly displayed on the black background of the display screen, and the position can be easily recognized. The camera device (103) transmits a video photographed wirelessly or by wire to the display device (101) of the main body, and identifies the position of the pointer (104) by image processing. Position detection by image processing may be performed inside the camera device (103), and the position information may be transmitted to the main body. An existing technique may be used for the position detection method. For example, when a ball painted in a specific color is used as the pointer (104), a circular area is obtained by extracting only the color range of the ball from the captured image. On the other hand, since the display screen is displayed in black, a display area can be obtained by edge extraction processing. Position detection can be realized by projective transformation of the center position of the circular area with respect to the display area.
The pointer (104) may be any pointer that can be recognized by the camera by the user (105). For example, the pointer (104) may have a colored ball attached to the tip of a stick. In addition, if a device that does not have a bar portion for pointing, such as a laser pointer, is used, there is no obstacle to visual recognition.

Next, details of an embodiment of the information input device according to the present invention will be described with reference to FIG. 2, and details of the information input method will be described.
The information input device of the present invention mainly comprises a display device (101), a camera device (103), and a pointer (104).
The display device (101) includes a polarizing filter 1 (201), a screen display unit (202), a main body side communication unit (203), an information processing unit (204), and an interface (205). The polarization filter 1 (201) is a filter for passing only a certain amount of polarized light out of the light emitted from the screen display unit (202) as described above. The screen display unit (202) is a display screen and is a device that displays information and video. The main body side communication unit (203) is a device for receiving video information transmitted from the camera device (103). The information processing unit (204) is a device that recognizes and calculates the position on the display screen indicated by the pointer (104) by image processing from the video information transmitted from the camera device (103). The interface (205) is a device for exchanging information with a device such as a PC connected to the outside, and has a role of transmitting the position information calculated by the information processing unit (204) to the external device.
The camera device (103) includes a polarizing filter 2 (206), an imaging unit (207), and a camera side communication unit (208). The polarizing filter 2 (206) is a filter that passes polarized light different from the polarizing filter 1 (201). The imaging unit (207) is a device for photographing the display screen and the pointer. The camera side communication unit (208) is a device for communicating with the main body side communication unit (203) and transmitting the image information captured by the imaging unit (207) to the display device (101). In FIG. 2, the information processing unit (204) and the interface (205) are provided in the display device (101), but may be provided in the camera device (103).
The camera device (103) photographs the display screen on the display device (101) and the pointer (104) operated by the user. At this time, due to the effects of the polarizing filter 1 (201) and the polarizing filter 2 (206), the display screen displayed by the screen display unit (202) appears black on the image captured by the imaging unit (207), and the pointer (104). Only is clearly distinguished. The camera device (103) transmits the image to the display device (101) side by wireless or wired communication between the camera side communication unit (208) and the main body side communication unit (203). The transmitted image information is subjected to image processing by the information processing unit (204), and the relative position on the display screen indicated by the pointer (104) is calculated. The interface (205) transmits the calculated position information to a PC or the like connected to the outside. Through the above procedure, the position indicated by the user is input to the device, and the information input is completed. Thus, if this application is used, the position of the pointer can be recognized by a simple configuration in which the polarizing film is provided in both the display and the imaging device.

  An example of image processing by the information processing unit (204) will be described. When the user moves the pointer along the display screen, in order to calculate the relative position of the pointer on the screen, the area of the display screen in the captured image and the position of the center point of the pointer are calculated. I just need to know. The display screen area can be easily obtained by edge extraction processing because only the display screen appears black. Further, since the pointer center point is clearly distinguished on a black background, it can be easily obtained by edge extraction processing and region division processing. By combining these information and the condition that the display screen is rectangular, even if the relationship between the camera and the display position is unknown, the relative position of the pointer on the screen can be calculated by projective transformation processing. For a specific method of projective transformation, see, for example, the Virtual Reality Society of Japan Transactions TVRSJ, Vol. 4, no. 4, pp. 607-616 (1999), Kato Hirokazu, Mark Billinghurst, Asano Koichi, Tachibana Keihachiro, co-authored. As described above, by using the present invention, the position information of the pointer can be acquired by imaging the screen regardless of the arrangement of the camera device (103) and the display device (101). In addition to the above, a color area extraction process for extracting a specific color is used as an alternative to the edge extraction process, and an existing technique for acquiring a display area by tracing the outline of this area to determine a straight line is used. You can also.

Next, FIG. 3 shows an embodiment of a communication terminal according to the present invention, and its features will be described.
As shown in the usage diagram (312), the communication terminal of the present invention shares information drawn on the electronic board system as if users who are remote from each other are using the same whiteboard, and further communicates with the other party. The video image of the video is mirror-converted and the position is adjusted to display the video track system with a high sense of realism while accurately matching the trajectory on the screen drawn by the other party and the movement on the other party's video. . As an example of the communication terminal, a case where the information input device is used will be described. The user (301) indicates the position to draw using the pointer (303), the position of the pointer (303) is tracked by the camera device (302) which is a part of the information input device, and information is input. The display device has a configuration in which an information input device and an image display device are integrated, and includes a polarizing filter (304) and a display screen (305). The display screen (305) is a semi-transparent half mirror, and a user's action is photographed by a user photographing camera (307) which is a photographing device. A projector (307) is used as a device for displaying an image on a display. Furthermore, if a speaker (308) and a microphone (309) are provided, a device for inputting and outputting audio can be provided. Inside the communication terminal is a controller (308) that manages these information input device, image display device, photographing device, audio input / output device, and network device, and communicates with other terminals via the network (309).

The details of an embodiment of the communication terminal according to the present invention will be described with reference to FIG.
The communication terminal of the present invention includes at least an information input device (401), an image display device (402), a photographing device (403), an image processing device (404), a voice input / output device (405), a controller (406), a network device (407).
The information input device (401) is a device that acquires a position pointed to by the user by the above-described various methods, and performs drawing and PC operation. For this, an ultrasonic method, an infrared method, a resistance film method, a capacitance method, or the like, which is an existing method, may be used, or the information input device shown in the present invention may be used. An image display device (402) is a display device for displaying image information of a shared electronic board and video of a communication partner. For this, a plasma display, a liquid crystal display, a rear projector or the like may be used. The photographing device (403) is a camera device for photographing a user. These information input device (401), image display device (402), and photographing device (403) may be combined. For example, the information input device shown in the present invention is composed of a camera for photographing the locus of the pointer and a display screen, so that the information input device (401) and the image display device (402) are integrated. ing. Further, as the photographing apparatus (403), a normal camera may be used, or a flat camera integrated with a display may be used. In this case, a large number of minute lenses are embedded in the display so that the whole image is taken as one image.

  The image processing device (404) adjusts an image for display on the image display device (402), and synthesizes the shared information screen of the electronic board and the captured video of the communication partner. When the camera image taken by the other party's terminal is displayed as it is on the user's display, the path drawn by the other party on the shared information screen on the electronic board does not match the drawing path on the other party's video. Occur. This is because the screen information of the shared electronic board is displayed in the correct orientation for each user, while the captured camera image is an image captured from the back side of the communication partner's board. This is because the orientations when they do not match. Therefore, in the present invention, a method is used in which the image captured by the communication partner is reversed left and right and converted into a mirror image by the image processing device (404) so that the direction coincides with the shared information screen on the electronic board. As a method for converting to a mirror image, in the case of an image shot with a normal camera, the image displayed by raster scan from the upper left of the screen to the right is scanned and displayed from the upper right of the screen to the left. That's fine. As a result, when the other user writes a picture on the electronic board, it looks as if the locus written on the back side of the glass is displayed as it is on the electronic board.

  In order to realize this, the camera parameter and the shooting environment parameter for displaying the video of the communication partner in actual size are transmitted between the communication terminals. The camera parameter is a numerical value such as a camera direction or zoom. This is information inside the camera and is acquired from the photographing apparatus (403). The shooting environment parameters indicate the positional relationship between the camera and the display, the display size, the display resolution, and the distance to the subject, and are information determined in advance at the time of design or acquired by the information input device (401). Is done. For example, in the case of a system that communicates with a remote terminal, assuming that the display is glass, even if the screen size of the terminal that communicates is different, the display size and the positional relationship between the camera and the display are transmitted as information. The image can be displayed in full size by adjusting the image. The distance between the subject and the camera may be acquired and transmitted using an infrared sensor or the like.

Further, when displaying by a rear projection type projector, the image processing apparatus (404) also performs a process of correcting geometric distortion caused by the positional relationship between the projector and the display display surface, so that the image is displayed on the display surface. The user's appearance can be displayed more accurately and naturally.
The voice input / output device (405) is a device that collects a user's voice using a microphone and reproduces a voice of a communication partner using a speaker.
The controller (406) is a device for managing all devices of the communication terminal.
The network device (407) is a communication terminal for the image information of the photographed user, the shared information screen, audio information, camera parameters for image adjustment and imaging environment parameters, and other information for network adjustment and control. It is a device for exchanging data.
By the method as described above, it is possible to realize a method in which the path of the shared screen drawn by the communication partner and the captured image of the communication partner are displayed so as to overlap with each other and communication is performed while maintaining a high sense of reality.

  In the above embodiment, the video conference system having the electronic board function has been described as an example. However, the application of the present invention is not limited to this, and a personal computer equipped with a camera, a portable information terminal, a portable It can also be used for a home game machine having a telephone or communication function.

It is explanatory drawing of one Example of the information input device used by this invention. It is the figure which showed one Example of the information input device used by this invention in detail. It is explanatory drawing of one Example of the communication terminal used by this invention. It is the figure which showed in detail the Example of the communication terminal used by this invention.

Explanation of symbols

101 ... Display device; 102, 305 ... Display screen; 103, 302 ... Camera device; 104, 303 ... Pointer; 105, 301 ... User; 106 ... Photographed image; 201, 304 ... Polarizing filter 1; 202 ... Screen display unit; 203 ... Main body side communication unit; Processing unit; 205 ... Interface; 206 ... Polarizing filter 2; 207 ... Imaging unit; 208 ... Camera side communication unit; 306, 403 ... Imaging device; 307 ... Projector; 308 ... Speaker; 309 ... Microphone; 310, 406 ... Controller; 312 ... usage diagram; 401 ... information input device; 402 ... image display device; 404 ... image processing device; 405 ... voice input / output device;

Claims (6)

  A display device that displays an image; a pointer that indicates a position on the screen; and a camera device that recognizes the position of the pointer; and the camera device and the display device each have a polarization filter that transmits different polarized light. Characteristic information input device.   The information input device according to claim 1, wherein the display device includes a linearly polarized light filter.   The information input device according to claim 1, wherein the camera device recognizes a color and a shape of a pointer to identify a position.   An imaging device for photographing a user, an image display device for displaying an image, an information input device for pointing a position on the screen and inputting a designated position, a network device for performing communication, image conversion and information An image processing apparatus that superimposes a screen and a captured image; an audio input / output apparatus that inputs and outputs audio; and a controller that controls and manages these apparatuses; the image processing apparatus is transmitted from a communication partner Using the shooting environment parameters, the shot image is converted to look the same as the actual size by scaling, projection, clipping, etc., and displayed on the image display device superimposed on the information screen Communication terminal.   5. The communication terminal according to claim 4, wherein the image processing apparatus performs mirror surface conversion on a captured image transmitted from a communication partner and adjusts the position pointed to by the communication partner and the screen position of the user to coincide with each other. . A communication method in a video conference system,
Video information of the camera that captured the participant, data information such as computer screens and numerical values, camera parameters such as camera orientation and zoom, positional relationship between the camera and display, display size, resolution, distance to the subject, etc. A communication method characterized by transmitting an imaging environment parameter indicating

Images