JP2005063225A - Interface method, system and program using self-image display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable an operator to designate a remote object in a remote image while watching a self-image superimposed on the remote-side image. <P>SOLUTION: A reverse image generation part 2 generates a reverse image 202 from an image 201 inputted in a user-side image input part 1. An instructed position detection part 3 detects a user designated position 203 in the reverse image 202. An image inputted by a remote-side image input part 4 is transmitted to the user side through a network 13. The reverse image 202 and the remote-side input image 204 are superimposed by an image superimposition part 5. A command selection part 6 selects and outputs command information 206 from the user instructed position 203 and the display position on a display 8 of a plurality of pieces of command information to be selected by the user 12. An information display part 7 superimposes and the command information 206 on the superimposed image 205 and displays them on the display 8. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an interface device that allows an image (self image) including a user (operator) to be an input image, and allows a user to select and execute a menu while viewing the self image by himself / herself.

  Conventionally, in an interface between a computer and a human, there is a method using a pointing device such as a keyboard or a mouse as an interface method for selecting a command on a screen based on a human operation. There is also a method for detecting an operation from image information from a camera. On the other hand, as an interface device for operating / displaying information on an object (device or the like) at a remote location through communication, there is a Web camera capable of controlling the remote camera via a network by computer software.

  As a first conventional method of such an interface device, there are a keyboard, a mouse, and a touch panel type display as a commercially available device that can be connected to a general computer. All of them detect the movement of the user's hand and convert it into input information.

  As a second conventional technique, there is a device for detecting motion by image processing by attaching a marker to the body as a product related to a motion capture system. For example, Vicon Motion Systems Ltd. The company's “3D motion analysis system Vicon” (distributor: NAC Image Technology Co., Ltd.) has a spherical reflective marker attached to multiple parts of the body, and each marker is processed by image processing from input images from multiple cameras. This is a method of calculating three-dimensional coordinates.

  In addition, as a third conventional method, there is a product related to Gesture Xtreme Software (for example, Soccer GX, Volleyball, etc.) of Vivid Group. This product superimposes the self-image on the CG background image, determines the contact between the CG ball and the user's body on the image, and realizes the interface between the user and the CG software such as games based on the result doing. Further, there is a method described in Non-Patent Document 1. In this method, a CG icon is displayed on the self-portrait, and a desired icon is selected (executed) based on an operation performed toward the camera.

As a fourth conventional technique, there is a technique described in Non-Patent Document 2 as a technique that enables communication between remote locations. This method is an interface that has the effect that users in various places are in the same room by superimposing user images at multiple points on the display including their own images and displaying the images to each user.
Saki et al., “A trial of action interface using PC video input”, Proceedings of the 12th Human Interface Symposium, pp. 213-218, 1996 Morikawa et al., “Video Dialogue that Enables Pointing to the Other's Space: Ultra-Mirror”, Society of Instrument and Control Engineers 50th Pattern Measurement Subcommittee Meeting, pp. 382-387, 2000

  However, the above conventional methods have the following problems.

  The first conventional method needs to touch the device when detecting the movement of the user's (operator's) hand, and requires a fixed and limited place for using the device. It is difficult to detect the operation at the place.

  The second conventional method is a non-contact type and can detect an operation at an arbitrary location within the range reflected in the camera image. However, since it is necessary to always wear some kind of marker or sensor on the body part, such as at home or at work. It lacks convenience as an interface device in normal life.

  The third conventional method can detect a user's movement by touching (superimposing) a finger or the like on a CG displayed on the same display while looking at the self-portrait, but indicates an object in real space that is reflected in the background. The application range is limited.

  In the fourth conventional method, by superimposing the image of a person or object in a remote place and the user's self-image, the effect of being in the same room can be obtained virtually. Users can interact with remote objects, such as operating devices and displaying information, because they can only observe the same video and do not have access to remote people or objects. I can't.

  In addition, in the fourth conventional method, the image of one user side is cut out using the chroma key method and the user's image portion is cut out and then superimposed on the image at a remote place. This can only be achieved with limited shooting locations and conditions such as a fixed background (blue background, etc.).

The object of the present invention is compared with the conventional method described above.
(1) The problem that it is a contact type, the problem that it is difficult to detect an operation in an arbitrary place,
(2) Problems lacking in convenience as an interface device,
(3) Problems with limited application range
(4) The problem that device operation and information display operation cannot be performed on an object in a remote place,
(5) Problems where shooting location and conditions are limited,
It is an object to provide an interface method, an apparatus, and a program that solve the above.

  In order to achieve the above object, the interface device of the present invention inputs an image on the operator side, generates a reversed image that is reversed left and right, and detects a position indicated by the operator on the reversed image. Means for inputting an image on the remote site side, means for detecting a real object corresponding to the detected position of the operator from the remote site input image, the inverted image and the remote site input Means for generating an image superimposed with the image, means for selecting a command designated by the operator on the superimposed image, and means for displaying information on the superimposed image based on the designated command. doing.

  Therefore, the operator can designate the remote object in the remote image as non-contact while viewing the superimposed self-image on the remote image, so that the problems (1), (2), (3) ( 4) can be solved.

  In an embodiment of the present invention, the means for superimposing and displaying the reverse image and the remote place side input image includes means for making the reverse image on the operator side translucent and superimposing it on the remote place side input image. .

  Since the entire video on the operator side is made translucent and superimposed, the operator can operate a remote object at an arbitrary background and in an arbitrary background. Therefore, the above problems (1), (2), (3) ) (4) (5) can be solved.

  In another embodiment of the present invention, the means for detecting an object corresponding to the detected position indicated by the operator from the remote side input image registers an arbitrary real object position in advance on the remote side input image. And means for detecting a real object designated by the operator from a plurality of pre-registered real objects.

  Therefore, since the initial value of the position information can be registered and updated, even if the position of the object on the remote side moves, if the initial value of the position information is corrected in advance, the user specifies the object at the position after the movement Therefore, the problems (1), (2), (3), (4), and (5) can be solved.

  According to another embodiment of the present invention, the means for detecting the object corresponding to the detected position indicated by the operator from the remote place side input image, the means for detecting the position of the real object on the remote place side, Means for detecting the real object position indicated by the operator from the obtained real object position and the position indicated by the operator on the operator side.

  Since the object position of the moving destination is detected even if the object on the remote side moves, the object specified by the user can be detected not only for the stationary object but also for the moving object. The problems (1), (2), (3), (4), and (5) can be solved.

  According to the present invention, for example, home appliances in a remote room can be remotely operated, or information on a remote object can be displayed on a screen. As a remote place, for example, operation and information display between the home and the office or between different rooms in the home are possible, and the use as a non-contact type interface device in such a scene is conceivable.

  According to the first, fifth, and ninth aspects of the invention, the operator can specify a remote object in the remote image while viewing the superimposed self-image on the remote image, so that user feedback can be obtained. The designated place (object) desired by the person can be designated at high speed and accurately. Further, since it can be designated by a non-contact operation, a designated place (object) in a remote image can be designated from a position away from the display which is an input device for the operation. As a result, the convenience of the interface is improved.

  In the second, sixth, and ninth aspects of the invention, the entire video on the operator side is made translucent and superimposed on the remote video, so that the background video portion other than the operator in the video on the operator side is also shown as being thin. Therefore, the user's selection operability at the time of designating an object to be operated on the remote side that is dark is hardly deteriorated. Therefore, the user can select the remote object in an arbitrary place and an arbitrary background. Operation becomes possible. For this reason, the convenience of the interface is improved.

  The inventions of claims 3, 7 and 9 can register and update the initial value of the position information. Therefore, even if the position of the object on the remote location side moves, if the initial value of the position information is corrected in advance, the position information is moved. The user can specify an object at a later position. For this reason, the convenience of the interface is improved.

  In the inventions of claims 4, 8 and 9, since the position of the destination object is detected even if the object on the remote side moves, the user can detect not only a stationary object but also a moving object. An object to be specified can be detected, and an operation can be performed on an arbitrary target object. For this reason, the convenience of the interface is improved.

  Next, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a configuration diagram of an interface apparatus according to the first embodiment of the present invention, and FIG. 2 is a flowchart showing the processing.

  The interface apparatus according to the present embodiment uses an image (self-image) including the user (operator) 12 as an input image, and the user 12 superimposes the image based on the operation of the user 12 while viewing the self-image on the display 8 by himself / herself. A command is executed for an object in a remote site side image (superposed), and a user side image input unit 1, a reverse image generation unit 2, a pointing position detection unit 3, a remote site side image input unit 4 and an image It comprises a superimposing unit 5, a command selecting unit 6, and an information display unit 7.

  As the user side image input unit 1, in this example, one camera is used as shown in FIG. The camera may be a commonly used video camera and may be black and white or color. However, when using the method using color information described below, a color camera is required.

  The inverted image generation unit 2 inputs the image (input image) 201 input by the image input unit 1 and generates an image (inverted image) 202 that has been horizontally reversed (step 101). A reverse image 202 can be generated in real time with general-purpose image processing software (for example, HALCON) for the input image 201 captured in the computer. Alternatively, it can be realized by a device that inputs an input image and generates a reverse image in real time (for example, a screen horizontal reversing device UPI-100LRF of Sakae Co., Ltd., or a camera-integrated SONY EVI-D100).

  The designated position detection unit 3 detects a position (user designated position) 203 that the user 12 points to with a body part such as a finger on the reverse image 202 obtained by the reverse image generation unit 2 (step 102).

  As a first implementation example of the indicated position detection unit 3, there is a method in which the user 12 has a marker and a marker in the reverse image 202 is detected by image processing. For example, first, a red object (other colors may be used, but less colors are preferable in the background image) is held or attached to the hand, and is moved by shaking in front of the camera 1. A red component is extracted from the RGB color matching information values of the obtained color image, and a labeling process (a general image processing method for grouping each group) is performed, and one red part having the largest area is selected. The center of gravity (x, y) of the position is obtained. The obtained (x, y) coordinates are set as the user designated position 203. Here, instead of a red object, a light emitting diode (LED) of red color (which may be other colors as well as the above object) is made to emit light, and is held or attached to the hand to extract the color. A method of performing is also possible. (X, y) is not the center of gravity position (the average value of the total values of the x coordinates of all the pixel values of the extracted portion and the average value of the total values of the y coordinates), but the center position (the maximum of the x coordinates of the extracted portion) Or the center value of the minimum value and the maximum value of the y coordinate and the center value of the minimum value).

  As a second example of realization of the indicated position detection unit 3, there is a method in which the user 12 directly detects the position of the hand in the inverted image 202 without having a marker by image processing. For example, first, a skin color component is extracted from the obtained color image from color information values such as RGB, and a labeling process is performed. A target hand is selected from the obtained plurality of skin color portions using constraint information such as the size and position of the hand. As a specific example of the method to be selected, if the user is usually wearing clothes, it is considered that the candidate for the skin color portion is likely to be both hands and face, and the largest area is the face. Since it is conceivable, a skin color portion having a large area in the second and third areas is selected as a candidate for the hand. When the user-specified positions are both hands, the center-of-gravity positions for the two skin color part candidates may be the user-specified positions for the left and right hands, respectively. Also, if the user-specified position is one, it is necessary to select one of the two candidates. For example, if the right hand is designated in advance, the candidate on the right hand side of the body (because the possibility of the right hand is high). The skin color portion of the right hand may be selected and the center of gravity position may be set as the user-specified position 203 of the right hand. The same applies to the case of the left hand.

  In these implementation examples of the indicated position detection unit 3, the accuracy can be improved by adding a process using the difference image before the detection process described above. For example, the difference image is created on the reverse image 202 between a background image captured in advance (image captured without a user) and a current image currently captured (image with a user). . The difference image can be created, for example, by obtaining the absolute value of the difference between the pixel value of the background image and the pixel value of the current image (at the same pixel position) for every pixel. By using this difference image, only a moving object such as a body is extracted in advance, and then the user performs image processing as described above (for example, a method of extracting a marker attached to the body or a method of extracting skin color). If the designated position 203 is detected and the coordinates thereof are acquired, it can be easily estimated that false detection of colors similar to the color of the marker or the skin color included in the background image can be reduced.

  Such an indicated position detection unit 3 does not target the entire body for command selection, and only a marker or a hand can be used as a contact determination part, so an icon for command selection or the like can be placed at an arbitrary position. Therefore, a high degree of freedom can be obtained with respect to the position and user operation.

  As the remote location side image input unit 4, one camera is used as shown in FIG. The camera may be a commonly used video camera and may be black and white or color. The remote site side input image 204 is sent to the user side via the network 13. The method of sending an image via the network 13 can be achieved by using an existing video data transmission technique using the general Internet, or by a wireless transmission such as a general video broadcast. It is also possible by transmission on the network. Furthermore, the output 204 of the remote image input unit 4 can be directly connected to the image superimposing unit 5 without going through the network 13. When the user side and the remote site side are relatively close to each other, a method of sending a video signal by such direct connection is also effective. In that case, the network 13 in the figure can be omitted.

  The image superimposing unit 5 superimposes the reverse image 202 obtained by the reverse image generating unit 2 and the remote site side image 204 obtained by the remote site side image input unit 4 on one image (step 103). At this time, the outline of the body of the user 12 (or only a partial image such as a finger necessary for operation input) may be cut out from the inverted image 202 in which the self-image of the user 12 is captured, and the cut out on the remote side image 204 Overlapping image parts. The user 12 can be cut out by, for example, image processing. As an image processing method of clipping, there is a clipping method of the user 12 by background difference. This is because, for example, a reverse image 202 (or an input image 201 on the user side) is captured and acquired in advance when a user 12 is not present, and when the user 12 is present, the background image and the image when the user 12 is present. This can be done by generating a difference image. The difference image can be created, for example, by obtaining the absolute value of the difference between the pixel value of the background image and the pixel value of the current image (at the same pixel position) for every pixel. Alternatively, it is possible to easily cut out only the hand portion by using the skin color recognition process. As a result, it is possible to generate a superimposed image as if the user 12 entered the remote site side image 204.

  The command selection unit 6 compares the user indication position 203 obtained by the indication position detection unit 3 with the display positions on the display 8 of a plurality of command information to be selected by the user 12, and determines the user from the positional relationship between the two. 12 searches for and selects command information 206 to be instructed (step 104). The display positions on the display 8 of a plurality of command information to be selected by the user 12 may be determined in advance and stored in advance on the remote side input image. For example, if there is a command having a display position within a certain distance range from the user-designated position, it may be selected. The display position of the command information 206 may be determined in advance for each command.

  The information display unit 7 superimposes the superimposed image 202 obtained by the image superimposing unit 5 and the command information 206 created by the command selecting unit 6 on the display 8 (step 105). The display 8 may be a general-purpose display used for a computer as long as it can display a computer screen and a camera image. The command information 206 is information related to the command such as the command display position and command content, and information on the command selected by the user 12. The display method is a command that can be selected by a computer generated CG. The (group) and the selected command may be displayed. Further, the user designated position 203 obtained by the designated position detection unit 3 may also be explicitly displayed in real time, for example, as a mouse cursor display. The content of the command may be a command for operating an object on the remote side or displaying information, or a command for performing an operation or information related to the room on the user side. If the command is for operating a remote object or displaying information, the remote image is displayed in the superimposed image 205, so that the user 12 can determine the operation and confirm the operation result. Therefore, an interface with high user convenience can be realized.

  According to the present embodiment, the user 12 can select a command by simply placing a finger or the like on the command information 207 displayed on the same screen while viewing the screen on which the self-image is displayed. Easy and error-free instructions are possible.

(Second Embodiment)
FIG. 3 is a block diagram of the image superimposing unit of the interface apparatus according to the second embodiment of the present invention, and FIG. 4 is a flowchart showing the processing.

  The interface apparatus according to the present embodiment uses an image (self-image) including the user (operator) 12 as an input image, and the user superimposes the self-image on the 12 display 8 based on the operation of the user 12 while viewing the self-image on the 12 display 8 himself. A command is executed with respect to an object in a remote place side image (superposed), and the image superimposing unit 5 shown in the first embodiment in FIG. 5A and the superimposition processing unit 5B. The functions of the other user side image input unit 1, inverted image generation unit 2, designated position detection unit 3, remote site side image input unit 4, command selection unit 6, and information display unit 7 are the same as those in the first embodiment. is there. Similar to the first embodiment, the remote side input image 204 may or may not be sent via the network 13 when being sent to the user side.

  The translucent processing unit 5A translucents the input reverse image 203 (step 103A). It is also possible to set the degree of translucency freely. The translucency method can be realized by providing transparency (information called alpha value) information for each pixel in the image data. The superimposition processing unit 5B generates a single image (superimposed image) 205 by superimposing (superimposing) the translucent inverted image on the remote site side image 204 as a background (step 103B). ). At the time of superimposing, the pixel information (color information in the case of a color image) is displayed for the pixel having a high transparency value in the translucent inverted image with the ratio of the transparency of the background pixel value. For example, if the transparency is 0.7, the background pixel information at the same position is 70% and the remaining 30% is the pixel information on the inverted image 205, and the respective values are added.

  According to the present embodiment, since the inverted image 202 including the self image of the user 12 is translucent and superimposed on the remote site side image 204, the user 12 has the object image in the remote site side input image 204 and the user 12 's image. Since you can see the self-image at the same time, you can realize an interface with high operability as if you are in the same room virtually while you are in a remote place, when you perform an operation to overlap your finger and the target object on the image . Further, since the transparency can be freely set, the transparency can be freely changed by the user 12 most easily when superimposed.

  Further, the translucent processing unit 5 </ b> A may perform the translucent processing on the remote place side input image 204 and superimpose the inverted image 202 on the remote image 202 instead of translucent the inverted image 202 alone. Of course, both the remote-side input image 204 and the reverse image 202 may be made semi-transparent and then superimposed. Accordingly, the transparency of each input image can be freely changed according to the state of each input image (such as a photographed illumination condition, a background image, an object / user color state, etc.) that is most easily operated by the user 12.

(Third embodiment)
FIG. 5 is a block diagram of the interface apparatus according to the third embodiment of the present invention, and FIG. 6 is a flowchart showing the processing.

  The interface apparatus according to the present embodiment uses an image (self-image) including the user (operator) 12 as an input image, and the user 12 superimposes the image based on the operation of the user 12 while viewing the self-image on the display 8 by himself / herself. A command is executed for an object in a remote site side image (superposed), and a user side image input unit 1, a reverse image generation unit 2, a pointing position detection unit 3, a remote site side image input unit 4 and an image The superimposing unit 5, the real object detection unit 9, the information display unit 7, and the real object registration unit 10 are configured.

  In this embodiment, in addition to the contents of the first embodiment, the command selection unit 6 shown in the first embodiment is equipped with a mechanism that can detect the real object 14. Specifically, the command selection unit 6 of the first embodiment includes a real object detection unit 9 and a real object registration unit 10. The real object registration unit 10 includes real object position data 10A and a real object registration processing unit 10B. The other user side image input unit 1, inverted image generation unit 2, designated position detection unit 3, remote location side image input unit 4, image superimposition unit 5, and information display unit 7 are the same as those in the first embodiment. Similar to the first and second embodiments, the remote side input image 204 may be sent via the network 13 or not when sent to the user side. The image superimposing unit 5 can also perform processing including translucency processing as in the second embodiment.

  The real object registration processing unit 10 </ b> B is an arbitrary real object 14 included on the remote site side input image 204 (for example, in the house, a television, an air conditioner, a computer, a clock, a window, a shelf, a chair, a desk, a drawer, , Documents, audio equipment, lighting equipment, etc.), the display positions on the remote site side input image 204 are obtained in advance, and are registered and stored in the real object position data 10A (step 106).

  The actual object position data 10A may be one center coordinate (x, y) of the object, but information indicating the range (for example, the start point coordinates (x1, y1) and end point coordinates (x2, y2) of the rectangular frame, the center A coordinate (x, y) and a radius value r, or a coordinate list of all dots in which an object exists may be provided. The center coordinates may be replaced with the center-of-gravity coordinates as in the processing in the designated position detection unit 3 of the first embodiment.

  The real object detection unit 9 compares the plurality of position data stored in the real object position data 10A with the user designated position 203 obtained by the designated position detection unit 3, and the command selection unit according to the first embodiment. Similarly to FIG. 6, based on the positional relationship between the two, the real object 14 to be instructed by the user 12 is detected and selected (step 107). For example, if there is position data of the real object 14 within a certain distance range from the user-designated position, it may be selected. Alternatively, when a range is provided as the real object position data 10A, the real object 14 in which the coordinates of the user-designated position are included in the range is detected and selected.

  FIG. 7 shows an example of the interface operation according to the present embodiment.

  Display positions are obtained in advance for real objects shown on the remote-side input image 204, and are registered in the real object position data 10A. An inverted image 202 obtained by inverting the user-side input image 201 is superimposed on the remote site-side input image 204, and when the user 12 operates, the user image on the superimposed image 205 is displayed in front of the real object on the remote site side. Looks like it works. When the user 12 is not instructing an action, nothing reacts, but it is determined that the real object is detected by the real object detection unit 9 by performing an operation of placing a finger (or a marker) on the real object that the user 12 wants to access. The position of the detected real object is specified. For example, an icon of the same shape or the same size as the real object (such as a color changed, semi-transparent or blinked, etc. if necessary) is superimposed on the real object. And a method of displaying a CG marker (changed color, semi-transparent or blinked, etc. as necessary) near the center of the real object or around the real object. . That is, the user actually touches a virtual icon on the real object position or a displayed (transparent or other color) CG icon. Further, related information (name, state, etc.) may be displayed around the detected real object. Further, as shown in the figure, a menu list to be subsequently selected is displayed CG around the detected real object position and the like, and the user 12 selects a similar selection method (that is, a menu for selecting a finger or the like from the menu list). It is also possible to select and execute the menu by a method of selecting by overlapping the icon.

  As a result, the user 12 can not only select a command by specifying a CG icon displayed on the image, but can also directly select the real object 14 at a remote location via the object image 208 in the image. Become. When combined with command selection by CG as in the first embodiment, for example, a command selection CG menu that can control a television switch, channel, etc. when a remote television or the like is pointed to is displayed on the real object. The user 12 can continue to select commands and perform various operations on a remote real object (television).

  When information to be presented to the user 12 is acquired on the real object side in a remote place, the information is displayed on or around the real object image on the superimposed image 205, or an icon (color is changed). It is also conceivable that a user, a semi-transparent object, a blinking object, etc.) are clearly indicated by being superimposed on the real object image to notify the user 12 who is viewing. Thereafter, as described above, when the user 12 selects the real object, necessary information and a CG menu can be displayed.

(Fourth embodiment)
FIG. 8 is a block diagram of an interface apparatus according to the fourth embodiment of the present invention, and FIG. 9 is a flowchart showing the processing.

  The interface apparatus according to the present embodiment uses an image (self-image) including the user (operator) 12 as an input image, and the user 12 superimposes the image based on the operation of the user 12 while viewing the self-image on the display 8 by himself / herself. A command is executed for an object in a remote site side image (superposed), and a user side image input unit 1, a reverse image generation unit 2, a pointing position detection unit 3, a remote site side image input unit 4 and an image The superimposing unit 5, the real object detection unit 9, the information display unit 7, the real object registration unit 10, and the real object position detection unit 11 are configured. The operation of the present invention will be described below.

  In this embodiment, in addition to the contents of the third embodiment, a mechanism capable of detecting the position of a real object in real time is provided on the remote side. Specifically, in addition to the third embodiment, an actual object position detection unit 11 is provided. Other user side image input unit 1, inverted image generation unit 2, designated position detection unit 3, remote site side image input unit 4, image superimposition unit 5, information display unit 7, and real object registration unit 10 are the third implementation. It is the same as the form. Similar to the first, second, and third embodiments, the remote-side image input unit 4 and the real object position detection unit 11 may or may not be connected to the user side via the network 13. In addition, the image superimposing unit 5 can be a processing method including translucent processing as in the second embodiment.

  The real object position detection unit 11 inputs the remote site side input image 204 sent from the remote site side image input unit 4 and performs a process of obtaining the position of the target object on the image (step 108). As a method for obtaining the object position, a sensor capable of recognizing the position of each target real object (widely marketed and researched and developed, including various sensors such as a magnetic sensor, an ultrasonic sensor, and an infrared sensor) is attached. There is a method for detecting the position of each object in real time. As another method for obtaining the object position, there is a method for obtaining the object position by image processing using the remote place side input image 204 being photographed. Specifically, the color and shape information of the target object may be registered in advance, and the registered object may be detected on the image by image processing such as pattern matching processing. The pattern matching processing method is equipped as one function of general-purpose image processing software, or is widely marketed and researched and developed, and can be easily used and realized. In the case of a method using image processing, the sensor 15 attached to the object 14 shown in FIG.

  According to the present embodiment, it is possible to constantly update the actual object position data, and even if the object is moved, the position information is also moved, so that the image displayed on the user side by the information display unit 7 In the above, even if the object moves, the user 12 can touch the moved object (current object) on the screen.

(Fifth embodiment)
In the above embodiment, the image flows from the remote site side to the user side and is operated on the user side. However, in any embodiment, two sets of this system are used and installed in opposite directions. An interface device capable of bidirectional operation can be realized. As a result, there is a user between remote locations, and it is also possible to operate and display information on objects that have moved to the background of each other's room. It is also possible to do this easily, so it is possible for both users to be in the same room and touch them while checking the objects that are also visible.

(Sixth embodiment)
In the above embodiment, two points are connected, but it is also possible to set a plurality of remote locations in this system. Similarly, it is possible to easily set a plurality of users. Furthermore, if bidirectional operation as in the fifth embodiment is possible, a user at an arbitrary point can perform device operation and information display (information acquisition) on a user or an object at an arbitrary point. As an image to be superimposed, it is possible to superimpose images at two necessary points and switch to another two points as necessary, or to superimpose and display images at three or more necessary points.

  Note that the interface method of the present invention records a program for realizing its function on a computer-readable recording medium in addition to that realized by dedicated hardware, and stores the program recorded on this recording medium. It may be read by a computer system and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in the computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short period of time, as in the case of transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.

It is a block diagram of the interface apparatus of the 1st Embodiment of this invention. It is a flowchart which shows the process of 1st Embodiment. It is a block diagram of the interface apparatus of the 2nd Embodiment of this invention. It is a flowchart which shows the process of 2nd Embodiment. It is a block diagram of the interface apparatus of the 3rd Embodiment of this invention. It is a flowchart which shows the process of 3rd Embodiment. It is a figure explaining the example of operation in 3rd Embodiment. It is a block diagram of the interface apparatus of the 4th Embodiment of this invention. It is a flowchart which shows the process of 4th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 User side image input part 2 Reverse image generation part 3 Pointed position detection part 4 Remote place side image input part 5 Image superimposition part 5A Translucent process part 5B Superimposition process part 6 Command selection part 7 Information display part 8 Display 9, 9 ′ Real object detection unit 10 Real object registration unit 10A Real object position data 10B Real object registration processing unit 11 Real object position detection unit 12 User 13 Network 14 Real object 15 Sensor 101 Reverse image generation process 102 Instructed position detection process 103 Image superposition process 103A Translucent processing 103B Image superimposition processing 104 Command selection processing 105 Information display processing 106 Real object data processing 107 Real object detection processing 108 Real object position detection processing 201 User side input image 202 Reverse image 203 Instructed position coordinates 204 Remote place side input Image 205 Superimposed image 206 Command information 07 Select command information 208 the object image

Claims (9)

An operator that is displayed on the image at a predetermined location on the screen while the operator views the input image, with the image including the operator photographed by the camera or an inverted image of the image as the input image. An interface method for recognizing an operation of an operator based on a position of the partial image on a screen by moving and overlapping a partial image of a part of the body or a partial image of a mounted marker,
Inputting the operator's image and generating a reversed image that is reversed left and right;
Detecting a position indicated by an operator on the reverse image;
Inputting the remote image,
Detecting a real object corresponding to the detected position indicated by the operator from the remote-side input image;
Generating an image in which the inverted image and the remote-side input image are superimposed;
Selecting a command designated by the operator on the superimposed image;
And displaying information on the superimposed image based on the designated command.   The step of superimposing and displaying the inverted image and the remote site side input image includes translucent the operator side inverted image and superimposing the remote site side input image on the remote site side image. Interface method.   The step of detecting the object corresponding to the detected position indicated by the operator from the remote side input image includes the step of preregistering an arbitrary real object position on the remote side input image, and the operator 3. The interface method according to claim 1, further comprising the step of: detecting a real object designated by a plurality of real objects registered in advance.   The step of detecting the object corresponding to the detected position indicated by the operator from the remote side input image includes the step of detecting the position of the real object on the remote side, and the obtained real object position, The interface method according to claim 1, further comprising: detecting an actual object instructed by the operator from a position instructed by the operator on the operator side. An operator that is displayed on the image at a predetermined location on the screen while the operator views the input image and includes an image including the operator photographed by the camera or an image obtained by inverting the image. An interface device for recognizing an operator's movement based on a position of the partial image on the screen by moving and overlapping a partial image of a part of the body or a partial image of the attached marker,
Means for inputting an image on the operator side and generating a reversed image that is reversed left and right;
Means for detecting a position indicated by an operator on the reverse image;
Means for inputting a remote image,
Means for detecting a real object corresponding to the detected position of the operator from the remote side input image;
Means for generating an image in which the reverse image and the remote-side input image are superimposed;
Means for selecting a command designated by the operator on the superimposed image;
An interface device having means for displaying information on the superimposed image based on the designated command.   6. The means for superimposing and displaying the reverse image and the remote site side input image includes means for translucent the operator side reverse image and superimposing it on the remote site side input image. Interface device.   The means for detecting an object corresponding to the detected position indicated by the operator from the remote side input image includes means for previously registering an arbitrary real object position on the remote side input image; The interface device according to claim 5, further comprising: means for detecting a real object designated by a plurality of real objects registered in advance.   The means for detecting the object corresponding to the detected position indicated by the operator from the remote place side input image, the means for detecting the position of the remote place side real object, and the obtained real object position, The interface device according to claim 5, further comprising means for detecting a real object designated by the operator from a position designated by the operator on the operator side.   An interface program for causing a computer to execute the interface method according to claim 1.

Images