JP6346585B2 - Operation support apparatus and program - Google Patents

Description

  The present invention relates to a technique for performing an operation on an operation target in an AR environment.

  There is a technology that provides an augmented reality (AR) environment that extends a real space by presenting virtual information superimposed on the real space. As information presentation using this AR, a presentation method using a head mounted display (HMD) has been proposed mainly in a wearable environment. This makes it possible to utilize AR that does not hold a physical operation device and covers a wide field of view.

  However, in AR by HMD, there is a problem that the user cannot directly operate an operation target in the AR environment (hereinafter referred to as AR operation target). Therefore, a remote controller built into the HMD is used to operate the HMD. However, the correspondence between the remote control operation and the AR operation target is difficult to understand, and it is difficult to operate intuitively.

  In order to solve such a problem of operability, for example, as disclosed in Non-Patent Document 1, a method of operating the AR space by operating a controller (finger) equipped with a marker has been studied. ing. Further, as disclosed in Non-Patent Document 2, a method of operating an operation target by hand gesture (hand movement) has also been proposed. As a result, a specific operation target can be operated intuitively to some extent by a gesture without a device.

SixthSense: a wearable gestural interface, SIGGRAPH Asia 2009, Yokohama, Japan, December 16-19, 2009. LeapLook: a free-hand gestural travel technique using the leap motion finger tracker, SUI'14, October 4-5, 2014, Honolulu, HI, USA

  It is considered that the AR operation target can be operated more intuitively by using the above-described conventional technology. However, when the above prior art is used in an AR environment, it is necessary to operate an AR operation target that does not exist in the real space with a finger or a gesture, so that it is difficult to obtain sufficient feedback for the operation. The physical problem remains that it is easy to get tired because it is necessary to always move hands and arms in the air for operation. In addition, when viewed from a third party, the arm or hand moves with respect to an empty space, so that the appearance of the operation looks strange (maybe seen) There are also some challenges.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a technique capable of efficiently operating an AR operation target in an AR environment.

According to the embodiment of the present invention, virtual information is superimposed on a real space to present an operation to an AR operation target that is an operation target existing in an augmented reality (AR) environment that extends the real space. An operation support device for
First input means for acquiring a position of an operation device that is a physical device including a touch panel ;
A specifying means for specifying an AR operation target based on the position of the operation device;
Means for displaying the specified AR operation target on the touch panel of the operation device;
Second input means for acquiring operation information given to the operation device by an operation on the touch panel on which the specified AR operation target is displayed;
The operation information is converted based on the operation information given to the operation device and the AR operation target specified by the specifying means, and the operation obtained by the conversion is applied to the specified AR operation target. Operating means ,
The operation support device includes a conversion database that defines an operation corresponding to an AR operation target and operation information,
The operation means determines an operation to be applied to the specified AR operation target by referring to the conversion database, and an operation support apparatus is provided.

  According to the embodiment of the present invention, a technique is provided that enables an AR operation target in an AR environment to be efficiently operated.

It is a lineblock diagram of operation support device 10 concerning an embodiment of the invention. 3 is a flowchart showing an operation procedure of an input unit 11. It is a figure which shows the example of input information. 3 is a flowchart showing an operation procedure of a specifying unit 12. It is a figure which shows a mode that AR operation object is specified. It is a figure which shows the example of the program corresponding to operation | movement of the specific part. 3 is a flowchart showing an operation procedure of the operation unit 13. It is a figure which shows the example of information of conversion DB14. The operation information when the operation device has a touch panel is also a diagram illustrating an application example to the AR operation target. It is a figure which shows the example of application of the operation information to a several operation target. It is a figure which shows the example of information of conversion DB14 in the case of using inclination and acceleration. 3 is a flowchart showing an operation procedure of the output unit 15.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is only an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

(Outline of the embodiment)
In the present embodiment, for the purpose of efficiently operating an AR operation target, an intuitive operation of the AR operation target can be performed by using a “physical AR space operation device” (hereinafter referred to as an operation device). An operation support device is provided. Specifically, the operation support apparatus acquires the position information of the operation device, matches the position information of the AR operation target in the AR environment, specifies the AR operation target operated by the operation device, and sets the operation device to the operation device. The given operation information is converted and applied according to the AR operation target, thereby enabling high-speed identification and operation of the AR operation target.

  As a result, the AR operation target can be intuitively operated as if using an existing device such as a smartphone, and both physical and psychological problems in the operation of the AR environment using an HMD (head mounted display) are realized. Is going to solve. Hereinafter, this embodiment will be described in more detail. In the present embodiment, it is assumed that the HMD is used as a display for presenting the AR environment to the user, but the present invention is also applicable to a display other than the HMD.

(Apparatus configuration, function overview)
FIG. 1 shows a configuration diagram of an operation support apparatus 10 according to the present embodiment. The operation support apparatus 10 is an apparatus that supports a user of a display (HMD) that provides an AR environment to operate an AR operation target in the AR environment.

  The operation support apparatus 10 may be an apparatus incorporated in the HMD, for example, or may be an apparatus provided outside the HMD. Moreover, the function of the operation assistance apparatus 10 may be provided as a part of function of HMD.

  Note that the HMD method for providing the AR environment in the present embodiment is not limited to a specific method. For example, in the present embodiment, it is possible to use an HMD in which the AR operation target is visible to the user while overlapping with the real space. Also, the AR method is not limited to a specific method. For example, either a method using a marker or a method not using a marker may be used.

  As illustrated in FIG. 1, the operation support apparatus 10 includes an input unit 11, an identification unit 12, an operation unit 13, a conversion DB (database) 14, and an output unit 15. The specifying unit 12 includes a plurality of specifying units 112.

  As described above, in this embodiment, an operation device is used to perform an operation on an AR operation target. As an example, the operation device is a smartphone, but the operation device is not limited to a specific device. The operation device and the operation support apparatus 10 can communicate with each other by wire or wireless, and the operation support apparatus 10 can acquire information on operations performed by the operation device from the operation device. Hereinafter, an outline of the operation of each unit of the operation support apparatus 10 will be described.

  The input unit 11 receives position information of the operation device and operation information given to the operation device. As an example, when the operation device is a device having a touch panel like a smartphone, the input unit 11 acquires operation information on the touch panel as operation information.

  The specifying unit 12 matches the position information of the operation device and the position information of the AR operation target, and specifies the AR operation target to be operated by the operation device. The multiple specifying unit 112 specifies a plurality of operation targets when there is specific operation information (for example, an operation for specifying multiple selection) given to the operation device.

  The operation unit 13 converts the operation information given to the operation device in accordance with the AR operation target specified by the specifying unit 12 or the like, and applies it to the AR operation target. For example, when the operation device is a touch panel device, the operation on the touch panel may be applied to the AR operation target as it is. When the operation information is converted by the operation unit 13, information stored in the conversion DB 14 is used.

  The output unit 15 presents the result of specifying the AR operation target in the specifying unit 12 and the result of applying the operation information to the AR operation target to the AR environment. That is, the AR environment including the operated AR operation target is displayed on the display 22 (eg, HMD). In addition, for example, when the operation device is a touch panel device, the output unit 15 displays the specified AR operation target on the touch panel in addition to the presentation of the identification result to the AR environment and the application result of the operation information. Application results may also be displayed on the touch panel.

  Note that functions for creating and presenting an AR environment (functions for creating and displaying an AR operation target, etc.) themselves are existing technologies and are not described in detail. The function itself for creating and presenting the AR environment may be included in the operation support apparatus 10 or may be included in an apparatus outside the operation support apparatus 10. In any case, the operation support apparatus 10 can hold or acquire information on an AR operation target in the AR environment (position information, shape information, etc. for each AR operation target).

  The operation support apparatus 10 according to the present embodiment, for example, executes a program describing the processing contents described in the present embodiment on a computer (including various apparatuses having a computer function) including a CPU and a memory. This can be realized. That is, the function of the operation support apparatus 10 is realized by executing a program corresponding to the process executed by the operation support apparatus 10 using hardware resources such as a CPU, a memory, and a hard disk built in the computer. Is possible. Further, the program can be recorded on a computer-readable recording medium (portable memory or the like), stored, or distributed. It is also possible to provide the program through a network such as the Internet or electronic mail.

  Hereinafter, the operation of each unit will be described in more detail.

(Input unit 11)
FIG. 2 shows an operation procedure of the input unit 11. As illustrated in FIG. 2, the input unit 11 acquires the operation device position (x, y, z) and operation information for the operation device from the operation device linked to the AR environment (steps S <b> 101 and S <b> 102). This is done continuously. Further, the acquired position information of the operation device is transferred to the specifying unit 12 (step S103), and the operation information is transferred to the operation unit 13 (step S104).

  FIG. 3 shows an example of input information. In the example illustrated in FIG. 3, time (time when the operation is performed), position, and operation information are input. In the example of FIG. 3, information input in a table format is assumed, but any format may be used as long as the information can be acquired. Further, the position of the operation device may be in any format as long as it can match the AR operation target (xi, yi, zi). As operation information for the operation device, for example, when the operation device is equipped with a touch panel, operations such as tap, swipe, and pinch can be accepted as input.

  As a specific method for acquiring the position of the operation device, for example, by installing a device capable of estimating the position and distance of the target device by the ToF (Time of Flight) method using an infrared laser such as Kinect in the HMD. The position information can be acquired. In addition, by mounting an AR marker in advance on the operation device and mounting a camera on the HMD, the AR marker is recognized, the device position is estimated, and the estimated position is provided to the operation support apparatus 10. Is possible. Conversely, it is possible to detect the distance between the HMD and the operation device by mounting a camera on the operation device side and detecting an AR marker mounted on the HMD. As described above, the position of the operation device may be acquired by any method as long as x, y, and z can be taken with reference to the position of the HMD (that is, the position of the user).

  As a specific method for linking the AR environment and the operation device, for example, wireless communication such as Bluetooth (registered trademark) can be used, or a wired connection may be used. As described above, any method may be used as long as the position information of the operation device based on the HMD can be detected and the operation information of the operation device can be applied to the AR operation target.

(Specific part 12)
The specifying unit 12 specifies an AR operation target based on the position information of the operation device obtained by the input unit 11. FIG. 4 shows an operation procedure of the specifying unit 12.

  As illustrated in FIG. 4, the specifying unit 12 reads information on the position of the operation device obtained by the input unit 11 and stored in a memory or the like (step S <b> 201). The specifying unit 12 calculates the distance from the operation device for each AR operation target i (steps S202 and S203).

  When the calculation of the distance to the operation device is completed for all AR operation targets, the AR operation target i having the minimum distance is specified (step S204). The specifying unit 12 passes the specified AR operation target information to the operation unit 13 (step S205).

  FIG. 5 illustrates an image for specifying an AR operation target using information on the position of the operation device. In the example illustrated in FIG. 5, among the AR operation targets 1 to 4, the AR operation target 1 having the smallest distance from the operation device is specified as an operation target by the operation device. In the example of FIG. 4, the AR operation target having the minimum distance is specified. However, when the AR operation target having the minimum distance is equal to or less than a certain distance, the AR operation target is set as the operation target. It may be specified as An example of the program in this case is shown in FIG. The program corresponds to the operation of the specifying unit 12.

  The variables in the program of FIG. 6 are defined as follows.

AR operation target i: objects [i]; // objects is an array Operation device position information: x, y, z;
Position information of AR operation target i: xi, yi, zi;
Acquisition of absolute value: Math. abs ();
Acquisition of minimum value: Math. min ();
Distance between AR operation target i and operation device: distances [i] // distances is an array coefficient: A, B, C;
Threshold value to be regarded as an operation target: Threshold.

  Note that it is also possible to first detect an AR operation target within a certain distance and select the AR operation target closest to the position of the operation device when there are a plurality of AR operation targets.

  In addition, specific operation information (operation for designating multiple selection) is given to the operation device, and the operation information is notified to the plurality of specification units 112, so that the plurality of specification units 112 can perform an AR operation target (operation target by the operation device). ) May be specified. For example, multiple AR operation targets can be selected by operations similar to range selection, multiple selection, all selection, etc. in a desktop computer environment, or AR operation targets within a certain distance from the position of the operation device can be automatically selected. All may be selected.

(Operation unit 13)
The operation unit 13 converts the operation information given to the operation device according to the AR operation target, and gives the converted operation to the specified AR operation target. FIG. 7 shows an operation procedure of the operation unit 13.

  As illustrated in FIG. 7, the operation unit 13 reads information about an AR operation target specified by the specifying unit 12 (step S301), and reads operation information for the operation device acquired by the input unit 11 (step S302). . The operation unit 13 refers to the conversion DB 14, identifies the AR operation target acquired in steps S301 and S302 and the operation corresponding to the operation information, and applies the operation to the AR operation target (steps S303 and S304).

  FIG. 8 shows an example of the conversion DB 14. In the example shown in FIG. 8, information is held in a tabular format, but may be held in any format such as text or xml as long as it can hold information on a conversion method. As illustrated in FIG. 8, for example, when the AR operation target is “link” and the operation information is “tap”, the operation “link selection” is applied to the “link”.

  As an example, FIG. 9 shows an application example of the operation information to the AR operation target when the operation device has a touch panel. In an operation device having a touch panel, the operation information given to the touch panel can be directly applied to an operation target in the AR environment. That is, as shown in FIG. 9, the AR operation target 1 is expanded by performing “pinch” (for example, an operation of pushing the touch panel surface with two fingers) (for example, the user zooms in on the target). It seems to have grown larger).

  Also, as shown in FIG. 10, when a plurality of AR operation targets are specified, the operation information given to the operation device can be applied to the plurality of operation targets at the same time. In the example of FIG. 10, a pinch operation is applied to the AR operation objects 1 and 3.

  In the present embodiment, the relationship between the AR operation target and the operation device is temporarily fixed, and even when the operation device position changes, an operation (instruction for instructing to continue the application of the operation information to the AR operation target) Called a fixed operation). In this case, the position of the operation device can be applied to the position of the AR operation target.

  In other words, the AR operation target is designated by the position of the operation device, and an operation for instructing to temporarily fix the relationship between the AR operation target and the operation device is performed. Thereafter, the user can perform an actual operation on the operation device at hand (not the position of the AR operation target). As a result, it is possible to achieve both a quick operation device specification and an intuitive operation feeling after the specification. After the operation is completed, the application of the operation information of the operation device to the AR operation target can be stopped by releasing the fixation.

  It is also possible to apply the position of the operation device to the AR operation target by instructing synchronization of position information by operation of the operation device after specifying the AR operation target by the position of the operation device. Thereby, it becomes possible to quickly change the presentation (display) position of the AR operation target in the AR space using the operation device.

  In addition, when tilt or acceleration (e.g., shaking the operation device) given to the operation device is given to the input unit 11 as operation information, the operation information is converted according to the AR operation target, and becomes the AR operation target. It is also possible to apply. An example of the conversion DB 14 for performing the conversion is shown in FIG. In the example illustrated in FIG. 11, for example, when “button” is specified as an AR operation target and “acceleration” is given as operation information, “tap” is applied as an operation on the “button”. Furthermore, the geographical position coordinates (for example, GPS coordinates) of the operation device may be acquired, and the operation applied to the AR operation target may be determined in consideration of the geographical position coordinates.

  Note that tilt, acceleration, and geographical position information may be combined. That is, any one or more of inclination, acceleration, and geographical position information may be used. Further, the tilt, acceleration, and the like may be used in combination with operation information on the touch panel shown in FIG.

(Output unit 15)
FIG. 12 shows an operation procedure of the output unit 15. As illustrated in FIG. 12, the output unit 15 outputs the result of specifying the specifying unit 12 to the display 22 and outputs the result of the operation on the AR operation target by the operation unit 13 (AR operation target after operation) to the display 22. (Steps S401 and S402).

  That is, on the display (HMD or the like) that displays the AR environment, the identification result in the identification unit 12 and the result of operation in the operation unit 13 are integrated and presented. Thereby, the user can see the AR environment after the operation by the operation device, for example, as shown in FIG.

  The output unit 15 has a function of outputting an AR environment including not only an AR operation target related to an operation but also an AR operation target not related to the operation.

  In this way, the user of the operation support apparatus 10 can operate the AR operation target in the AR environment using the operation device while viewing the information displayed on the display 22.

  For example, when using an operation device equipped with a touch panel, the specified AR operation target may be projected onto the touch panel. As a result, the AR operation target can be operated on the touch panel, and the AR operation target can be operated with a feeling of operation of a device that has been widely used such as a touch panel. The process of projecting the specified AR operation target on the touch panel is performed by, for example, sending information on the projection image from the operation support apparatus 10 to the operation device, and displaying the projection image (AR operation target) by the operation device.

(Summary of embodiment)
As described above, according to the present embodiment, an AR operation target that is an operation target existing in an Augmented Reality (AR) environment that extends the real space by presenting virtual information superimposed on the real space. An operation support apparatus that supports an operation on the basis of the position of the operation device, which is a physical device, input means for acquiring operation information given to the operation device, and the position of the operation device, Based on the specifying means for specifying the AR operation target, the operation information given to the operation device, and the AR operation target specified by the specifying means, the operation information is converted, and the operation obtained by the conversion is An operation support apparatus is provided that includes an operation unit applied to the specified AR operation target.

  The operation support apparatus includes, for example, a conversion database that defines an operation corresponding to an AR operation target and operation information, and the operation means applies to the specified AR operation target by referring to the conversion database. The operation to be performed may be determined.

  The operation device is a device including a touch panel, for example, and the operation information is operation information indicating an operation on the touch panel. Further, as the operation information, any one or more of the tilt, acceleration, and geographical position information of the operation device may be used.

  When a specific operation is given to the operation device, the specifying unit simultaneously specifies a plurality of AR operation targets, and the operation unit performs an operation based on operation information for the operation device, the plurality of AR operation targets. It is good also as applying to.

  For example, the specifying unit can specify an AR operation target whose distance from the operation device is smaller than a predetermined threshold.

(Effect of embodiment)
By using the operation support apparatus 10 according to the present embodiment, the AR operation target can be intuitively operated as if using an existing device such as a smartphone, so that the learning time for the operation is shortened. In addition, since a large operation such as a gesture is not necessary, the operation operation can be reduced. In addition, in the operation of the AR environment using the HMD, there is a psychological barrier that the appearance of the operation when viewed from a third party looks strange (is seen). Can be reduced.

(Section 1)
An operation support apparatus that supports an operation to an AR operation target that is an operation target existing in an augmented reality (AR) environment that extends the real space by presenting virtual information superimposed on the real space,
A position of an operation device, which is a physical device, and an input means for acquiring operation information given to the operation device;
A specifying means for specifying an AR operation target based on the position of the operation device;
The operation information is converted based on the operation information given to the operation device and the AR operation target specified by the specifying means, and the operation obtained by the conversion is applied to the specified AR operation target. Operation means
An operation support apparatus comprising:
(Section 2)
The operation support device includes a conversion database that defines an operation corresponding to an AR operation target and operation information,
The operation means determines an operation to be applied to the specified AR operation target by referring to the conversion database.
The operation support apparatus according to item 1, wherein the operation support apparatus is described above.
(Section 3)
The operation device is a device including a touch panel, and the operation information is operation information indicating an operation on the touch panel.
The operation support device according to item 1 or 2, characterized by the above.
(Section 4)
As the operation information, any one or more of the tilt, acceleration, and geographical position information of the operation device is used.
The operation support device according to any one of Items 1 to 3, wherein
(Section 5)
When a specific operation is given to the operation device, the specifying unit simultaneously specifies a plurality of AR operation targets, and the operation unit performs an operation based on operation information for the operation device, the plurality of AR operation targets. Apply to
The operation support device according to any one of Items 1 to 4, wherein the operation support device is characterized in that
(Section 6)
The specifying unit specifies an AR operation target whose distance from the operation device is smaller than a predetermined threshold.
The operation support apparatus according to any one of claims 1 to 5, characterized in that:
(Section 7)
The program for functioning a computer as each means in the operation assistance apparatus of any one of Claim 1 thru | or 6.
The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

10 Operation Support Device 11 Input Unit 12 Identification Unit 13 Operation Unit 14 Conversion DB
15 Output unit 12 Multiple specifying unit 21 Position, operation information 22 Display

Claims (5)

An operation support apparatus that supports an operation to an AR operation target that is an operation target existing in an augmented reality (AR) environment that extends the real space by presenting virtual information superimposed on the real space,
First input means for acquiring a position of an operation device that is a physical device including a touch panel ;
A specifying means for specifying an AR operation target based on the position of the operation device;
Means for displaying the specified AR operation target on the touch panel of the operation device;
Second input means for acquiring operation information given to the operation device by an operation on the touch panel on which the specified AR operation target is displayed;
The operation information is converted based on the operation information given to the operation device and the AR operation target specified by the specifying means, and the operation obtained by the conversion is applied to the specified AR operation target. Operating means ,
The operation support device includes a conversion database that defines an operation corresponding to an AR operation target and operation information,
The operation support apparatus determines an operation to be applied to the specified AR operation target by referring to the conversion database . The operation support apparatus according to claim 1, wherein any one or more of tilt, acceleration, and geographical position information of the operation device is used as the operation information. When a specific operation is given to the operation device, the specifying unit simultaneously specifies a plurality of AR operation targets, and the operation unit performs an operation based on operation information for the operation device, the plurality of AR operation targets. operation support apparatus according to claim 1 or 2, characterized in that to apply to. The operation support apparatus according to any one of claims 1 to 3 , wherein the specifying unit specifies an AR operation target whose distance from the operation device is smaller than a predetermined threshold. The program for functioning a computer as each means in the operation assistance apparatus of any one of Claims 1 thru | or 4 .

Images