JP6792666B2 - Terminal device, terminal device control method, computer program - Google Patents

Description

The present invention relates to a character motion control technique.

Conventionally, there is a technique for generating and displaying a two-dimensional image of a character moving in a three-dimensional virtual space (Patent Document 1).

Japanese Unexamined Patent Publication No. 2013-232205

Tango, [online], [Searched on January 10, 2017], Internet, <URL: https: // get. google. com / tango />

In the technique disclosed in Patent Document 1, the movement of a character cannot be controlled based on the position of the character in the virtual space corresponding to the predetermined position in the real space. The present invention provides a technique for controlling the movement of a character based on the position of the character in a virtual space corresponding to a predetermined position in the real space.

The present invention is a terminal device including an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control means to perform the display operation of
A display control means for synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display .
It is characterized by including an audio output means for outputting an audio associated with the specified distance .

According to the configuration of the present invention, the movement of a character can be controlled based on the position of the character in the virtual space corresponding to the specified position in the real space.

The schematic block diagram of the terminal apparatus according to an embodiment. The figure for demonstrating the movable area and the route of a character. A flowchart of a process for displaying a composite image in which a captured image in the real space and a two-dimensional image of a character are combined.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a schematic configuration diagram of a terminal device 10 according to the present embodiment. The terminal device 10 is a portable communication device such as a smartphone. The speaker 101 converts the voice signal output from the control unit 100 into voice. The microphone 102 converts voice into a voice signal and outputs it to the control unit 100. The image pickup unit 103 has a lens and a CCD sensor, acquires image information, and outputs the image information to the control unit 100. The GPS processing unit 104 receives a signal from a GPS satellite, and outputs position information including latitude, longitude, and altitude to the control unit 100 based on the received signal. It should be noted that the configuration may be such that the position information is output based on the signal from the satellite of another positioning system such as the quasi-zenith satellite instead of the GPS satellite. In the present embodiment, the display 105 is a touch-sensitive display, and in addition to presenting information to the user, it also functions as an input unit for user operation. If the display is not a touch-sensitive display, an input unit is provided separately. The depth sensor 106 measures the distance (depth) to the object and outputs the measurement result to the control unit 100 as depth information. The communication unit 107 performs communication processing via a mobile communication network, a wireless LAN, or the like. The geomagnetic sensor 108 detects the geomagnetism and outputs directional information indicating the current azimuth of the terminal device 10 in the reference direction to the control unit 100. The acceleration sensor 109 detects the acceleration and outputs the acceleration information to the control unit 100. The gyro sensor 110 detects the angular velocity information and outputs the angular velocity information to the control unit 100.

The control unit 100 is a control unit for the entire terminal device 10, and includes one or more processors and a memory unit. The memory unit has a RAM used by the processor for temporarily storing data, and a flash memory in which the operating system and applications executed by the processor and the data to be retained even when the power is turned off are stored.

The control unit 100 generates location description information based on, for example, the technique described in Non-Patent Document 1. The location description information is information for determining the three-dimensional spatial position of the terminal device 10 and the imaging direction of the imaging unit 103 from the image captured by the imaging unit 103. First, the generation of the place description information described in Non-Patent Document 1 will be described. When the user of the terminal device 10 starts acquiring the location description information, the user of the terminal device 10 changes the position and the imaging direction while capturing an image by the imaging unit 103. While the location description information is acquired, the control unit 100 keeps determining the position of the terminal device 10 and the direction of the imaging direction based on the acceleration information and the angular velocity information, with the position at the start as the origin. Further, the control unit 100 obtains feature points in the image captured by the image pickup unit 103. At this time, the control unit 100 determines the position and imaging direction of the terminal device 10 in the three-dimensional space and the position of the same feature point in the image, that is, the screen coordinates. As a result, the control unit 100 generates location description information indicating at which position in the image captured by the imaging unit 103 each feature point can be seen at each position in the three-dimensional space and each imaging direction. Therefore, the terminal device 10 compares the already generated location description information with the feature points of the image captured by the imaging unit 103, and thereby, the three-dimensional spatial position of the terminal device 10 at that time and the imaging unit 103. It is possible to determine the imaging direction of. It is assumed that the origin of the three-dimensional space position is the position when the place description information is started. Further, the control unit 100 determines the three-dimensional spatial position of the terminal device 10 and the imaging direction of the imaging unit 103 by comparing the location description information already stored with the feature points of the image captured by the imaging unit 103. At the same time as the determination, the location description information already stored is updated based on the feature points of the image captured by the imaging unit 103.

As described above, since the location description information is information indicating the relationship between the position and imaging direction of the terminal device 10 and the position of each feature point of each object in the image captured by the imaging unit 103 in the screen coordinates. , It is possible to generate three-dimensional space information indicating the coordinates of the three-dimensional space position of each feature point based on the place description information. That is, for the information corresponding to the combination of different positions and imaging directions in the place description information, the information in the depth direction is obtained by considering the movement of the same feature point in the screen coordinates, and the three-dimensional space of the feature point is obtained. The position can be derived.

Such location description information is stored in the memory unit of the control unit 100. Further, in the memory unit of the control unit 100, character model data (data defining a three-dimensional geometric shape of the character, texture data, a plurality of types of motion data, etc.), text data, and voice data are stored.

Next, FIG. 3 describes a process for displaying a composite image obtained by combining the image captured in the real space by the imaging unit 103 and the two-dimensional image of the character drawn by the terminal device 10 on the display 105 of the terminal device 10. This will be described using the flowchart of. The process according to the flowchart of FIG. 3 is a process for displaying a composite image for one frame. However, by repeating the process according to the flowchart of FIG. 3, a composite image of a plurality of frames can be displayed on the display 105. Further, it is assumed that the computer program and data for causing the control unit 100 to execute the process according to the flowchart of FIG. 3 are included in the above application.

In step S301, the control unit 100 stores the captured image captured by the imaging unit 103 in the memory unit.

In step S302, the control unit 100 determines that the current position of the character in the virtual space corresponding to the real space is within the "region R within the specified distance D from the position in the virtual space corresponding to the specified position in the real space". Judge whether or not.

The control unit 100 moves the character in the virtual space by updating the position of the character in the virtual space periodically or irregularly. When moving the character, the character is made to perform a walking / running motion according to the corresponding motion data. In the virtual space, as shown in FIG. 2A, a range 210 in which the character 200 can move and a route 220 in which the character 200 can move are designated in advance as shown in FIG. 2B. It shall be. The coordinate positions of the range 210 (for example, the coordinate positions of the four corners of the range 210) and the coordinate positions of the route 220 (for example, the coordinate positions of points existing at equal intervals on the route 220) are obtained by using the above-mentioned location description information. This is in the coordinate system to which the "three-dimensional spatial position of the device 10 and the imaging direction of the imaging unit 103" belong.

Various methods can be considered for generating the movable range. For example, the coordinate position of the movable range (path) in the virtual space may be manually specified by the user, or the coordinate position corresponding to the movable range in the two-dimensional map information is converted into the above coordinate system. You may ask for it. In addition, the above three-dimensional space information is generated, and the floor and walls are measured by the depth sensor 106 to generate a three-dimensional structure model representing the three-dimensional geometric structure in the real space. In the three-dimensional structure model, the wall is located in the floor area. The part in front of is may be a movable range.

Further, the position in the virtual space corresponding to the specified position in the real space is, for example, when the user specifies the position by tapping on the display 105, the position in the real space corresponding to the specified position is used. It is a position in the virtual space corresponding to the position. The position in the virtual space corresponding to the position in the real space corresponding to the tap position on the display 105 is, for example, a projection transformation in which the tap position is "projected on the projection plane in order to draw a two-dimensional image of the character. It is a three-dimensional position on the floor surface converted by the inverse transformation of. Further, the depth information corresponding to the tap position is acquired by using the depth sensor 106, and the position on the back side (imaging direction side of the imaging unit 103) by the distance indicated by the depth information from the tap position is set in the real space. It may be a position in the virtual space corresponding to the specified position of. Further, the above-mentioned distance D may be a different value for each "position in the virtual space corresponding to the specified position in the real space", or may be a different value for each "position in the virtual space corresponding to the specified position in the real space". It may be a fixed value regardless of the "position of". In the former case, the distance D is stored in the memory unit in association with "a position in the virtual space corresponding to a predetermined position in the real space". The "position in the virtual space corresponding to the specified position in the real space" may be registered in the memory unit in advance, or may be downloaded from an external device such as a server.

If the current position of the character in the virtual space corresponding to the real space is in the area R, the process proceeds to step S303, and if the current position of the character in the virtual space corresponding to the real space is outside the area R, the process proceeds. Proceeds to step S304.

In step S303, the control unit 100 causes the character to perform a predetermined operation. For example, even if the motion of the character (motion performed by moving parts such as hands and feet) is controlled based on the motion data associated with "the position in the virtual space corresponding to the specified position in the real space". Alternatively, the position and posture of the character may be changed, such as jumping, running around, or lying down according to the operating rules associated with "the position in the virtual space corresponding to the specified position in the real space".

For example, the specified operation may be switched for each attribute information of the user of the terminal device 10. For example, when the user's attribute information indicates "male", a female character is displayed (in female motion), the character's utterance is output from the speaker 101 by female voice, or a female tone text is output. It may be displayed in a balloon. Further, comments from other users exchanged on SNS may be displayed in balloons, or the comments may be output from the speaker 101 as voice utterance contents of the character.

Furthermore, the display content of the character or text and the content of the voice may be changed depending on at least one of the time zone, the number of times the character is displayed, the season and the time. For example, if the current time measured (or received from the outside) by the control unit 100 is in the morning time zone (for example, from 6:00 to 12:00), a process related to the morning (for example, "Good morning" is said. Along with reproducing the motion, the speaker 101 may output a voice saying "Good morning"). Further, for example, when displaying the image of the character for the first time, for example, the motion of saying "Nice to meet you" may be reproduced and the voice of "Nice to meet you" may be output from the speaker 101. Also, for example, if the current date and time is Christmas day, a character wearing Santa's clothes may be displayed. In addition, it recognizes a message input by the user to the terminal device 10, extracts a keyword from the recognized message, reproduces a motion associated with the extracted keyword, displays a text, and outputs a voice. You may do it. When the user emits a voice into the microphone 102, the voice can be recognized as a message by the voice recognition technology. Further, when the user inputs a text on the display 105, the text can be recognized as a message by the character recognition technology.

Also, when the audio output is stopped as in the manner mode, the text display is permitted without performing the audio output, and when the audio output is permitted as in the audio output mode, the text is displayed. May be prohibited and audio output may be permitted.

In this way, the character operation rules, text, and audio output conditions according to various conditions may be managed as a database file in an external device such as a server. In that case, this database file may be periodically or from the external device. By downloading irregularly, the database file held by the terminal device 10 can be partially updated or added, or replaced with another database file. It is also possible to access an external device from the terminal device 10 and edit a database file (character motion, text content, voice content, motion reproduction condition, text or voice output condition, etc.). Further, the speaker 101 may output a voice based on the voice data of the character associated with the "position in the virtual space corresponding to the predetermined position in the real space".

In step S304, the technique described in Non-Patent Document 1 is used, and the location description information is compared with the feature points on the captured image stored in the memory unit in step S301. It recognizes the three-dimensional spatial position of the device 10 and the imaging direction of the imaging unit 103 (that is, the position and orientation of the terminal device 10). Then, the control unit 100 sets each of the recognized three-dimensional position and the imaging direction as the position and the posture of the virtual viewpoint. Then, the control unit 100 arranges a character based on the model data in a movable area in the virtual space or on a movable path, and draws a two-dimensional image of the arranged character as viewed from a virtual viewpoint.

In step S305, the control unit 100 displays on the display 105 a composite image obtained by synthesizing the two-dimensional image of the character drawn in step S304 on the captured image stored in the memory unit in step S301.

The location description information created in advance for facilities such as museums may be uploaded to an external device such as a server in advance. In this case, the terminal device 10 can download the location description information corresponding to the position information from the external device by transmitting the position information by the GPS processing unit 104 to the external device. The download can be started, for example, by the user touching the download icon displayed on the display 105 of the terminal device 10. When there are a plurality of location description information corresponding to the location information, the terminal device 10 downloads the plurality of location description information as candidates, displays them on the display 105, and instructs the user to select one. You may accept it.

Further, the terminal device 10 periodically uploads the captured image or the feature points in the captured image to an external device such as a server, and the external device periodically uploads the captured image or the feature points in the captured image to the terminal based on the captured image or the feature points uploaded from the terminal device 10. The location description information including the position of the device 10 may be transmitted to the terminal device 10.

Further, the terminal device 10 does not display the character image for the specified time or longer when the position information by the GPS processing unit 104 is separated from the area of the place description information, or when the usage time of the place description information exceeds the specified time. If the conditions such as the above are satisfied, the location description information stored in the memory unit may be deleted.

Data other than the location description information (such as the model data described above) may also be acquired from an external device via the communication unit 107. Further, the two-dimensional image of the character may be drawn based on both the position and the posture of the virtual viewpoint, or may be drawn based on the position or the posture.

Further, in the present embodiment, the terminal device 10 determines the position and direction of the terminal device 10 in the three-dimensional space by using the location description information based on the feature points included in the captured image. However, for example, transmitters that transmit signals indicating the respective positions may be installed at a plurality of positions in a predetermined area. The transmitter can be, for example, a speaker that outputs sound outside the human audible range. In this case, the microphone 102 of the terminal device 10 converts the sound of the speaker into an audio signal, and the terminal device 10 determines where in the three-dimensional space the sound signal is located. As a result, when the terminal device 10 determines the position and direction of the terminal device 10 based on the feature points of the captured image, the range of the place description information to be taken into consideration can be narrowed down, and the position and direction of the terminal device 10 can be determined. The time required for this can be shortened. Further, once the position and direction of the terminal device 10 are determined, the terminal device 10 determines the position and direction of the terminal device 10 after the change even if the position and direction of the terminal device 10 are changed by the acceleration information and the angular velocity information. You can continue to judge. As a result, for example, after determining the position and direction of the terminal device 10, even if the user puts the terminal device 10 in a pocket, a bag, or the like and the state in which the captured image cannot be obtained continues, the user can use the imaging unit 103. When the acquisition of the captured image is started and the position and direction of the terminal device 10 are determined based on the location description information, the location description to be taken into consideration is based on the position and direction of the terminal device 10 determined by the acceleration information and the angular velocity information. The range of information can be narrowed down. Further, the place description information may include position information such as the longitude and latitude of the area corresponding to the place description information. In that case, if it is determined by the position information or the like by the GPS processing unit 104 that the terminal device 10 is in or near the area corresponding to the location description information, the terminal device 10 is in or near the area. The user may be notified (notified / vibrated on the screen) that he / she is in the vicinity.

Further, in the above description, the two-dimensional image of the character is drawn by the terminal device 10, but an external device such as a server performs a process of drawing the two-dimensional image of the character and draws the two-dimensional image on the terminal device 10. You may send a two-dimensional image. In that case, the terminal device 10 transmits information (position and orientation of the virtual viewpoint, character position, motion information, etc.) necessary for drawing a two-dimensional image of the character to the external device.

Further, the above-mentioned distance D may take a plurality of distance values (D1, D2, ...: D1> D2> ...). Then, when the current position of the character in the virtual space is within the specified distance Di (i = 1, 2, ...) From the position in the virtual space corresponding to the specified position in the real space, the distance Di You may want the character to perform the associated action. For example, when the current position of the character is within the specified distance D1 from the position in the virtual space corresponding to the specified position in the real space, the voice of the character associated with the distance D1 is output from the speaker 101. When the vocal motion of the character is reproduced and the distance is within the specified distance D2 from the position in the virtual space corresponding to the specified position in the real space, the text associated with the distance D2 is displayed on the display 105 and the character is displayed. You may try to reproduce the utterance motion of. In this way, the motion, text, and voice data of the character may be stored in the memory unit or the external device for each distance value of the distance D.

Further, when the current position of the character in the virtual space belongs to a plurality of regions R, it is associated with the region R having the highest priority among the preset priorities for each region R. Processing (control of character motion, display of text, output of voice) may be performed. In addition, when the current position of the character in the virtual space belongs to a plurality of areas R, the process associated with the area R closest to the current position of the character in the virtual space (control of character motion, text). Display and audio output) may be performed.

Further, the above-mentioned distance D may be changed according to at least one of time, day, day of the week, month, and season.

Further, the "current position of the character" may be the "position of the area around the character". In that case, if a part or all of the character's peripheral area overlaps with the area R, the process proceeds to step S303, and if the character's peripheral area does not overlap with the area R, the process proceeds to step S304.

Claims (11)

A terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control means to perform the display operation of
A display control means for synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display .
A terminal device including an audio output means for outputting audio associated with the specified distance . A terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control means to perform the display operation of
A display control means for synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
An audio output means for outputting audio associated with the region having the highest priority among the plurality of regions to which the character's position in the virtual space belongs.
Terminal device comprising: a. A terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control means to perform the display operation of
A display control means for synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
A voice output means for outputting voice associated with the region closest to the position among the plurality of regions to which the character's position in the virtual space belongs.
Terminal device comprising: a. A terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control means to perform the display operation of
A display control means for synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
With
The terminal device, characterized in that the specified distance differs for each position in the virtual space corresponding to the specified position. In the voice output mode, the voice output means outputs voice, and the control means prohibits the display of text.
The terminal device according to any one of claims 1 to 3 , wherein in the manner mode, the voice output means does not output voice and the control means displays text . The terminal device according to any one of claims 1 to 3 , wherein the audio output means changes the content of the audio to be output according to at least one of a time zone, a season, and a time period . A control method for a terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control process to perform the display operation of
A display control step of synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display .
A method for controlling a terminal device, which comprises a voice output process for outputting voice associated with the specified distance . A control method for a terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control process to perform the display operation of
A display control step of synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
A voice output process that outputs voice associated with a region having the highest priority among a plurality of regions to which the character's position in the virtual space belongs.
A method for controlling a terminal device, which comprises. A control method for a terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control process to perform the display operation of
A display control step of synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
A voice output process that outputs voice associated with the region closest to the position among the plurality of regions to which the character's position in the virtual space belongs.
A method for controlling a terminal device, which comprises. A control method for a terminal device having an imaging means and a display.
When the position of the character in the virtual space corresponding to the real space is within a specified distance from the position in the virtual space corresponding to the specified position different from the position of the imaging means in the real space, it is predetermined. Control process to perform the display operation of
A display control step of synthesizing an image of the display operation on an image captured by the imaging means and displaying it on the display.
With
A method for controlling a terminal device, wherein the specified distance differs for each position in the virtual space corresponding to the specified position. A computer program for causing a computer of a terminal device to function as each means of the terminal device according to any one of claims 1 to 6 .

Images