JP6216398B2 - Image generating apparatus and image generating method - Google Patents

Description

  The present invention relates to an apparatus and a method for generating an image to be displayed on a head mounted display.

  A head-mounted display connected to a game machine is attached to the head, and a game is played by operating a controller or the like while viewing a screen displayed on the head-mounted display. In a normal stationary display connected to a game machine, the user's field of view extends outside the display screen, so it may not be possible to concentrate on the display screen or lack of immersion in the game. . On the other hand, when the head mounted display is mounted, since the user does not see any video other than the video displayed on the head mounted display, the feeling of immersion in the video world is increased and the entertainment property of the game is further enhanced.

  In addition, when a panoramic image is displayed on the head-mounted display and a user wearing the head-mounted display rotates his / her head, a 360-degree panoramic image and a virtual space are displayed. The operability of applications such as games will be improved.

  However, in order to display a panoramic image or a virtual space on the head mounted display, calculation power is required on the head mounted display side, which increases processing load.

  The present invention has been made in view of these problems, and an object thereof is to provide an image generation apparatus and an image generation method capable of efficiently generating an image to be displayed on a head-mounted display.

  In order to solve the above-described problem, an image generation apparatus according to an aspect of the present invention is an image generation apparatus that provides a cloud service to a head-mounted display via a network, and receives a plurality of captured images having different shooting directions. A captured image receiving unit, a panoramic image processing unit that generates a panoramic image by combining a plurality of captured images with different shooting directions, a format conversion unit that converts the panoramic image into an image format suitable for the specifications of the head mounted display, A panoramic image transmission unit that transmits the panoramic image after the format conversion to the head mounted display.

  Another embodiment of the present invention is also an image generation device. This device is an image generation device that provides a cloud service to a head-mounted display via a network, and obtains information on the position of a user wearing the head-mounted display acquired by a position sensor mounted on the head-mounted display. A positional information receiving unit for receiving, a positional relationship processing unit for determining the positional relationship of the characters of the plurality of users based on the positional relationship between the plurality of users, and the plurality of users based on the positional relationship of the determined characters Are arranged in a virtual space, a three-dimensional rendering unit that renders the virtual space, and a drawing data transmission unit that transmits information of the rendered virtual space to the head mounted display.

  Yet another embodiment of the present invention is an image generation method. This method is an image generation method for providing a cloud service to a head-mounted display via a network, and includes a captured image receiving step for receiving a plurality of captured images with different shooting directions, and a plurality of captured images with different shooting directions. A panorama image processing step for combining and generating a panorama image, a format conversion step for converting the panorama image into an image format suitable for the specifications of the head mounted display, and a panorama image for transmitting the panorama image after the format conversion to the head mounted display Sending step.

  Yet another embodiment of the present invention is also an image generation method. This method is an image generation method for providing a cloud service to a head mounted display via a network, and information on the position of a user wearing the head mounted display acquired by a position sensor mounted on the head mounted display is obtained. A positional information receiving step for receiving, a positional relationship processing step for determining the positional relationship of the characters of the plurality of users based on the positional relationship between the plurality of users, and the plurality of users based on the determined positional relationship of the characters Are arranged in a virtual space, and a three-dimensional rendering step for rendering the virtual space, and a drawing data transmission step for transmitting information on the rendered virtual space to the head mounted display are included.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the image displayed on a head mounted display can be produced | generated efficiently.

It is an external view of a head mounted display. It is a functional lineblock diagram of a head mount display. It is a block diagram of an image generation system. It is a functional block diagram of a panoramic image generation device. It is a block diagram of the system which shares an image | video by several head mounted displays. It is a functional block diagram of a virtual space drawing processing apparatus. It is a sequence diagram explaining a virtual space drawing process procedure. It is a flowchart explaining in detail the procedure of the positional relationship process with the other user of FIG. It is a figure explaining the avatar of the 1st user who appeared in virtual space. It is a figure explaining the avatar of the 2nd user who appeared in virtual space. It is a figure explaining the avatar of the 3rd user who appeared in virtual space. It is a figure explaining the avatar of the 4th user who appeared in virtual space. It is a figure explaining another example which makes a user's avatar appear in virtual space. It is a figure explaining the example which makes a user's avatar appear on a game screen. It is a figure explaining the score calculation method of a user's positional relationship in case there is no geographical correlation between a virtual space and the real world. It is a figure explaining the score calculation method of a user's positional relationship in case there is no geographical correlation between a virtual space and the real world. It is a figure explaining the score calculation method of a user's positional relationship in the case where there is a geographical correlation between the virtual space and the real world. It is a figure explaining the score calculation method of a user's positional relationship in the case where there is a geographical correlation between the virtual space and the real world.

  FIG. 1 is an external view of the head mounted display 100. The head mounted display 100 includes a main body portion 110, a forehead contact portion 120, and a temporal contact portion 130.

  The head mounted display 100 is a display device that is worn on the user's head and enjoys still images and moving images displayed on the display, and listens to sound and music output from the headphones.

  The position information of the user can be measured by a position sensor such as GPS (Global Positioning System) incorporated in or externally attached to the head mounted display 100. Further, posture information such as the orientation and inclination of the head of the user wearing the head mounted display 100 can be measured by a posture sensor built in or externally attached to the head mounted display 100.

  The main body 110 includes a display, a position information acquisition sensor, a posture sensor, a communication device, and the like. The forehead contact unit 120 and the temporal contact unit 130 include a biological information acquisition sensor that can measure biological information such as a user's body temperature, pulse, blood component, sweating, brain waves, and cerebral blood flow.

  The head mounted display 100 may further be provided with a camera that captures the eyes of the user. The camera mounted on the head mounted display 100 can detect the user's line of sight, pupil movement, blinking, and the like.

  Here, a method for generating an image displayed on the head mounted display 100 will be described. However, the image generating method according to the present embodiment is not limited to the head mounted display 100 in a narrow sense, but a glasses, a glasses type display, a glasses type camera, The present invention can also be applied when headphones, headsets (headphones with microphones), earphones, earrings, ear-mounted cameras, hats, hats with cameras, hair bands, and the like are attached.

  FIG. 2 is a functional configuration diagram of the head mounted display 100.

  The control unit 10 is a main processor that processes and outputs signals such as image signals and sensor signals, commands and data. The input interface 20 receives operation signals and setting signals from the touch panel and the touch panel controller, and supplies them to the control unit 10. The output interface 30 receives the image signal from the control unit 10 and displays it on the display. The backlight 32 supplies a backlight to the liquid crystal display.

  The communication control unit 40 transmits data input from the control unit 10 to the outside through wired or wireless communication via the network adapter 42 or the antenna 44. The communication control unit 40 also receives data from the outside via wired or wireless communication via the network adapter 42 or the antenna 44 and outputs the data to the control unit 10.

  The storage unit 50 temporarily stores data, parameters, operation signals, and the like processed by the control unit 10.

  The GPS unit 60 receives position information from a GPS satellite and supplies it to the control unit 10 in accordance with an operation signal from the control unit 10. The radio unit 62 receives position information from the radio base station and supplies it to the control unit 10 in accordance with an operation signal from the control unit 10.

  The attitude sensor 64 detects attitude information such as the orientation and inclination of the main body 110 of the head mounted display 100. The posture sensor 64 is realized by appropriately combining a gyro sensor, an acceleration sensor, an angular acceleration sensor, and the like.

  The external input / output terminal interface 70 is an interface for connecting peripheral devices such as a USB (Universal Serial Bus) controller. The external memory 72 is an external memory such as a flash memory.

  The clock unit 80 sets time information according to a setting signal from the control unit 10 and supplies time data to the control unit 10.

  The control unit 10 can supply the image and text data to the output interface 30 to be displayed on the display, or can supply the image and text data to the communication control unit 40 to be transmitted to the outside.

  FIG. 3 is a configuration diagram of the image generation system according to the present embodiment. The image generation system includes a head mounted display 100, a cloud server 200, and a camera 300.

  The head-mounted display 100 is an example of a head-mounted display device that a user can wear on the head and view images such as still images and moving images. The head mounted display 100 has a communication function such as a wired LAN, a wireless LAN, Bluetooth (trademark), and a mobile phone line, and can input and output data with the outside. The head mounted display 100 performs data communication with the cloud server 200 via the network 400. Further, the position of the user can be measured by a GPS sensor mounted on the head mounted display 100, and the user position information can be transmitted to the cloud server 200. In addition, the orientation and inclination of the head mounted display 100 can be measured by the orientation sensor mounted on the head mounted display 100, and the orientation information can be transmitted to the cloud server 200.

  The camera 300 is an example of an image data acquisition device that can capture a two-dimensional or three-dimensional still image or moving image. The camera 300 has a communication function such as a wired LAN, a wireless LAN, Bluetooth (trademark), and a mobile phone line, and can input and output data with the outside. The camera 300 performs data communication with the cloud server 200 via the network 400. In addition, the camera 300 has a function of recording device information such as exposure, lens information, and posture at the time of shooting, and can transmit device information together with the shot image data to the cloud server 200. In addition, the camera 300 may have a function of measuring the distance to a subject photographed by a distance sensor using a built-in or external ultrasonic wave or laser, and transmits distance information to the cloud server 200 together with photographed image data. can do.

  The cloud server 200 processes the image data acquired by the camera 300 into a form suitable for displaying on the head mounted display 100 and transmits the processed image data to the head mounted display 100.

  For example, the cloud server 200 may convert data optimized for each head-mounted display such as resolution, display method, and image quality so as to support head-mounted displays with different hardware specifications. The cloud server 200 may process the image data captured by the camera 300 so as to be optimal for display on a head-mounted display. The cloud server 200 may process image data that can be displayed three-dimensionally from a plurality of captured images captured by a plurality of cameras that capture a two-dimensional image. The cloud server 200 may calculate distance data from a plurality of captured images captured by a plurality of cameras that capture a two-dimensional image to a subject. The cloud server 200 may process the image data that can be displayed two-dimensionally by combining the distance data acquired by the distance sensor and the image data. The cloud server 200 may generate panoramic image data by connecting data captured by the camera 300.

  As described above, since various processes can be performed on the image on the cloud server 200 side by using the cloud service, the hardware configurations of the head mounted display 100 and the camera 300 can be made simple and inexpensive. By transmitting an image captured by the camera 300 to the cloud server 200 and receiving and displaying the optimized data by the head mounted display 100, the user can view a panoramic image of a remote place in real time.

  FIG. 4 is a functional configuration diagram of the panoramic image generation apparatus 500. The functional configuration of the panoramic image generation apparatus 500 is implemented in the cloud server 200. A part of the functional configuration of the panoramic image generation apparatus 500 can be mounted on the head mounted display 100 or the camera 300.

  The captured image receiving unit 510 receives a plurality of captured images having different shooting directions from the camera 300 and stores them in the image data storage unit 520. The panoramic image processing unit 530 generates a panoramic image by stitching together a plurality of photographed images with different photographing directions stored in the image data storage unit 520 by stitching processing.

  The format conversion unit 540 converts the resolution, aspect ratio (aspect ratio), 3D display data format, etc. of the panoramic image in accordance with the specifications of the head mounted display 100.

  The panorama image transmission unit 550 transmits the panorama image data converted by the format conversion unit 540 to the head mounted display 100.

  When the camera 300 is an imaging device capable of panoramic shooting, the captured image receiving unit 510 receives a panoramic image from the camera 300 and stores it in the image data storage unit 520. In this case, since the stitching process is unnecessary, the format conversion unit 540 may read the panoramic image from the image data storage unit 520 and convert the image format in accordance with the specifications of the head mounted display 100.

  When the camera 300 includes a distance sensor that measures the distance to the subject, the captured image receiving unit 510 receives distance information from the camera 300 together with the two-dimensional panoramic captured image, and stores the two-dimensional panoramic image in the image data storage unit 520. Save image and distance information. The panoramic image processing unit 530 performs image analysis, calculates parallax using distance information from the two-dimensional panoramic image, and generates a three-dimensional panoramic image including a left-eye image and a right-eye image. The format conversion unit 540 converts the resolution, aspect ratio (aspect ratio), display data format, and the like of the 3D panoramic image in accordance with the specifications of the head mounted display 100. The panoramic image transmission unit 550 transmits the three-dimensional panoramic image to the head mounted display 100.

  In addition, the panoramic image processing unit 530 of the cloud server 200 may apply various effects to the real image captured by the camera 300. For example, effects such as computer graphics (CG) tone, animation tone, and sepia tone are applied to a live-action image. Further, the panorama image processing unit 530 may generate a CG such as a game character and render it superimposed on a real image captured by the camera 300. Thereby, the cloud server 200 can distribute the video content obtained by combining the photographed image and the CG to the head mounted display 100.

  FIG. 5 is a configuration diagram of a system for sharing a video by a plurality of head mounted displays. The video sharing system includes a plurality of head mounted displays 100 a and 100 b and a cloud server 200. The head mounted displays 100 a and 100 b perform data communication with the cloud server 200 via the network 400.

  The head mounted displays 100 a and 100 b transmit user location information in the real world to the cloud server 200. The cloud server 200 transmits image data reflecting the user's positional relationship to the head mounted displays 100a and 100b.

  For example, when a plurality of users are sitting around a table in one room, the cloud server 200 reflects the actual position information on the position information of the avatar in the virtual world based on the position information of each user. Determine the position and draw each user's avatar in the virtual space. In addition, when multiple users wear head-mounted displays in front of a movie theater or an exhibit, videos with different viewpoint positions may be distributed to the user's head-mounted display according to the user's position in the real world. . When a plurality of users are at different locations, the distance between the user locations may be reflected in the positional relationship of the avatar and the display size of the avatar.

  FIG. 6 is a functional configuration diagram of the virtual space drawing processing apparatus 600. The functional configuration of the virtual space rendering processing device 600 is implemented in the cloud server 200. A part of the functional configuration of the virtual space rendering processing apparatus 600 can be mounted on the head mounted display 100.

  The user position / orientation receiving unit 610 includes information on the position of the user acquired by the GPS sensor built in the head mounted display 100 and the orientation of the user's head acquired by the posture sensor built in the head mounted display 100. Is received from the head mounted display 100, and the user position and orientation information is stored in the user database 660 in association with the user identification information.

  The positional relationship processing unit 620 reads information on the positions and orientations of a plurality of users from the user database 660 and determines the positional relationship between the plurality of users. For example, if two users face each other in the real world, the position and direction of the avatar are determined so that the two avatars face each other in the virtual space. In addition, when the distance between the user A and the user B in the real world is larger than the distance between the user A and the user C, the avatar of the user A is more avatar of the user C than the avatar of the user B even in the virtual space. It is arranged at a close position. When user A and user B face each other, user A's avatar and user B's avatar face each other. When the users A, B, and C surround one object and face each other, the avatars of the users A, B, and C also determine the directions so as to surround and face the object in the virtual space.

  The line-of-sight direction setting unit 630 sets the camera position of each user, in other words, the line-of-sight direction in the virtual space, according to the actual direction of the user. For example, when a plurality of users surround a single target object, the direction in which the user is facing the real target object is set as the line-of-sight direction in the user's virtual space.

  The three-dimensional rendering unit 640 reads virtual space information and character data from the virtual space information storage unit 670, and performs three-dimensional rendering of the virtual space viewed from the line-of-sight direction of each user. Each user's avatar is synthesized in the virtual space. The drawing data transmission unit 650 transmits virtual space drawing data to the head mounted display 100.

  FIG. 7 is a sequence diagram illustrating the virtual space drawing processing procedure.

  The head mounted display 100 initializes various variables (S10). A GPS sensor or posture sensor mounted on the head mounted display 100 measures the position and orientation of the user (S12). The communication control unit 40 of the head mounted display 100 transmits the user position and orientation information to the cloud server 200.

  The positional relationship processing unit 620 of the cloud server 200 maps the user character to a position in the virtual space based on the user position information (S14). The positional relationship processing unit 620 calculates the relative positional relationship with other users already mapped in the virtual space, and determines the position and orientation for displaying the user's character (S16). The line-of-sight setting unit 630 calculates the camera direction of the user, that is, the line-of-sight direction in the virtual space (S18). The three-dimensional rendering unit 640 performs three-dimensional rendering of the virtual space viewed from the viewpoint direction of the user (S20). The drawing data transmission unit 650 transmits three-dimensional rendering drawing data to the head mounted display 100.

  The head mounted display 100 displays the three-dimensional drawing data (S22). If the end condition is satisfied (Y in S24), the process ends. If the end condition is not satisfied (N in S24), the process returns to step S12 and the subsequent processing is repeated.

  FIG. 8 is a flowchart for explaining in detail the procedure of the positional relationship process with another user in step S16 of FIG.

  First, it is determined whether or not an avatar already exists in the virtual space (S30). If there is no avatar yet in the virtual space (N in S30), the avatar of the user appears in the virtual space (S42).

  If an avatar already exists in the virtual space (Y in S30), a positional relation score is calculated (S32). For example, the position relationship score is obtained based on the distance between users in the real world. Using the information on the position and direction of each user in the real world, the relative positional relationship of avatars, that is, the distance between avatars and the direction that one avatar takes with respect to another avatar are determined.

  An example of the positional relationship score calculation in step S32 will be described with reference to FIGS. Although an example of the score calculation method will be described here, the score calculation method is not limited to this.

  12 and 13 are diagrams for describing a score calculation method for a user's positional relationship when there is no geographical correlation between the virtual space and the real world.

  First, based on the latitude and longitude data of each user's real-world position, the distance and direction of another user based on each user are obtained. The position of the user in the real world is acquired using GPS data. FIG. 12A shows the latitude and longitude data of the real world positions of the user 1, the user 2, and the user 3.

  FIG. 12B shows the distance and direction of the user 2 and the user 3 with respect to the user 1 in the real world. The distance of the user 2 with respect to the user 1 is Lr_12, and the direction is Ar_12. The distance of the user 3 with respect to the user 1 is Lr_13, and the direction is Ar_13. Here, the direction is defined as a clockwise angle with respect to the latitude line.

  Similarly, FIG.12 (c) shows the distance and direction of the user 2 and the user 1 on the basis of the user 3 in the real world. The distance of the user 2 with respect to the user 3 is Lr_32, and the direction is Ar_32. The distance of the user 1 with respect to the user 3 is Lr_31, and the direction is Ar_31.

  If there is only one avatar in the virtual space in advance, the direction calculation can be omitted.

  Next, based on the absolute coordinates of the position of each user in the virtual space, the distances and directions of other users based on each user are obtained. FIG. 13A shows absolute coordinates of positions of the virtual spaces of the user 1, the user 2, and the user 3.

  FIG. 13B shows the distance and direction of the user 2 and the user 3 with respect to the user 1 in the virtual space. The distance of the user 2 with respect to the user 1 is Lv_12, and the direction is Av_12. The distance of the user 3 with respect to the user 1 is Lv_13, and the direction is Av_13.

  Similarly, FIG.13 (c) shows the distance and direction of the user 2 and the user 1 on the basis of the user 3 in virtual space. The distance of the user 2 with respect to the user 3 is Lv_32, and the direction is Av_32. The distance of the user 1 with respect to the user 3 is Lv_31, and the direction is Av_31.

  Next, the distance / direction between users in the real world is compared with the distance / direction between users in the virtual space, and a correlation coefficient is calculated.

The correlation coefficient Rmn of the positional relationship between the real world and the virtual space for the user m and the user n is defined by the following equation.
Rmn = | Lv_mn−Lr_mn | × | Av_mn−Ar_mn |
Here, | a | is an operation that takes the absolute value of the value a.

  The correlation coefficient Rmn is used as the score of the user's positional relationship (the smaller the correlation coefficient, the higher the score, and the higher the correlation coefficient, the lower the score), and the correlation coefficient Rmn is below a predetermined threshold value. If there is, it is considered that the positional relationship between the real world and the virtual space is the same.

  14 and 15 are diagrams illustrating a score calculation method for a user's positional relationship when there is a geographical correlation between the virtual space and the real world.

  First, based on the latitude / longitude data of each user's real world location and the latitude / longitude data of the real world geographical reference point, each user's reference is based on the real world geographical reference point (here, Tokyo). Find the distance and direction. The position of the user in the real world is acquired using GPS data. FIG. 14A shows latitude and longitude data of the user 1, the user 2, the user 3, and the real world location in Tokyo.

  FIG. 14B shows the distance and direction of the user 2 with reference to Tokyo in the real world. The distance of the user 2 with respect to Tokyo in the real world is Lr_02 and the direction is Ar_02. Similarly, FIG.14 (c) shows the distance and direction of the user 1 on the basis of Tokyo in the real world. The distance of the user 1 with respect to Tokyo in the real world is Lr_01, and the direction is Ar_01. FIG. 14D shows the distance and direction of the user 3 with reference to Tokyo in the real world. The distance of the user 3 with respect to Tokyo in the real world is Lr_03 and the direction is Ar_03.

  Next, based on the absolute coordinates of the virtual space position of each user and the absolute coordinates of the geographical reference point of the virtual space, the distance of each user based on the geographical reference point of the virtual space (here, Tokyo) And seek directions. FIG. 15A shows the absolute coordinates of the positions of the user 1, the user 2, the user 3, and the virtual space in Tokyo.

  FIG. 15B shows the distance and direction of the user 2 with reference to Tokyo in the virtual space. The distance of the user 2 with reference to the virtual space Tokyo is Lv_02, and the direction is Av_02. Similarly, FIG.15 (c) shows the distance and direction of the user 1 on the basis of Tokyo in virtual space. The distance of the user 1 relative to the virtual space Tokyo is Lv_01, and the direction is Av_01. FIG. 15D shows the distance and direction of the user 3 with reference to Tokyo in the virtual space. The distance of the user 3 relative to the virtual space Tokyo is Lv_03, and the direction is Av_03.

  Next, the distance / direction between the reference point and the user in the real world and the distance / direction between the reference point and the user in the virtual space are compared, and a correlation coefficient is calculated.

The correlation coefficient R0m of the positional relationship between the real world and the virtual space for the reference point and the user m is defined by the following equation.
R0m = | Lv — 0m−Lr — 0m | × | Av — 0m—Ar — 0m |

  The correlation coefficient R0m is used as a score of the user's positional relationship (the smaller the correlation coefficient, the higher the score, and the higher the correlation coefficient, the lower the score), so that the correlation coefficient R0m is equal to or less than a predetermined threshold. If there is, it is considered that the positional relationship between the real world and the virtual space is the same.

  Refer to FIG. 8 again. It is determined whether the real world positional relationship is the same as the avatar positional relationship in the virtual space (S34). This is because the avatar may move in the virtual space and deviate from the actual positional relationship. The correlation coefficient Rmn (or R0m) obtained in step S32 is compared with the threshold value Rth. If Rmn <Rth (or R0m <Rth) for all users, it is determined that the real world positional relationship and the avatar positional relationship in the virtual space are the same. When the positional relationship between the real world and the avatar in the virtual space is maintained within a predetermined range (Y in S34), the user's avatar appears in the virtual space (S42).

  When the positional relationship between the real world and the avatar in the virtual space is not maintained (N in S34), it is determined whether there is a place in the virtual space that is close to the actual positional relationship between the users (S36). In the comparison between the correlation coefficient and the threshold value performed in step S34, if Rmn <Rth (or R0m <Rth) for one or more users, it is determined that there is a place close to the actual positional relationship between the users in the virtual space. . If there is a place in the virtual space that is close to the actual positional relationship between users (Y in S36), the place with the highest score is determined as the closest place (S38). For example, in the comparison between the correlation coefficient and the threshold value performed in step S34, the position is determined using only user data satisfying Rmn <Rth (or R0m <Rth). If there is no place close to the actual positional relationship between users in the virtual space (N in S36), an appropriate place, for example, a random position or the last position is determined (S40). In any case, the user's avatar appears at the determined location (S42).

  9A to 9D are diagrams illustrating an example in which a user's avatar appears in the virtual space. Here, an example will be described in which avatars 700a to 700d of four users A to D appear in the virtual space. Assume that the four users A to D are in different places in the real world.

  As shown in FIG. 9A, it is assumed that user A first logs in to the virtual space, and avatar 700a appears in the virtual space.

  In the real world, it is assumed that the remaining users B, C, and D are close to the user A in this order. For example, for user A in Tokyo, user B is in Yokohama, user C is in Kobe, and user D is in Hong Kong.

  When the avatars of the user B, the user C, and the user D appear in the virtual space, the avatar 700b of the user B appears first, as shown in FIG. As shown in FIG. 9C, the avatar 700c of the user C appears, and finally the avatar 700d of the user D appears as shown in FIG. 9D. As described above, the avatar of the user near the user who first appears the avatar appears first, and the avatar of the user far away appears slowly. Thus, by changing the appearance speed of each user's avatar according to the actual distance between users, the sense of distance in the real world can be reflected in the virtual space.

  In the above description, the case where the users A, B, C, and D are located away from each other is considered. However, when the users A, B, C, and D are in the same room, for example, the users A, B, C, and D The direction of D may be reflected in the direction of the avatars 700a, 700b, 700c, and 700d in the virtual space. For example, when the user A and the user B face each other across the table, the user A avatar 700a and the user B avatar 700b are arranged at positions facing each other on the table in the virtual space.

  FIG. 10 is a diagram illustrating another example in which a user's avatar appears in the virtual space. 9A to 9D, based on the actual positional relationship between the users B, C, and D with respect to the user A who first made the avatar 700a appear in the virtual space, the avatars 700b, 700c, and 700d of the users B, C, and D are displayed. In FIG. 10, the avatars 700b, 700c, and 700d of the users B, C, and D appear on the vehicle.

  As shown in FIG. 10, the avatar 700b of the user B closest to the user A appears on the bicycle, and the avatar 700c of the user C closest to the user A appears on the car. The avatar 700d of the user D who is far away appears on the plane. In this way, the sense of distance in the real world can be reflected in the virtual space by changing the avatar's vehicle according to the distance to the user who first introduced the avatar.

  As described with reference to FIGS. 9A to 9D and FIG. 10, by changing the appearance of other users' avatars according to the distance to the user who previously made the avatar appear in the virtual space, the user can feel the real sense of distance. It can be felt in the virtual space displayed on the head mounted display 100. Note that both the avatar appearance speed and the vehicle type may be changed according to the actual distance between users.

  FIG. 11 is a diagram illustrating an example in which a user's avatar appears on the game screen. The distance from a reference point on the earth (for example, Tokyo) to the actual position of the user is reflected in the game. However, the reference point is made random or changed according to conditions so as not to be disadvantageous in the game depending on the actual position of the user.

  As shown in FIG. 11, by mapping the positions of the users A, B, C, D on the earth to the positions in the game while maintaining the mutual positional relationship as much as possible, the users A, B, C, D The avatars 700a, 700b, 700c, and 700d are displayed. Since the locations where the positions in the game can be taken are limited, the actual positional relationship between the users A, B, C, and D needs to correspond exactly to the positional relationship in the virtual space of the avatars 700a, 700b, 700c, and 700d. There is no limitation as long as the sense of distance between users in the real world reflects to some extent the avatar's positional relationship in the virtual space.

  As described above, according to the present embodiment, by incorporating a head mounted display equipped with a sensor capable of detecting the position and orientation into the cloud service, the actual positional relationship and orientation of the user wearing the head mounted display can be determined. Reflecting in the virtual space, it is possible to improve the immersive feeling and operability of the screen displayed on the head mounted display. Moreover, the hardware configuration of the client-side head mounted display can be simplified by using a cloud server.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. . Such a modification will be described.

  10 control unit, 20 input interface, 30 output interface, 32 backlight, 40 communication control unit, 42 network adapter, 44 antenna, 50 storage unit, 60 GPS unit, 62 wireless unit, 64 attitude sensor, 70 external input / output terminal interface 72 external memory, 80 clock unit, 100 head mounted display, 110 main body unit, 120 forehead contact unit, 130 side head contact unit, 200 cloud server, 300 camera, 400 network, 500 panoramic image generation device, 510 shooting Image receiving unit, 520 image data storage unit, 530 panoramic image processing unit, 540 format conversion unit, 550 panoramic image transmission unit, 600 virtual space rendering processing device, 610 User position / orientation receiving unit, 620 position relation processing unit, 630 line-of-sight direction setting unit, 640 three-dimensional rendering unit, 650 drawing data transmission unit, 660 user database, 670 virtual space information storage unit.

Claims (10)

An image generation device that provides an image to a head mounted display ,
A position information acquisition unit that acquires information on the position of the user wearing the head mounted display;
A positional relationship processing unit that determines the positional relationship of the characters of the plurality of users based on the positional relationship between the plurality of users;
A plurality of user characters arranged in a virtual space based on the determined character positional relationship, and a three-dimensional rendering unit for rendering the virtual space,
The positional relationship processing unit determines a distance between the characters of the plurality of users in the virtual space according to an actual distance between the plurality of users .
The three-dimensional rendering unit changes an aspect in which each user's character appears in a virtual space according to an actual distance between the plurality of users . An orientation information receiving unit that receives information on the orientation of the head of the user wearing the head mounted display acquired by the attitude sensor mounted on the head mounted display;
The image generation apparatus according to claim 1, wherein the positional relationship processing unit determines the orientation of each user's character in the virtual space based on the orientation of each user. The 3-dimensional rendering unit, according to claim, characterized in that changing the manner in accordance with the actual distance of each user for the user who is introduced the character in the virtual space ahead of the user character appears in the virtual space 1 Or the image generation apparatus of 2. The image according to any one of claims 1 to 3, wherein the three-dimensional rendering unit changes the order in which each user's character appears in the virtual space as an aspect in which each user's character appears in the virtual space. Generator. The image according to any one of claims 1 to 3, wherein the three-dimensional rendering unit changes a speed at which each user's character appears in the virtual space as an aspect in which each user's character appears in the virtual space. Generator. The said three-dimensional rendering part changes the kind of vehicle when each user's character appears in virtual space as a mode that each user's character appears in virtual space, The one of Claim 1 to 5 characterized by the above-mentioned. The image generating apparatus described in 1. The positional relationship processing unit determines the position of each user's character in the virtual space by mapping the actual position of each user to a position in the virtual space while maintaining the actual positional relationship of each user. the image generating apparatus according to claim 1, wherein 6. The positional relationship processing unit determines the position of each user's character in the virtual space by reflecting the distance from the actual reference position to the actual position of each user, or sets the actual reference position randomly, The image generation apparatus according to claim 7 , wherein the image generation apparatus is changed according to a predetermined condition. An image generation method for providing an image to a head mounted display ,
A position information acquisition step of acquiring information of a position of a user wearing the head mounted display;
A positional relationship processing step for determining the positional relationship of the characters of the plurality of users based on the positional relationship between the plurality of users;
A three-dimensional rendering step of arranging the plurality of user characters in a virtual space based on the determined positional relationship of the characters and rendering the virtual space;
The positional relationship processing step determines a distance between the characters of the plurality of users in the virtual space according to an actual distance between the plurality of users .
In the three-dimensional rendering step, an aspect in which each user's character appears in a virtual space is changed according to an actual distance between the plurality of users . An image generation program for providing an image to a head mounted display ,
A position information acquisition step of acquiring information of a position of a user wearing the head mounted display;
A positional relationship processing step for determining the positional relationship of the characters of the plurality of users based on the positional relationship between the plurality of users;
Arranging the plurality of user characters in the virtual space based on the determined positional relationship of the characters, and causing the computer to execute a three-dimensional rendering step of rendering the virtual space;
The positional relationship processing step determines a distance between the characters of the plurality of users in the virtual space according to an actual distance between the plurality of users.
The three-dimensional rendering step changes the manner in which each user's character appears in a virtual space in accordance with an actual distance between the plurality of users .

Images