JP7492833B2 - Program, system, and method for providing content using augmented reality technology - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act December 23, 2019, https://apps. apple. com/jp/app/id1481536391 January 17, 2020, https://play. google. com/store/apps/details? id=com.id showroom. show stage

The present invention relates to a program, a system, and a method for providing content using Augmented Reality (AR) technology.

Conventionally, various digital contents (hereinafter, sometimes referred to as AR contents) that apply augmented reality technology have been provided (see, for example, Patent Document 1 below). In such contents, an augmented reality image is generated and displayed in which an image of a virtual space is superimposed on an image (image of real space) captured by a camera of a device such as a smartphone. Also, typically, the virtual camera corresponding to the viewpoint in the virtual space is controlled so as to be linked to the real camera (so as to follow the movement of the real camera).

JP 2019-197499 A

However, in conventional AR content, there are cases where an appropriate augmented reality image is not generated simply by controlling a virtual camera to be linked to a real camera. For example, depending on the position of the virtual camera in a virtual space, objects such as characters contained in the virtual space may not be displayed appropriately.

One of the objects of the embodiments of the present invention is to support the generation and display of appropriate augmented reality images. Other objects of the embodiments of the present invention will become apparent by reference to the entire specification.

A program according to one embodiment of the present invention is a program for providing content using augmented reality technology, which causes one or more computers to execute a process of acquiring an image of real space via a real camera, a process of controlling a virtual camera included in a virtual space so as to link with the real camera, and a process of generating and displaying an augmented reality image in which the image of the virtual space via the virtual camera is superimposed on the image of the real space, and the process of controlling the virtual camera includes adjusting the relative position of the virtual camera with respect to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space.

A system according to one embodiment of the present invention is a system for providing content using augmented reality technology, comprising one or more computer processors, in which the one or more computer processors execute, in response to execution of readable instructions, a process of acquiring an image of real space via a real camera, a process of controlling a virtual camera included in a virtual space so as to be linked to the real camera, and a process of generating and displaying an augmented reality image in which the image of the virtual space via the virtual camera is superimposed on the image of the real space, and the process of controlling the virtual camera includes a process of adjusting the relative position of the virtual camera with respect to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space.

A method according to one embodiment of the present invention is executed by one or more computers and is a method for providing content using augmented reality technology, comprising the steps of acquiring an image of real space via a real camera, controlling a virtual camera included in a virtual space so as to link with the real camera, and generating and displaying an augmented reality image in which the image of the virtual space via the virtual camera is superimposed on the image of the real space, and the step of controlling the virtual camera includes adjusting the relative position of the virtual camera with respect to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space.

Various embodiments of the present invention assist in the generation and display of suitable augmented reality images.

1 is a diagram showing a schematic configuration of a network including a content providing terminal 10 according to an embodiment of the present invention. 11 is a flow diagram illustrating a process executed by the content providing terminal 10 when playing back a live space video. FIG. 1 is a diagram for explaining how a virtual space (objects) is arranged. FIG. 1 is a diagram for explaining a virtual space. 11 is a flow diagram illustrating a process executed by the content providing terminal 10 during playback of a live space video. FIG. 2 is a diagram illustrating the bounding boxes of each mesh that makes up a character. 11A and 11B are diagrams for explaining control of the virtual camera when the virtual camera is located in a low-angle area. 13 is a diagram illustrating an augmented reality image 50 in a state in which the virtual camera VP is located at a position higher than the waist height h of the character CH. A figure showing an example of an augmented reality image 50 in a state in which the virtual camera VP has moved upward from the reference point RP corresponding to the position of the camera 131 and is at a position higher than the waist height h of the character CH. 13A and 13B are diagrams for explaining control of the virtual camera when the virtual camera is located in a sinking region and not immediately after the virtual camera has moved upward in response to entering a low-angle region. A diagram illustrating an augmented reality image 50 in a state where the virtual camera VP is positioned outside the bounding box BB. A diagram illustrating an augmented reality image 50 in a state in which the virtual camera VP has moved in the opposite direction of the line of sight from a reference point RP corresponding to the position of the camera 131 and is located outside the bounding box BB. 13A and 13B are diagrams illustrating control of the virtual camera when the virtual camera is located in a sinking region and immediately after the virtual camera has moved upward in response to entering a low-angle region.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same reference numerals may be used to denote the same or similar components.

Figure 1 is a schematic diagram showing the configuration of a network including a content providing terminal 10 according to one embodiment of the present invention. As shown in the figure, the terminal 10 is communicatively connected to a content management server 30 via a communication network 20 such as the Internet. The content providing terminal 10 has a function for providing AR content using augmented reality (AR) technology, and is an example of a device that implements all or part of the system of the present invention.

The content providing terminal 10 is configured as a general computer, and as shown in FIG. 1, is equipped with a computer processor 11 including a CPU and a GPU, a main memory 12, an input/output I/F 13, a communication I/F 14, a storage (memory device) 15, and various sensors 16, and each of these components is electrically connected via a bus or the like (not shown).

The computer processor 11 loads various programs stored in the storage 15 or the like into the main memory 12 and executes various instructions contained in the programs. The main memory 12 is composed of, for example, a DRAM or the like.

The input/output I/F 13 includes, for example, an information input device such as a touch panel or a controller, an audio input device such as a microphone, and an image input device such as a camera 131. The input/output I/F 13 also includes an image output device (display device) such as a display, and an audio output device such as a speaker.

The communication I/F 14 is implemented as hardware such as a network adapter, various communication software, or a combination of these, and is configured to enable wired or wireless communication.

The storage 15 is configured, for example, by a magnetic disk or a flash memory. The storage 15 stores various programs including an operating system, various data, and the like. The programs stored in the storage 15 can be downloaded and installed from an application market, for example. In this embodiment, the programs stored in the storage 15 include a content providing program 40 according to one embodiment of the present invention. The content providing program 40 is a program for causing the content providing terminal 10 to function as a device for providing AR content using augmented reality technology.

The various sensors 16 include, for example, motion sensors composed of an acceleration sensor, a gyro sensor (angular velocity sensor), a geomagnetic sensor, etc. The content providing terminal 10 can determine various movements of the terminal 10 itself based on information detected by these motion sensors.

The content management server 30 has a configuration as a general computer. The content management server 30 stores and manages various information related to the AR content provided by the content providing terminal 10. For example, the content management server 30 manages information related to the content for each user, and various data for providing the content (e.g., the content itself and data on the virtual space, etc.). The content management server 30 can be configured using multiple server devices, each of which has a configuration as a general computer.

For example, in response to starting the content providing program 40, the content providing terminal 10 acquires information and various data related to the corresponding user's content from the content management server 30. Also, for example, while the content providing program 40 is running, the content providing terminal 10 acquires various data from the content management server 30 and transmits the status of content provision to the user (e.g., the progress of a game) to the content management server 30. Note that in this embodiment, the content providing terminal 10 can be configured to provide AR content without communicating with the content management server 30, in which case the content management server 30 may not be necessary.

Next, the functions of the content providing terminal 10 of this embodiment will be described. As shown in FIG. 1, the computer processor 11 of the content providing terminal 10 is configured to function as an image generating and displaying unit 111 and a content control unit 112 by executing instructions contained in a program (e.g., a content providing program 40) loaded into the main memory 12.

The image generation and display unit 111 is configured to execute various processes related to the generation and display of images related to the content. For example, the image generation and display unit 111 is configured to acquire an image of real space via the camera 131, and generate and display an augmented reality image in which an image of the virtual space via a virtual camera included in the virtual space is superimposed on the acquired image of real space. The virtual space is configured as, for example, a three-dimensional virtual space.

In this embodiment, the virtual camera included in the virtual space is controlled to be linked to the real camera 131. For example, the image generation and display unit 111 controls the virtual camera to move in the virtual space according to the movement (changes in position and tilt (attitude), etc.) of the camera 131 (terminal 10). Such control of the virtual camera can also be said to be control for establishing a correspondence between the real space and the virtual space, and is sometimes called tracking. The virtual camera can also be said to be a viewpoint in the virtual space.

In addition, the image generation and display unit 111 is configured to adjust the position of the virtual camera relative to the camera 131 so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space. For example, the image generation and display unit 111 is configured to adjust the position of the virtual camera relative to the camera 131 by setting an offset for the virtual camera linked to the camera 131 (setting the position of the virtual camera to a position a specific distance away in a specific direction from a reference point corresponding to the position of the camera 131).

The content control unit 112 executes various processes related to the control of the content. For example, if the content is a video using augmented reality technology (e.g., a video in which a virtual object such as a character appears in real space), the content control unit 112 is configured to request a video specified by the user from among multiple videos to the content management server 30, and to receive and play the video from the server 30. Also, for example, if the content is a game using augmented reality technology, the content control unit 112 is configured to control the progress of the game in response to operational inputs by the user (game player).

In this way, the content providing terminal 10 in this embodiment generates and displays an augmented reality image in which an image of a virtual space via a virtual camera linked to the camera 131 is superimposed on an image of a real space via the camera 131, and adjusts the position of the virtual camera relative to the camera 131 so that the virtual camera is positioned outside a specified area based on a specified object included in the virtual space, thereby preventing the generation and display of an inappropriate image due to the virtual camera being positioned within such an area. In other words, the content providing terminal 10 supports the generation and display of an appropriate augmented reality image.

In this embodiment, the adjustment of the position of the virtual camera relative to the camera 131 can be realized in various ways. For example, the image generation and display unit 111 can be configured to adjust the position of the virtual camera relative to the camera 131 (e.g., move the virtual camera outside the specified area) in response to the virtual camera linked to the camera 131 entering a specified area so that the virtual camera is located outside the specified area. Such a configuration makes it possible to adjust the position of the virtual camera in response to the virtual camera entering the specified area.

Furthermore, the image generation and display unit 111 may be configured to adjust the position of the virtual camera relative to the camera 131 so that the virtual camera is located outside a predetermined area, and then return the position of the virtual camera relative to the camera 131 to the position before the adjustment, as long as the virtual camera is located outside the predetermined area. For example, the image generation and display unit 111 is configured to set an offset for the virtual camera linked to the camera 131, and then gradually/temporarily eliminate the offset (moving the virtual camera gradually/temporarily toward a reference point corresponding to the position of the camera 131). This configuration makes it possible to return the virtual camera to its normal position after adjusting its position relative to the camera 131.

In this embodiment, the specified object includes various types of objects contained in the virtual space, including, but not limited to, characters, man-made objects (buildings, etc.), and natural objects (trees, etc.). For example, if the specified object is configured as an object such as a character that moves within the virtual space, the specified area based on the specified object may also move within the virtual space.

In this embodiment, the specified region may include various regions based on a specified object. For example, the specified region includes a region surrounding the specified object. The region is configured, for example, as a bounding box that defines the boundary of the specified object (e.g., an axis-aligned bounding box (AABB) or an oriented bounding box (OBB)), or a region based on the bounding box (a region that is slightly larger/smaller than the bounding box). This configuration prevents the virtual camera from being positioned within the region surrounding the specified object (e.g., the virtual camera becoming embedded in the specified object), resulting in an inappropriate augmented reality image (e.g., the specified object not being displayed properly, etc.).

The image generation and display unit 111 may be configured to move the position of the virtual camera relative to the camera 131 in the direction opposite to the line of sight of the virtual camera in response to the virtual camera entering an area surrounding a specific object. This configuration moves the virtual camera along a straight line corresponding to the line of sight of the virtual camera, thereby reducing changes in the image via the virtual camera, and as a result, reducing the discomfort that may be caused to the user by the movement of the virtual camera.

Also, for example, the specified area includes an area below a specified position of a specified object. For example, the specified position includes the waist position of a specified object configured as a character. For example, the image generation and display unit 111 can be configured to move the position of the virtual camera relative to the camera 131 in an upward direction (e.g., vertically upward or diagonally upward, etc.) in response to the virtual camera entering an area below the specified position of the specified object. This configuration prevents an inappropriate augmented reality image from being generated due to the virtual camera being positioned in an area below the specified position of the specified object (for example, a low-angle image from a viewpoint lower than the waist position of the character, etc.).

The image generation and display unit 111 may be configured to move the position of the virtual camera relative to the camera 131 in the horizontal direction when the virtual camera enters an area surrounding a specific object as the virtual camera's position relative to the camera 131 moves upward in response to the virtual camera's entry into an area below the specific object's specific position. This configuration prevents the virtual camera, which moves out of the area surrounding the specific object, from re-entering an area below the specific object's specific position.

In this embodiment, the control of the virtual camera may be changed depending on the type of the specified object. For example, when the specified object is a first type of object (e.g., a character for which a low angle is not desirable), the image generation and display unit 111 may be configured to adjust the relative position of the virtual camera so that the virtual camera is positioned outside the area below the specified position, but when the specified object is a second type of object (e.g., a character for which a low angle is not a problem), the image generation and display unit 111 may be configured not to adjust the relative position of the virtual camera. This configuration enables the virtual camera to be controlled depending on the type of the specified object, and as a result, the load on the terminal 10 may be reduced.

Next, a specific example of one aspect of the content providing terminal 10 of this embodiment having such functions will be described. In this example, the content providing terminal 10 provides the user with a live space video corresponding to a live performance (sometimes called a concert, gig, performance, show, etc.) held in a virtual space. The video is configured to include an augmented reality image in which an image of the virtual space is overlaid on an image of the real space.

In this example, the live space video is live-streamed from the content management server 30 at a specified date and time (date and time of the live performance). For example, a user can watch a live space video for which they have previously purchased a ticket at the date and time of the performance via the content providing terminal 10. Note that instead of or in addition to live streaming of such videos, a pre-stored video may be streamed in response to a user's request.

Figure 2 is a flow diagram illustrating the processing executed by the content providing terminal 10 when playing a live space video. These processing are executed in response to a user request to play a specific video (enter a virtual live space (live venue)). As shown in the figure, the terminal 10 first acquires object information in the corresponding live space video (step S100). The object information is information about objects included in the virtual space in the live space video, and is acquired from the content management server 30. The objects included in the virtual space include characters (predetermined objects) as performers, and the object information includes a 3D model of the character. The objects may include various types of objects other than characters (for example, objects corresponding to equipment such as a stage or screen).

Next, the terminal 10 recognizes a plane in the image (captured image) of real space input via the camera 131 (step S110), and places the virtual space (object) in the real space based on the recognized plane (step S120).

Figure 3 is a diagram for explaining how a virtual space (object) is arranged. Image 45 at the top of Figure 3 is a photographed image of real space input via camera 131, and in this example includes the top surface of table TB. Image 50 at the bottom of Figure 3 is an augmented reality image in which a virtual space including character CH as an object is arranged on photographed image 45. Augmented reality image 50 can also be said to be an image of virtual space superimposed on an image of real space. In this example, the top surface of table TB in photographed image 45 is recognized as a flat surface, and in augmented reality image 50, character CH is standing on the top surface of table TB.

FIG. 4 is a diagram for explaining the virtual space. As shown in the figure, in this example, a three-dimensional coordinate system with the right direction as the X axis, the upward direction as the Y axis, and the forward direction as the Z axis is defined as the world coordinate system in the virtual space. The XZ plane of the world coordinate system is then associated with a plane recognized in the captured image of the real space, and the character CH is placed on the XY plane. In addition, a virtual camera VP, which is the viewpoint in the virtual space, is placed at a predetermined position, and the position of the virtual camera VP and the direction of the line of sight VL are controlled so as to be linked to the movement of the camera 131 (terminal 10). Such control of the virtual camera VP is performed based on the captured image via the camera 131 and information detected by a motion sensor. In addition, a local coordinate system based on the positions of the character CH and the virtual camera VP can also be managed in the virtual space.

Returning to the flow diagram of FIG. 2, once the virtual space (objects) have been placed in the real space in this manner, the terminal 10 then plays the live space video (step S130). Playback of the video continues until a predetermined end condition is met (NO in step S140). During video playback, an augmented reality image is repeatedly generated and displayed in which an image of the virtual space taken by a virtual camera linked to the camera 131 is overlaid on an image of the real space taken by the camera 131. In the image of the virtual space, the character is drawn based on its 3D model, and moves based on the movement data (such as animation information of the 3D model) received from the content management server 30. During video playback, sound received from the content management server 30 is also output.

In this example, while the video is being played, the virtual camera linked to camera 131 is controlled so as to be outside a predetermined area based on the character. FIG. 5 is a flow diagram illustrating the processes executed by the content providing terminal 10 while the video is being played. These processes are executed repeatedly at a predetermined timing, for example, on a frame-by-frame basis corresponding to the generated and displayed augmented reality image. As shown in the figure, the terminal 10 first obtains the waist height of the character and the position of the bounding box of each mesh that constitutes the character (step S200).

In this example, the height of the character's waist is determined based on the bone information included in the character information; specifically, the height of the bottom end of the bone corresponding to the spine among the multiple bones included in the character is determined as the height of the character's waist (Y coordinate in the world coordinate system). Note that in other examples of this embodiment, instead, the position of the character's center of gravity is used as the position of the waist, or the waist height is preset as a fixed value for each character.

In this example, the bounding boxes of each mesh that makes up the character are defined as axis-aligned bounding boxes (AABB) and are identified based on the mesh information of the character. For example, if a character is composed of two meshes, a first mesh corresponding to the head and a second mesh corresponding to the torso, two bounding boxes are identified, a first bounding box BB1 that surrounds the head and a second bounding box BB2 that surrounds the torso, as shown in FIG. 6, and the positions (coordinates in the world coordinate system) of these bounding boxes BB are obtained.

Returning to the flow diagram of FIG. 5, the terminal 10 then acquires the position and line of sight direction of the virtual camera (step S210). As described above, the virtual camera is controlled to be linked to the camera 131, and in step S210, the position and line of sight direction of the virtual camera at that time (coordinates and direction vector in the world coordinate system) are acquired.

Then, the terminal 10 determines whether or not the virtual camera is located within a position adjustment required area (predetermined area) where its position needs to be adjusted (step S220). In this example, the position adjustment required areas are a "low-angle area" that is lower than the height of the character's waist, and a "sunken area" that corresponds to the inside of the bounding box of each mesh that constitutes the character. If the virtual camera is located within the position adjustment required area (YES in step S220), the position of the virtual camera is adjusted as described below.

First, if the virtual camera is located within the low-angle area (YES in step S230), the terminal 10 moves the virtual camera upward (step S240). In this example, specifically, based on the difference between the height of the character's waist and the height of the virtual camera, the virtual camera moves upward (vertically upward) so that the height is slightly higher than the height of the character's waist. Note that in other examples of this embodiment, the amount of movement of the virtual camera in this case may be a fixed value.

Figure 7 is a diagram for explaining the control of the virtual camera when it is located in a low-angle area. As shown, when the virtual camera VP (Figure 7(A)), which is located at a position higher than the waist height h of the character CH, moves in conjunction with the camera 131 and enters a position lower than the waist height h of the character CH (Figure 7(B)), the virtual camera VP moves upward from the reference point RP corresponding to the position of the camera 131 and becomes higher than the waist height h of the character CH (Figure 7(C)). Note that the direction and distance from the reference point RP corresponding to the position of the camera 131 to the virtual camera VP are managed as an offset set for the virtual camera VP.

8 illustrates an example of an augmented reality image 50 in a state where the virtual camera VP is at a position higher than the waist height h of the character CH (FIG. 7(A)), and FIG. 9 illustrates an example of an augmented reality image 50 in a state where the virtual camera VP, which has entered a position lower than the waist height h of the character CH, has moved upward from the reference point RP corresponding to the position of the camera 131 to a position higher than the waist height h of the character CH (FIG. 7(C)). When the virtual camera VP moves upward from the reference point RP corresponding to the position of the camera 131, the positional relationship between the image in the real space and the image in the virtual space is shifted in the vertical direction. As a result, in the augmented reality images 50 in FIG. 8 and FIG. 9, the display of the character CH in the virtual space is the same, while the display of the table TB in the augmented reality image 50 in FIG. 9 is located higher than the table TB in the augmented reality image 50 in FIG. 8. Thus, in this example, although a vertical shift occurs in the positional relationship between the image in the real space and the image in the virtual space, the generation and display of inappropriate images including low-angle characters is suppressed.

Returning to the flow diagram of FIG. 5, if the virtual camera is located within the embedded region (YES in step S250), the terminal 10 further determines whether or not it is immediately after the virtual camera has moved upward in response to entering the low-angle region (step S240) (step S260). If it is not immediately after such upward movement of the virtual camera (NO in step S260), the terminal 10 moves the virtual camera in the opposite direction to the current line of sight (step S270). In this example, the virtual camera is moved in the opposite direction to the line of sight so as to be slightly outside the embedded region (boundary box) based on the distance from the current position of the virtual camera to the intersection between the straight line in the opposite direction to the virtual camera's line of sight and the embedded region (boundary box). Note that in other examples of this embodiment, the amount of movement of the virtual camera in this case may be a fixed value.

Figure 10 is a diagram for explaining the control of the virtual camera when the virtual camera is located within the embedded area (boundary box) and has not just moved upward in response to entering the low-angle area. As shown in the figure, when the virtual camera VP (Figure 10 (A)) located outside the bounding box BB moves in conjunction with the camera 131 and enters the inside of the bounding box BB (Figure 10 (B)), the virtual camera VP moves from the reference point RP corresponding to the position of the camera 131 in the opposite direction to the line of sight VL, and becomes a position outside the bounding box BB (Figure 10 (C)). As described above, the direction and distance from the reference point RP corresponding to the position of the camera 131 to the virtual camera VP are managed as an offset set for the virtual camera VP.

11 illustrates an augmented reality image 50 in a state where the virtual camera VP is located outside the bounding box BB (FIG. 10(A)), and FIG. 12 illustrates an augmented reality image 50 in a state where the virtual camera VP, which has entered inside the bounding box BB, has moved in the opposite direction of the line of sight from the reference point RP corresponding to the position of the camera 131 to a position outside the bounding box BB (FIG. 10(C)). When the virtual camera VP moves in the opposite direction of the line of sight from the reference point RP corresponding to the position of the camera 131, the positional relationship between the image in real space and the image in virtual space shifts in the line of sight. As a result, while the display of the character CH in the virtual space is the same in the augmented reality images 50 of FIGS. 11 and 12, the display of the table TB in the real space is such that the table TB in the augmented reality image 50 of FIG. 12 is positioned closer to the viewer (enlarged) than the table TB in the augmented reality image 50 of FIG. 11. Thus, in this example, although there is a deviation in the line of sight between the positional relationship between the image in real space and the image in virtual space, the generation and display of inappropriate images from a virtual camera embedded inside the character is suppressed.

Returning to the flow diagram of FIG. 5, on the other hand, if the virtual camera is located in the embedded area and immediately after the virtual camera has moved upward in response to entering the low-angle area (step S240) (YES in step S260), the terminal 10 moves the virtual camera horizontally (step S280). In this example, the virtual camera is moved horizontally to a position slightly outside the embedded area (bounding box) based on the distance from the current position of the virtual camera to the intersection between the embedded area (bounding box) and a line corresponding to the horizontal component of a line pointing in the opposite direction to the line of sight of the virtual camera. Note that in other examples of this embodiment, the amount of movement of the virtual camera in this case may be a fixed value.

13 is a diagram for explaining the control of the virtual camera when the virtual camera is located within the embedded area (boundary box) and immediately after the virtual camera moves upward in response to entering the low-angle area. As shown in the figure, when the virtual camera VP is at a position lower than the height h of the character's waist (FIG. 13(A)), it moves upward from the reference point RP corresponding to the position of the camera 131 to a position higher than the height h of the character's waist. Then, when the position of the virtual camera VP that has moved upward is inside the bounding box BB (FIG. 13(B)), the virtual camera VP moves from its position at that time (the position immediately after the upward movement) in a direction corresponding to the horizontal component in the opposite direction to the line of sight VL, and becomes a position outside the bounding box BB (FIG. 13(C)). As described above, the direction and distance from the reference point RP corresponding to the position of the camera 131 to the virtual camera VP are managed as an offset set for the virtual camera VP.

For example, if the line of sight of a virtual camera positioned in a low-angle area is close to upward, moving the virtual camera in the opposite direction of the line of sight (close to downward) may result in the virtual camera re-entering the low-angle area. However, in this example, the virtual camera moves horizontally in such a case, and so re-entering the low-angle area can be avoided.

In this way, when the virtual camera is located within an area requiring position adjustment, the position of the virtual camera is adjusted and an offset is set for the virtual camera, and the virtual camera is controlled to move in conjunction with camera 131 while maintaining the offset (while maintaining the positional relationship with the reference point corresponding to the position of camera 131).

Returning to the flow diagram of FIG. 5, on the other hand, if the virtual camera is not located within the area requiring position adjustment (NO in step S220), the terminal 10 determines whether an offset has been set for the virtual camera (step S290).

If an offset has been set for the virtual camera (YES in step S290), the terminal 10 moves the virtual camera in the direction of the reference point corresponding to the position of the camera 131 (step S300). In this example, the virtual camera moves a predetermined distance from its current position in the direction of the reference point. Thereafter, as long as the virtual camera is located outside the area requiring position adjustment, the virtual camera will repeatedly move the predetermined distance toward the reference point corresponding to the camera 131, so that the position of the virtual camera will eventually become the position of the reference point, and the offset will be eliminated.

In the above example, the need for upward movement control of the virtual camera in response to entry into the low-angle area may be determined depending on the type of character (e.g., the type of clothing worn, etc.). In this case, information regarding the type of character (or the need for such movement control) may be included in the object information obtained from the content management server 30.

In the above example, the low-angle area and the embedded area are set as areas requiring position adjustment where the position of the virtual camera needs to be adjusted, but only one of these areas may be set as an area requiring position adjustment. In other examples of this embodiment, various areas other than the low-angle area and the embedded area may be applied as areas requiring position adjustment.

In the above example, when the virtual camera is located in a low-angle area, the virtual camera is moved vertically upwards, but it may also be moved diagonally upwards. Also, the low-angle area is an area lower than the height of the character's waist, but it may also be an area having a cone/pyramid shape with the apex at the position of the character's waist.

In the above example, when the virtual camera is located in the embedded area, the virtual camera is moved in the direction opposite to the line of sight, but it may be moved in another direction.

In the above example, a live space video was given as an example of AR content, but the AR content in this embodiment is not limited to this and can be configured as various content such as other videos or games.

The content providing terminal 10 of this embodiment described above generates and displays an augmented reality image in which an image of a virtual space via a virtual camera linked to the camera 131 is superimposed on an image of a real space via the camera 131, and adjusts the position of the virtual camera relative to the camera 131 so that the virtual camera is positioned outside of a predetermined area (e.g., a low-angle area and a recessed area) based on a predetermined object (e.g., a character) included in the virtual space, thereby preventing the generation and display of an inappropriate image due to the virtual camera being positioned within such an area. In other words, the content providing terminal 10 supports the generation and display of an appropriate augmented reality image.

In another embodiment of the present invention, at least some of the functions of the content providing terminal 10 may be realized by cooperation between the terminal 10 and the content management server 30, or may be realized by the content management server 30.

The processes and procedures described in this specification may be realized by software, hardware, or any combination thereof, other than those explicitly described. For example, the processes and procedures described in this specification may be realized by implementing logic equivalent to the processes and procedures in a medium such as an integrated circuit, volatile memory, non-volatile memory, or magnetic disk. In addition, the processes and procedures described in this specification may be implemented as a computer program equivalent to the processes and procedures, and executed by various computers.

Although the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple components, and/or multiple modules. Also, the software and hardware elements described herein may be realized by integrating them into fewer components or breaking them down into more components.

In this specification, even if a component of the invention is described as either singular or plural, or described without being limited to either singular or plural, the component may be either singular or plural unless the context requires otherwise.

REFERENCE SIGNS LIST 10 Content providing terminal 20 Communication network 30 Content management server 11 Computer processor 111 Image generating and displaying unit 112 Content control unit 13 Input/output I/F
131 Camera 15 Storage 40 Content providing program 45 Photographed image 50 Augmented reality image BB Bounding box RP Reference point VL Line of sight VP Virtual camera

Claims (6)

A program for providing content using augmented reality technology, comprising:
A process of acquiring an image of a real space via a real camera;
A process of controlling a virtual camera included in a virtual space so as to be linked to the real camera;
and generating and displaying an augmented reality image by superimposing an image of the virtual space via the virtual camera on an image of the real space;
the process of controlling the virtual camera includes adjusting a position of the virtual camera relative to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space;
the predetermined area includes a first area surrounding the predetermined object and a second area below a predetermined position of the predetermined object,
the predetermined position is a waist position of the predetermined object configured as a character,
The process of controlling the virtual camera includes:
moving a position of the virtual camera relative to the real camera in an upward direction in response to entry of the virtual camera linked to the real camera into the second area; and
moving a position of the virtual camera relative to the real camera in a horizontal direction when the virtual camera enters the first area with an upward movement of the position of the virtual camera relative to the real camera in response to the virtual camera entering the second area,
program. the process of controlling the virtual camera includes adjusting a position of the virtual camera relative to the real camera in response to entry of the virtual camera linked to the real camera into the predetermined area so that the virtual camera is located outside the predetermined area.
The program of claim 1. the process of controlling the virtual camera includes adjusting a relative position of the virtual camera with respect to the real camera so that the virtual camera is located outside the predetermined area, and then returning the relative position of the virtual camera with respect to the real camera to a position before the adjustment as long as the virtual camera is located outside the predetermined area.
The program of claim 1 or 2. the process of controlling the virtual camera includes, when the predetermined object is a first type object, adjusting a relative position of the virtual camera so that the virtual camera is located outside the second area, and, when the predetermined object is a second type object, not adjusting the relative position of the virtual camera so that the virtual camera is located outside the second area.
The program of any one of claims 1 to 3 . 1. A system for providing content using augmented reality technology, comprising one or more computer processors, the system comprising:
The one or more computer processors, in response to execution of the readable instructions,
A process of acquiring an image of a real space via a real camera;
A process of controlling a virtual camera included in a virtual space so as to be linked to the real camera;
and generating and displaying an augmented reality image by superimposing an image of the virtual space via the virtual camera on the image of the real space;
the process of controlling the virtual camera includes adjusting a position of the virtual camera relative to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space;
the predetermined area includes a first area surrounding the predetermined object and a second area below a predetermined position of the predetermined object,
the predetermined position is a waist position of the predetermined object configured as a character,
The process of controlling the virtual camera includes:
moving a position of the virtual camera relative to the real camera in an upward direction in response to entry of the virtual camera linked to the real camera into the second area; and
moving a position of the virtual camera relative to the real camera in a horizontal direction when the virtual camera enters the first area with an upward movement of the position of the virtual camera relative to the real camera in response to the virtual camera entering the second area,
system. 1. A method, executed by one or more computers, for providing content using augmented reality technology, comprising:
acquiring an image of a real space via a real camera;
controlling a virtual camera included in a virtual space so as to be linked to the real camera;
and generating and displaying an augmented reality image by superimposing an image of the virtual space via the virtual camera on the image of the real space,
the step of controlling the virtual camera includes adjusting a position of the virtual camera relative to the real camera so that the virtual camera is positioned outside a predetermined area based on a predetermined object included in the virtual space;
the predetermined area includes a first area surrounding the predetermined object and a second area below a predetermined position of the predetermined object,
the predetermined position is a waist position of the predetermined object configured as a character,
The step of controlling the virtual camera includes:
moving a position of the virtual camera relative to the real camera in an upward direction in response to entry of the virtual camera linked to the real camera into the second area; and
moving a position of the virtual camera relative to the real camera in a horizontal direction when the virtual camera enters the first area with an upward movement of the position of the virtual camera relative to the real camera in response to the virtual camera entering the second area,
Method.

Images