JP7521072B1 - VIRTUAL SPACE CONTENT DELIVERY SYSTEM, VIRTUAL SPACE CONTENT DELIVERY PROGRAM, AND VIRTUAL SPACE CONTENT DELIVERY METHOD - Google Patents

Abstract

[Problem] To enable more viewer users to participate. [Solution] A virtual space content distribution system including a viewer user terminal capable of displaying images of virtual space content and assigned to each viewer user, and a server computer connected to the viewer user terminal, wherein when the viewer user terminal displays a viewer avatar viewpoint image, which is a viewpoint image of the virtual space corresponding to the viewer avatar, the viewer user terminal generates the viewer avatar viewpoint image based on virtual space data of the virtual space transmitted from the server computer. [Selected Figure] Figure 10

Description

The present invention relates to a virtual space content distribution system capable of distributing virtual space content in a virtual space, a virtual space content distribution program, and a virtual space content distribution method.

Conventionally, a distribution system is known in which animation of a character object (avatar) of a broadcast user (performer) that moves in a virtual space in conjunction with the movements of the broadcast user (performer) is generated in a server computer connected to a viewer user's terminal via a computer network, video content including such animation is live-distributed, and the viewer user can also participate as an audience member in the virtual space in which the avatar of the broadcast user (performer) is participating, upon request from the viewer user (see, for example, Patent Document 1).

JP 2012-120098 A

However, in Patent Document 1, video content including animation is generated on a server computer, so when a large number of viewer users participate, the processing capacity of the server computer is exceeded, causing problems in the distribution of the video content, and it is therefore not possible to allow many viewer users to participate.

The present invention has been made in response to these problems, and aims to provide a virtual space content distribution system, a virtual space content distribution program, and a virtual space content distribution method that allow more viewer users to participate.

The virtual space content delivery system according to claim 1 comprises:
A virtual space content distribution system for distributing virtual space content (e.g., a virtual live performance) in a virtual space (e.g., a virtual live performance venue),
a viewer user terminal (e.g., viewer terminal 300) that is a terminal (e.g., a smartphone P, a computer PC) capable of outputting an image of the virtual space content and that can be used by a viewer user who performs an operation to operate a viewer avatar that can participate in the virtual space that is the target of the virtual space content in the virtual space ;
A server computer (e.g., a distribution server computer 100) that is connected to the viewer user terminal via a communication network (e.g., the Internet) and is capable of executing at least processes related to the virtual space (e.g., a virtual space update process, a virtual space update data generation process, etc.) and processes related to the distribution of the virtual space content (e.g., a virtual space update data distribution process, a generated audio distribution process, etc.);
Including,
There are a plurality of viewer users, and each viewer user is assigned a viewer user terminal (for example, as shown in FIG. 1, viewer users A to C are assigned viewer terminals 300a to 300c),
each of the viewer user terminals, when displaying a viewer avatar viewpoint image which is a viewpoint image of the virtual space corresponding to the viewer avatar, generates the viewer avatar viewpoint image based on virtual space data (e.g., virtual space update data) of the virtual space transmitted from the server computer ;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
the virtual space data includes performer avatar motion data for generating motions of the performer avatar;
The viewer user terminal is capable of generating the movement of the performer avatar in a plurality of generation modes having different processing loads when generating the movement of the performer avatar based on the performer avatar movement data.
It is characterized by the following.
According to this feature, the generation of viewer avatar viewpoint images, which requires a large processing load, is performed on the viewer user terminal side, so that more viewer users can participate. Also, performer users can participate as performer avatars, and the movements of the performer avatars can be reflected in the viewer avatar viewpoint images, so that the interest of the virtual space content can be further increased. Furthermore, by changing the generation mode, viewer avatar viewpoint images corresponding to the movement status of the performance avatar can be generated, and differences in the processing capabilities of viewer user terminals can also be accommodated.

The virtual space content delivery system of claim 2 is the virtual space content delivery system according to claim 1 ,
The virtual space content is characterized in that the performer avatar performs a performance (e.g., singing or dancing) in the virtual space.
According to this feature, since a performer avatar performs, the interest of the virtual space content can be further increased.

The virtual space content delivery system of claim 3 is the virtual space content delivery system according to claim 1 or 2,
Regarding a common viewpoint image (e.g., a virtual camera viewpoint image) that is commonly displayed on each of the viewer user terminals, rather than a viewpoint image corresponding to each of the viewer avatars, the server computer can generate it and distribute it to the viewer user terminal (for example, as shown in FIG. 25, during a period in which only the virtual camera viewpoint image is viewable, such as during the pre-show, opening performance, and flying performance periods in Example 2, the distribution server computer 100 generates the virtual camera viewpoint image and distributes it to the viewer terminal 300).
It is characterized by the following.
According to this feature, it is possible to prevent unnecessary increase in the processing load of the entire system due to duplicated processing being executed at the viewer user terminal.

The virtual space content delivery system of claim 4 is the virtual space content delivery system according to claim 3 ,
During a specific period in which either the viewer avatar viewpoint video or the common viewpoint video can be selected and displayed on the viewer user terminal, the viewer user terminal generates and displays the common viewpoint video (for example, as shown in FIG. 26, during a program period excluding the MC period and the flying performance period in Example 2, during a period in which both the viewer user viewpoint video and the virtual camera viewpoint video can be viewed, the virtual camera viewpoint video is generated on the viewer terminal 300 side).
It is characterized by the following.
According to this feature, it is possible to prevent problems such as interruption of the video when switching the viewpoint video.

The virtual space content delivery system of claim 5 is the virtual space content delivery system according to claim 3 ,
In a special period in which the processing load on the server computer is large, the viewer user terminal also generates and displays the common viewpoint video (for example, in a costume change performance period or a double-ego performance period in the second embodiment in which the processing load on the distribution server computer 100 is large, the virtual camera viewpoint video is generated on the viewer terminal 300 side).
It is characterized by the following.
This feature makes it possible to prevent the processing load on the server computer from temporarily becoming extremely large.

A virtual space content delivery system according to claim 6 is the virtual space content delivery system according to claim 1 or 2,
The server computer transmits advance data including basic data of the virtual space and basic data of avatars participating in the virtual space (e.g., advance data including virtual space data, performer avatar data, viewer avatar data, etc.) to each of the viewer user terminals before starting the virtual space content (e.g., a portion in which the delivery server computer 100 delivers advance data during a advance data DL period).
It is characterized by the following.
This feature makes it possible to prevent the occurrence of a problem in which the viewer avatar viewpoint video is not displayed properly at the start of virtual space content.

The virtual space content delivery system of claim 7 is the virtual space content delivery system according to claim 6 ,
The server computer does not transmit the virtual space data to the viewer user terminals for which it cannot confirm that the advance data has been received, but transmits a common viewpoint video to be displayed in common on each of the viewer user terminals (for example, a virtual camera viewpoint video generated by the distribution server computer 100 is distributed to a viewer terminal 300 for which the advance data download completion check timing has not yet been completed, so that only the virtual camera viewpoint video can be displayed on the viewer terminal 300 during the entire period of the virtual live).
It is characterized by the following.
This feature prevents the processing load on the server computer from increasing excessively due to the transmission of basic data, etc. after the start of the virtual space content, while also preventing the viewer user from being unable to view the virtual space content.

The virtual space content delivery system of claim 8 is the virtual space content delivery system according to claim 1 or 2,
The viewer user terminals include a first viewer user terminal (e.g., viewer terminal 300a of viewer user A in modification 2-2) and a second viewer user terminal (e.g., viewer terminal 300b of viewer user B in modification 2-2),
The server computer can transmit the virtual space data to the second viewer user terminal that is different from that transmitted to the first viewer user terminal (for example, the difference between virtual space update data A including information on special action A transmitted to viewer terminal 300a and virtual space update data B not including information on special action A transmitted to viewer terminal 300b).
It is characterized by the following.
According to this feature, it is possible to increase the degree of freedom of the virtual space data, and to generate different viewer avatar viewpoint images for the first viewer user terminal and the second viewer user terminal.

The virtual space content delivery system of claim 9 is the virtual space content delivery system according to claim 8 ,
The second viewer user terminal is characterized in that it is capable of generating an image of a second viewer avatar performing an action (e.g., a special action in variant example 2-2) different from the action of the first viewer avatar of the first viewer user using the first viewer user terminal based on virtual space data different from the virtual space data transmitted to the first viewer user terminal.
According to this feature, an image of a second viewer avatar performing actions different from those of the first viewer avatar can be displayed on the second viewer user terminal, thereby increasing the second viewer user's interest in the virtual space content.

The virtual space content delivery program according to claim 10 ,
A virtual space content distribution program (e.g., a program that operates in the distribution server computer 100 to transmit virtual space update data, and a program that operates in the viewer terminal 300 to generate a viewer avatar viewpoint video) for distributing virtual space content (e.g., a virtual live performance) in a virtual space (e.g., a virtual live performance venue) over a network (e.g., the Internet) using at least a server computer (e.g., a distribution server computer 100),
The avatars (e.g., viewer avatars) that can participate in the virtual space that is the subject of the virtual space content include viewer avatars that operate in the virtual space by operating a viewer user terminal (e.g., viewer terminal 300) that is a terminal available to a viewer user and is connected to the server computer via the network,
There are a plurality of viewer users, and each viewer user is assigned a viewer user terminal (for example, as shown in FIG. 1, viewer users A to C are assigned viewer terminals 300a to 300c),
a step of generating, in each of the viewer user terminals, a viewer avatar viewpoint image, which is a viewpoint image of the virtual space corresponding to the viewer avatar, based on virtual space data of the virtual space (e.g., virtual space update data) transmitted from the server computer (e.g., a viewer viewpoint image generating process) ;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
The virtual space data is data for generating movements of the performer avatar, and is characterized in that it includes movement data for the performer avatar that can generate the movements of the performer avatar in a plurality of generation modes that have different processing loads when generating the movements of the performer avatar on the viewer user terminal .
According to this feature, the generation of viewer avatar viewpoint images, which requires a large processing load, is performed on the viewer user terminal side, so that more viewer users can participate. Also, performer users can participate as performer avatars, and the movements of the performer avatars can be reflected in the viewer avatar viewpoint images, so that the interest of the virtual space content can be further increased. Furthermore, by changing the generation mode, viewer avatar viewpoint images corresponding to the movement status of the performance avatar can be generated, and differences in the processing capabilities of viewer user terminals can also be accommodated.

The virtual space content delivery method according to claim 11 ,
A virtual space content distribution method for distributing virtual space content (e.g., a virtual live performance) in a virtual space (e.g., a virtual live performance venue) via a network (e.g., the Internet) using at least a server computer (e.g., a distribution server computer 100), comprising:
The avatars (e.g., viewer avatars) that can participate in the virtual space that is the subject of the virtual space content include viewer avatars that operate in the virtual space by operating a viewer user terminal (e.g., viewer terminal 300) that is a terminal available to a viewer user and is connected to the server computer via the network,
There are a plurality of viewer users, and each viewer user is assigned a viewer user terminal (for example, as shown in FIG. 1, viewer users A to C are assigned viewer terminals 300a to 300c),
a step (e.g., a viewer viewpoint image generation process) of generating, in each of the viewer user terminals, a viewer avatar viewpoint image which is a viewpoint image of the virtual space corresponding to the viewer avatar, the viewer avatar viewpoint image based on virtual space data (e.g., virtual space update data) of the virtual space transmitted from the server computer ;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
The virtual space data is data for generating movements of the performer avatar, and is characterized in that it includes movement data for the performer avatar that can generate the movements of the performer avatar in a plurality of generation modes that have different processing loads when generating the movements of the performer avatar on the viewer user terminal .
According to this feature, the generation of viewer avatar viewpoint images, which requires a large processing load, is performed on the viewer user terminal side, so that more viewer users can participate. Also, performer users can participate as performer avatars, and the movements of the performer avatars can be reflected in the viewer avatar viewpoint images, so that the interest of the virtual space content can be further increased. Furthermore, by changing the generation mode, viewer avatar viewpoint images corresponding to the movement status of the performance avatar can be generated, and differences in the processing capabilities of viewer user terminals can also be accommodated.

The present invention may have only the invention-specific matters described in the claims of the present invention, or may have the invention-specific matters described in the claims of the present invention as well as configurations other than the invention-specific matters.

1 is a block diagram showing an example of a system configuration of a virtual space content delivery system according to a first embodiment of the present invention. 1 is a diagram illustrating an example of the configuration of a delivery server computer in a virtual space content delivery system according to a first embodiment of the present invention. 3 is a diagram showing various data stored in a storage device of a delivery server computer according to the first embodiment of the present invention; FIG. 2 is a diagram showing an example of the configuration of a performer terminal in the virtual space content distribution system according to the first embodiment of the present invention; FIG. 2 is a diagram illustrating an example of the configuration of a viewer terminal in the virtual space content delivery system according to the first embodiment of the present invention. 2 is a diagram illustrating an example of the configuration of an administrator terminal in the virtual space content delivery system according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of the configuration of event data in the virtual space content delivery system according to the first embodiment of the present invention; FIG. A figure showing an example of the configuration of performer user data in the virtual space content distribution system of Example 1 of the present invention. 3 is a diagram showing an example of the configuration of viewer user data in the virtual space content delivery system according to the first embodiment of the present invention; FIG. 2 is an explanatory diagram showing data exchanged between each terminal and a server computer constituting the virtual space content distribution system according to the first embodiment of the present invention; FIG. 1 is a schematic diagram showing the facilities of a studio used by a performer user in a virtual space content delivery system according to a first embodiment of the present invention. FIG. 1 is a diagram showing a virtual live performance venue, which is a virtual space used in a virtual space content distribution system according to a first embodiment of the present invention. FIG. 1 is a diagram showing an example of a display on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention; 1 is a diagram showing an example of a display on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention; 1 is a diagram showing an example of a display on a performer terminal constituting a virtual space content distribution system according to a first embodiment of the present invention. FIG. 1A to 1C are diagrams illustrating examples of display by zooming operations on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. 1A to 1C are diagrams illustrating examples of display by zooming operations on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. 1 is a diagram showing an example of a display on a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention; A figure showing an example of the display of a flying performance period on a viewer terminal constituting the virtual space content distribution system of Example 1 of the present invention. 1 is a flow diagram showing an example of a viewer viewpoint video control process executed in a viewer terminal constituting the virtual space content delivery system according to the first embodiment of the present invention. FIG. 11 is an explanatory diagram showing the contents of a virtual live performance delivered by a virtual space content delivery system according to a second embodiment of the present invention. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal constituting a virtual space content delivery system according to a second embodiment of the present invention; FIG. 13 is a diagram showing a participant user table used in a delivery server computer constituting a virtual space content delivery system according to a second embodiment of the present invention. FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal constituting a virtual space content delivery system according to a second embodiment of the present invention; FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a second embodiment of the present invention. FIG. 11 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal constituting a virtual space content delivery system according to a second embodiment of the present invention; FIG. 11 is an explanatory diagram showing the relationship between the period type of a virtual live show and the entity that generates the virtual camera viewpoint video in a virtual space content distribution system according to a second embodiment of the present invention. FIG. 10 is an explanatory diagram showing the generation status of a virtual camera viewpoint image and a viewer viewpoint image in a distribution server computer and a viewer terminal that constitute a virtual space content distribution system of Example 2 of the present invention. FIG. FIG. 11 is a flow diagram showing a viewer viewpoint video generation process in a modified example of the second embodiment of the present invention. FIG. 13 is a diagram showing the contents of virtual space update data transmitted from a delivery server computer in a modified example of the second embodiment of the present invention. FIG. 11 is an explanatory diagram showing the contents of a virtual live performance delivered by a virtual space content delivery system according to a third embodiment of the present invention. 13A and 13B are diagrams illustrating the behavior of a viewer user in a virtual live event according to a third embodiment of the present invention. 1A is an explanatory diagram showing data sent and received from a distribution server computer constituting a virtual space content distribution system of Example 3 of the present invention to a viewer terminal, and FIG. 1B is an explanatory diagram showing differences in the timing of actions of each viewer user during a virtual live performance. 1A is a diagram showing a participant user table used in a distribution server computer constituting a virtual space content distribution system of Example 3 of the present invention, and FIG. 1B is a diagram showing music list data stored in the distribution server computer. FIG. 13 is an explanatory diagram showing the transmission status of each piece of information by the distribution server computer during each period of the virtual live in Example 3 of the present invention. 13 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a third embodiment of the present invention. FIG. FIG. 11 is a flow diagram showing the contents of a delay actor determination process executed in a delivery server computer constituting a virtual space content delivery system according to a third embodiment of the present invention. 13 is an explanatory diagram showing data exchanged between a delivery server computer and a viewer terminal that constitute a virtual space content delivery system according to a third embodiment of the present invention. FIG. 13A is a flow diagram showing the contents of viewer perspective video generation process 1 executed in a viewer terminal constituting a virtual space content distribution system of Example 3 of the present invention, and FIG. 13B is a flow diagram showing the contents of viewer perspective video generation process 2. 13A and 13B are examples of the display of a viewer's viewpoint video of a virtual live performance on a viewer terminal constituting a virtual space content delivery system according to a third embodiment of the present invention. 13A and 13B are examples of the display of a viewer's viewpoint video of a virtual live performance on a viewer terminal constituting a virtual space content delivery system according to a third embodiment of the present invention. FIG. 13 is a diagram showing a virtual live concert venue in a modified example of the present invention. FIG. 13 is a diagram showing a participation position selection screen for a virtual live performance venue in a modified example of the present invention. FIG. 13A is a diagram showing a movable range of a viewer user in a modified example of the present invention, and FIG. 13B is a diagram showing an example of a display displayed on a viewer terminal in the modified example. 11 is an explanatory diagram showing a situation in which the movement of a viewer avatar is adjusted in a modified example of the present invention. A diagram showing the relationship between the duration of a virtual live show and the periods during which performance content and non-performance content are performed in a modified example of the present invention. A diagram showing the relationship between the duration of a virtual live show and the actions that a viewer avatar can take in a modified example of the present invention. FIG. 11 is an explanatory diagram showing data exchanged between a distribution server computer and a viewer terminal in a modified example of the present invention.

The following describes the form for implementing the virtual space content distribution system of the present invention based on the embodiment and the drawings. Note that the same reference numerals are used to designate identical or similar components in multiple drawings, and duplicated explanations will be omitted. The virtual space content distribution program of the present invention may be a program used by the entire system consisting of the server computer and each terminal that constitute the virtual space content distribution system, which will be described later, or it may be a program that operates only on a portion of the devices that constitute the virtual space content distribution system, such as the server computer or each terminal, and each process described below is executed by a program stored in the device on which the process is performed.

Figure 1 is a block diagram showing an example of the system configuration of a virtual space content distribution system in Example 1, which is an embodiment of the present invention. In the system of Example 1 shown in Figure 1, a virtual live performance in which a performer avatar 1 corresponding to a performer user performs a live event in a virtual live venue shown in Figure 12 is distributed as virtual space content. Note that in the following, distribution means transmission to multiple people, and is synonymous with transmission.

As shown in FIG. 1, the virtual space content distribution system of this embodiment 1 is mainly composed of a distribution server computer 100 capable of executing various processes related to the virtual live venue shown in FIG. 11, which is a virtual space, and various processes related to the distribution of the virtual live event, an administrator terminal 150 used by an event manager M of an event management organization that manages and operates the virtual live event to manage the distribution of the virtual live event, a performer terminal 200 used by a performer user, and a viewer terminal 300 that can be used by a viewer user who virtually participates in and watches the virtual live event, and the distribution server computer 100, the administrator terminal 150, the performer terminal 200, and the viewer terminal 300 are connected to each other so that data communication can be performed via the Internet network N, which is an open computer network.

In this embodiment 1, the administrator terminal 150 and the performer terminal 200 are also connected to the distribution server computer 100 via the Internet network N, but the present invention is not limited to this. For example, if the studio shown in FIG. 11 is a studio operated by an event management organization that provides studios to performer users, and the distribution server computer 100 is installed in these studios, the administrator terminal 150 and the performer terminal 200 may be locally connected to the distribution server computer 100 via data communication, or may be connected via a local area network LAN within the studio facility.

As shown in FIG. 1, there are multiple viewer users, such as viewer user A, viewer user B, viewer user C, etc., and viewer terminals 300 include viewer terminals 300a, 300b, 300c, etc. that can be used by each of viewer users A, B, C, etc. Although there are usually multiple viewer users, there may be one or four or more, and the number may be determined appropriately depending on the virtual space, such as a virtual live venue. Hereinafter, viewer user A, viewer user B, viewer user C, etc. may be collectively referred to as viewer users, and viewer terminals 300a, 300b, 300c, etc. may be collectively referred to as viewer terminals 300.

In addition, in this embodiment 1, an example is given in which there is one performer user, but the present invention is not limited to this, and the virtual live performance may be a joint event in which multiple performer users who are working individually virtually participate, or a group event in which a group of multiple people who are working together virtually participate. In this way, when multiple performers are appearing, performer terminal 200a, performer terminal 200b, performer terminal 200c, etc. may be provided for each performer user, or when a group of multiple people are performing, one performer terminal 200 may be used by multiple people.

In addition, in FIG. 1, the distribution server computer 100 is illustrated as a single unit, but the present invention is not limited to this. The distribution server computer 100 may be configured with multiple server computers, such as a first server computer that mainly performs virtual space processing and a second server computer that mainly performs distribution processing, or may be configured as a cloud server, etc., and the number of server computers may be changed appropriately depending on the scale of the event, the number of participating listener users, etc.

Similarly, while the administrator terminal 150 is illustrated as a single unit in FIG. 1, the present invention is not limited to this, and it goes without saying that there may be multiple administrator terminals 150 depending on the number of performer users and viewer users virtually participating in the live performance.

<Distribution server computer>
Fig. 2 is a diagram showing the configuration of the distribution server computer 100 used in this embodiment 1. As described above, the distribution server computer 100 used in this embodiment 1 is a normal server computer with relatively excellent processing power installed in a studio operated by an event management organization, and as shown in Fig. 2, has a processor (CPU) 101 connected to a data bus 108, a memory (RAM) 102, a storage 103 such as a hard disk, a communication interface (I/F) 104 capable of two-way communication by TCP/IP protocol via the Internet, which is an open computer network, and an input/output interface (I/F) 105 to which an input device such as a keyboard can be connected.

In this embodiment 1, an input/output interface (I/F) 105 is provided to enable local operation, but if remote operation is possible via a communication interface (I/F) 104, the input/output interface (I/F) 105 may not be provided.

In addition to an operation system (server OS) for providing server functions (not shown), various data and programs are stored in the storage 103 of the distribution server computer 100 as shown in FIG. 3. Specifically, mainly stored are an event management program 110 for providing an event management function for managing live events, a virtual space generation program 120 for providing a virtual space for a virtual live venue, a performer user management program 130 for providing a performer user management function for managing performer users, and a viewer user management program 132 for providing a viewer user management function for managing viewer users.

The event management program 110 is a program for managing a virtual live event, which is an event held at a virtual live venue (see FIG. 12), in cooperation with the administrator terminal 150, and is capable of managing the progress of the virtual live event using event data 111 (see FIG. 7), time schedule data (TS data) 112, and music data 113 stored in the storage 103 together with the event management program 110.

An example of the event data 111 used in this embodiment 1 is shown in FIG. 7. The event data 111 stores information such as the event name, the file name of the time schedule (TS) data, a performer ID uniquely assigned to the performer user who is the performer, the file name of the music list, the file name of the participating viewer list in which the information of special viewer users who are permitted to participate virtually is registered in the third area of the virtual audience area described below, and the file name of the participating viewer list in which the information of viewer users who have reserved virtual participation is registered in the fourth area of the virtual audience area described below, in association with the event ID uniquely assigned to the event.

The time schedule data (TS data) 112 is data that describes the order of progression of the virtual live event to be held and the programs such as songs in chronological order, and the virtual live event is managed so that it progresses based on the time schedule described in the time schedule data (TS data) 112. Furthermore, the time schedule data (TS data) 112 and event data 111 can be changed, updated, or added by the event manager M operating the manager terminal 150.

The song data 113 is data on the performance parts (so-called karaoke) of each song sung by the performer user, and is played by the audio control program 124 (described later) based on the time schedule data (TS data) 112 by the event management program 110, and is transmitted to the performer terminal 200 and the viewer terminal 300 (see FIG. 11).

The virtual space generation program 120 is a program for providing a virtual live venue, which is the virtual space shown in FIG. 12, and has the function of providing a virtual live venue in which performer avatars and viewer avatars are virtually participating, using virtual space data 121 describing the structure of the virtual live venue, which is stored in storage 103 together with the virtual space generation program 120, object data 122 for placing various objects (including virtual placement object 43, which is a star decoration on the stage, and virtual aerial objects 50-52, which are star objects in the sky) to be placed in the virtual space, and performer avatar data included in performer user data 131 described below and viewer avatar data included in viewer user data 133.

As shown in FIG. 12, the viewpoint image control program 123 is a program that provides a function to generate viewpoint images from virtual cameras C1-C4 virtually set in the virtual live venue and an airborne virtual camera (not shown), and also provides a function to display images from the viewpoints of viewer avatars 11-14 virtually participating in the third area of the virtual live venue and viewer avatars 21-32 virtually participating in the fourth area on the viewer terminal 300. Note that virtual camera C2 is a virtual camera set to provide an image from the viewpoint of performer avatar 1 on the stage of the virtual live venue, and the viewpoint image from virtual camera C2 is projected onto a screen S installed in the studio by a projector 211 (described later) connected to the performer terminal 200 in cooperation with the performer terminal 200.

The audio control program 124 is a program for controlling the audio (including music) distributed during the virtual live event, and specifically provides a function for playing music data 113 and a function for synthesizing the audio produced by the performer user with the played music and distributing it to the administrator terminal 150 and the viewer terminal 300.

The comment control program 125 cooperates with a viewer comment control program stored in the viewer terminal 300 described below to provide a function for displaying various comments entered by each viewer user in the viewer terminal 300 during a virtual live event in chronological order on the performer terminal 200 and the viewer terminal 300.

The gift control program 126 is a program for controlling gifts given from viewer users to performer users in cooperation with a viewer program stored in the viewer terminal 300 described below, and specifically provides functions such as a function for the viewer user to purchase gifts based on information on the unit purchase price set for various gifts stored in the gift data 134, a processing function for the viewer user to give gifts purchased using the gift object data included in the gift data 134 to the performer user (including processing for presentation when giving the gift), and a processing function for changing the owner of a gift given from the viewer user to the performer user.

The performer user management program 130 provides management functions and authentication functions for information regarding performer users based on the performer user data 131 shown in FIG. 8, which is stored in the storage 103 together with the performer user management program 130.

The performer user data 131 used in this Example 1 is illustrated in FIG. 8. The performer user data 131 stores various information, such as the performer user's account (email address), name, authentication information, the file name of the avatar data of the avatar used in the virtual space, and the file name of the item list in which the items owned are registered, in association with a performer user ID uniquely assigned to the performer user. Although not illustrated in FIG. 8, information on virtual value such as points owned by each performer user may also be stored as the performer user data 131.

Performer users can be authenticated by matching their performer user ID, account, and authentication information, and items (gifts) that each performer user has received from viewers and that they come to own can be identified from an item list.

The viewer user management program 132 provides functions such as management of information about viewer users and authentication functions based on the viewer user data 133 shown in FIG. 9, which is stored in the storage 103 together with the program.

The viewer user data 133 used in the present embodiment 1 is illustrated in FIG. 9. The viewer user data 133 stores various information associated with a viewer user ID uniquely assigned to the viewer user, such as the viewer user's account (email address), authentication information, the file name of the avatar data of the avatar to be used in the virtual space, owned points which are the number of points that can be used in the virtual space, the file name of an item list in which items (gifts) purchased using points are registered, and personal information such as name, date of birth, and telephone number. Although not shown in FIG. 9, a nickname (comment name) displayed together with a comment is also registered in the viewer user data 133, and the nickname (comment name) is displayed together with the comment. Points can be increased, for example, by purchasing points from a specified operating company.

<Performer terminal>
Fig. 4 is a diagram showing an example of the configuration of a performer terminal 200 in the virtual space content delivery system of this embodiment 1. In this embodiment 1, as shown in Fig. 11, the performer terminal 200 is installed in a control room adjacent to a studio where a performer user performs his/her acting motions, and uses a normal computer with relatively excellent processing power, and as shown in Fig. 4, has a processor (CPU) 201 connected to a data bus 208, a memory (RAM) 202, a storage 203 such as a hard disk, a communication interface (I/F) 204 capable of two-way communication by TCP/IP protocol via the Internet, which is an open computer network, an image processing unit 206 including a graphics processing unit (GPU) to which display devices A210 to C212 are connected, and an input/output interface (I/F) 205 to which various input/output devices are connected.

The input/output interface (I/F) 205 is connected to the following input devices: a motion sensor 220 including multiple wearable sensors 220C1-220C5 (see FIG. 11) worn by the performer user on the left and right limbs and waist; a facial expression input device 221 consisting of an imaging camera for inputting the performer's facial expressions; a voice input device 222 consisting of a sound-collecting microphone worn on the performer user's head; and an operation input device 224 such as a keyboard or touch panel that can be operated by an assistant operator O who assists the performer user. Note that by providing these input devices with a controller that the performer user can hold to perform various operations, the performer himself may be able to perform various operations without the assistance of the assistant operator O.

In this embodiment 1, the performer user performs performance actions in accordance with the music (including musical actions such as singing and dancing), and an assistant operator O assists with operations during the performance. However, there may be multiple assistant operators O, or the operation contents may be set in advance in the performer terminal 200, for example, in a sequence program, so that an assistant operator O is not provided.

The input/output interface (I/F) 205 is connected to an audio output device 223 including high-performance earphones (ear monitors) worn by the performer user as output devices and speakers placed in the control room, so that the sound of the music being played sent from the distribution server computer 100 is output to the performer user via the high-performance earphones (ear monitors), and the music audio is output from the speakers along with the performer's voice, allowing an assistant operator O, etc., to check the status of the music, including the voice produced by the performer user, in the control room.

In this embodiment 1, high-performance earphones (in-ear monitors) are used to avoid inconvenience caused by the reproduced music audio being input from the sound collection microphone worn by the performer user, but the present invention is not limited to this, and the music audio may be output from speakers in the studio, etc., as long as it is possible to avoid inconvenience caused by the input of the music audio.

Note that, in this embodiment 1, an example is shown in which the sound collection microphones are worn by the performers, but these sound collection microphones may also be stationary microphones that are installed on the floor, walls, or ceiling of the studio.

Here, we will explain the motion sensor 220 used in the performer terminal 200 of this Example 1. Any motion sensor can be used as long as it can properly detect (measure) the body movements (movements) of the performer user. In this Example 1, in order to enable more accurate and shorter-period detection of the performer's movements, multiple wearable sensors 220C1 to 220C5 are used that are worn by the performer user on the body. Note that the motion sensor 220 may be of a type that does not require the performer to wear any equipment, such as a LiDER that uses laser light or the like.

In this embodiment 1, in order to reduce the burden on the performer user's head caused by wearing sensors when performing a performance, in particular the burden on the head, the performer user's head movements are detected by image recognition using images captured by the imaging camera constituting the facial expression input device 221, as described below, and five wearable sensors 220C1 to 220C5 are used to detect movements other than the head. However, for example, wearable sensors may also be provided on the performer user's head, or more wearable sensors (seven or more) may be worn to detect more detailed movements.

As shown in FIG. 11, the mounting sensors 220C1 to 220C5 detect their own position and orientation in cooperation with base station 220a and base station 220b, which are installed in a studio room separated from the adjacent adjustment room by a glass window.

Base station 220a and base station 220b can use, as an example, a multi-axis laser emitter. After emitting a flashing light for synchronization, base station 220a scans the laser light, for example, around a vertical axis. Base station 220b scans the laser light, for example, around a horizontal axis.

Each of the mounting sensors 220C1 to 220C5 may include multiple optical sensors that detect the incidence of blinking light and laser light from the base station 220a and base station 220b.

Each of the mounting sensors 220C1 to 220C5 can detect its own position and orientation based on the time difference between the incidence of the blinking light and the incidence of the laser light, the light reception time at each optical sensor, the incidence angle of the laser light detected by each optical sensor, and other information as necessary. For example, ViveTracker or a base station provided by HTC CORPORATION can be suitably used.

Base station 220a and base station 220b emit flashing light and perform laser light scanning at regular intervals, so the detection information of each worn sensor 220C1-220C5 is updated at each interval. Detection information indicating the position and orientation of each motion sensor calculated in each of these worn sensors 220C1-220C5 is input to the performer terminal 200 via short-range wireless communication and transmitted to the distribution server computer 100 as performer avatar information together with facial motion information (see FIG. 10).

Note that in this embodiment 1, an example is shown in which two base stations, base station 220a and base station 220b, are used, but the present invention is not limited to this, and three or more base stations may be used.

Motion sensor 220 may also be configured such that each of a number of motion sensors worn by the performer user is equipped with a number of infrared LEDs or visible light LEDs, and the position and orientation of each of the motion sensors is detected by detecting the light from these LEDs with an infrared camera installed on the floor or wall of the studio.

Moreover, in order to reduce the burden on the performer user, the motion sensor 220 may be replaced with a motion sensor using a lighter reflective marker instead of the wearable sensors 220C1-220C5. In other words, the motion sensor may detect the performer user's movements by photographing reflective markers attached to various parts of the performer user's body with adhesive tape or the like to generate photographic data, and then detecting the position and orientation of the reflective markers through image processing of the photographic data. In this case, a cameraman who photographs the performer user may be positioned in the studio, and the cameraman may photograph the performer user from a direction suitable for detection, thereby enabling the motion of the performer user to be detected well.

Motion sensor 220 may also be a suit with a built-in inertial sensor, such as a suit with a built-in inertial sensor for the MVN motion capture system marketed by Xsens.com, and the movements of the performer user may be detected by analyzing the sensor signal output from the inertial sensor.

Any facial expression input device 221 can be used as long as it can properly detect the movement of the head, including the face, of the performer user. In this embodiment 1, specifically, it is configured with an imaging camera (digital camera) arranged so as to be able to continuously capture head images, including the face, of the performer user, and the head movement and facial movement (facial expression) of the performer user captured by the imaging camera are detected by image recognition, and the facial motion information, which is the detected head movement and facial movement (facial expression), is transmitted to the distribution server computer 100 as performer avatar information together with body motion information, as shown in FIG. 10.

The facial expression input device 221 is not limited to the imaging camera (digital camera) used in the first embodiment. For example, the facial expression input device 221 may be a 3D camera capable of detecting the depth of a person's face, or a mobile terminal such as a smartphone equipped with a LiDER device. In this case, the performer user may wear such a mobile terminal.

As described above, the performer terminal 200 of this embodiment 1 has three display devices A210 to C212. Display device A210 is an LCD monitor or the like installed in the control room as shown in FIG. 11. Display device B211 is a projector that projects an image onto a screen S installed in the studio. Display device C212 is a large vertical display installed adjacent to the screen S in the studio.

Display device A210, which is placed in the studio's adjustment room, displays images from each of the virtual cameras C1 to C4, as well as viewpoint images from a virtual camera moving in the air during the flying performance period described below, as well as the contents of comments and a message input window into which assistant operator O inputs any message he or she wishes to convey to the performer user.

On the other hand, a screen S projected from a projector, which is a display device B211 installed in the studio, displays an image from the viewpoint of a virtual camera C2 (see FIG. 12), which is the viewpoint of the performer, for example, as shown in FIG. 15, including avatars of viewers virtually participating in the live performance in the third and fourth areas, allowing the performer user to check the status of the viewer users virtually participating in the virtual live performance through the image.

In addition, comments from viewer users, as well as comments and messages entered by the assistant operator O, are displayed on the display device C212 installed in the studio, allowing the performer user to check the comments from viewer users and messages from the assistant operator O even during the performance.

In addition to an operating system (OS) for operating the computer that is the performer terminal 200, the storage 203 stores a performer program that is executed by the processor (CPU) 201 etc. to provide various functions in cooperation with the distribution server computer 100, including the function of allowing the performer user to control the movement of the performer avatar 1 in a virtual live performance, which is virtual space content.

As shown in FIG. 4, the performer programs include a performer avatar control program, a performer viewpoint image control program, a performer viewpoint image control program, a performer voice control program, a performer comment control program, performer authentication data, etc.

The performer avatar control program is a program that provides the function of primarily scanning the body motion of the performer user using the motion sensor 220 and scanning the facial motion (facial expression) of the performer user using the facial expression input device 221 to generate performer avatar information for operating the performer avatar and transmit it to the distribution server computer 100.

The performer's viewpoint image control program is a program that provides the function of generating and outputting the viewpoint image of virtual camera C2, which is the performer's viewpoint, based on the virtual space update data distributed from the distribution server computer 100.

The performer's voice control program is a program that provides the function of outputting the playback sound of a song based on the playback music data distributed from the distribution server computer 100 via the audio output device 223, converting the audio input from the audio input device 222 into data and sending it to the distribution server computer 100 as performer voice data, and generating and outputting a monitor voice consisting of the playback sound of the song and the performer's voice.

The performer comment control program is a program that provides the function of displaying each comment by the viewer user that is distributed from the distribution server computer 100.

The performer authentication data is used to authenticate the identity of the performer user in a communication connection with the distribution server computer 100.

The performer avatar control program includes a motion parameter generation program capable of generating parameter information for the positions and rotation angles of the bones that make up the rig data (sometimes called "skeleton data") that indicates the skeleton of the performer avatar included in the performer avatar data, as well as an image recognition processing program for image recognition of the performer user's head position and facial expression from the image captured by the imaging camera.

In addition, in this embodiment 1, the performer terminal 200 is illustrated as a form in which a computer serving as the performer terminal 200 installed in a studio operated by an event management organization is used jointly by multiple performer users to hold a live event, and each performer user can input performer authentication data to use the computer as the performer terminal 200 of that performer user. However, the present invention is not limited to this, and the performer terminal 200 may be used only by the performer user who owns it, such as when a performer user appears in a virtual live event from his or her home, and the performer terminal 200 is a computer installed in the performer user's home and dedicated to the performer user's own use, and in such cases, storage of performer authentication data is not necessarily required.

In this way, by using the performer terminal 200 constituting the virtual space content distribution system of this embodiment 1, the performer user can move the performer avatar 1 placed on the virtual stage G in the virtual live venue in conjunction with the performer user's own movements, and can have the performer user's own facial expressions reflected in the facial expressions of the performer avatar 1. In addition, the performer user can distribute his or her own voice as the voice of the performer avatar 1 to viewers via the distribution server computer 100.

<Viewer terminal>
Fig. 5 is a diagram showing an example of the configuration of a viewer terminal 300 in the virtual space content delivery system of the present embodiment 1. In the present embodiment 1, the viewer terminal 300 is a smartphone P carried by a viewer user or a regular desktop computer (PC) installed at a viewer user's home or the like, and as shown in Fig. 5, has a processor (CPU) 301 connected to a data bus 308, a memory (RAM) 302, a storage 303 such as a hard disk or non-volatile memory, a communication interface (I/F) 304 capable of two-way communication by TCP/IP protocol via the Internet, which is an open computer network, an image processing unit 306 including a graphics processing unit (GPU) to which a display device 310 is connected, and an input/output interface (I/F) 305 to which various input/output devices are connected.

The display device 310 may be any device capable of displaying various viewpoint images generated based on the virtual space data of the virtual live venue, which is a virtual space updated by the virtual space update data (see FIG. 10) distributed from the distribution server computer 100, by the viewer viewpoint image control program included in the viewer program stored in the storage 303, as described below, and the flying viewpoint images distributed from the distribution server computer 100 during the flying performance period described below. If the viewer terminal 300 is a smartphone P, the display device 310 corresponds to the display of the smartphone P, and if the viewer terminal 300 is a normal computer (PC), the display device 310 corresponds to a stationary display constituting the computer (PC). Note that these displays are not limited to those that display planar images (2D images), but may be, for example, those that can display stereoscopic images (3D images) by displaying right-eye images and left-eye images, such as a head-mounted display (HMD) that can be worn by the viewer.

The input/output interface (I/F) 305 is connected to an operation input device 321 consisting of one or more devices that allow the viewer to perform various operations as input devices, and an audio output device 322 such as earphones or speakers that can output live audio, etc., distributed from the distribution server computer 100.

It goes without saying that the audio output device 322 need not be a separate device, but may be integrated with the display device 310, such as the head-mounted display (HMD) described above.

The operation input device 321 may be any device that allows the viewer to perform various operations such as motion operations including movement of the viewer avatar, zoom operations, operations related to comments, operations related to gifts, etc. For example, it may be composed of at least one of a transparent touch panel formed on the display surface of a smartphone, a keyboard constituting a computer, a game controller, etc.

In this embodiment 1, as described below, the virtual audience area in the virtual live venue is divided into a third area near the virtual stage G in which only viewer users who have been specially permitted by the performer user can virtually participate, and a fourth area around the third area in which general viewer users can virtually participate, as shown in FIG. 12. Viewer users who virtually participate in the fourth area may have an operation input device 321 such as a touch panel, keyboard, or game controller as described above, but viewer users who virtually participate in the third area possess an operation input device 321 such as an MVN motion capture suit with the above-mentioned inertial sensor built in, and are viewer users who can precisely move the viewer avatar with their own movements, just like the performer users.

However, even viewer users who are permitted to virtually participate in the third area cannot change the facial expression of the viewer avatar, but the present invention is not limited to this, and the facial expressions of these viewer avatars may be changed, for example, as described above, by the viewer user using the camera function of a smartphone or a camera connected to a computer to recognize the facial expression of the viewer user, even if it is at a different level from the change level of the performer user.

The various input/output devices connected to the input/output interface (I/F) 305 may be built-in or external, and in the case of external devices, they may be connected in either a wired or wireless manner.

In addition to an operating system (OS) for operating the viewer terminal 300, such as a smartphone or computer, the storage 303 also stores a viewer program as a viewing application that is executed by the processor (CPU) 301 or the like to provide various functions, including a live viewing function for viewer users in a virtual live, which is virtual space content, in cooperation with the distribution server computer 100.

As shown in FIG. 5, the viewer program includes a viewer viewpoint image control program that provides the function of generating each viewpoint image of the virtual live to be displayed on the display device 310, a viewer audio control program that provides the function of outputting live audio from the audio output device 322 based on the live audio (data) of the virtual live distributed from the distribution server computer 100, a viewer comment control program that displays comments based on comment data distributed from the distribution server computer 100 and provides functions related to comments such as comments entered by the viewer and transmitted to the distribution server computer 100, and viewer authentication data for authenticating the identity of the viewer user in a communication connection with the distribution server computer 100.

Note that, although not shown in FIG. 5, in addition to the above-mentioned programs, there is included a gift control program that controls gifts, and an area designation program for reserving an area (position) in which a viewer avatar will virtually participate in the virtual live venue in advance before the start of the live event, as shown in variant example 3-1 (see FIGS. 42 and 43) described below, but other programs may also be included.

In this embodiment 1, the comments entered by the viewer are short messages in text format, but are not limited to these short messages in text format and may be, for example, messages in still image format, messages in video format, or any other electronic message format. The comments are displayed on the display device 310, superimposed on each viewpoint video.

In this way, by using the viewer terminal 300 constituting the virtual space content distribution system of this embodiment 1, a viewer user can move and operate the viewer avatar virtually participating in the virtual live venue, and by changing the viewpoint of the viewer avatar, the viewpoint image displayed on the display device 310 can be changed, and the zoom state (viewpoint state) can also be changed, allowing the viewer avatar to operate and cheer on the performer avatar 1.

Note that a viewer avatar that virtually participates in the third area is only allowed to move within the third area, and cannot move to the fourth area. On the other hand, a viewer avatar that virtually participates in the fourth area is only allowed to move within the fourth area, and cannot move to the third area. However, the present invention is not limited to this, and a viewer avatar that virtually participates in three areas may be allowed to move to the fourth area. Note that a viewer avatar naturally cannot move onto the virtual stage G, etc., so visibility of the performer avatar 1 is not obstructed by the viewer avatar moving onto the virtual stage G.

In addition, by displaying viewpoint images on these display devices 310 and outputting live audio, users can enjoy the virtual live performance with the feeling that they are actually present at the virtual live performance venue.

Although we will not go into detail here, the comment function provided by the viewer comment control program allows viewers to enjoy the virtual live show while checking comments entered by themselves and other viewer users, and the gift function provided by the gift control program allows viewers to give items they own to the performer avatar 1 to liven up the virtual live show.

<Administrator terminal>
Fig. 6 is a diagram showing an example of the configuration of the administrator terminal 150 in the virtual space content delivery system of this embodiment 1. In this embodiment 1, the administrator terminal 150 uses a normal computer installed in a control room adjacent to the studio together with the performer terminal 200, and as shown in Fig. 6, has a processor (CPU) 151 connected to a data bus 158, a memory (RAM) 152, a storage 153 such as a hard disk, a communication interface (I/F) 154 capable of two-way communication by TCP/IP protocol via the Internet, which is an open computer network, an image processing unit 156 including a graphics processing unit (GPU) to which a display device 160 is connected, and an input/output interface (I/F) 155 to which various input/output devices are connected.

As described below, the display device 160 is configured as one or more stationary displays capable of individually displaying the viewpoint images of each of the virtual cameras C1 to C4 described below and the virtual camera moving in the air during the flying performance period, which are generated based on the virtual space data of the virtual live venue, which is a virtual space updated by the virtual space update data (see FIG. 10) distributed from the distribution server computer 100, by the administrator viewpoint image control program included in the administrator program stored in the storage 153, as well as comments based on the comment data distributed from the distribution server computer 100.

The input/output interface (I/F) 155 is connected to an operation input device 161 consisting of one or more devices that allow the event manager M to perform various operations as input devices, and an audio output device 162 such as earphones, headphones, or speakers that can output live audio, etc., distributed from the distribution server computer 100.

The operation input device 161 can be any device that allows the event manager M to perform various operations such as input and settings related to the virtual live event, and may be composed of at least one of, for example, a keyboard constituting a computer, a transparent touch panel formed on the surface of a display, a viewpoint switching controller, etc.

The various input/output devices connected to the input/output interface (I/F) 155 may be connected in either a wired or wireless manner.

In addition to an operating system (OS) for operating the computer that is the administrator terminal 150, the storage 153 also stores an administrator program that is executed by the processor (CPU) 151 etc. to provide various functions in cooperation with the distribution server computer 100, including a distribution management function for the distribution of virtual live shows, which are virtual space content.

As shown in FIG. 6, the administrator programs include an administrator viewpoint video control program, an administrator audio control program, an administrator comment control program, a viewer management program, a performer management program, etc.

The administrator viewpoint image control program is a program that provides the function of generating viewpoint images from each virtual camera to be displayed on the display device 160, as well as the function of changing and switching between each viewpoint.

The administrator audio control program is a program that provides the function of outputting live audio from the audio output device 162 based on the audio data of the virtual live broadcast distributed from the distribution server computer 100.

The administrator comment control program is a program that displays comments based on comment data distributed from the distribution server computer 100, and provides comment-related functions such as prohibiting comments from being distributed and selecting viewers to prevent them from being distributed.

The viewer management program is a program for managing viewer users who virtually participate in a virtual live performance, and the performer management program is a program for managing performer users who virtually participate in a virtual live performance.

Note that although not shown in FIG. 6, the system includes an authentication program for authenticating whether the administrator who is the operator is the actual person, and a schedule program for editing the time schedule and song order of the virtual live performance, but may also include other programs.

In this way, the event manager M can operate the manager terminal 150 constituting the virtual space content distribution system of this embodiment 1 to implement various settings for executing the event, such as the program, order of songs, settings related to the performance, the time schedule, and the trajectory of the virtual camera moving in the air during the flying performance period, and the virtual live performance is managed based on the settings made in this way.

<Virtual live venue>
A virtual live performance venue, which is the virtual space used in Example 1, is shown in Fig. 12. As shown in Fig. 12, the virtual live performance venue, like a live performance venue in real space, is a virtual space having a virtual stage G where performer users appear as performer avatars 1 and a virtual audience area where audience users virtually participate as audience avatars.

As shown in FIG. 12, virtual stage G has a relatively large trapezoidal floor surface, and performer avatar 1 can move on virtual stage G in the same way as in a live venue in real space. On the side of virtual stage G opposite the virtual audience seats, as shown in FIG. 12, a virtual stage wall is formed that is divided into three parts: a center, a right side, and a left side, and virtual displays 40, 41, and 42 are provided in front of these virtual stage walls, and images and videos for the performance are virtually displayed on the virtual displays 40, 41, and 42.

A star virtual placement object 43 related to the performer user is placed between the virtual displays 40, 41, and 42, and large star virtual aerial objects 50 to 52 are placed in a second area, which is an area on the virtual stage G above the first area surrounded by the virtual stage walls, as shown in FIG. 12. The virtual aerial objects 50 to 52 are capable of moving within the second area.

A virtual audience area is provided in front of the virtual stage G, where viewer users can virtually participate in the virtual live venue as viewer avatars. As shown in FIG. 12, this virtual audience area is made up of a third area, which is a special area close to the center of the virtual stage G where the performer avatar 1 appears, and a fourth area, which is formed to surround the third area and is located at a greater distance from the center of the virtual stage G than the third area.

In these third areas, special viewer users who are permitted to participate virtually can participate virtually up to a certain upper limit. In this embodiment 1, as shown in FIG. 12, for example, viewer avatars 11 to 14 of four viewer users who have met a certain virtual participation record and have been permitted to participate virtually by the performer user are placed.

The fourth area also contains viewer avatars 21-32 of general viewer users who have booked virtual participation before the start of the virtual live event. Note that in FIG. 12, viewer avatars 21-32 are shown in a simplified manner for convenience, but are the same avatars as viewer avatars 11-14.

In addition, each viewer avatar 11 to 14 arranged in the third area can be moved within the third area by the viewer user operating the viewer terminal, and each viewer avatar 21 to 32 arranged in the fourth area can be moved within the fourth area by the viewer user operating the viewer terminal.

As shown in FIG. 12, four virtual cameras C1 to C4 are virtually positioned (set) in the virtual live venue. Virtual camera C1 is virtually positioned in front of performer avatar 1 so as to face the performer avatar 1, and the viewpoint image captured by virtual camera C1 is a viewpoint image viewed from a position close to the front of performer avatar 1, as shown in FIG. 13(a).

The virtual camera C2 is a camera virtually placed (set) above the head of the performer avatar 1, and the viewpoint image captured by the virtual camera C2 is a performer viewpoint image viewed from the performer avatar 1 toward the virtual audience area, as shown in FIG. 15.

Virtual camera C3 is a virtual camera virtually placed on the virtual display 41 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C3 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in Figure 13 (c).

Virtual camera C4 is a virtual camera virtually placed on the virtual display 42 diagonally behind the performer avatar 1, and the viewpoint image from virtual camera C4 is a viewpoint image looking at the virtual audience area from diagonally behind the performer avatar 1, as shown in FIG. 13(b).

In addition, in this embodiment 1, in addition to the fixedly placed (set) virtual cameras C1 to C4, an air-moving virtual camera (not shown) is virtually placed so that it can move through the air in the virtual live venue space, which is a virtual space. The aerial trajectories along which these air-moving virtual cameras move and the viewpoint directions (angles) at each position on the aerial trajectory are preset by the administrator terminal 150, and during the flying performance period described below, the performer avatar 1 virtually flies so as to follow the aerial trajectory of the air-moving virtual camera, thereby generating a viewpoint image in which the flying performer avatar 1 is captured by the air-moving virtual camera.

In this embodiment 1, as shown in FIG. 12, four virtual cameras C1 to C4 are virtually arranged as fixedly arranged (set) virtual cameras, but the present invention is not limited to this. The number of these fixedly arranged (set) virtual cameras may be five or more. Conversely, without providing fixedly arranged (set) virtual cameras, only the viewpoint images seen from the viewpoints corresponding to each avatar virtually participating in the virtual live venue may be displayed on the performer terminal 200 and the viewer terminal 300.

In addition, in this embodiment 1, an example is given of a configuration in which there is one virtual camera moving in the air, but the present invention is not limited to this, and a configuration in which there are multiple virtual cameras moving in the air, or conversely, a configuration in which no virtual camera moving in the air is virtually placed, may also be used.

<Studio>
11 is a diagram showing the studio used in Example 1, and as described above, the studio is adjacent to the control room separated by a glass window, and the performer user performs each action corresponding to singing in the studio, while the assistant operator O assists with the operation of the performer terminal 200 installed in the control room. Around the performer user wearing the mounting sensors 220C1 to 220C5, base stations 220a and 220b are mounted on a stand, and an imaging camera constituting the facial expression input device 221 is mounted on a stand in a position almost in front of the performer user.

A screen S is provided on the wall facing the performer user in the studio, and a projector (display device B211) placed on the ceiling of the studio projects the viewpoint image of virtual camera C2 onto screen S, so that the viewpoint image seen from performer avatar 1 as looking at the virtual audience area is displayed on screen S, as shown in FIG. 15. This allows the performer user to proceed with the live performance while keeping track of the behavior of the viewer avatars virtually participating in the virtual audience area (the audience's reaction to the live performance).

A large vertical display serving as display device C212 is placed to the side of screen S, and is configured to display comments from viewers and messages that the assistant operator O wishes to convey to the performer user. Note that display device C212 may also be configured to display information (performer support information) such as the lyrics and lines of the song sung by the performer during the live performance.

<Operation of Virtual Space Content Distribution System>
FIG. 10 is an explanatory diagram for explaining the operation of the virtual space content delivery system of the first embodiment, specifically, the flow of delivery of video and audio of a virtual live performance.

First, the video system will be described. As mentioned above, the distribution server computer 100 is capable of providing a virtual space of a virtual live venue based on the virtual space generation program 120 and the avatar data of the performer users and the viewer users, and the actions of each avatar of these performer users and viewer users are reflected in the actions of the performer avatar 1, and the actions (operations) of the viewer users are reflected in the actions of the viewer avatar by updating the virtual space data describing the state of the virtual space with the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300.

Although not shown in FIG. 10, when the performer terminal 200 and the viewer terminal 300 communicate with the distribution server computer 100 to virtually participate in the virtual live performance, virtual space data of the virtual live performance venue is delivered in advance from the distribution server computer 100 and stored in the performer terminal 200 and the viewer terminal 300. As described below, the virtual space data thus prestored is updated sequentially by virtual space update data successively delivered from the distribution server computer 100, so that the latest state of the virtual live performance venue can be identified on the performer terminal 200 and the viewer terminal 300.

Specifically, the movements and facial expressions of the performer user in the studio are scanned at predetermined time intervals by the body motion scan and face motion scan functions of the performer avatar control program described above, and performer avatar information is generated and transmitted to the distribution server computer 100.

On the other hand, viewer users who are permitted to virtually participate in the third area use motion sensors, just like the performers, and are scanned at predetermined time intervals using the body motion scan and face motion scan functions to generate viewer avatar information that is then sent to the distribution server computer 100.

In addition, a viewer user who is permitted to virtually participate in the fourth area can use the touch panel, keyboard, or controller of the viewer terminal 300 to perform various operations such as movement operations, operations to change the viewpoint direction, operations to change the viewpoint state (zoom), operations to raise hands, operations to clap hands, operations to jump, etc., to cause the viewer avatar to perform the corresponding action. Then, of these operations, viewer avatar information is generated based on the operations related to the actions of the viewer avatar and is transmitted to the distribution server computer 100.

In addition, viewer users virtually participating in the third area can also use a controller or the like to perform various operations such as movement, changing the viewpoint (zoom), raising their hands, clapping their hands, jumping, etc. In this case, menu items corresponding to each operation are displayed, and the menu items corresponding to each operation can be selected using a controller or the like.

In this way, the viewer terminal 300 executes an avatar action operation process that accepts operations corresponding to each action to be performed by the viewer avatar. The avatar action operation process also includes processes such as displaying and erasing menu items corresponding to each of the above-mentioned actions.

In this way, based on the performer avatar information transmitted from the performer terminal 200 and the viewer avatar information transmitted from the viewer terminal 300, the distribution server computer 100 executes a virtual space update process that updates the virtual space data, thereby reflecting the actions (operations) of the performer user and the viewer user in each avatar virtually participating in the virtual live venue.

Then, a virtual space update data generation process is executed to generate virtual space update data based on the virtual space data before the update and the virtual space data after the update, and the generated virtual space update data is distributed to the performer terminal 200 and the viewer terminal 300.

As shown in FIG. 10, in the performer terminal 200 and the viewer terminal 300 to which the virtual space update data has been delivered, the pre-stored virtual space data is updated based on the delivered virtual space update data, and a performer perspective video generation process and a viewer perspective video generation process are executed using the updated virtual space data, thereby generating performer perspective video and viewer perspective video based on the updated virtual space data, and the generated performer perspective video and viewer perspective video are displayed on the performer terminal 200 and the viewer terminal 300 (viewer perspective video output process).

In this way, in this embodiment 1, the viewpoint video of each viewer user who is virtually participating in the virtual live venue through a viewer avatar is generated and displayed on the viewer terminal 300 of each viewer user, and by generating the viewer avatar viewpoint video of each viewer user on the distribution server computer 100, it is possible to avoid a situation in which the processing load on the distribution server computer 100 becomes significantly large, which would result in many viewer users being unable to participate in the virtual live, and it is also possible to avoid a situation in which distribution becomes difficult due to an increased processing load.

In this embodiment 1, a viewpoint image in which the almost front of the performer avatar 1 is displayed large, which is the viewpoint image seen from the viewer avatar 13 close to the performer avatar 1, as shown in FIG. 13(d), is generated and displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 13 who is virtually participating in the virtual stage G in the third region close to the performer avatar 1, while a viewpoint image in which the almost front of the performer avatar 1 is displayed large, which is the viewpoint image seen from the viewer avatar 13 close to the performer avatar 1, as shown in FIG. 13(d), is displayed on the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 28 who is virtually participating in the virtual stage G in the fourth region far from the performer avatar 1, for example, at a position behind the viewer avatar 13, as shown in FIG. 13(e).

In addition, the display device 310 of the viewer terminal 300 of the viewer user of the viewer avatar 32 who is virtually participating at the edge position of the virtual stage G in the fourth area, which is set to be U-shaped when viewed from above, displays a viewpoint image as seen from the viewer avatar 32 positioned diagonally forward of the performer avatar 1, as shown in Figure 13 (f), in which the performer avatar 1 appears relatively small through the viewer avatar 14 who is virtually participating in the third area.

The display examples shown in FIG. 13 all illustrate the case where the viewer terminal 300 is a stationary computer (PC) and the display device 310 is a stationary display. However, for example, if the viewer terminal 300 is a smartphone P or a tablet, the viewpoint image of the virtual camera C1 is displayed as shown in FIG. 14(a), and if the viewer terminal 300 is the viewer terminal 300 of the viewer avatar 13, it is displayed as shown in FIG. 14(b).

Next, regarding the audio system, in the distribution server computer 100, music data 113 is played by the audio control program 124 based on time schedule data (TS data) 112, and is transmitted to the performer terminal 200 as a played music piece (data) (music playback process).

On the performer terminal 200, the music audio from the played music (data) sent from the distribution server computer 100 is output to the performer user through high-performance earphones (ear monitors) worn by the performer user (played music output process), and when the performer user sings along with the output played music, the performer voice input from the sound collection microphone (audio input device 222) is digitized and sent to the distribution server computer 100 as the performer voice (data).

The distribution server computer 100 generates distribution audio (data) from the performer audio (data) received from the performer terminal 200 and the music played in the music playback process described above (distribution audio generation process), and distributes the generated distribution audio (data) to the viewer terminal 300 (generated audio distribution process).

In the viewer terminal 300, the distribution audio (data) distributed from the distribution server computer 100 is output from an audio output device 322 such as an earphone or speaker, allowing the viewer to hear the live audio as if the performer avatar 1 were singing in the virtual live venue.

In addition, as described above, in the performer terminal 200, the performer's voice (data) is sent to the distribution server computer 100, and as shown in FIG. 10, the monitor voice generation process and the monitor voice output process are executed, so that the music voice of the played music and the performer's voice are output from the speakers located in the control room, allowing the assistant operator O, etc. to check the singing status of the performer user in the control room.

<Viewer's viewpoint video control processing>
Next, a viewer's viewpoint video control process executed in the viewer terminal 300 based on a viewer's viewpoint video control program included in the viewer's program will be described with reference to FIG.

In the viewer viewpoint video control process, first, it is determined whether or not the performer avatar 1 is in a flying performance period in which it is virtually flying, as shown in FIG. 19 (step S1). Whether or not the performance period is in a flying performance period can be identified from performance status data indicating that the performance period is in a flying performance period, sent from the distribution server computer 100. The distribution server computer 100 transmits performance status data indicating that the performance period is in a flying performance period to the viewer terminal 300 when the flying performance period is reached based on the time schedule data.

If it is during the flying performance period (Y in step S1), proceed to step S21. If it is not during the flying performance period (N in step S1), it is determined whether viewpoint-related operations including viewpoint movement operations (including avatar movement operations and viewpoint changes due to motion operations) and zoom operations are disabled (step S2).

If the viewpoint-related operation is disabled (Y in step S2), proceed to step S7. If the viewpoint-related operation is not disabled (N in step S2), determine whether or not there is an operation that changes the viewpoint, such as an operation to move the avatar or an operation to move the motion (step S3), and if there is an operation (Y in step S3), change the viewer viewpoint according to the operation, and if there is no operation (N in step S3), proceed to step S5 and determine whether or not there is a zoom operation (step S5).

If a zoom operation is performed (Y in step S5), the image area displayed on the display device 310 is changed to an image area corresponding to the operation, and if a zoom operation is not performed (N in step S5), it is determined whether or not a motion tracking setting is set (step S7).

This motion tracking setting can be set at the viewer's terminal 300 at will, for example, before the start of a live performance or during a live performance, depending on the type of device the viewer uses for operation and his/her level of proficiency in operation. Specifically, the motion tracking setting is set when, for example, the viewer is not familiar with operating the operation input device 321 such as a touch panel, keyboard, or game controller, or when the viewer is not familiar with changing the viewpoint by moving or moving the avatar, or when, as described below, the virtual participation position of the viewer's avatar is far from the performer avatar 1 and the viewer is zoomed in and displayed in an enlarged manner, so that even the slightest movement of the performer avatar 1 prevents the performer avatar 1 from being displayed properly, and the viewer is unable to change the viewpoint in response to the movement or movement of the performer avatar 1.

If there is no motion tracking setting (N in step S7), proceed to step S11, whereas if there is a motion tracking setting (Y in step S7), it is further determined whether the motion tracking condition is met (step S8).

In this embodiment 1, the motion tracking condition is a condition under which it is almost certain that the performer avatar 1 will no longer be displayed clearly on the display device 310. Specifically, if the head (or face) of the performer avatar 1 is within a specified range around the display area of the display device 310, it is highly likely that the movement or motion of the performer avatar 1 will cause the head (face) to move outside the display area, causing it to no longer be displayed clearly, and therefore the motion tracking condition is determined to be met.

In this way, in this embodiment 1, by determining that the motion tracking condition is met before the head (face) of the performer avatar 1 disappears from the display area, it is possible to significantly reduce the occurrence of the head (face) of the performer avatar 1 disappearing from the display area, but the present invention is not limited to this, and these motion tracking conditions may be the head (face) moving outside the display area, or other conditions may be used as the motion tracking conditions. In other words, any condition can be used as the motion tracking condition as long as it can prevent the performer avatar 1 from disappearing from the display area properly.

If the action-following condition is met (Y in step S8), after disabling the viewpoint-related operation (step S12), the viewpoint image displayed in the display area of the display device 310 is automatically changed to a motion-following viewpoint in which the head (face) of the performer avatar 1 is within a non-metrology range surrounded by a predetermined range around the display area, and the viewpoint image seen from the motion-following viewpoint is displayed in the display area of the display device 310 (step S13).

On the other hand, if the motion tracking condition is not met (N in step S8), it is further determined whether or not the viewpoint-related operation is disabled (step S9). If the viewpoint-related operation is disabled (Y in step S9), the disabled viewpoint-related operation is enabled (step S10) and the process proceeds to step S11. If the viewpoint-related operation is not disabled (N in step S9), the process proceeds to step S11 without going through step S10.

In step 11, since the motion tracking setting is not set or the motion tracking condition is not met, the viewpoint image of the viewer avatar continues to be displayed in the display area of the display device 310 (step S11).

Also, as mentioned above, if the flying performance period in which the performer avatar 1 virtually flies during a live performance has begun, step S1 is judged as Y and the process proceeds to step S21, where it is determined whether or not it is time to start the flying performance period (step S21).

If it is time to start the flying performance period (Y in step S21), the flying viewpoint video delivered from the delivery server computer 100 is displayed (step S22) in place of the viewer avatar's viewpoint video or the action-tracking viewpoint video that was displayed in the display area of the display device 310 before the start of the flying performance period, and then viewpoint-related operations are disabled (step S23) and the process returns to step S1.

In this embodiment 1, since the same flying viewpoint video is displayed on the viewer terminals 300 of all viewer users during the flying performance period, the flying viewpoint video is not generated on each viewer terminal 300 side like the viewer avatar viewpoint video that is different for each viewer user, but is generated on the distribution server computer 100 side and distributed to the viewer terminal 300. This allows the flying viewpoint video, which involves airborne movement of the viewpoint and therefore imposes a large rendering load on the viewpoint video, to be displayed well even on viewer terminals 300 with low processing capabilities, but the present invention is not limited to this, and these flying viewpoint videos may be generated on the individual viewer terminals 300 side, similar to the viewer avatar viewpoint video.

On the other hand, if it is not the start timing of the flying performance period (N in step S21), it is further determined whether it is the end timing of the flying performance period (step S24).

Whether or not it is the end timing of the flying performance period may be determined based on data indicating the end contained in the flying viewpoint video data transmitted from the distribution server computer 100, or may be determined based on the aforementioned performance status data that is different from the flying viewpoint video.

If it is not the end of the flying performance period (N in step S24), that is, if it is during the flying performance period, proceed to step S26, update the flying viewpoint video, and then return to step 1; on the other hand, if it is the end of the flying performance period (Y in step S24), enable viewpoint-related operations (step S25), proceed to step S11, display the viewer avatar's viewpoint video in the display area of display device 310 instead of the flying viewpoint video that was displayed in the display area of display device 310 during the flying performance period, and then return to step S1.

In this way, the viewer's viewpoint video control process shown in FIG. 20 is executed in the viewer's terminal 300 in this embodiment 1, and the various viewpoint videos displayed on the viewer's terminal 300 are explained using FIG. 16 to FIG. 19.

Figure 16 is a diagram showing the display contents of the display device 310 when a zoom operation is performed on the viewer terminal 300 of this Example 1. In the default state where no zoom operation is performed, a viewpoint image of the image area centered on the upper body of the performer avatar 1 is displayed as shown in Figure 16 (a). In this default state, when the viewer user performs a zoom-up operation, for example, by touching the "+" display (not shown) if the viewer terminal 300 is a smartphone P, or by operating the "+" key on the keyboard if the viewer terminal 300 is a computer (PC), a viewpoint image of the image area centered on the head (face) of the performer avatar 1 is displayed as shown in Figure 16 (b), for example.

On the other hand, if the viewer user performs a zoom-down operation, for example, by touching the "- " display (not shown) if the viewer terminal 300 is a smartphone P, or by operating the "- " key on the keyboard if the viewer terminal 300 is a computer (PC), then a viewpoint image of the image area including the entire body of the performer avatar 1 will be displayed, as shown in FIG. 16(c), for example.

Note that in Figure 16, in order to make the zoom function easier to understand, the explanation was given using the viewpoint image of virtual camera C1 rather than the viewpoint of the viewer avatar, but as shown in Figure 17, zooming up and down can also be performed on the viewpoint of the viewer avatar in the same way.

Although not described in detail in the viewer viewpoint image control process of FIG. 20, for example, when performer avatar 1 appears on virtual stage G for the first time when a live performance starts, the viewpoint of the viewer avatar is not directed toward performer avatar 1, and therefore there is a possibility that the period during which performer avatar 1 does not appear on the viewer terminal 300 will be long. Therefore, in this way, the period during which performer avatar 1 first appears may be set as an appearance performance period, and during this performance period, the viewpoint image displayed on the viewer terminal 300 may be automatically switched to the viewpoint image of virtual camera C1, etc., instead of the viewpoint image of the viewer avatar, and each viewer user may be allowed to perform zooming in, etc. on the viewpoint images of virtual camera C1.

Figure 17 shows the viewpoint image of viewer avatar 28, who is virtually participating in the fourth area, as an example of the viewpoint image of a viewer avatar. As shown in Figure 17, the viewpoint image of viewer avatar 28 includes the heads of viewer avatar 13 and viewer avatar 14, who are virtually participating in the third area, and the image shows performer avatar 1 seen through the heads of viewer avatar 13 and viewer avatar 14.

Therefore, the viewer user of viewer avatar 28 can, for example, perform a zoom-in operation to change the image area so that the upper body of performer avatar 1 is displayed at the center, as shown in FIG. 17(b), thereby preventing viewer avatar 13 and viewer avatar 14 from being displayed and allowing the viewer user to enjoy the live performance.

However, in the zoomed-in state shown in FIG. 17(b), for example, as the live performance progresses and the performer avatar 1 begins to move and make large movements, if the viewer user is unable to properly change the viewpoint of the viewer avatar 28 in accordance with the movements of the performer avatar 1, the performer avatar 1 may not be displayed properly, as shown in FIGS. 18(d)-(f). In that case, if the viewer user has to pay attention to changing the viewpoint, it may become difficult to concentrate on the live performance, and the viewer may not be able to fully enjoy the live performance.

However, in this embodiment 1, when the viewer user sets the motion-following setting as described above, as shown in Figures 18(a) to (c), the head of the performer avatar 1 enters within a specified peripheral area of the display area, the motion-following condition is met, the viewpoint is automatically changed to the motion-following viewpoint, and the viewpoint image of the motion-following viewpoint is displayed. This means that the viewer user does not need to pay attention to the viewpoint operation, and can concentrate on the live performance and enjoy it to the fullest.

As the live performance progresses and the flying performance period begins, as shown in Figure 19, the viewpoint images of the individual viewer avatars are automatically changed to flying viewpoint images seen from a virtual camera moving in the air, without any operation by the viewer user.

Specifically, before the performer avatar 1 begins the virtual flight, an image is displayed in which a wing object grows on the back of the performer avatar 1. Then, as the performer avatar 1 looks up at a virtual aerial star object 50 that is virtually placed in the second area of the virtual live venue, the viewpoint of the viewer avatar also moves in a similar manner. At this time, light effects 61 appear around the performer avatar 1.

After that, when the performer avatar 1 starts virtual flight, the viewpoint image displayed on the display device 310 automatically switches from the viewpoint image of the viewer avatar up to FIG. 19(b) to a flying viewpoint image seen from a virtual camera moving in the air, as shown in FIG. 19(c).

During the flying performance period, the airborne virtual camera moves through the second area based on a preset aerial trajectory as described above, and the flying viewpoint image seen from this airborne moving virtual camera is displayed on the display device 310 as shown in FIG. 19(d).

<Modification of the First Embodiment>
In the above-mentioned first embodiment, the general action (first action) is an action in which the performer avatar 1 stands without moving to the virtual stage G, the special action (second action) is an action in which the performer avatar 1 performs a virtual flight, and the specific action (third action) is an action in which the performer avatar 1 moves rapidly in the lateral direction. However, the present invention is not limited to this, and the specific action may be an action according to the content to be distributed. For example, the general action (first action) may be an action (normal action) that the performer avatar performs for the longest period in the distributed content, the special action (second action) may be an action that is performed in the shortest special period in the distributed content or the least frequently performed action, and the specific action (third action) may be an action that is performed for a shorter period than the general action (first action) but for a longer period than the special action, or an operation that is performed less frequently than the general action (first action) but more frequently than the special action. Note that the special action (second action) may include the specific action (third action).

In addition, in the above-mentioned Example 1, an embodiment is exemplified in which there is both a function of switching the viewpoint corresponding to a special action (second action) and a function of following a specific action (third action), but the present invention is not limited to this, and the virtual space content delivery system may have only one of these functions.

In the above-mentioned Example 1, an example was given of a form in which the viewpoint of the viewer user is changed and followed in accordance with the movements of the performer avatar, but the present invention is not limited to this. For example, the change and following of the viewpoint may be performed in accordance with specific effects such as smoke or fireworks (pillars of fire) devices, special lighting, and deformation of the virtual stage G, which are effects on the virtual stage G.

Next, the virtual space content distribution system of the second embodiment will be described below with reference to Figs. 21 to 28. Note that the same components as those of the first embodiment will be denoted by the same reference numerals and the description will be omitted.

The configuration of the virtual space content delivery system of Example 2 is the same as that of Example 1 shown in FIG. 1, but the progress of the virtual live performance is different, and the period during which the viewpoint images from the virtual cameras C1 to C4 and the airborne moving virtual camera (hereinafter collectively referred to as virtual camera viewpoint images) are delivered and the entity that generates these virtual camera viewpoint images are different from those of Example 1, which is a characteristic feature of Example 2.

Fig. 21 is a diagram showing the progress of a virtual live performance distributed in the virtual space content distribution system of Example 2. In the virtual live performance of Example 2, a "live virtual space participation period" and a "pre-performance speech" period are set as periods before the start of the virtual live performance, and during the live virtual space participation period, viewer users can virtually participate in the virtual live performance venue, which is the virtual space that is the target of the virtual live performance.

The period up to the pre-data download completion check timing, which is set just before the end of the "pre-show speech" period, is the pre-data download (DL) period, during which virtual space data (basic data) of the virtual live venue, avatar data of performer avatar 1, and avatar data of other viewer users who have participated virtually are downloaded (DL) as pre-data from the distribution server computer 100 to the viewer terminal 300 of the viewer user who has participated virtually during the period during which participation in the live virtual space is possible.

In addition, at the pre-data DL completion check timing, which is the end timing of the pre-data download (DL) period, it is checked whether the pre-data download (DL) has been completed at each viewer terminal 300.

During the "pre-show introduction" period, explanations about the virtual live show are given through video footage, etc. Therefore, by making the "pre-show introduction" period a period during which advance data can be downloaded (DL), each viewer terminal 300 can also download (DL) from the distribution server computer 100 the avatar data of viewer avatars who virtually participate in the live virtual space just before the end of the participation period.

As shown in FIG. 21, the virtual live performance of Example 2 progresses as follows: "Opening performance" → "First MC" → "First performance" → "Second MC" → "Second performance" ... "Opening performance" is a performance in which performer avatar 1 appears on virtual stage G. In the "First MC", performer avatar 1 appears on virtual stage G, greets the audience through talk, and introduces the first performance to be performed first. The "First performance" is a performance that includes singing and dancing by performer avatar 1 (performer), and also includes a flying performance in the middle of the performance, in which the performer avatar 1 virtually flies through the virtual live venue and is virtually captured by a virtual camera moving in the air. In the "Second MC", performer avatar 1 (performer) mainly introduces the second performance through talk. The "second performance" includes singing and dancing by performer avatar 1 (performer), as well as a costume change performance using footage virtually captured by virtual camera C1 of the performer avatar 1 changing costumes, and an avatar splitting into multiple people using footage virtually captured by virtual camera C1.

The processing load on the distribution server computer 100 during each period in which these effects and performances are performed is the lowest at "low" during the period in which participation in the live virtual space is possible, while it is the highest at "high" during the period in which the special effects "flying effect", "costume change effect", and "double-ego effect" are performed. The processing load on the distribution server computer 100 during the period of the "first performance" from the "introduction", "opening effect", "first MC", and "flying effect" is "medium", and the processing load is also "medium" during the period of the "first performance" after the "flying effect", the period of the "second performance" until the "second MC" and "costume change effect", the period of the "second performance" from the "costume change effect" to the "double-ego effect" is also "medium", and the period of the "second performance" after the "double-ego effect" is also "medium".

Next, regarding each period in the virtual live in Example 2, the viewpoint images that can be viewed on the viewer terminal 300 will be explained. During the period during which participation in the live virtual space is possible, from virtual participation until the start of the "pre-show speech," only the viewer avatar viewpoint images can be displayed.

When the "pre-show speech" begins, only the virtual camera viewpoint image, specifically the viewpoint image of virtual camera C1, can be displayed. In this way, the viewpoint image during the "pre-show speech" is limited to the virtual camera viewpoint image even though it is before the start of the virtual live performance. This is because the pre-show speeches also include explanations of points to note about the virtual live performance, and the purpose is to prevent viewer users from not viewing the pre-show speech taking place on virtual stage G due to the viewer avatar viewpoint image being displayed on the viewer terminal 300.

As described below, these virtual camera viewpoint images of the "introduction" are generated by the distribution server computer 100 and distributed to each viewer terminal 300, so they can be displayed even on viewer terminals 300 that have not yet completed downloading the advance data.

For viewer terminals 300 for which the download of advance data has been completed at the above-mentioned advance data DL completion check timing, during the period of the "first performance" up to the "first MC" and the "flying performance", as well as the period after the "flying performance", it is possible to display viewer avatar viewpoint video in addition to virtual camera viewpoint video.

On the other hand, for viewer terminals 300 that have not completed downloading the advance data at the above-mentioned advance data DL completion check timing, the viewer avatar viewpoint video cannot be generated accurately, so the virtual camera viewpoint video generated by the distribution server computer 100 is distributed, making it possible to display only the virtual camera viewpoint video during the entire virtual live performance, and it is not possible to display the viewer avatar viewpoint video.

In addition, among the images of the "Flying Performance", "Costume Change Performance", and "Double Ego Performance", which have a "high" processing load on the distribution server computer 100, the virtual camera viewpoint images of virtual camera C1 for the "Costume Change Performance" and "Double Ego Performance", like the virtual camera viewpoint images during the second performance period other than these "Costume Change Performance" and "Double Ego Performance", are generated on each viewer terminal 300 rather than on the distribution server computer 100, thereby reducing the processing load on the distribution server computer 100 for generating the virtual camera viewpoint images of virtual camera C1 for these "Costume Change Performance" and "Double Ego Performance".

The "flying performance" period is also a period in which the processing load on the distribution server computer 100 is high, so it is considered that the "flying performance" is generated on the viewer terminal 300, similar to the "costume change performance" and "doppelganger performance". In this case, detailed trajectory data of the movement trajectory of the virtual camera moving in the air within the virtual live venue must be distributed in advance to each viewer terminal 300 to generate the video. Therefore, in Example 2, the "flying performance" video is generated and distributed on the distribution server computer 100 side, but the present invention is not limited to this, and the "flying performance" video may also be generated on each viewer terminal 300 in order to reduce the processing load on the distribution server computer 100.

Here, we briefly explain the participant user table used in the distribution server computer 100 of Example 2 using Figure 23.

The participant user table is a data table used to identify the participation status of viewer users in a virtual live event and the download status of advance data for each virtual live event registered in the event data shown in FIG. 7, and as shown in FIG. 23, it is a table that can store data such as participation identification data (participation identification flag) that can identify the participation status of the participant (viewer user) in association with a participant ID (viewer user ID), the participation area of the participant (viewer user), and advance data DL identification data (advance data DL identification flag) that can identify the download status of advance data.

In addition, the participant ID field will contain all the viewer user IDs of the people who are expected to participate and are described in the file for each participant group stored in the event data in association with the virtual live.

When a participant ID (viewer user ID) is registered, the participation identification data (participation identification flag) is registered as "0", which corresponds to a non-participating state, and is updated to "1", indicating participation, when the participation of each participant (viewer user) is permitted by authentication. Therefore, the participation identification data (participation identification flag) makes it possible to identify, among the viewer users who are scheduled to participate, those who are participating and those who were unable to participate for some reason.

The participation area data is data that can identify whether the participation area of each participant (viewer user) is the third area or the fourth area, and is registered when the participation of each participant (viewer user) is permitted by authentication.

The pre-data DL identification data (pre-data DL identification flag) is updated based on a check as to whether the download (DL) of the pre-data is complete at the pre-data DL completion check timing described above, and if it is complete, it is updated from "0", indicating incomplete, to "1", indicating complete. Therefore, by using this pre-data DL identification data (pre-data DL identification flag), it is possible to easily identify all participants (viewer users) who have not completed the download (DL) of their pre-data among the participants (viewer users) who are virtually participating.

Next, the flow of downloading advance data by a viewer user virtually participating in a virtual live performance venue in Example 2 will be described with reference to FIG. 22.

First, the viewer user performs an operation to join the virtual live event that he or she plans to virtually participate in, such as selecting a participation menu or operating a specific key on the keyboard to which the participation operation is assigned, in the viewer program (viewing application) running on the viewer terminal 300.

By executing this participation operation, a participation request including the viewer user ID and authentication data for authenticating the viewer user is sent from the viewer terminal 300 to the distribution server computer 100.

The distribution server computer 100 that receives the participation request executes a participant authentication process on the condition that it is a period during which participation in the live virtual space is possible. In the participant authentication process, it is determined whether the viewer user is a prospective participant of the corresponding virtual live depending on whether the viewer user ID included in the received participation request is registered as a participant ID in the participant user database corresponding to the virtual live to be participated in, and the viewer user is authenticated by comparing the authentication data described in the authentication information file identified from the authentication information file data stored in association with the viewer user ID in the viewer user data shown in FIG. 9 with the authentication data included in the participation request.

If the viewer user is a prospective participant and has completed authentication using the authentication data, the participant identification data (participation identification flag) in the participant user table is updated to "1", which corresponds to "participation".

Then, the distribution server computer 100 transmits (responds to) advance data DL instruction information to the viewer terminal 300 that sent the participation request, in order to have the viewer terminal 300 download advance data corresponding to the virtual live event to be participated in. By receiving the advance data DL instruction information, the viewer terminal 300 can confirm that participation has been permitted.

In response to receiving the advance data DL instruction information, the viewer terminal 300 identifies the advance data to be downloaded and transmits an advance data DL request to the distribution server computer 100, requesting the download of the identified advance data.

In response to receiving this advance data DL request, the distribution server computer 100 executes the advance data transmission process.

In this advance data transmission process, the requested advance data is transmitted to the viewer terminal 300 that sent the advance data DL request. As mentioned above, the advance data includes virtual space data (basic data) of the virtual live venue, avatar data of the performer avatar 1, and avatar data of other viewer users who are already participating virtually.

In addition, in the advance data transmission process, the avatar data of the viewer user who sent the advance data DL request is additionally distributed as advance data to the viewer terminals 300 other than the viewer terminal 300 that sent the advance data DL request, but of viewer users whose participation has already been confirmed by the participant data table. As a result, the avatar data of all viewer users who are virtually participating in the virtual live show is distributed in advance to and stored in the viewer terminals 300 of each viewer user who is virtually participating in the virtual live show, along with the virtual space data (basic data) and avatar data of the performer avatars.

Then, at the pre-data DL completion check timing described above (see FIG. 21), it is checked whether or not all of the pre-data has been downloaded, and if the pre-data download has been completed, the pre-data DL identification data (pre-data DL identification flag) in the participant user table is updated from "0", indicating incomplete, to "1", indicating complete.

As described above, for participants (viewer users) whose advance data download has not been completed and whose advance data DL identification data (advance data DL identification flag) remains at "0," indicating incomplete download, only the virtual camera viewpoint image can be displayed.

Next, we will use Figures 24 to 27 to explain how the subject generating the various viewpoint videos displayed on the viewer terminal 300 changes during each period of the virtual live broadcast.

First, during the live virtual space participation period when only the viewer avatar viewpoint video can be displayed on the viewer terminal 300, as shown in FIG. 24, the virtual space data already stored in the viewer terminal 300 is updated sequentially by receiving the advance data based on the virtual space update data transmitted from the distribution server computer 100, including the performer avatar action data for generating the actions of the performer avatar 1 that perform actions linked to the actions of the performer user, and the viewer viewpoint video generation process is executed in the viewer terminal 300 based on the updated virtual space data to generate the viewer avatar viewpoint video, which is output (displayed) by the video output process. Note that although the performer terminal 200 is omitted in FIG. 24, the virtual space update data is also transmitted to the performer terminal 200 as shown in FIG. 10 in Example 1.

Furthermore, during each of the periods of "Pre-show", "Opening performance", and "Flying performance" in which only virtual camera viewpoint images can be displayed on the viewer terminals 300, as shown in FIG. 25, the distribution server computer 100 transmits virtual space update data only to the performer terminals 200 in the virtual space update data distribution process. Then, following the virtual space update data distribution process, a virtual camera viewpoint image generation process is executed in which a virtual camera viewpoint image is generated, and a virtual camera viewpoint image data distribution process is executed in which the image data of the virtual camera viewpoint image generated in the virtual camera viewpoint image generation process is distributed to all viewer terminals 300 participating in the virtual live.

The virtual camera viewpoint video data distributed from the distribution server computer 100 in these virtual camera viewpoint video data distribution processes is displayed by executing a video output process on the viewer terminal 300 that receives the virtual camera viewpoint video data.

In other words, as shown in FIG. 27, during the periods of "pre-show," "opening performance," and "flying performance," in which only virtual camera viewpoint images can be displayed, the virtual camera viewpoint images are generated by the distribution server computer 100.

In addition, during the performance period, excluding the MC period and the flying performance period, during which the viewer terminal 300 can select and display either the viewer avatar viewpoint image or the virtual camera viewpoint image, as shown in FIG. 26, the distribution server computer 100 executes a virtual space update data distribution process, and virtual space update data is distributed to the viewer terminal 300 of each viewer user participating in the virtual live performance, thereby executing the virtual space update process on the viewer terminal 300.

Then, in the viewer terminal 300, it is determined whether or not the viewer avatar viewpoint video has been selected as the video to be displayed. If the viewer avatar viewpoint video has been selected, the viewer avatar viewpoint video is generated by executing the viewer viewpoint video generation process, similar to the flow shown in FIG. 24, and the generated viewer avatar viewpoint video is displayed by the video output process.

On the other hand, when the viewer avatar viewpoint video is not selected as the video to be displayed, i.e., when the virtual camera viewpoint video is selected as the video to be displayed, the virtual camera viewpoint video generation process is executed in the viewer terminal 300 to generate the virtual camera viewpoint video, and the generated virtual camera viewpoint video is displayed by the video output process.

The viewer user may be able to select which of the virtual cameras C1, C3, or C4 the virtual camera viewpoint image generated in this virtual camera viewpoint image should be, or the distribution server computer 100 may instruct all viewer terminals 300.

In other words, as shown in FIG. 27, during the performance period excluding the MC period and the flying performance period when either the virtual camera viewpoint video or the viewer avatar viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated on the viewer terminal 300. In this way, during the period when either the virtual camera viewpoint video or the viewer avatar viewpoint video can be selected and displayed on the viewer terminal 300, switching between the virtual camera viewpoint video and the viewer avatar viewpoint video is performed at the viewer user's choice, so that switching between these videos can be performed without problems such as video interruption and while preventing an increase in the processing load on the distribution server computer 100 that accompanies switching between these videos, compared to when the virtual camera viewpoint video data is displayed by distribution from the distribution server computer 100.

Meanwhile, during each of the periods of "pre-show", "opening performance", and "flying performance", when only the virtual camera viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated only on the distribution server computer 100 and distributed to each viewer terminal 300. In this way, during the period when only the virtual camera viewpoint video can be displayed on the viewer terminal 300, the virtual camera viewpoint video is generated only on the distribution server computer 100, so that it is possible to prevent unnecessary increases in the processing load of the entire system caused by the process of generating the virtual camera viewpoint video being executed redundantly on the viewer terminal 300.

Here, the situation when the viewpoint video is switched on the viewer terminal 300 will be explained with reference to FIG. 28, taking as an example the period of "flying effect", during which only the virtual camera viewpoint video can be displayed.

For example, as shown in FIG. 28, on a viewer terminal 300 on which viewer avatar viewpoint video is displayed during the "first MC" period and the first performance period up to the "flying performance", if the distribution server computer 100 starts distributing the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from the start of the "flying performance" period in which only the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) can be displayed, the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will not be displayed on the viewer terminal 300 during the period from the start of these distributions until the virtual camera viewpoint video is displayed, which could result in a problem in which the viewer is unable to view the first video of the "flying performance".

For this reason, as shown in FIG. 28, at the start of a "flying performance" in which the viewpoint image is forcibly changed from the viewer avatar viewpoint image to the virtual camera viewpoint image (viewpoint image from a virtual camera moving in the air), the distribution server computer 100 of Example 2 transmits a BF advance notification to the viewer terminal 300 at a specific timing prior to the start of the "flying performance", thereby notifying the viewer terminal 300 of the distribution start (buffering start) timing for starting advance distribution of the virtual camera viewpoint image (viewpoint image from a virtual camera moving in the air) before the start of the flying performance period.

In response to receiving the BF advance notification, the viewer terminal 300 identifies the timing at which distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will begin from the distribution server computer 100, and begins buffering the received virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from that timing.

The timing at which distribution of these virtual camera viewpoint images (viewpoint images from virtual cameras moving in the air) begins, i.e., the timing at which buffering begins, can be determined appropriately taking into account differences in the image storage capabilities of the viewer terminals 300, but can be set to a period equivalent to several tens of frames of image, for example.

In this way, the distribution server computer 100 of Example 2 starts generating and distributing virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) of the flying performance before the flying performance period starts, and the viewer terminal 300 buffers the distributed virtual camera viewpoint video, so that when the flying performance period starts, the video can be instantly changed from the viewer avatar viewpoint video that was displayed before the flying performance period started to the virtual camera viewpoint video, thereby preventing the occurrence of a problem in which the viewer is unable to view the initial video of the "flying performance".

Furthermore, when the period of the "flying performance" ends, because the viewer avatar viewpoint video was selected before the start of the "flying performance", the viewpoint video switches from the flying performance's virtual camera viewpoint video (viewpoint video from the virtual camera moving in the air) to the viewer avatar viewpoint video when the period of the "flying performance" ends. However, even in this case, if the viewer avatar viewpoint video is generated and displayed from the time when the period of the "flying performance" ends, there is a risk that a malfunction will occur in which the viewer avatar viewpoint video is not displayed on the viewer terminal 300 during the period until the viewer avatar viewpoint video is generated and displayed, just as at the start of the "flying performance".

For this reason, even when the period of these "flying effects" ends, the distribution server computer 100 of Example 2 notifies the viewer terminal 300 in advance of the end timing of the "flying effects" period when distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) will end, by sending an end TM (timing) advance notification to the viewer terminal 300 as shown in FIG. 28.

In response to receiving the end TM (timing) advance notification, the viewer terminal 300 identifies in advance the timing at which the distribution of the virtual camera viewpoint video (viewpoint video from a virtual camera moving in the air) from the distribution server computer 100 will be stopped, and starts the viewer viewpoint video generation process at a specified timing prior to that timing, thereby pre-generating the viewer avatar viewpoint video.

The distribution server computer 100 starts distributing the virtual space update data from the time of sending the end TM (timing) advance notification, and the viewer terminal 300 executes virtual space processing to update the virtual space data using the received virtual space update data, so that at the specified timing mentioned above, a viewer avatar viewpoint image that reflects the situation of the virtual live venue at that time can be generated.

In this way, the distribution server computer 100 of Example 2 transmits an end TM (timing) advance notification to the viewer terminal 300 before the end of the flying performance period, enabling the viewer terminal 300 to generate the viewer avatar viewpoint video in advance before the end of the flying performance period, thereby preventing the viewer terminal 300 from failing to display the viewer avatar viewpoint video.

Note that in Figure 28, the "flying performance" has been explained as an example, but similar control is also performed during the "pre-show" and "opening performance" periods, when the viewpoint image is forcibly changed from the viewer avatar viewpoint image to the virtual camera viewpoint image.

As described above, in the virtual space content distribution system of Example 2, there are multiple viewer users, and each viewer user is assigned a viewer terminal 300. When the viewer terminal 300 displays the viewer avatar viewpoint video, which is a viewpoint video of the virtual live venue (virtual space) corresponding to the viewer avatar, the viewer avatar viewpoint video is generated based on the virtual space update data (virtual space data) transmitted from the distribution server computer 100. Since the generation of these viewer avatar viewpoint videos, which have a large processing load, is performed by the viewer terminal 300, more viewer users can participate in the virtual live.

The virtual space content of Example 2 is content that allows a performer user to participate in a virtual live venue as performer avatar 1, and is content in which performer avatar 1 performs musical actions (performances) such as singing and dancing in the virtual live venue, thereby enhancing the interest of the virtual space content.

In addition, since the virtual space update data (virtual space data) of Example 2 includes performer avatar movement data for generating the movements of performer avatar 1, the movements of performer avatar 1 can be accurately reflected in the viewer avatar viewpoint image generated in the viewer terminal 300.

In addition, the common viewpoint video displayed in common on each viewer terminal 300 in Example 2, such as the "pre-show speech," "opening performance," and "early performance" videos, is generated in the distribution server computer 100 and distributed to each viewer terminal 300, so that it is possible to prevent unnecessary increases in the processing load of the entire system due to duplicate processing being executed on each viewer terminal 300.

In addition, in the viewer terminal 300 of Example 2, during the performance period (specific period) other than the "MC period" and "flying performance" periods, during which either the viewer avatar viewer's ...

In addition, during special periods such as "costume change performance" and "doppelganger performance" in which the processing load on the distribution server computer 100 in Example 2 is high, each viewer terminal 300 also generates and displays virtual camera viewpoint images (common viewpoint images), thereby preventing the processing load on the distribution server computer 100 from temporarily becoming extremely high.

In addition, the distribution server computer 100 of Example 2 transmits advance data including basic data of the virtual live venue (virtual space) and basic data of the performer avatar 1 and viewer avatars who will virtually participate in the virtual live venue to each viewer terminal 300 before the virtual live starts, thereby preventing the occurrence of a problem in which the viewer avatar's viewpoint image is not displayed properly at the start of the virtual live.

In addition, the distribution server computer 100 of Example 2 transmits virtual camera viewpoint video (common viewpoint video) to viewer terminals 300 for which it cannot confirm that they have completed receiving the advance data. In this way, it is possible to prevent the processing load on the distribution server computer 100 from increasing excessively due to the transmission of basic data, etc. after the start of the virtual live broadcast, while also preventing viewer users from being unable to watch the virtual live broadcast.

<Modification of the second embodiment>
FIG. 29 is a flowchart showing the process of generating a viewer's viewpoint video in the modification 2-1 of the second embodiment.

In the above-mentioned Example 2, a form was shown in which the viewer avatar viewpoint images including the movements of the performer avatar 1 were always generated uniformly, but the present invention is not limited to this, and these viewer avatar viewpoint images may be generated in multiple forms with different processing loads in the image generation, depending on the selection of the viewer user at the viewer terminal 300.

In other words, as shown in Example 1, the viewer terminal 300 used by the viewer user may be a stationary personal computer (PC) with high processing power, or a smartphone terminal or tablet terminal with relatively lower processing power than a personal computer (PC). It is assumed that terminals with different processing powers will be used. If the viewer avatar viewpoint video is always generated uniformly on terminals with different processing powers, the viewer avatar viewpoint video cannot be generated properly on terminals with low processing power, and there is a risk that a mismatch will occur between the viewer avatar viewpoint video and the audio, or the viewer avatar viewpoint video will become a discontinuous and unnatural video. For example, when generating a viewer avatar viewpoint video including performer movements, the viewer may be allowed to select, as appropriate, according to the processing power of the terminal he or she uses, whether to generate a viewer avatar viewpoint video including high-definition performer movements, which requires a high processing load for generating the video, or to generate a viewer avatar viewpoint video including low-definition performer movements, which requires a low processing load for generating the video.

Specifically, the viewer viewpoint video generation process executed on the viewer terminal 300 may be as shown in the flow diagram of FIG. 29. In the viewer viewpoint video generation process, first, the performer movement generation mode selected by the viewer user on the viewer terminal 300 is identified (step S101). Then, it is determined whether the identified performer movement generation mode is a high processing load mode (step S102). If the high processing load mode is selected (Y in step S102), a viewer avatar viewpoint video including high-definition performer movements is generated (step S103). On the other hand, if the high processing load mode is not selected (N in step S102), a viewer avatar viewpoint video including low-definition performer movements is generated (step S104).

Note that, although FIG. 29 illustrates an example of two modes, a high processing load mode and a low processing load mode, the present invention is not limited to this, and three or more processing load modes may be used.

In addition, in the above-mentioned Example 2 (similar to Example 1), an example is given in which all actions, including the movement of the viewer avatar of the viewer user, are reflected in each viewer avatar by distributing virtual space update data, but the present invention is not limited to this. For example, in a case where the actions of these viewer avatars are actions that are normally not executable but are possible when a special fee is paid, for example, an operation to perform a special action that allows the performer avatar 1 to appeal to the performer avatar's own viewer avatar, the special action is an action directed at the performer user and not at other viewer users, so that the virtual space update data including these special actions is not distributed to the viewer terminals 300 of other viewer users, and it is possible to prevent an excessive increase in the processing load of the distribution server computer 100 by making these special actions executable.

In this way, if virtual space update data including special actions is not distributed to the viewer terminals 300 of other viewer users, as shown in variant example 2-2 of FIG. 30, when viewer user A performs an operation corresponding to a special action, causing viewer user A's viewer avatar to perform a special action, virtual space update data A corresponding to viewer user A's viewer avatar performing a special action is transmitted from the distribution server computer 100 to viewer user A's viewer terminal 300, but virtual space update data B in which viewer user A's viewer avatar is not performing a special action is distributed to the viewer terminal 300 of viewer user B, who is different from viewer user A.

As a result, different virtual space update data is delivered from the delivery server computer 100 to the viewer terminal 300 of viewer user A and the viewer terminal 300 of viewer user B, which increases the degree of freedom of the virtual space update data and makes it possible to generate different viewer avatar viewpoint images on the viewer terminal 300 of viewer user A and the viewer terminal 300 of viewer user B.

In addition, viewer terminal 300 of viewer user A can generate and display an image of viewer avatar of viewer user A performing a special action different from the actions that viewer avatar of viewer user B is capable of performing, based on virtual space update data A that is different from virtual space update data B distributed to viewer terminal 300 of viewer user B, thereby increasing viewer user A's interest in the virtual live performance, which is virtual space content.

In addition, in the above-mentioned Example 2 (which is also the same as Example 1), the viewer avatars of the viewer users virtually participating in the fourth area are illustrated as normal avatars in which the avatar appearance differs for each viewer user, but the present invention is not limited to this. For example, in order to reduce the processing load when generating various viewpoint images in a virtual live performance, the viewer avatars of the viewer users virtually participating in the fourth area may be simplified avatars in which the avatar appearance is simplified.

Note that FIG. 30 illustrates an example of a special action that becomes possible when a special fee is paid, but for example, in the case of using a simplified avatar as described above, normal actions other than the movement of the simplified avatar, such as waving, clapping, and changing the direction of the viewer avatar, may be prevented from being included in the virtual space update data distributed to other viewer users, just like special actions, thereby further reducing the processing load on the distribution server computer 100.

Next, the virtual space content distribution system of the third embodiment will be described below with reference to Figs. 31 to 41. Note that the same components as those of the first and second embodiments will be given the same reference numerals and will not be described.

The configuration of the virtual space content distribution system of Example 3 is the same as that of Examples 1 and 2 shown in FIG. 1, but the progress of the virtual live performance differs from that of Examples 1 and 2, and the chemilight action (see FIG. 33) of swinging a rod-shaped chemical light K (hereinafter abbreviated as chemilight K) that emits light by mixing the loaded medicine can be performed, which is a content-compatible action that is not executable as a viewer avatar action in Examples 1 and 2. The control related to these chemilight actions differs from that of Examples 1 and 2, which is a characteristic feature of Example 3.

Figure 31 is a diagram showing the progress of a virtual live performance distributed in a virtual space content distribution system of Example 3. In the virtual live performance of Example 3, a "live virtual space participation period" and a "pre-show speech" period are set as periods before the start of the virtual live performance, and viewer users can virtually participate in the virtual space that is the subject of the virtual live performance during the live virtual space participation period.

The period from the live virtual space participation period to the end of the advance data download (not shown), which is set just before the end of the "pre-show speech" period, is the advance data download (DL) period, as in Example 2, and the virtual live venue virtual space data (basic data), avatar data of performer avatar 1, and avatar data of other viewer users who participated virtually during the live virtual space participation period are downloaded (DL) as advance data from the distribution server computer 100 to the viewer terminal 300 of the viewer user who participated virtually during the live virtual space participation period.

In addition, during the "pre-show" period, explanations about the upcoming virtual live show will be provided through video footage, etc., just as in Example 2.

As shown in FIG. 31, the virtual live performance in Example 3 progresses as follows: "Opening performance" → "First MC" → "First performance" → "Second MC" → "Second performance" → "Third MC" → "Third performance", etc. The "Opening performance", "First MC", and "Second (3rd) MC" are the same as those in Example 2. The "First performance", "Second performance", "Third performance", etc. include singing and dancing by performer avatar 1 (performer).

Just before the end of each period of the "First MC," "Second MC," and "Third MC," the execution timing of the delay action determination process (see FIG. 37) for determining the participant (viewer user) who will perform the delay action in the chem-light action is set, so that a different participant (viewer user) is determined as the participant (viewer user) who will perform the delay action in each of the "First Performance," "Second Performance," "Third Performance," etc.

As shown in FIG. 31, the actions that the performer avatar 1 can perform during each period in which each performance or show is performed in the virtual live show of Example 3 include normal actions of moving parts of the avatar such as the hands, feet, and head, as well as actions of moving around the virtual stage G and talking during the opening performance in which the performer avatar 1 appears. Note that in FIG. 31, normal actions that the performer avatar 1 and the viewer avatars are capable of are omitted due to space limitations.

In addition, during the MC periods of "First MC", "Second MC", "Third MC", etc., in addition to normal actions, actions related to moving on the virtual stage G and talking can be performed, as in the "Opening performance", and during the performance periods of "First performance", "Second performance", "Third performance", etc., in addition to normal actions, singing and dancing can be performed. Note that dancing actions may include actions of moving on the virtual stage G.

In each period of the virtual live in Example 3, the actions that can be performed by the viewer avatar of a participant (viewer user) who virtually participates in the virtual live venue include normal actions of moving parts of the avatar such as hands, feet, and head, as well as asynchronous chemilight actions in which the timing and speed of waving the chemilight K of individual participants (viewer users) are not synchronized, as performed by viewer avatar 12 and viewer avatar 13 in FIG. 32, during the period before the live starts (live virtual space participation period and opening pre-show period), the opening performance period, and each MC period, and actions of moving within the virtual live venue, as performed by viewer avatar 11 and viewer avatar 14 in FIG. 32(A). Note that in FIG. 32, the viewer avatar participating in the fourth area is shown in a simplified manner for convenience, but is an avatar similar to viewer avatars 11 to 14 and is capable of performing the same actions as viewer avatars 11 to 14.

As described below, the non-synchronized chemilight action is the action of shaking the chemilight K from side to side at any speed (cadence) of speed 1 to speed 5 that is individually selected as the default by the viewer user on each viewer terminal 300.

In addition to normal actions, the viewer avatars of participants (viewer users) virtually participating in each performance period of "First Performance", "Second Performance", "Third Performance", etc. can perform synchronized chemilight actions, such as the actions of viewer avatars 11 to 14 as shown in FIG. 32(B), in which the timing and speed of the viewer avatars of each participant (viewer user) shaking the chemilight K are synchronized based on various information (see FIG. 33) transmitted from the distribution server computer 100. In other words, in Example 3, during each performance period in which the performer avatar is performing a performance, the movement of the viewer avatars during each performance period is prohibited, thereby preventing the movement of these viewer avatars during each performance period from having a negative impact on the virtual live performance, for example, preventing the viewing of other viewer avatars from being hindered due to collisions with other viewer avatars, or preventing the performers from being able to concentrate due to such movements.

The Chemilight action is not an action that can be performed by the viewer avatars of all participants (viewer users) who virtually participate in the virtual live, but is only an action that can be performed by the viewer avatars of participants (viewer users) who have purchased the Chemilight K virtual item in advance. Therefore, even if a viewer avatar is virtually participating in the virtual live, if the viewer avatar does not possess Chemilight K, the viewer avatar cannot perform the Chemilight action, and therefore there will be viewer avatars that do not perform the Chemilight action during each performance period.

Using Figures 40 and 41, we will explain the situation in which movement, which is an action of the viewer avatar, is prohibited on the personal computer (PC) that the viewer user is using as the viewer terminal 300.If the viewer user performs an operation to display the action menu during the period before the start of a live performance or during each MC period when the performer avatar 1 is not performing, the avatar action operation process is executed on the viewer terminal 300, and the action menus for "Move" and "Chemical Light" are displayed in an active (enabled) state, as shown in Figure 40 (B), thereby notifying the viewer that both the actions of "Move" and "Chemical Light" can be performed, and the viewer user can have the viewer avatar perform either the action of "Move" or "Chemical Light" by selecting and inputting.

In addition, in Figures 40 and 41, in order to simplify the explanation, items such as normal operations other than the "Move" and "Chemilite" operations have been omitted, and it goes without saying that items other than these "Move" and "Chemilite" items may be included.

On the other hand, if a viewer user performs an operation to display the action menu during the performance period in which performer avatar 1 is performing, as described above, movement is prohibited, and as a result, the avatar action operation process displays only the "Chemilite" action menu in an active (enabled) state, as shown in FIG. 41, while the "Move" action menu is displayed in an inactive (disabled) state, allowing the viewer user to cause the viewer avatar to perform the "Chemilite" action by selecting and inputting, but not the "Move" action.

These action restrictions on the viewer avatar by controlling the display of the action menu may be performed by executing an avatar action operation process on the viewer terminal 300 side, and judging whether the current time is during the performance period or not based on BPM information and start of swing information. Alternatively, the distribution server computer 100 may generate action restriction information for each action of the viewer avatar at each point during the virtual live performance, which can specify whether the action can be performed or not, and distribute this information to each viewer terminal 300, so that the viewer terminal 300 can perform action restrictions based on the action restriction information. Alternatively, instead of transmitting this action restriction information, the viewer terminal 300 may temporarily accept the action corresponding to the action, and the distribution server computer 100 may determine whether the accepted action is valid or invalid.

Next, in order to perform synchronized chemi-lighting operations as described above, information transmitted from the distribution server computer 100 to each viewer terminal 300 is shown in Fig. 31 and Fig. 33 (A). Specifically, the distribution server computer 100 transmits BPM (beats per minute) information, chemi-lighting operation information, and delay operation information, as shown in Fig. 33 (A), along with the virtual space update data described in Example 1.

In addition to the BPM information, chemilight action information, and delayed action information, as shown in Figures 31 and 35, swing start information that can identify the timing at which each viewer avatar performing the chemilight action will start swinging the chemilight K at the start of each performance is transmitted from the distribution server computer 100 to each viewer terminal 300 at a timing a predetermined period (the period during which the viewer avatar's action is stopped) before the start of each performance, and each viewer avatar performing the chemilight action starts swinging the chemilight K based on the timing identified by the swing start information (in Example 3, the timing is approximately the same as the start timing of the performance), so that the chemilight actions of each viewer avatar are synchronized.

The swing start information in Example 3 is information that specifies the timing a predetermined period after the timing of receiving the swing start information, that is, information that can identify the length of the predetermined period, and each viewer terminal 300 that receives the swing start information can start a timer count from the timing of receiving the information and identify the timing of starting to swing the Chemilight K (the start of the performance), which is the timing a predetermined period after the timing of receiving the information, based on the timing when the timer expires.

In Example 3, as shown in FIG. 31, during the period before the swing start information is transmitted a predetermined time before the start of the performance, the movement of the viewer avatar performed by the viewer C avatar and the asynchronous chemi-lighting action performed by the viewer A avatar and the viewer B avatar at the default swing speed are possible; however, during the predetermined period from the transmission of the swing start information to the start of the performance, which is the start timing of the synchronized chemi-lighting action specified by the swing start information, all actions of the viewer avatar are stopped (prohibited). This enables each viewer avatar to appropriately begin synchronized chemi-lighting action, and avoids the viewer feeling uncomfortable when suddenly transitioning from an asynchronous chemi-lighting action to a synchronized chemi-lighting action with a different swing speed, etc. In addition, in Example 3, the BPM (beats per minute) information described below is distributed in advance from the timing of sending the start of the dance information, which is a certain period before the start of the performance, so that chemi-light movements synchronized with the music can start properly from the beginning, and this BPM information can be prevented from being erroneously reflected in non-synchronized chemi-light movements.

BPM (beats per minute) information indicates the number of beats in a song that occurs within one minute; for songs with a fast tempo, the BPM value is high, and for songs with a slow tempo, the BPM value is low. The BPM (beats per minute) information is written as data that can identify the interval between beats and the timing of the beats corresponding to these BPM values, and by identifying the timing of the beats from the BPM information as the BPM reference timing (see Figure 33), and operating so that one cycle of swinging the Chemilight K back and forth is completed at the interval between these beats, the speed of the Chemilight movement can be matched to the speed (tempo) of the song being performed in each performance, and the speed of the Chemilight movement of each viewer avatar can be adjusted to the same speed.

In Example 3, as described above, the BPM information is transmitted in advance from the transmission timing, with the start of the swing information being transmitted a predetermined period before the start of the performance, but the BPM information transmitted during this predetermined period is not reflected in the asynchronous chemi-light movements of the viewer avatar, and the viewer terminal 300 can identify the manner of synchronized chemi-light movements in advance, prior to the start of the performance, from the beats and the periods between beats identified from this BPM information transmitted in advance. In this way, by the viewer terminal 300 being used to identify the manner of synchronized chemi-light movements in advance, these synchronized chemi-light movements can be executed accurately from the start of the performance.

The chemilight action information is information for identifying viewer avatars that are performing the chemilight action among the viewer avatars virtually participating in the virtual live. In other words, as described above, viewer avatars that have not purchased chemilight K in advance are not capable of performing the chemilight action, and there are also viewer avatars that have purchased chemilight K in advance but are not performing the chemilight action. Therefore, for all viewer avatars virtually participating in the virtual live, data describing whether or not they are performing the chemilight action is transmitted to each viewer terminal 300 as chemilight action information. In this way, when each viewer terminal 300 generates a viewer avatar viewpoint image or a virtual camera viewpoint image, the presence or absence of the chemilight action is identified for each viewer avatar based on the chemilight action information, and is reflected in the viewer avatar's action.

The delayed action information is information for identifying viewer avatars performing the chem-light action that perform the delayed A or delayed B action shown in FIG. 33 (B), and is data describing whether all viewer avatars (viewer users) performing the chem-light action belong to the no-delay group, the delayed A group, or the delayed B group.

In other words, for a viewer avatar (viewer user) that is identified as belonging to the "no delay" group by the delay action information, for example, the chemilight action performed by viewer avatar 13 in FIG. 33(B) will begin one cycle of chemilight action (one action cycle of swinging the chemilight K left and right) without any delay (deviation) from the BPM reference timing that corresponds to the beat timing identified by the BPM information.

Furthermore, for viewer avatars (viewer users) identified by the delay action information as belonging to the "Delay A" group, the viewer avatars will begin one cycle of chemilight action (a swinging action cycle from side to side) at a timing delayed by 0.1 seconds from the BPM reference timing corresponding to the beat timing identified by the BPM information, such as the chemilight action performed by viewer avatar 12 in FIG. 33(B).

Furthermore, for viewer avatars (viewer users) identified by the delay action information as belonging to the "Delay B" group, the viewer avatar will begin one cycle of chemilight action (a swinging action cycle from side to side) at a timing delayed by 0.2 seconds from the BPM reference timing corresponding to the beat timing identified by the BPM information, such as the chemilight action performed by viewer avatar 11 in FIG. 33(B).

The reason for transmitting such delayed action information to have viewer avatar 12 and viewer avatar 11 perform chemilight actions that are delayed (shifted) by 0.1 seconds and 0.2 seconds from the BPM reference timing is that, for example, if all the chemilight actions performed by the viewer avatars were to perfectly match (synchronize) based on the swing start information and BPM information, the chemilight actions would become unrealistic, which could significantly reduce the realism of the virtual live performance. Therefore, in order to increase the realism of the chemilight actions performed by the viewer avatars, the timing of the chemilight actions for some viewer avatars is deliberately shifted by a range of 0.2 seconds, which is a predetermined range of seconds that does not go out of sync.

In the third embodiment, the timing of the shift is delayed from the BPM reference timing because it is easy to identify the timing of the shift from the BPM information on the viewer terminal 300. However, the present invention is not limited to this. For example, the timing of the group of delay B may be set to 0.1 seconds before the BPM reference timing instead of a delay, so as to reduce the difference from the BPM reference timing.

In addition, in Example 3, the maximum deviation is set to 0.2 seconds, which is a predetermined range that does not cause out-of-sync, but the present invention is not limited to this, and these maximum deviations can be any appropriate period that does not make the listener feel out of sync, and are not limited to a fixed number of seconds. In other words, the maximum deviation can be a variable number of seconds, so that if the music has a fast tempo, the maximum deviation will be a small number of seconds, and if the music has a slow tempo, the maximum deviation will be a large number of seconds. In such a case, for example, the number of seconds can be set to a number corresponding to a value obtained by multiplying the inverse of the tempo (BPM) of the music by a predetermined percentage (e.g., 10%).

In addition, in Example 3, an example is given in which the chemilight operation information and the delay operation information are treated as separate pieces of information (data), but the present invention is not limited to this, and these may be treated as a single piece of data.

In addition, in Example 3, the timing of the shift is set to two, 0.1 seconds and 0.2 seconds, to further improve the realism of the chemilight operation, but the present invention is not limited to this, and the timing of the shift may be simplified to just one, or conversely, may be set to three or more to further improve the realism.

In addition, in Example 3, the number of timing deviations is the same for each piece, but the present invention is not limited to this. For example, the number of timing deviations may be reduced or "none" for a piece of music with a very fast tempo, and the number of timing deviations may be increased for a piece of music with a slow tempo. In this way, the number of timing deviations may be changed depending on the piece.

Next, FIG. 34(A) shows a participant user table used in the distribution server computer 100 of the third embodiment. The participant user table of the third embodiment, like the participant user table of the second embodiment, is a table that can store, in association with a participant ID (viewer user ID), participation identification data (participation identification flag) that can identify the participation state of the participant (viewer user), the participation area of the participant (viewer user), participant coordinates that can identify the position in the virtual live venue where the viewer avatar of the participant (viewer user) is currently located, chemilight item purchase data that can identify whether or not the virtual item chemilight K has been purchased, chemilight operation status data that can identify whether or not chemilight operation is in progress, default speed data that can identify the type of default speed of the chemilight operation, and delay type data that can identify which of the above-mentioned "no delay", "delay A", and "delay B" groups the participant belongs to.

The participant ID field registers all viewer user IDs of prospective participants described in the files of each participant group stored in the event data corresponding to the virtual live. In addition, the participation identification data (participation identification flag) is registered as "0", corresponding to a non-participating state, when the participant ID (viewer user ID) is registered, and is updated to "1", indicating participation, when the participation of each participant (viewer user) is permitted by authentication. Therefore, in the example shown in Figure 34 (A), it is possible to identify that viewer user "MID-00007", whose participation identification data (participation identification flag) is "0", is not participating in the virtual live for some reason.

The participation area data is data that can identify whether the participation area of each participant (viewer user) is the third area or the fourth area, and is registered when the participation of each participant (viewer user) is permitted by authentication. The participant coordinate data is data that is updated sequentially in the virtual space update process, and is coordinate data of the location where the viewer avatar of each participant (viewer user) is located within the virtual live venue. Therefore, from the coordinate data, the location within the virtual live venue where each viewer avatar is located at that time can be easily identified.

The chemilite item purchase data is updated from "0" indicating that the item has not been purchased to "1" indicating that the item has been purchased in response to the purchase. Therefore, as described above, only the viewer avatar of a viewer user whose chemilite item purchase data is "1" can perform chemilite actions, and as operations corresponding to the execution of these chemilite actions are performed, the validity or invalidity of the operation is determined in the distribution server computer 100 based on the chemilite item purchase data.

The chemilite action status data is data that can identify whether or not the viewer avatar of each participant (viewer user) is performing a chemilite action, and when an operation corresponding to the execution of a chemilite action is executed on the viewer terminal 300 and the operation is determined to be valid, the data is updated from "0", indicating ineffective, to "1", and when an operation corresponding to the non-execution (end) of the chemilite action is executed on the viewer terminal 300, the data is updated from "1", indicating execution, to "0", indicating ineffective.

The default swing speed data is data that can specify the speed at which the chemilite K is swung when performing an asynchronous chemilite action, and corresponds to the speed selected by the viewer user when performing the chemilite action during a period such as before the start of the first performance. Specifically, in Example 3, the viewer user can select from five default speeds ranging from a low speed 1 (P1) to a high speed 5 (P5), and information on the default speeds selected by the viewer user on the viewer terminal 300 is notified to and registered in the distribution server computer 100.

Therefore, the speed at which the viewer avatar swings the chemilight K during a period when asynchronous chemilight operation is possible is based on the default speed data associated with the viewer user ID of the viewer user to which the viewer avatar corresponds.

The delay type data is data that identifies which group each viewer user's viewer avatar belongs to, the "no delay", "delay A", or "delay B" group described above. If it belongs to "no delay", "N" is registered, if it belongs to "delay A", "A" is registered, and if it belongs to "delay B", "B" is registered.

Here, we will explain the process by which a viewer avatar is assigned to either the "No Delay", "Delay A", or "Delay B" group using Figures 31 and 37.

These groupings of "No Delay", "Delay A", and "Delay B" are performed by the delivery server computer 100 executing the delay performer determination process shown in FIG. 37(A).

The timing at which the delay performer determination process is executed in the distribution server computer 100 is not just once, but is executed at multiple times immediately before the start of each performance, as shown in FIG. 31. Therefore, in Example 3, the viewer avatars belonging to "No Delay", "Delay A", and "Delay B" change for each performance.

In this way, in the delayed performer determination process that is executed immediately before the start of each performance, first, as shown in FIG. 37(A), all viewer avatars that are performing the chem-lighting action are identified (step S201). Specifically, it is sufficient to identify all participant IDs (viewer user IDs) that have "1" registered as the chem-lighting action status data at that time in the participant user table shown in FIG. 34(A).

Then, from the viewer avatars performing the identified chemilighting actions, a viewer avatar performing a "delay A" chemilighting action and a viewer avatar performing a "delay B" chemilighting action are selected based on the determination ratio in the determination table shown in FIG. 37 (B) (step S202).

As shown in FIG. 37(B), the determination ratios in Example 3 are exemplified as follows: viewer avatars performing the "Delay A" chemilight action make up 20% of all viewer avatars performing the chemilight action, and viewer avatars performing the "Delay B" chemilight action make up 10% of all viewer avatars performing the chemilight action. In this way, the larger the deviation, the lower the determination ratio, which is preferable because it makes the deviation more realistic; however, these determination ratios may be determined as appropriate. However, it is preferable that the combined ratio of "Delay A" and "Delay B" does not exceed 50%.

Note that in Example 3, an example is shown in which the determination ratio is one, but the present invention is not limited to this. For example, multiple determination tables with different determination ratios may be provided, and the determination ratio may be changed by changing the determination table to be used depending on the music in the program.

The specific method of selection is not limited, but for example, a determination number 1 is calculated by multiplying the total number M of viewer avatars performing the chemilight action identified in step S201 by 10%, which is the determination ratio corresponding to "Delay B", and a determination number 2 is calculated by multiplying the total number M of viewer avatars performing the chemilight action identified in step S201 by 10%, which is the determination ratio corresponding to "Delay B", and a lottery is held to determine a viewer avatar that will perform the chemilight action of "Delay B" from among the viewer avatars performing the chemilight action identified in step S201 until the determination number 1 is reached, a lottery is held to determine a viewer avatar that will perform the chemilight action of "Delay A" from among the undetermined viewer avatars until the determination number 2 is reached, and the remaining viewer avatars are determined as viewer avatars that will perform the chemilight action of "no delay".

Then, the delay action type data in the participant user table is updated based on the above-mentioned determination result (step S203).

Then, new delay operation information is generated based on the updated delay operation type data and distributed to each viewer terminal 300 (step S204).

Next, FIG. 34(B) shows the song list data used in the delivery server computer 100 of the third embodiment. Note that the song list data is stored in the delivery server computer 100 as one file data.

As shown in FIG. 34(B), the song list data stores information such as "song name," "song ID," and "BPM" in association with the type of performance, and it is possible to identify which songs will be used in each performance based on the information on the type of performance.

In other words, in the example shown in FIG. 34(B), it can be determined that the song with song ID "GK-XXXX" will be performed in the first performance, the song with song ID "GK-YYYY" will be performed in the second performance, and the song with song ID "GK-ZZZZ" will be performed in the third performance.

It can also be seen from the BPM data that the music performed in the first performance has a relatively slow tempo of 82 beats per minute, the music performed in the second performance has a relatively fast tempo of 123 beats per minute, and the music performed in the third performance has an even faster tempo of 136 beats per minute. Note that while the BPM data in Figure 34(B) is expressed as integer numerical values, the present invention is not limited to this, and these values may be numerical values including decimal points.

Therefore, as shown in FIG. 35, in the first performance, BPM information corresponding to 82 beats per minute, which is the BPM of the song ID "GK-XXXX", is distributed to each viewer terminal 300 as BPM information corresponding to the first performance, in the second performance, BPM information corresponding to 123 beats per minute, which is the BPM of the song ID "GK-YYYY", is distributed to each viewer terminal 300 as BPM information corresponding to the second performance, and in the third performance, BPM information corresponding to 136 beats per minute, which is the BPM of the song ID "GK-ZZZZ", is distributed to each viewer terminal 300 as BPM information corresponding to the third performance. In other words, in each performance, different BPM information corresponding to the song being performed is distributed.

As shown in FIG. 35, the BPM information for the first performance, the BPM information for the second performance, and the BPM information for the third performance all begin to be distributed from the timing of sending the swing start information a predetermined period before the start of the first performance, the second performance, and the third performance, but this predetermined period is treated as a viewer avatar motion stop period T during which the motion of the viewer avatar is stopped, and is therefore not reflected in the chemilight motion, which is the motion of the viewer avatar.

Here, the flow when a viewer user virtually participates in a virtual live venue in Example 3 will be explained with reference to FIG. 36.

First, the viewer user performs an operation to join the virtual live show that he or she plans to virtually participate in, for example, by selecting a participation menu or operating a specific key on the keyboard to which the participation operation is assigned, in the viewer program (viewing application) running on the viewer terminal 300.

By executing this participation operation, a participation request including the viewer user ID and authentication data for authenticating the viewer user is sent from the viewer terminal 300 to the distribution server computer 100.

The distribution server computer 100 that receives the participation request executes a participant authentication process on the condition that it is a period during which participation in the live virtual space is possible. In the participant authentication process, it is determined whether the viewer user is a prospective participant of the corresponding virtual live depending on whether the viewer user ID included in the received participation request is registered as a participant ID in the participant user database corresponding to the virtual live to be participated in, and the viewer user is authenticated by comparing the authentication data described in the authentication information file identified from the authentication information file data stored in association with the viewer user ID in the viewer user data shown in FIG. 9 with the authentication data included in the participation request.

If the viewer user is a prospective participant and authentication using the authentication data is complete, the participation identification data (participation identification flag) in the participant user table is updated to "1" corresponding to "participation", and the distribution server computer 100 then returns participation completion information to the viewer terminal 300 that sent the participation request. In response to receiving the participation completion information returned in this manner, the viewer terminal 300 displays a message indicating that virtual participation has been completed, thereby notifying the viewer that virtual participation has been completed. This allows the viewer user to, for example, purchase virtual items (including Chemilite K) that are available for purchase at the virtual live show in which virtual participation has been completed.

Following the participation completion notification process, the distribution server computer 100 executes a pre-data transmission process to transmit the pre-data to the viewer terminal 300 of the viewer user who has newly participated virtually. The viewer terminal 300 that receives the pre-data transmitted in this manner stores the received pre-data, and then generates viewer perspective video based on the stored pre-data and begins displaying (outputting). This allows the viewer user to view video of the virtual live venue as seen from the viewer avatar of their own virtual participation.

Next, the flow of information between the distribution server computer 100 and the viewer terminal 300 after virtual participation will be explained using Figure 38.

In FIG. 38, the flow up to the virtual space update process in the distribution server computer 100 is the same as in Example 1, so the explanation will be omitted here. However, the operations in the avatar action operation process in the viewer terminal 300 include operations related to chemilight actions.

In the distribution server computer 100, the participant user table update process is executed after the virtual space update process, and the participant user table, specifically, for example, participant coordinate data, chemilight operation status data, default swing speed data, etc., may be updated.

Then, the distribution server computer 100 determines whether or not it is currently in the period for transmitting BPM information. Note that, as described below, BPM information is also distributed (transmitted) before the start of each program period (see FIG. 35).

When it is the period for transmitting BPM information, the chemilight operation information, BPM information, and delayed operation information are distributed (transmitted) to the performer terminal 200 and each viewer terminal 300. The chemilight operation information, BPM information, and delayed operation information distributed in this manner are updated and stored in the performer terminal 200 and each viewer terminal 300.

In the third embodiment, even during each performance period, the viewer terminal 300 is capable of newly performing a chemilighting action on the viewer avatar. However, during each performance period, as shown in FIG. 31, the delay operator determination process is not executed, so that the viewer user who newly performs a chemilighting action during the performance period is uniquely assigned "no delay", and together with the chemilighting action information reflecting the information of the viewer user who newly performed the chemilighting action, delay action information that identifies the viewer user as belonging to the "no delay" group is transmitted to the performer terminal 200 and each viewer terminal 300. However, during each performance period, as shown in FIG. 31, the processing load of the distribution server computer 100 is relatively high, so in order to prevent a further increase in the processing load of the distribution server computer 100 due to the increase in the processing load associated with the above-mentioned process, the viewer terminal 300 may be disabled from newly performing a chemilighting action on the viewer avatar.

Next, in the distribution server computer 100, the virtual space update data generation process and the virtual space update data distribution process are executed in the same manner as in Example 1, and the virtual space update data is distributed (transmitted) to the performer terminal 200 and each viewer terminal 300.

In response to receiving the virtual space update data distributed (transmitted) in this manner, each viewer terminal 300 executes a virtual space update process, and the update contents of the received virtual space update data are reflected in the virtual space data stored in each viewer terminal 300, after which viewer perspective video generation process 1 shown in Figure 39 (A) is executed.

Figure 39 (A) is a flow diagram showing the process of viewer's viewpoint video generation process 1 executed in the viewer terminal 300 of Example 3. In this viewer's viewpoint video generation process 1, first, a process of generating a normal viewer's viewpoint video based on the updated virtual space data is performed (step S300).

Next, based on the chemi-lighting action information and delay action information received and updated from the distribution server computer 100, all viewer avatars performing chemi-lighting actions and their delay types are identified (step S301).

Then, the operation mode based on the delay type identified in step S301 and the received BPM information is reflected in the chem-lighting operation of each viewer avatar currently performing the chem-lighting operation (step S302).

In this way, by reflecting the changes made in step S302, a viewer avatar viewpoint image is generated that includes a viewer avatar performing synchronized chemilighting actions based on the BPM information (step S303).

The viewer avatar perspective images, in which the chemilight actions of each viewer avatar are synchronized, generated by viewer perspective image generation process 1 are output (displayed) on each viewer terminal 300 by viewer perspective image output process.

Note that even if BPM information is being distributed, if the viewer avatar is not moving during this period, the viewer avatar's movement is stopped before the swing start timing, and the viewer avatar is not performing the chemilight movement that is the subject of the BPM information, so the viewer's viewpoint image will be generated and displayed as a viewer's avatar viewpoint image that includes the viewer avatar that is not moving.

On the other hand, when it is not the period for transmitting BPM information, the chemilight action information and default swing speed information are distributed (transmitted) to the performer terminal 200 and each viewer terminal 300. The chemilight action information and default swing speed information distributed in this manner are updated and stored in the performer terminal 200 and each viewer terminal 300.

The default swing speed information is based on the default swing speed data in the participant user table and includes data that can identify the default speed for each viewer avatar performing the chemi-lighting action. This allows each viewer terminal 300 that receives the default swing speed information to identify the speed (speed 1 to speed 5) at which each viewer avatar swings the chemi-light K in the chemi-lighting action for all viewer avatars performing the chemi-lighting action.

In addition, the default swing speed may be changed by a viewer user during each MC period, for example, and these changes are updated on the viewer terminals of other viewer users by sequentially distributing the default swing speed information.

Next, in the distribution server computer 100, the virtual space update data generation process and the virtual space update data distribution process are executed in the same manner as in Example 1, and the virtual space update data is distributed (transmitted) to the performer terminal 200 and each viewer terminal 300.

In response to receiving the virtual space update data distributed (transmitted) in this manner, each viewer terminal 300 executes a virtual space update process, and the update contents of the received virtual space update data are reflected in the virtual space data stored in each viewer terminal 300, after which viewer viewpoint video generation process 2 shown in FIG. 39 (B) is executed.

Figure 39 (B) is a flow diagram showing the process of viewer's viewpoint video generation process 2 executed in the viewer terminal 300 of Example 3. In this viewer's viewpoint video generation process 2, first, a process of generating a normal viewer's viewpoint video based on the updated virtual space data is performed (step S400).

Next, based on the chemi-lighting action information and default swing speed information received and updated from the distribution server computer 100, all viewer avatars performing chemi-lighting actions and their default swing speeds are identified (step S401).

Then, the motion mode based on the default swing speed identified in step S401 is reflected in the chemi-lighting motion of each viewer avatar currently performing the chemi-lighting motion (step S402).

In this way, by reflecting the reflection in step S402, a viewer avatar viewpoint image is generated that includes a viewer avatar performing an asynchronous chemilighting action in which the chemilight K is swung at a swing speed uniquely selected by each viewer user as specified by the default swing speed information (step S403).

These viewer avatar perspective images, in which the chemilight actions of each viewer avatar are not synchronized, generated by viewer perspective image generation process 2 are output (displayed) on each viewer terminal 300 by viewer perspective image output process.

As described above, in the virtual space content distribution system of Example 3, the viewer avatar can perform a chemilight action, which is a content-related action, and when multiple viewer avatars perform chemilight actions, the distribution server computer 100, which serves as a content-related action adjustment means for synchronizing the timing of the chemilight actions of each viewer avatar, distributes chemilight action information, BPM information, and delay action information, and the viewer terminal 300 receives and updates and stores this chemilight action information, BPM information, and delay action information, and executes viewer viewpoint video generation process 1 based on the stored information, so that the chemilight actions of each viewer avatar are synchronized, thereby increasing the interest of each viewer user participating in the virtual space.

In addition, in the virtual space content distribution system of Example 3, it is possible to adjust the operation timing of some of the viewer avatars among multiple viewer avatars performing chemilight operations to "delay A" or "delay B" operation modes, which are non-synchronized timings that differ within a range of 0.2 seconds from the synchronized synchronization timing, thereby improving the realism of the chemilight operations in the virtual live space and further increasing the interest of the viewer user.

Furthermore, in the virtual space content distribution system of Example 3, as in Example 1, a viewer avatar viewpoint image including an image of a viewer avatar performing synchronized chemilight actions can be generated and displayed on the viewer terminal 300, so that more viewer users can participate in the virtual live event, and even when more such viewer users participate, the interest of each viewer user participating in the virtual live space can be increased.

In addition, in the virtual space content distribution system of Example 3, the virtual space content includes performance content such as singing and dancing to music performed by performer avatars who can virtually participate in the virtual live space, and the BPM information, which is information for synchronizing the chemilight movements, is information corresponding to the tempo of the music, which is the performance content, so that the chemilight movements can be synchronized to the music performed by the performer avatars, thereby further increasing the interest of the virtual space content.

In addition, in the virtual space content distribution system of Example 3, the distribution server computer 100 includes, as the action synchronization information, BPM information corresponding to the tempo of the music, and swing start information that serves as start information that can identify the start time of the chemi-light action, so that the chemi-light action can be made to correspond to the tempo of the music.

In addition, in the virtual space content distribution system of Example 3, as shown in FIG. 35, when the music of the first performance, which is the first performance content, is progressing, first performance-specific BPM information corresponding to the tempo (BPM) of the music of the first performance is distributed from the distribution server computer 100, and when the music of the second performance, which is the second performance content, is progressing, second performance-specific BPM information corresponding to the tempo (BPM) of the music of the second performance is distributed from the distribution server computer 100, so that chemilight movements with different swing speeds corresponding to differences in the tempo of the music of the performance contents can be performed.

In addition, in the virtual space content distribution system of Example 3, BPM information and start of dance information are distributed in advance from the distribution server computer 100 a predetermined period before the start of a song during each performance period, so that it is possible to prevent the chemilight operation from becoming out of sync at the start of a song.

In addition, in the virtual space content distribution system of Example 3, the viewer terminal 300 does not reflect the BPM information transmitted before the start of the music in the chemi-lighting movements of the viewer avatar, so that it is possible to prevent a decrease in interest in the virtual space content due to unnatural chemi-lighting movements before the start of the music.

Furthermore, in the virtual space content delivery system of Example 3, of the viewer avatars performing the chemilighting action, 20% of the viewer avatars can be determined to be in the "Delay A" group, in which the start timing of the chemilighting action is the first non-synchronized timing delayed by 0.1 seconds from the BPM reference timing, and 10% of the viewer avatars can be determined to be in the "Delay B" group, in which the start timing of the chemilighting action is the second non-synchronized timing delayed by 0.2 seconds from the BPM reference timing. In this way, by providing multiple delay modes, the realism of the chemilighting action can be further improved.

In addition, the virtual space content distribution system of Example 3 includes performance content in which a song accompanied by singing and dancing is performed as the main content (first content) of the virtual live, which is the virtual space content, and MC (talk) content, which is non-performance content different from the performance content, and the viewer avatar can perform chemi-lighting actions corresponding to the song performed in the performance content during each performance period in which the performance content is distributed, while during the MC period in which the MC (talk) is distributed, the viewer avatar can perform movement actions that do not correspond to the talk during the MC period. In this way, the actions of the viewer avatar are adjusted to chemi-lighting actions in sync with the song when the performance content is distributed, while movement actions that do not correspond to the talk are possible when the talk content is distributed during each MC period, thereby increasing the interest of each viewer user participating in the virtual live space.

In addition, in the virtual space content distribution system of Example 3, the main content (first content) is performance content performed by performer avatar 1 who can virtually participate in the virtual live space, and includes singing and dancing to a song performed by performer avatar 1, so the chemilight movements of the viewer avatar are adjusted to movements corresponding to the song and dance that are the performance content, thereby further increasing the interest of the virtual space content.

In addition, in the virtual space content distribution system of Example 3, the content distributed during each MC period when MC (talk) is distributed does not include performance content in which a musical piece is performed by performer avatar 1, so it is possible to appropriately prevent movement actions that do not correspond to the performance content from being performed during the performance period when the performance content is being distributed.

In addition, in the virtual space content distribution system of Example 3, the performance content includes a song with a song ID of "GK-XXXX" (first period content) that is distributed during a first performance period that is a first period, and a song with a song ID of "GK-YYYY" (second period content) that is distributed during a second performance period that is a second period different from the first period, and as BPM information that serves as the operation information for synchronizing the chemi-light operation of the viewer avatar, BPM information corresponding to the first performance is transmitted during the first period, and BPM information corresponding to the second performance that can specify a tempo different from the BPM information corresponding to the first performance is transmitted during the second period, so that the chemi-light operation can be adjusted to correspond to the difference in tempo of the songs that are the performance contents distributed during different periods.

The virtual space content distribution system of Example 3 includes, as virtual space content, performance content (first content) that becomes the main content when performer avatar 1, who virtually participates in the virtual live space, performs a musical piece, and non-performance content (second content) that does not become the main content when performer avatar 1 does MC (talk) without performing a musical piece, and is equipped with a motion restriction function that restricts movement, which is a predetermined motion among the movements of the viewer avatar, during each performance period when the performance content is distributed, but does not restrict movement during each MC period when the non-performance content (MC) is distributed. Therefore, since the movement of the viewer avatar is restricted during the period when the performance content is distributed, it is possible to prevent the interest of the performance content from being reduced by these movements. On the other hand, since movement is not restricted during each MC period when the non-performance content, MC (talk), is distributed, it is also possible to prevent the movement of the viewer avatar from being excessively restricted.

In addition, in the virtual space content distribution system of Example 3, as shown in Figures 40 and 41, during each performance period when performance content is being distributed and during each MC period when performance content is not being distributed, an operation of selecting the "Chemilite" item, which is an operation corresponding to the chemilite action that is a specific action of the viewer avatar, is possible, and when this operation is executed, synchronized chemilite actions are possible without restriction (prohibition) during each performance period, and non-synchronized chemilite actions are possible without restriction (prohibition) during each MC period. In this way, a viewer user can have the viewer avatar perform a chemilite action by selecting the "Chemilite" item, which is a specific operation, during both each performance period when performance content is being distributed and each MC period when non-performance content is being distributed.

In addition, in the virtual space content distribution system of Example 3, the predetermined period before the start of each performance period during which performance content is distributed, that is, the predetermined period from when the start of swing information is transmitted until the start of each performance period, is also considered to be a performance period, and all actions of the viewer avatar, including movement, are restricted (prohibited) in the same way as for the performance period. In this way, movements such as movement of the viewer avatar are restricted (prohibited) even during the predetermined period before the performance period begins, so that it is possible to more accurately prevent a decrease in interest in the performance content due to these movements and other actions being performed up until the start of the performance period.

Note that in the above-mentioned Example 3, an example is given in which all actions of the viewer avatar other than movement are restricted (prohibited) during a specified period before the start of the performance period, but the only action that is restricted (prohibited) may be movement.

In addition, in the virtual space content distribution system of Example 3, during each performance period in which the performance content is distributed, chemical light operation corresponding to the tempo of the musical piece that is the performance content is possible, thereby increasing the interest of the viewer users participating in the virtual live space.

<Modification of the third embodiment>
In the above Example 3, as in Examples 1 and 2, an example was given of a form in which a viewer avatar can move freely within the third or fourth area in which they are virtually participating. However, doing so may give rise to problems such as errors in the virtual space data due to, for example, different viewer avatars participating in the fourth area colliding virtually with each other or overlapping in the same virtual position.

To solve such problems, for example, a modification 3-1 shown in Figures 42, 43, and 44 may be used. Specifically, the third and fourth areas are divided into smaller areas as shown in Figure 42, and the viewer user is asked which of these areas they will virtually participate in the virtual live show in when pre-participation in the virtual live show (when reserving virtual participation) or when virtually participating in the virtual live show, as shown in Figure 42. When the viewer user virtually participates with a viewer avatar, they can only participate in the area selected in advance, and movement is restricted (prohibited) to only within the selected area. This prevents viewer avatars from colliding with each other virtually or overlapping in the same virtual position, which could cause errors in the virtual space data, while also preventing the movement of the viewer avatar from being overly restricted.

Specific examples of the restriction (prohibition) of these movements include, for example, in the case of a viewer user who has selected the "H5" area in FIG. 43 as the virtual participation position, when the viewer user performs an operation to move to any of the areas surrounding "H5", such as "G4-G6", "H4", "H6", or "I4-I6", as shown in FIG. 44, the movement is determined by the viewer terminal 300 to be a movement to a prohibited movement area, and the movement is restricted (prohibited) by displaying a warning message on the viewer terminal 300, such as "You cannot move to the specified position because you are outside the movable range", as shown in FIG. 44(B). On the other hand, movement within the "H5" area is permitted without displaying the warning message, allowing the viewer user to move the viewer avatar within the "H5" area where he or she has virtually participated.

The range in which movement is prohibited is not limited to the range of virtual participation described above. For example, in a situation where a group of multiple people is virtually participating in a virtual live show and the locations of the virtual participants are adjacent to each other, the range in which the viewer avatars participating in this group are virtually participating may be set as a movable range based on information indicating that they are a group.

Moreover, although the above-mentioned modified example 3-1 is described as a modified example of the third embodiment, it is applicable not only to the third embodiment but also to the first and second embodiments.

In addition, in the above-mentioned Example 3, a predetermined period from the timing of sending the start swing information to the timing of the start of the performance is set as a viewer avatar operation suspension period T in which the movement of the viewer avatar is prohibited, but the present invention is not limited to this. For example, as shown in variant example 3-2 in FIG. 45, during the predetermined period from the timing of sending the start swing information to the timing of the start of the performance, an adjustment operation period may be set in which an adjustment operation is performed for a viewer avatar performing a non-synchronized chemi-lighting action, to transition the action state at the time of sending the start swing information to the action state at the start of a synchronized chemi-lighting action.

Furthermore, for these adjustment actions, the posture of the viewer avatar at the time the swing start information is sent can be compared with the posture at the start of the chemilight action, and actions that compensate for the posture difference can be generated using, for example, a known motion morphing program.
In addition, in the above modification 3-2, the timing at which the synchronized chemi-lighting action starts is used as an example for explanation, but these adjustment action periods may be performed at the timing at which the synchronized chemi-lighting action ends. In other words, an adjustment action that compensates for the difference in posture between the viewer avatar's posture when the synchronized chemi-lighting action ends and the viewer avatar's posture when the non-synchronized chemi-lighting action starts may be performed by setting an adjustment action period.

In other words, as shown in variant example 3-2, when starting or ending a synchronized chemi-lighting action, an adjustment action is generated to adjust the difference between the viewer avatar's posture before the action begins and the posture at the start of the synchronized chemi-lighting action, and the difference between the posture at the end of the synchronized chemi-lighting action and the posture of the viewer avatar after the action ends, thereby preventing the viewer avatar's movements from becoming discontinuous and unnatural.

In addition, in the above Example 3, as shown in FIG. 31, each MC period is not included in the program period, but the present invention is not limited to this. As shown in Variation 3-3 in FIG. 46, depending on the performer user, these MC (talk) periods may also be included in the program, in which case movements and other actions may be prohibited during these MC periods as part of the program period. Note that when MC (talk) is included in the performer's program in this way, these MC (talk) periods are also included in the program period, and thus MC BPM information corresponding to the MC (talk) is sent as BPM information, just as in the case of music, and the chemilight action is adjusted to an action with a tempo corresponding to the MC (talk).

In other words, the first content included in the virtual space content includes content of the performer avatar 1 speaking (talking), so the chemilight action of the viewer avatar can be adjusted to an action corresponding to the speaking (talking).

When an MC (talk) is included in the performer's repertoire as in variant example 3-3 above, movement of the viewer avatar and other actions are always prohibited. Therefore, for example, as shown in FIG. 46, during the waiting period from when the viewer virtually joins before the start of the live performance until the virtual live performance begins, during a specific MC period within the MC period, or during a rest period for the performers to rest during the virtual live performance, the period during which archived video content of past virtual live performances and the like is displayed on virtual displays 40, 41, and 42 is set as a period not included in the repertoire period, and movement and other actions are permitted. In this case, the performer's MC (talk) content is included in the performance content along with singing and dancing, and archived video content and other content displayed on virtual displays 40, 41, and 42 are non-performance content.

In addition, in the above Example 3, movement is given as an example of an action that is restricted (prohibited) by the viewer avatar during each performance period, but the present invention is not limited to this, and in addition to these movements, as shown in Variation 3-4 of FIG. 47, jumping (jumping) and high-fiving with other viewer avatars, which are actions that affect other viewer avatars, may also be restricted (prohibited) during each performance period.

In other words, the actions of viewer avatars that are restricted (prohibited) during each performance period, which is the period during which performance content is distributed, include actions and jumps that affect other viewer avatars virtually participating in the virtual live space, thereby preventing the performance content from losing interest due to actions and jumps that affect other viewer avatars. Note that the targets of the actions may include performer avatars in addition to viewer avatars.

The actions of the viewer avatar that are restricted (prohibited) during these performances are not limited to those shown in FIG. 47, but may include any action that is considered to cause discomfort to the viewer avatar during each performance period.

Similarly, in the above Example 3, an example is given of a form in which the content-related action that the viewer avatar can perform during each performance period is only one action, which is the chemilight action of waving the chemilight K from side to side, but the present invention is not limited to this, and for example, these actions may include multiple actions, such as, as shown in FIG. 47, in addition to the chemilight action of waving the chemilight K from side to side, a chemilight action of waving the chemilight K back and forth and a wave action in which the viewer avatar swings both hands raised above from side to side, so that each of these three content-related actions can be performed in the performance to which each content-related action corresponds.

The content-related actions that can be performed during these performances are not limited to the three actions shown in FIG. 47. Any action that is pre-approved as an action that does not cause discomfort to all viewer avatars during each performance period can be added to the content-related actions.

As for the chemilight action of waving the chemilight K back and forth and the wave action, similar to the chemilight action of waving the chemilight K left and right, multiple viewer avatars performing these actions can synchronize their movements at a tempo that corresponds to the music being performed based on the BPM information and the start of the movement information.

In other words, the chemilight action includes multiple chemilight actions with different action patterns, such as shaking left and right and shaking back and forth, and the virtual space content distribution system of variant example 3-4 is capable of synchronizing the chemilight actions of multiple viewer avatars for any of the chemilight actions. In this way, the chemilight action has multiple action patterns, and it is possible to synchronize the chemilight actions of these multiple action patterns, which further increases the interest of each viewer user participating in the virtual live space.

In addition, when these multiple content-related actions are executable, during periods before the start of a live performance or during MC periods other than the performance period, it is possible to make all types of content-related actions executable, as shown in FIG. 47, for example, and it is also possible to make multiple content-related actions executable in one performance, rather than just one type, such as making the chemilight action of shaking the chemilight K from side to side possible during the performance period corresponding to the wave action.

In addition, in the above-mentioned Example 3, the delay performer determination process is executed in the distribution server computer 100, and the distribution server computer 100 determines the viewer avatar that will perform the delay action in the synchronized chemi-light action. However, the present invention is not limited to this. For example, as shown in Variation 3-5 of FIG. 48, the delay performer determination process may be executed in the viewer terminal 300, and the viewer avatar that will perform the delay action may be determined in the viewer terminal 300.

In other words, in variant example 3-5, the viewer terminal 300 can generate viewer avatar viewpoint video in which the timing of the chemi-lighting action of each viewer avatar is synchronized, the distribution server computer 100 can distribute chemi-lighting action information to each viewer terminal 300 as related action execution information that can identify the viewer avatar performing the chemi-lighting action, and the viewer terminal 300 can adjust the chemi-lighting action to "delay A" or "delay B", which are non-synchronized timings, for some of the viewer avatars performing the chemi-lighting action identified from the chemi-lighting action information. In this way, the distribution server computer 100 does not need to distribute delayed action information, and the processing load on the distribution server computer 100 for distributing this delayed action information can be reduced.

In addition, in variant example 3-5, the execution timing and processing contents of the delayed performer determination process executed by the viewer terminal 300 may be the same as the execution timing and processing contents executed by the distribution server computer 100 shown in FIG. 37. In this case, the determination table may be received as advance data from the distribution server computer 100 and stored, and the viewer avatar performing the chemilight action at the time of execution of the process may be identified from the chemilight action information distributed from the distribution server computer 100. The determination results are reflected in a delayed performer table for storing and updating the determination results, not in a participant user table, in association with the viewer user IDs of all participants included in the chemilight action information, and the viewer viewpoint video generation process 1 may be executed based on the determination results identified from these delayed performer tables.

In addition, in variant example 3-5, the delayed performer determination process is executed individually in each viewer terminal 300, so that a viewer avatar determined to be the delayed performer for "Delay A" in viewer user A's viewer terminal 300 may be determined to be the delayed performer for "Delay B" in viewer user B's viewer terminal 300, and the viewer user's viewpoint video including the synchronized chemilight action generated in each viewer terminal 300 will be a different video for each viewer terminal 300.

In addition, in Example 3, the timing for starting the chemilight operation based on the start of the swing information is set to the start timing of the song that is the performance content (the start timing of the performance period), but the present invention is not limited to this, and the timing for starting the chemilight operation based on the start of the swing information may be set to a timing different from the start timing of the performance period or the start timing of the song, for example, a certain period of time has passed since the start timing of the performance period, or the start timing of the song after the intro or other part has passed and singing begins.

The present invention has been described above based on each embodiment and modified example, but the specific configuration is not limited to the above-mentioned embodiments and modified examples, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

For example, as shown in variant example 3-1, in the case where each viewer user is made to select the virtual participation position of the viewer avatar in advance, for example, for a viewer user who selects areas A1 to C6 on the right side as facing the virtual stage G in the fourth area, a viewpoint image from the right front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period, for a viewer user who selects areas D4 to H6 on the center as facing the virtual stage G in the fourth area, a viewpoint image from the center front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period, and for a viewer user who selects areas I1 to K6 on the left side as facing the virtual stage G in the fourth area, a viewpoint image from the left front position of the virtually flying performer avatar 1 is generated and displayed as the viewpoint image during the flying performance period. In this way, the viewpoint image during the virtual flight, which is a special action, may be a viewpoint image corresponding to the position (area) selected by each viewer user depending on the position selected by the viewer user.

In addition, in the above embodiments and modified examples, the viewer avatar viewpoint video is generated and displayed on the viewer terminal 300 side, but the present invention is not limited to this. For example, in the case of a live event with few virtual participants, the viewer avatar viewpoint video may also be generated and distributed on the distribution server computer 100 side. In addition, the viewer user may be able to select whether the viewer avatar viewpoint video is generated on the viewer terminal 300 side or on the distribution server computer 100 side depending on the processing capacity of the terminal he or she owns and the available data communication environment, or the distribution server computer 100 may identify the processing capacity of the viewer terminal 300 with which it is connected for communication, and based on the identified processing capacity, determine whether to generate the viewer avatar viewpoint video on the viewer terminal 300 side or on the distribution server computer 100 side.

In addition, in each of the above embodiments and variants, the distribution server computer 100 is illustrated as being installed in a studio operated by the event management organization that hosts the virtual live show, but the present invention is not limited to this. As the distribution server computer 100, a server computer owned by the company that rents out the server computer in a data center or the like may be used, or as described above, a server computer provided by a cloud service may be used. The installation form of these server computers may be any form that provides the functions of the distribution server computer 100.

In addition, in each of the above embodiments and variations, a performer user performs performance actions such as singing and dancing alone; however, the present invention is not limited to this, and the performer users may be a group of multiple people. When a virtual live performance is performed by a group of multiple people, the multiple people may share one performer terminal 200, or a performer terminal 200 may be provided for each member of the group. When the performer users are a group of multiple members, for example, a viewpoint tracking target designation process may be executed by at least one of the distribution server computer 100 and the viewer terminal 300, allowing the viewer user to select a member to follow the viewpoint.

In addition, in each of the above embodiments and modified examples, a performer user sings, dances, and performs other performance actions alone; however, the present invention is not limited to this, and the singer who sings and the actor who dances may be separate persons. In this case, the singer's performer terminal and the actor's performer terminal may be separate, and the singer may sing in a location other than the studio where the actor performs the performance actions.

In addition, in each of the above embodiments and variations, the Internet network is used as an example of a computer communication network connecting the distribution server computer 100, the administrator terminal 150, the performer terminal 200, and the viewer terminal 300, but the present invention is not limited to this, and these communication networks may be configured as local area networks that are capable of data communication only within a specific area, or may include a local area network or local data communication as part of the network.

In addition, in the above embodiments and variants, the viewer terminal 300 is exemplified as a smartphone P or a computer (PC), but the present invention is not limited to this, and these computers may be portable notebook computers, tablet terminals, e-book readers, wearable computers, game consoles, and various other information processing devices capable of outputting at least 2D video and audio.

In addition, in each of the above embodiments and variants, a performer user performs in a studio, including singing, but the present invention is not limited to this. For example, a performer user may use a performer terminal 200 installed in a home or the like to perform in a virtual live performance from a home or the like.

In addition, in the above embodiments and variants, a virtual live performance has been given as an example of virtual space content, but the present invention is not limited to this, and the virtual space content may be any content that is beneficial to viewer users, such as a fan meeting where performer users gather with their fans, or a talk session, and is related to a virtual space in which performer avatar 1 and viewer avatars virtually participate.

In addition, in each of the above embodiments and modified examples, the viewpoint switching of the four virtual cameras C1 to C4 as fixedly positioned (set) virtual cameras and the airborne moving virtual camera is exemplified as a form in which the viewpoint switching can be performed in advance by setting and operating the administrator terminal 150, but the present invention is not limited to this. For example, the event manager M may perform these viewpoint switching in real time on the administrator terminal 150 using a viewpoint switching controller or the like constituting the operation input device 161.

In addition, in each of the above embodiments and modified examples, virtual cameras C1 to C4 fixedly placed (set) in the virtual space are used as virtual cameras, but the present invention is not limited to this. For example, in addition to or instead of the virtual cameras C1 to C4, a movable virtual moving camera may be placed in the virtual live venue, and the position and imaging direction of the virtual moving camera in the virtual space may be controlled by the event manager M or a cameraman through operation of the manager terminal 150 to produce the viewpoint image of the virtual moving camera.

In addition, in each of the above embodiments and variants, an example is given of a form in which the viewpoint image of the virtual camera C2 is displayed on the screen S so that the facial expressions of the performer user can be detected and reflected in the facial expressions of the performer avatar, but the present invention is not limited to this. For example, if the performer user is a high-performance head-mounted display that has a sensor function for detecting the facial expressions of the performer user, the performer user may wear this high-performance head-mounted display while performing the performance, which may improve the performer user's sense of immersion in the virtual live performance. In addition, if the facial expressions of the performer user do not need to be reflected in detail in the facial expressions of the performer avatar, the performer user may wear a normal head-mounted display while performing the performance.

In addition, in the above Example 3, the timing of the chemilight operation is exemplified by setting delay A, which is 0.1 seconds behind the BPM reference timing, and delay B, which is 0.2 seconds behind the BPM reference timing, but the present invention is not limited to this, and all viewer avatars performing the chemilight operation may perform the chemilight operation in synchronization with only the same BPM reference timing, without setting delay A or delay B. In other words, in the present invention, synchronization does not only mean synchronization with only the reference timing, but also includes an operation mode in which the timing of the chemilight operation of some viewer avatars differs within a predetermined range of 0.2 seconds that can be recognized as approximately the same timing, and therefore the synchronization timing includes not only the synchronization timing, which is the BPM reference timing, but also the delay A and delay B, which are pseudo-synchronization timings.

In addition, in the above Example 3, as shown in Figures 31 and 35, an example is given in which the viewer avatar motion suspension period T is the period from when the start of swinging information is transmitted to the start timing of each performance period, but the present invention is not limited to this. As an example of a form in which these viewer avatar motion suspension periods T are not provided, the viewer avatar may be able to perform chemilight motion throughout the entire period of each MC period.

In addition, in each of the above embodiments and variations, the performer user sings or performs singing actions (performance actions) in real time during the distribution of the virtual live performance, allowing the performer user to proceed with the live performance while checking the reactions of the viewer users who are virtually participating in the virtual live performance, but the present invention is not limited to this, and the performer user's singing or singing actions (performance actions) may not be performed in real time, but may be performed in advance, and the singing voice and singing actions (performance actions) may be recorded in the distribution server computer 100 as performer avatar information, and the recorded singing voice and singing actions (performance actions) may be played back as the live performance progresses, thereby holding a virtual live performance.

In addition, instead of the performer user's singing or singing actions (performance), the singing or singing actions (performance actions) of a virtual performer created using computer graphics or the like can be recorded as performer avatar information on a distribution server computer, and a virtual live performance can be held by playing back the recorded singing voice and singing actions (performance actions) as the live performance progresses.

In this way, when singing or singing actions (performance actions) are performed in advance to record and play back performer avatar information on the distribution server computer 100 to distribute a virtual live performance, the configuration of the virtual space content distribution system at the time of distribution will be made up of the distribution server computer 100, the administrator terminal 150, and the viewer terminal 300, and will not include the performer terminal 200, which is a performer user terminal that can be used by a performer user to operate the performer avatar 1 in the virtual space.

1 Performer avatar 100 Distribution server computer 150 Administrator terminal 200 Performer terminal 300 Viewer terminal

Claims (11)

A virtual space content distribution system for distributing virtual space content in a virtual space, comprising:
a viewer user terminal capable of outputting a video of the virtual space content and usable by a viewer user who performs an operation to operate a viewer avatar that can participate in a virtual space that is a target of the virtual space content in the virtual space ;
a server computer connected to the viewer user terminal via a communication network and capable of executing at least processes related to the virtual space and processes related to the distribution of the virtual space content;
Including,
The viewer users are multiple, and the viewer user terminal is assigned to each viewer user;
each of the viewer user terminals, when displaying a viewer avatar viewpoint image which is a viewpoint image of the virtual space corresponding to the viewer avatar, generates the viewer avatar viewpoint image based on virtual space data of the virtual space transmitted from the server computer ;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
the virtual space data includes performer avatar motion data for generating motions of the performer avatar;
The viewer user terminal is capable of generating the movement of the performer avatar in a plurality of generation modes having different processing loads when generating the movement of the performer avatar based on the performer avatar movement data.
A virtual space content delivery system comprising: 2. The virtual space content delivery system according to claim 1 , wherein the virtual space content is content in which the performer avatar performs a performance in the virtual space. The virtual space content distribution system described in claim 1 or 2, characterized in that a common viewpoint image that is displayed in common on each of the viewer user terminals, rather than a viewpoint image corresponding to each of the viewer avatars, can be generated by the server computer and distributed to the viewer user terminal. The virtual space content distribution system described in claim 3, characterized in that during a specific period in which the viewer user terminal can select and display either the viewer avatar viewpoint video or the common viewpoint video, the viewer user terminal generates and displays the common viewpoint video. 4. The virtual space content delivery system according to claim 3 , wherein during a special period when the processing load on the server computer is large, the common viewpoint video is also generated and displayed by the viewer user terminal. The virtual space content distribution system described in claim 1 or 2, characterized in that the server computer transmits advance data including basic data of the virtual space and basic data of avatars participating in the virtual space to each of the viewer user terminals before starting the virtual space content. The virtual space content distribution system described in claim 6, characterized in that, for viewer user terminals where it is not possible to confirm that the advance data has been completely received, the server computer does not transmit the virtual space data to the viewer user terminals, but transmits a common viewpoint video that is displayed in common on each of the viewer user terminals. The viewer user terminals include a first viewer user terminal and a second viewer user terminal,
3. The virtual space content delivery system according to claim 1 , wherein said server computer is capable of transmitting to said second viewer user terminal said virtual space data different from that transmitted to said first viewer user terminal. The virtual space content distribution system described in claim 8, characterized in that the second viewer user terminal is capable of generating an image of a second viewer avatar that performs actions different from the actions of a first viewer avatar of a first viewer user using the first viewer user terminal based on virtual space data different from the virtual space data transmitted to the first viewer user terminal. A virtual space content distribution program for distributing virtual space content in a virtual space via a network using at least a server computer, comprising:
the avatars capable of participating in the virtual space that is the subject of the virtual space content include viewer avatars that operate in the virtual space by operating a viewer user terminal that is a terminal available to a viewer user and is connected to the server computer via the network;
The viewer users are multiple, and the viewer user terminal is assigned to each viewer user;
a step of generating, in each of the viewer user terminals, a viewer avatar viewpoint image, which is a viewpoint image of the virtual space corresponding to the viewer avatar, based on virtual space data of the virtual space transmitted from the server computer;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
The virtual space data is data for generating movements of the performer avatar, and includes performer avatar movement data that can generate the movements of the performer avatar in a plurality of generation modes that have different processing loads when generating the movements of the performer avatar on the viewer user terminal.A virtual space content distribution program characterized in that the virtual space data includes performer avatar movement data that can generate the movements of the performer avatar in a plurality of generation modes that have different processing loads when generating the movements of the performer avatar on the viewer user terminal. A virtual space content distribution method for distributing virtual space content in a virtual space via a network using at least a server computer, comprising:
the avatars capable of participating in the virtual space that is the subject of the virtual space content include viewer avatars that operate in the virtual space by operating a viewer user terminal that is a terminal available to a viewer user and is connected to the server computer via the network;
The viewer users are multiple, and the viewer user terminal is assigned to each viewer user;
a step of generating, in each of the viewer user terminals, a viewer avatar viewpoint image, which is a viewpoint image of the virtual space corresponding to the viewer avatar, based on virtual space data of the virtual space transmitted from the server computer;
The virtual space content is content in which a performer user can participate in the virtual space as a performer avatar,
A virtual space content distribution method characterized in that the virtual space data is data for generating movements of the performer avatar, and includes performer avatar movement data that can generate the movements of the performer avatar in a plurality of generation modes that have different processing loads when generating the movements of the performer avatar on the viewer user terminal .

Images