WO2018008434A1

WO2018008434A1 - Musical performance presentation device

Info

Publication number: WO2018008434A1
Application number: PCT/JP2017/023262
Authority: WO
Inventors: 昌利 ▲高▼木
Original assignee: 株式会社エム・ティー・ケー
Priority date: 2016-07-05
Filing date: 2017-06-23
Publication date: 2018-01-11
Also published as: JP2018005019A

Abstract

The musical performance presentation device is characterized by comprising a control means (14) for generating virtual reality data to be projected by a VR projection means (30) from virtual reality content data (22) and musical performance reaction information, wherein the control means (14) generates the virtual reality data and causes the VR projection means (30) to project the data such that a player perceives that he/she is playing music using an input means (40) in a real space, or an audience can perceive that they are listening to the musical performance in a real space.

Description

Performance performance device

The present invention relates to a performance performance device that realizes performance in a virtual reality space.

Conventionally, as a performance device, a device capable of performing while realizing that a performer is performing in a virtual reality space has been proposed. For example, there is a performance device disclosed in Patent Document 1.

The performance device of Patent Document 1 performs automatic performance by reproducing viewpoint data detection means for directly or indirectly detecting a user's viewpoint, display means mounted in the vicinity of the user's eyes, and automatic performance data. The viewpoint detection means detects the automatic performance means to be performed, the performance image generation means for generating a performance image in synchronization with the automatic performance by the automatic performance means, and the drawing angle of the performance image generated by the performance image generation means. And a control unit that controls the display unit to display the performance image with the drawing angle changed according to the user's viewpoint. The performance image is a virtual player performing a real or virtual musical instrument.

JP 2007-271698 A

However, in a conventional performance apparatus, for example, in a band performance, only a specific part is played, and the performer can perform in a virtual reality space while recognizing that the band is performing in a real space. could not.

The present invention has been made in view of such circumstances, and in the virtual reality space, as one of the performers and viewers, along with other performers and viewers, the performers can play musical instruments and the like in the real space. It is an object of the present invention to provide a performance performance device that can be recognized as if the user is performing using the keyboard, or can be recognized as if the viewer is viewing the performance in a real space.

The performance performance apparatus according to claim 1, wherein VR performance means for visually and auditorily projecting virtual reality data for forming a virtual reality space composed of video and sound in front of the performer or viewer. And virtual reality content data, which is individual content data related to video and sound, constituting the virtual reality space to be projected by the VR projection means, and performance response information consisting of the performance information of the performer or the response information of the viewer are input. And a control means for generating virtual reality data to be projected onto the VR projection means from the virtual reality content data and the performance response information. The control means uses the input means as if the player is in real space. It can be recognized as if it is playing or the viewer is as if It generates virtual reality data that can be recognized as to view played in real space, characterized in that to the projection in VR projection means.

The performance performance apparatus according to claim 2 is characterized in that the virtual reality content data includes content data relating to video and sound of other performers or other viewers input in advance.

The performance performance device according to claim 3 is characterized in that the virtual reality content data includes content data related to video and sound of other performers or other viewers generated in real time.

The performance performance device according to claim 4 includes content data for generating virtual reality space having an atmosphere matched with existing music information previously generated as sound or live music information performed by the performer in the virtual reality content data. It is characterized by that.

According to the present invention, in the virtual reality space, as a performer or viewer, it is recognized as if the performer is playing with an instrument or the like in a real space together with other performers and viewers. Or it can be perceived as if the viewer is watching the performance in real space.

It is a block diagram which shows an example of a structure of the performance performance apparatus which concerns on this invention. It is a block diagram which shows an example of the player apparatus of the performance performance apparatus. It is a block diagram which shows an example of the VR content of the performance apparatus. It is a block diagram which shows an example of the viewer apparatus of the performance performance apparatus.

Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a performance performance device according to the present invention. FIG. 2 is a block diagram showing an example of a player device of the performance device. FIG. 3 is a block diagram showing an example of VR content of the performance device. FIG. 4 is a block diagram showing an example of a viewer device of the performance device.

The performance performance device 1 according to the present invention is for realizing a performance in a virtual reality space. In the virtual reality space, a performance can be performed while feeling that the performer is performing. You can enjoy the music as if you were in a concert venue. Furthermore, the performance performance device 1 performs, for example, in the performance of a band, even if one of the performers is in a different location from the other performers, The purpose is to make you feel as if you are performing (ensemble) in the same place (virtual reality space) with other band members (other performers). In the case of a viewer, for example, the purpose is to allow the viewer to feel as if he / she is watching and listening to the performance as a concert participant at the concert venue, even if he / she is not at the concert venue.

As shown in FIG. 1, the performance device 1 is composed of a player device 10 and a viewer device 100, and if necessary (details will be described later) by connecting to the information communication network 5 or the like. It is connected to other player devices 10 and viewer devices 100.

As shown in FIG. 1, the player device 10 (viewer device 100) includes a VR performance processing terminal 12 and VR projection means 30 worn by a player P who is a player (for example, a viewer A who is a concert audience). It is comprised from the input means 40 which inputs information. The performance presentation processing terminal 12 includes a control unit 14 that controls the entire player device 10 (viewer device 100), and a storage unit 20 that stores data such as virtual reality content data (VR content data) 22. It is configured. The performance presentation processing terminal 12 can perform various controls such as an electronic computer, a portable information terminal (for example, a smartphone or a tablet terminal), a game machine, and the like, can output video and audio, and can receive information from the input means 40. It can be obtained.

The VR projection means 30 is for visually and auditorily projecting virtual reality data for forming a virtual reality space composed of video and sound in front of the player P and the viewer A who are performers. As shown in FIGS. 2 and 4, for example, an HMD (Head Mounted Display) 32 that is a head-mounted display is used. In the description of the present application, the VR projection unit 30 will be described using the HMD 32. However, the VR projection unit 30 is not limited to the HMD 32, and any smartphone having a function as the VR projection unit 30 may be used. It is not limited by the means. In the examples of FIGS. 2 and 4, the HMD 32 is described as including an acoustic device such as headphones, but an acoustic device different from the HMD may be used. It is not limited by.

The virtual reality content data 22 stored in the storage means 20 is individual content data related to video and sound that constitutes a virtual reality space to be projected by the VR projection means 30. The input means 40 inputs performance response information including performance information of the player P who is a player and reaction information of the viewer A to the control means 14 (details will be described later). Further, the control unit 14 has a function of generating virtual reality data to be projected on the VR projection unit 30 from the virtual reality content data 22 and the performance response information.

Specifically, the virtual reality content data 22 is roughly divided into, for example, video data 24 and audio data 26 when roughly classified. Then, by combining the video data 24 and the audio data 26, the virtual reality space can be experienced by the VR projection means 30, and becomes basic information for generating virtual reality data to be projected.

More specifically, the video data 24 of the virtual reality content data 22 includes, for example, background / stage apparatus data 24a, audience data 24b, band member data 24c, player data 24d, and the like. The background / stage apparatus data 24a is, for example, a fictitious virtual concert venue, a real concert venue, a baseball stadium, a stadium, etc., as information on a background (place, etc.) image that can form a virtual reality space in three dimensions. In addition, it is information on fictional and real scenes such as the coast and the plain. Further, the background / stage apparatus data 24a includes, for example, information on a stage apparatus that can form a virtual reality space in three dimensions and a video of the surrounding atmosphere, such as spotlights, large screens, smoke, flames, and illuminations Information about fictitious and real devices.

The audience data 24b includes, for example, appearance information of a person other than the player P and the band member who can appear three-dimensionally in the virtual reality space, information on things worn by the person such as penlights and towels, Information on the person's reaction, such as applause and cheers. The audience data 24b includes information on each gender, and may be information on a real person or information on a fictitious person, or may include real animals and fictional animals.

The band member data 24c and the player data 24d include the appearance information of the player P himself / herself and other performers who can appear three-dimensionally in the virtual reality space, the musical instrument to be played, the manner and expression of performance, vocals, If it is a chorus, it is information of a singing state, and if it is a dancer, it is information of the dance (information of movement). The band member data 24c and the player data 24d may be information on a real person or information on a fictitious person, or a photograph of a face of a real person inserted in a video of a fictitious body. The combination is arbitrary. Further, the band member data 24c and the player data 24d can be used as the audience data 24b, and the usage method itself is arbitrary. When the band member data 24c and the player data 24d are regarded as performers, they are, for example, vocals, guitars, basses, drums, keyboards, choruses, dancers, DJs, orchestras, and the like.

More specifically, the sound data 26 of the virtual reality content data 22 includes, for example, music data 26a and environmental sound data 26b. The music data 26a is, for example, performance information of existing music, and is information on performance sounds and singing voices of musical instruments. The music data 26a is, for example, information such as vocals, guitars, basses, drums, and keyboards as musical instruments, which are sounds formed in band performance, and each of them is individually turned off (player track off, etc.). It can also be turned on or turned on.

Further, the music data 26a can also include performance motion data, which is information on the operations of the player P and other performers in the performance for each music. In the case of an existing music piece, performance motion data can be provided in advance. However, when the player P performs live and inputs or in the case of a music piece without performance motion data, the tempo of the music piece is provided. Etc. can be used so that general-purpose performance motion data can be used. Such general-purpose performance motion data can also be stored as music data 26a.

The environmental sound data 26b is, for example, in the case of a concert, information on ambient sounds at the concert venue, and information on sound due to audience response without the video data 24 is also environmental sound data 26b.

The virtual reality content data 22 may include content data related to video and sound of other performers or other viewers input in advance, and other performers generated in real time or It is also possible to include content data related to the video and sound of other viewers, and further generate a virtual reality space with an atmosphere that matches existing music information previously generated as sound or live music information performed by the performer It is also possible to include content data for this purpose.

Note that the above-described example of the virtual reality content data 22 is merely an example, and is not limited as long as it contributes to the achievement of the object of the present application, that is, video and audio information constituting the virtual reality space. In the description of each figure, the virtual reality content data 22 is stored in the storage means 20 in advance. However, the virtual reality content data 22 is not necessarily stored in a fixed manner, and the external storage means is connected to the performance performance processing terminal. 12, the virtual reality content data 22 can be added to the storage means 20 or replaced. Moreover, it is also possible to add to the storage means 20 from the outside via the information communication network 5 as shown in FIG. The information communication network 5 is, for example, an electric communication line such as the Internet network. However, the information communication network 5 may simply be connected with a cable that directly connects the player device 10 and the viewer device 100 to each other. If the viewer apparatus 100 can be connected to the outside, it is not limited by the form.

Then, the control means 14 can recognize that the player P who is a performer is performing using the input means 40 in the real space, or the viewer P is viewing the performance in the real space. Virtual reality data that can be recognized by the VR projection unit 30 is generated.

According to the performance performance device 1 of the present application configured as described above, in the virtual reality space, as one of the performers and viewers, the player P (performer) is as if the player P (performer) is real. It can be recognized as if the user is performing using the musical instrument 44 or the like in the space, or the viewer A can be recognized as if the user is viewing the performance in the actual space.

Next, an embodiment in which the player P performs using the performance device 1 will be described. In this embodiment, in a band performance, a player P who is one of the performers plays his / her part with his / her musical instrument, as if it is the same place (virtual reality) as other band members (other performers). A method of use that allows the user to feel as if they are performing in a space is described with reference to FIG.

First, the player P uses the performance presentation processing terminal 12 to determine what kind of music, what kind of band member (other performers, etc.) and what kind of place (concert venue, etc.) Which part (which may be a vocal) is assigned is set by the input device 42 (keyboard, mouse, controller, etc.). The player P connects and sets the musical instrument 44 or the microphone 46 used by the player P to the performance presentation processing terminal (control means 14).

The interface between the musical instrument 44 and the performance presentation processing terminal 12 (control means 14) may be in any form as long as it is electronically connectable, and may be wired or wirelessly connected. The signal system may be a digital system or an analog system. For example, the musical instrument 44 and the performance presentation processing terminal can be connected by USB or by the Midi method, and an analog audio signal from the musical instrument 44 is sent to the control means 14 of the performance presentation processing terminal 12. You may make it send. When the performance presentation processing terminal 12 is a game machine, performance response information as performance information is input to the performance presentation processing terminal 12 (control means 14) via a controller which is an input device 42. You may do it.

Next, the player P wears the HMD 32, which is the VR projection means 30, so that the virtual reality space can be experienced through video and sound. Then, when the player P operates the performance presentation processing terminal 12 to start the performance, the virtual reality space set by the player P is projected onto the HMD 32, and the player P appears to be at a concert venue or the like set by the player P. It becomes a state like this. In addition, other performers and spectators appear in front of the eyes via the HMD 32. Then, the player P starts to play or sing the music in accordance with other imaginary players. The player P sends performance response information, which is performance information, to the control means 14 of the performance presentation processing terminal 12 via these input means 40 by playing the instrument 44 or singing via the microphone 46. Then, the control means 14 generates virtual reality data that can be recognized as if the player P is playing using the input means 40 (for example, the musical instrument 44) in a real space, and projects it onto the HMD 32 that is the VR projection means 30. Let

In the virtual reality content data 22 used, for example, when the information that the guitar part of the music is turned off is used, the performance performance information is sent to the control means 14 using the guitar played by the player P as the instrument 44. As a result, the virtual reality space projected on the HMD 32 can be recognized as if the player P is playing with his / her part. The other players are projected as if they are playing with their instruments and parts, and the player P feels as if they are having a session with other players in the real space, in the virtual reality space. Can play.

In addition, by setting a concert venue and audience at the performance presentation processing terminal 12, the performance is performed with other performers at the set concert venue, and performing while feeling the audience's reaction in front of the set audience. It is possible to perform in a virtual reality space with the same feeling.

If the musical piece to be played is prepared in advance in the virtual reality content data 22, the performance of other performers and the state of the audience's reaction can be determined based on the prepared virtual reality content data 22. However, if the performance of other performers or the reaction of the audience is not prepared as virtual reality content data 22 or is not sufficient information, the control means 14 determines from the performance of the player P. However, it is also possible to analyze the tempo of music and make it appear in the virtual reality space. Even when music prepared in advance in the virtual reality content data 22 is used, it is also possible to freely set the combination of the band configuration, the presence / absence of the audience, the appearance, and the like.

Furthermore, by using a motion capture unit that can capture the operation of the player P as the input device 42, the performance operation of the player P can be sent to the control unit 14 and reflected in the virtual reality space. As the motion capture means, for example, sensor information of the HMD 32 such as an acceleration sensor, a gyro sensor, a proximity sensor, and a line-of-sight detection sensor provided in the HMD 32 can be used. Then, the player P himself / herself can see the performance of his / her performance in the virtual reality space with the eyes of another player. Of course, it is possible to allow the player P to see a state other than himself / herself in the virtual reality space with the gaze of another player, the gaze of the audience, or another gaze.

In the above description, the other performers and spectators are fictitious ones prepared in advance in the virtual reality content data 22, but other performances connected by the information communication network 5 as shown in FIG. It is also possible to connect to the player device 10 of the person in real time and to make a session as if playing in the same place while being in a remote place in real time. In this case, not all performers actually perform, and some performers may be fictitious prepared in advance in the virtual reality content data 22 as described above. In addition, a spectator who is a viewer A, which will be described later, is connected in real time to the viewer apparatus 100 connected by the information communication network 5, and seems to be listening to the performance at the same place even though it is a remote place in real time. It is also possible to make it.

In addition, virtual reality data forming a virtual reality space generated by the performance of the player P can be stored as it is or distributed through the information communication network 5, and the virtual reality data can be reproduced on other media. It is possible to store and distribute in two or three dimensions as much as possible. Further, for example, it is possible to score by comparing the performance of the player P with the information of the existing performance.

Next, an example in which the viewer A views the performance using the performance device 1 will be described. In the present embodiment, a usage method in which the user can experience a virtual reality space as if viewing a band performance at an actual concert venue will be described with reference to FIG.

First, the viewer A uses the performance presentation processing terminal 12 to set what kind of band performance he / she wants to view at the concert venue, using the input device 42 (keyboard, mouse, controller, etc.). Further, the viewer A attaches the microphone 46, the penlight 48, etc., which are input means 40 for inputting performance response information as his / her reaction information, as necessary, to the performance stage processing terminal 12 (control means 14). Connect to and set.

Next, the viewer A wears the HMD 32 that is the VR projection means 30 to make the virtual reality space feelable by video and sound. Then, when the viewer A operates the performance presentation processing terminal 12 to start the performance, the virtual reality space set by the viewer A is projected onto the HMD 32, and the band performance set by the viewer A is the concert venue set by himself / herself. You can watch it as if you were in a situation. The viewer A raises a voice in accordance with the band performance and inputs the voice to the microphone 46, or shakes another input device 42 (keyboard, mouse, controller, etc.) or a penlight 48 to respond to his / her reaction. When input, performance response information, which is reaction information, is sent to the control means 14 of the performance performance processing terminal 12 via these input means 40. Then, the control means 14 can recognize that the viewer A is reacting in the real space using the input means 40 (for example, the microphone 46), and the virtual reality data is obtained in accordance with the reaction of the viewer A. It can also be generated and projected on the HMD 32 which is the VR projection means 30.

Also, by using a motion capture means that can capture the motion of the viewer A as the input device 42, it is possible to send the motion of the viewer A's reaction to the control means 14 and reflect it in the virtual reality space. As the motion capture means, for example, sensor information of the HMD 32 such as an acceleration sensor, a gyro sensor, a proximity sensor, and a line-of-sight detection sensor provided in the HMD 32 can be used. Then, it is possible for the viewer A himself / herself to see the state of his / her viewer as the viewer in the virtual reality space. Of course, it is possible to allow the viewer A to see a state other than himself / herself in the virtual reality space with the gaze of the performer, the gaze of another spectator (viewer), or another gaze.

In the above description, the performer and other spectators (viewers) are fictitious prepared in advance by the virtual reality content data 22, but are connected by the information communication network 5 as shown in FIG. It is possible to connect the player device 10 and the viewer device 100 of the performer in real time, and to feel as if they are performing in front of their eyes while being in a remote place in real time. In this case, not only all performers and spectators (viewers) are actually performing or watching, but some performers and spectators (viewers) have virtual reality content data as described above. An imaginary thing prepared in advance in 22 may be used.

Further, the virtual reality data forming the virtual reality space generated while the viewer A is observing can be stored as it is or distributed through the information communication network 5. This virtual reality data Can be stored and distributed two-dimensionally or three-dimensionally so as to be reproducible on other media.

The present invention is capable of various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for explaining the present invention and do not limit the scope of the present invention. In other words, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications within the scope of the claims and within the scope of the equivalent invention are considered to be within the scope of the present invention.

This application is based on Japanese Patent Application No. 2016-133087 filed on Jul. 5, 2016. The specification, claims, and entire drawings of Japanese Patent Application No. 2016-133087 are incorporated herein by reference.

As described above, according to the present invention, in the virtual reality space, as a performer or viewer, the performer performs as if using a musical instrument or the like in an actual space together with other performers or viewers. It is possible to provide a performance device that can be recognized as if the viewer is viewing the performance in a real space.

DESCRIPTION OF SYMBOLS 1 ... Performance performance apparatus 5 ... Information communication network 10 ... Player apparatus 12 ... Performance performance processing terminal 14 ... Control means 20 ... Storage means 22- ... Virtual reality content data (VR content data)
24 .... Video data 24a ... Background / stage apparatus data 24b ... Audience data 24c ... Band member data 24d ... Player data 26 ... Audio data 26a ... Music data 26b ... environmental sound data 30 ... VR projection means 32 ... HMD
40 .... Input means 42 ... Input device 44 ... Musical instrument 46 ... Mic 48 ... Penlight 100 ... Viewer device

Claims

In performance performance equipment that realizes performance in virtual reality space,
VR projection means for visually and auditorily projecting virtual reality data for forming the virtual reality space composed of video and sound in front of the performer or viewer;
Virtual reality content data, which is individual content data related to video and sound, constituting the virtual reality space to be projected on the VR projection means;
Input means for inputting performance response information including performance information of the performer or response information of the viewer;
Control means for generating the virtual reality data to be projected on the VR projection means from the virtual reality content data and the performance response information,
The control means can recognize the player as if he / she was playing using the input means in a real space, or recognize the virtual as if the viewer was watching a performance in the real space. A performance performance apparatus, characterized in that reality data is generated and projected by the VR projection means.
The performance presentation device according to claim 1, wherein the virtual reality content data includes the content data related to video and sound of other performers or other viewers input in advance.
The performance presentation apparatus according to claim 1, wherein the virtual reality content data includes the content data relating to video and sound of other performers or other viewers generated in real time.
The virtual reality content data includes the content data for generating the virtual reality space having an atmosphere that matches existing music information generated in advance as sound or live music information performed by the performer. Item 1. The performance performance device according to item 1.