JP5803956B2 - Karaoke system and karaoke device - Google Patents

Description

  The present invention relates to a karaoke system for synthesizing and reproducing a plurality of in-performance images photographed in synchronism with music reproduction and a karaoke apparatus used therefor.

  A normal musical piece is a form in which a plurality of performance parts (singing parts) are performed in synchronization, and in recent karaoke systems, one specified performance part is excluded for the purpose of singing a specific part or practicing musical instrument performance. Minus-one performances that play only with other performance parts are widespread.

  In Patent Document 1, the state of a player who is performing the minus one performance is photographed with a camera or the like on the karaoke apparatus side, and the images and images (hereinafter referred to as images) during the performance are transmitted to a server via a communication network such as the Internet. A technique for recording, accumulating, and synthesizing so as to collaborate by extracting images during performance corresponding to the same music piece is disclosed. As a result, it is possible to synthesize a co-starring image that appears as if the performers of different performance parts are co-starring.

JP 2012-73428 A

  Conventionally, when the co-star images are combined into a plurality of images, the co-star images are combined in an arrangement in which the extracted images during performance are arranged in time series in the respective recording orders. However, when a musical piece is co-starred in an actual live performance or the like, there are many cases where the standing position of the musical instrument player corresponding to each part is customarily determined. As described above, the co-star images in which the playing images of each part are arranged in the order of recording as described above, while there is customary arrangement regularity of each part according to the genre and organization form of the music to be played, etc. I felt uncomfortable when I saw it.

  The objective of this invention is providing the karaoke system and karaoke apparatus which can implement | achieve natural arrangement | positioning on a musical instrument performance in a co-starring image.

In order to achieve the above object, the present invention automatically designates the part type of a minus one performance part to be subjected to minus one performance for a predetermined musical piece by image recognition from an external image of a separately photographed instrument. A karaoke apparatus comprising: a part type image designation means; and a part image photographing means for photographing an image during performance of the music piece of a player of the minus one performance part; and the part image photographing means of the karaoke apparatus, A karaoke system comprising: a server having part image recording means for recording an image during performance in association with song identification information of the predetermined music piece and part type information of the minus one performance part; Either the device or the server performs the minus one performance by designating the music piece and the minus one performance part. In addition, the playing image that has been recorded corresponding to the piece identification information of the designated music and corresponding to the part type information of the part other than the designated minus one playing part is extracted from the part image recording means. And a co-star image synthesizing unit that synthesizes the co-star images arranged in a grid pattern in accordance with a predetermined arrangement rule, and the karaoke apparatus displays the co-star image synthesized by the co-star image synthesizing unit. The co-starring image display means stores the arrangement pattern in which the extracted performance images are associated with combinations between the performance parts according to the number and type of the performance images. Refer to the table, and arrange and synthesize so as to change the position and size in the display area of the co-star image according to the number and part type information It is characterized in.

In order to achieve the above object, the present invention automatically designates the part type of a minus one performance part to be subjected to minus one performance for a predetermined musical piece by image recognition from an external image of a separately photographed instrument. Part type image designating means, part image photographing means for photographing a performance image of the music of the performer of the minus one performance part, and the performance image captured by the part image photographing means for an external server Is a karaoke apparatus having a performance image transmission means for transmitting the recording information in association with the music identification information of the predetermined music and the part type information of the minus one performance part, and When the one performance part is designated, the server corresponds to the song identification information of the designated song and the designated player is designated. Corresponding to the part type information part other than Nasuwan performance part to extract the performance in an image that has already been recorded, the performance in image thereof extraction, between each performance part in accordance with the type and number of playing in the image Referring to the table storing the arrangement pattern corresponding to the combination, it is arranged in a grid pattern while changing the position and size in the display area of the co-star image according to the number and the part type information. were synthesized, the played image is characterized by comprising a co-star image display means for displaying received from the server.

  According to the present invention, it is possible to realize a natural arrangement on musical instrument performance in a co-star image.

It is explanatory drawing which shows the main structures of a karaoke apparatus. It is a functional block diagram which shows the main structures of the control system of a control apparatus. It is a figure which shows the specific process of the characteristic process in a karaoke system. It is a figure which shows the musical instrument image table used for a musical instrument image recognition process. It is a flowchart which shows the specific control procedure of a musical instrument image recognition process. It is a figure which shows the basic form of the arrangement | positioning rule of each performance part at the time of a collaboration image being synthesize | combined. It is a flowchart which shows the processing content each performed by CPU of a karaoke apparatus and CPU of a server at the time of performing a minus one performance. It is a figure which shows the data format of each information transmitted / received and recorded in a flow, and those relationship. It is a figure which shows the example of the arrangement | positioning pattern in the case of changing suitably the position and magnitude | size in those collaboration images according to the kind and number of performance parts.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a case will be described in which the user mainly plays the electric guitar by himself / herself along with the performance of the karaoke apparatus.

<Main configuration>
FIG. 1 is an explanatory diagram showing a main configuration of a karaoke system including a karaoke apparatus according to the present embodiment. As shown in FIG. 1, a karaoke apparatus 10 includes a monitor television (hereinafter abbreviated as a monitor) 13 for displaying lyrics telop, co-starring images described later, various operation screens, etc. on a CRT, a small monitor 14, and the like. Karaoke music composed of a musical instrument connection board 8 for connecting various electronic musical instruments (only an electric guitar is shown) including an electric guitar 4, an electric bass, an electronic drum, a keyboard, etc. (Referred to as “song” as appropriate) and music selection control such as reservation of playback of the music, transmission of a request signal to the server 58 including a request for transmission of the selected music, and the music indicated by the request signal Communication control such as reception of response signals including corresponding music data, etc., musical instrument of the type to be played minus minus (to be described later) A control device 20, such as specifying the guitar 4), is provided.

  Further, in this example, the karaoke apparatus 10 includes a mixing circuit 9 that mixes a musical instrument performance signal input from the musical instrument connection board 8, an audio signal input from the microphones 17 and 18, and a music reproduction signal. An amplifier 16 that performs volume balance between sound and music, echo adjustment, delay adjustment, amplification of mixing signal, pitch control of reproduced music, control of treble and bass, and the amplified signal output from this amplifier 16 As a floor type set of speakers 11 and 11, a set of speakers 12 and 12 for hanging the ceiling, a remote control 30 for remotely operating the control device 20, and a performance of a singer or a minus one performance to be described later And a camera 25 (corresponding to part image photographing means) for photographing a person.

  Although not shown, the musical instrument connection board 8 has input terminals (phone jacks) for connecting the output terminals (phone plugs) of the electric guitar 4 and an electronic drum, keyboard, electric bass, etc. (not shown), and input terminals. And an output terminal for outputting performance signals input to the. The musical instrument connection board 8 automatically detects that each musical instrument is connected by connecting the output terminal of each musical instrument to the input terminal, selects one performance signal of the connected musical instrument, and selects the selected performance signal. The performance signal can be output to an input terminal 44 of the control device 20 (see FIG. 2 described later). For example, a performance signal output from the electric guitar 4 is input to the musical instrument connection board 8 via a phone jack 4 a (see FIG. 2 described later) for connecting a phone plug of the electric guitar 4, and is output from the output terminal of the musical instrument connection board 8. The output performance signal is input to the input terminal 44 of the control device 20.

<Control system>
The configuration of the control system of the control device 20 will be described with reference to FIG. FIG. 2 is a functional block diagram showing the main configuration of the control system of the control device 20. The control device 20 includes a communication terminal 40 that connects the LAN line 15 to the device housing, an input terminal 44 that is connected to the output terminal of the musical instrument connection board 8, and an audio output terminal 41 that is connected to the audio input terminal of the amplifier 16. The video output terminal 42 connected to the video input terminal of the monitor 13 and the video output terminal 43 connected to the video input terminal of the monitor 14 are provided. The control device 20 is provided with a CPU 45 that executes various controls according to a control program. The CPU 45 includes a RAM 46 for temporarily storing data transmitted from the remote controller 30, song name data as song identification information of the selected song, song title data of a reserved song, a program executed by the CPU 45, and necessary A ROM 47 in which various data tables are stored is connected.

  Further, the CPU 45 has a video RAM 48 in which character video data for displaying lyrics telop and various message videos on the monitors 13 and 14, music data transmitted from an external server 58, and video data of co-star images ( A LAN board 50 for receiving various data such as lyrics data and video data via the communication terminal 40 and the communication network 15, and music data received by the LAN board 50 and moving images of co-star images. A RAM 49 for temporarily storing data and the like is connected.

  Further, the CPU 45 is connected to a MIDI sound source board 51 that inputs MIDI data included in the song data read from the RAM 49 and outputs a sound source signal from a sound source specified by the input MIDI data. . Further, the CPU 45 receives the output sound source signal and converts it into a signal that can be amplified by the amplifier 16, and a CD-ROM player 53 that reads background video data indicating a general background video separately. And are connected.

  Further, the CPU 45 is connected to a video control circuit 54 that performs video control for fitting and displaying the above-mentioned co-star image received from the server 58 and a minus one performance image captured by the camera 25.

  Further, the CPU 45 includes a conversion circuit 55 that converts an optical signal from the remote controller 30 received by the light receiving unit 38 of the control device 20 into a digital signal, and various buttons and keys 39 (tenkeys) provided on the housing of the control device 20. From the camera 25, a display circuit 56 that outputs a display signal to the LED 37 that lights when the music selection button is pressed, an input circuit 57 that inputs a switching signal generated when the various buttons and key 39 are pressed, and so on. Is connected to an input circuit 59 for inputting the photographic signal. The whole of the karaoke apparatus 10 and the server 58 corresponds to the karaoke system (karaoke system 1) described in each claim.

<Features of this embodiment>
As described above, the karaoke system 1 of this embodiment includes the karaoke apparatus 10 and the server 58 that are connected to each other via the communication network 15 so as to be able to send and receive information. In addition, a normal musical piece is a form in which a plurality of performance parts are played in synchronization, and in the karaoke system 1 of the present invention, a minus one performance in which only the other performance parts other than one performance part designated by the user are played together. I do. Then, the part type of the minus one performance part to be subjected to the minus one performance is designated by manual operation of the remote controller 30 of the karaoke apparatus 10 for a predetermined music specified separately, and the performance of the minus one performance part is designated by the camera 25. Take a picture of the person performing. When the performance is finished, the moving image data of the captured performance image is transmitted to the external server 58, and is associated with the song name of the designated music and the part type information of the minus one performance part. The moving image data is recorded in a large-capacity storage device 58a (equivalent to part image recording means) such as an HDD provided in the server 58.

  By recording and accumulating images during performance corresponding to each part type in the server 58 in this way, when performing a minus one performance by specifying a song name and a minus one performance part again after the next time, a co-star image (hereinafter referred to as a collaboration image). Image, and appropriately abbreviated as a collaboration image). Specifically, the moving image data of the performance image already recorded corresponding to the part type information of the part other than the designated minus one performance part corresponding to the music title of the designated music is stored in the large capacity of the server 58. A collaboration image can be synthesized by combining the extracted moving image data of the performance image extracted from the storage device 58a. This combining process itself may be performed by either the karaoke apparatus 10 or the server 58, but in this example, it is performed by the server 58. The collaborative image is displayed on the monitors 13 and 14 on the karaoke apparatus 10 side, so that the performer of the minus one performance is performed together with the performers of other parts (hereinafter referred to as a collaboration performance, and appropriately abbreviated as a collaboration performance). The visual effect is obtained.

  As a comparative example, a technique of synthesizing a collaboration image in an arrangement in which the extracted images during performance are arranged in time series in respective recording orders has been taken. However, when a musical piece is collaborated in an actual live performance or the like, there are many cases where the standing position of the performer corresponding to each part is conventionally determined.

  For example, when actually performing rock music in a band format, the vocal is located in the center of the stage, the bass is on the left side, the guitar is on the right side, and the drum is on the back side of the vocal. Each is generally located. The reason for this customary arrangement of each part is, for example, the historical theory that the band first became famous for the arrangement of such a part organization (such as the Beatles), and the drum that plays the role of rhythm. There are various theories such as the ease of taking the base eye contact, the high ears of the guitar with the right ear, and the theory that it is easier to hear the bass bass with the left ear. In (especially rock music), the positions of various musical instruments displayed in an image obtained by photographing the performance state are visible to the audience as a state in which the left and right and the vertical relationship with each other are determined.

  In this way, the performance images of each part are arranged in the recording order as described above, while there is customary arrangement regularity of each part according to the genre of the music to be played and the organization of the performance parts. I felt uncomfortable when I saw the collaboration image.

  On the other hand, in the karaoke system 1 of the present embodiment, the playing image extracted from the large-capacity storage device 58a of the server 58 is displayed in accordance with a customary arrangement rule according to the genre or organization form of the music as described above. The collaborative images arranged in a shape are synthesized.

<Specific processing contents>
In FIG. 3, the specific process of the said characteristic process in the karaoke system 1 of this embodiment is shown. First, as shown in FIG. 3A, the user designates the performance part by causing the camera 25 to photograph the musical instrument of the performance part to be subjected to the minus one performance. At this time, a real-time photographed image of the camera 25 is displayed together with a display prompting the user to shoot a musical instrument that is the subject of the minus one performance on the monitors 13 and 14. A musical instrument (the electric guitar 4 in this example) is positioned in the shooting frame of the video to be shot. As a result, the control device 20 to which the external image signal of the musical instrument is input via the camera 25 recognizes the type of the musical instrument through image recognition processing (see FIGS. 4 and 5 described later), and the corresponding performance part (musical instrument name). Is set as the type of the minus one performance part. At this time, a song name is also designated on a separate operation screen (not shown). As a result of the reservation operation performed in this way, the karaoke apparatus 10 transmits a request signal including the music title and information on the performance part to be subjected to minus one performance to the server 58 (see A in FIGS. 7 and 8 described later).

  Receiving the request signal, the server 58 extracts from the large-capacity storage device 58a the performance-in-progress images of the parts other than the designated performance part corresponding to the designated music title, and synthesizes the collaboration image using them. This composition may be performed in advance by the server 58. The moving image data of the collaboration image is returned as a response signal to the karaoke apparatus 10 together with the song data (MIDI data) corresponding to the designated song name (see B in FIGS. 7 and 8 described later).

  The karaoke apparatus 10 that has received the response signal from the server 58 reproduces the song data together with the moving image data of the collaboration image received from the server 58 as shown in FIG. In the example of the collaborative image shown in the figure, only the already-recorded vocal playing image is synthesized, and the arrangement location (upper right) corresponding to the target guitar for the minus one performance is blank.

  During the performance, as shown in FIG. 3C, the performance image of the performer of the guitar 4 in this example photographed by the camera 25 is displayed in the blank arrangement position. During this time, a performance image captured by the camera 25 is recorded, and when the performance of the music is finished, the video data of the recorded performance image is transmitted to the server 58 and stored in the mass storage device 58a. It is recorded (see D in FIGS. 7 and 8 to be described later). Thereby, when the same music is reserved for minus one performances other than vocals and guitars after the next time, a collaboration image of vocals and guitars is synthesized and displayed.

  Here, the image recognition processing performed when the performance part is set in FIG. 3A will be described with reference to FIGS. The control device 20 includes the musical instrument image table shown in FIG. 4 stored in the ROM 47, the CD-ROM 53, or the like. In this musical instrument image table, appearance images of various types of musical instruments that can be subjected to minus one performance are stored in association with the names of the musical instruments. Each appearance image is an appearance image of a typical model that is generally distributed in the corresponding musical instrument, or stores appearance images of several models that significantly show the appearance characteristics of the instrument. (In the figure, one each is shown). In the example of this embodiment, the karaoke device 10 side performs image recognition processing for identifying the type of instrument by comparing which appearance image in the instrument image table is the closest to the appearance image of the instrument photographed by the camera 25. To do.

  FIG. 5 is a flowchart showing a specific control procedure of the instrument image recognition process. In FIG. 5, first, in step S <b> 110, the CPU 45 of the karaoke apparatus displays the instrument photographing line on the monitors 13 and 14.

  Next, the process proceeds to step S120, and the CPU 45 of the karaoke apparatus photographs an image in the musical instrument photographing line with the camera 25.

  Next, the process proceeds to step S130, and the CPU 45 of the karaoke apparatus superimposes and displays the musical instrument photographing line video and the camera image on the monitors 13 and 14. Thus, the user can position the musical instrument within the shooting range of the camera 25 that is fixed.

  Next, the process proceeds to step S140, and it is determined whether or not the appearance image of the instrument photographed by the camera 25 matches with any appearance image in the instrument image table with sufficient certainty. Such determination of coincidence between two images is performed by a known image recognition method. If there is an appearance image in the musical instrument image table that matches the camera image with sufficient certainty, the determination is satisfied, and the routine goes to Step S150.

  In step S150, the CPU 45 of the karaoke apparatus specifies that the instrument name corresponding to the instrument appearance image in the instrument image table that matches the camera image is the performance part of the minus one performance part, and completes the image recognition of the instrument image. This flow is finished.

  On the other hand, in the determination in step S140, if none of the appearance images in the musical instrument image table matches the camera image with sufficient certainty, the determination is not satisfied, and the same procedure is repeated by returning to step S120.

  Through such a musical instrument image recognition process, the karaoke apparatus 10 automatically designates the part type of the minus one performance part from the appearance image of the musical instrument. Note that the vocal or chorus may be designated as the minus one performance part, and in this case, it is designated by manual operation of the remote controller 30 or the like.

  In addition, the arrangement of the performance parts when the collaboration image is synthesized by the server 58 follows the basic form of the arrangement rule as shown in FIG. 6, for example. In the example shown in the figure, the entire collaboration image is divided into a grid pattern of 2 rows × 3 columns. The base is in the first row and the first column, the vocal in the first row and the second column, the guitar in the first row and the third column, and the second row. A chord is assigned to the keyboard in the first column, the drum in the second row and the second column, and the chorus in the second row and the third column. With this arrangement, the bass is placed on the left side, the guitar is placed on the right side, and the drum is placed on the back side in a pseudo manner, as if it were the actual live performance stage. Close placement relationship.

  In the example of this embodiment, the position of the performance image of each performance part in the display area of the collaboration image is fixed to the specified position in the basic form as it is, and is displayed in the same size. For performance parts for which no images are being recorded on the server 58 yet, the part names as shown in the figure may be displayed at the corresponding positions, or the images of the corresponding musical instruments may be displayed, or as blanks. Also good. As described above, on the server 58 side, the position of the performance part that is the subject of the minus one performance this time is blank, and other than that, the performance image of the performance part that can be extracted from the large-capacity storage device 58a is combined. Synthesize (may be synthesized in advance). On the karaoke apparatus 10 side, the playing image of the minus one performance photographed by the camera 25 is inserted into the corresponding position in the collaboration image and reproduced. The fitting of the other image into one image and the fitting of the screen may be performed using a technique such as video overlay or superimposing, and will not be described in detail because it is well known.

  The basic form of the arrangement of each performance part shown in FIG. 6 is an example corresponding to the rock music in which the reserved music is played in a band format. However, according to other music genres and part organization, You may prepare the basic form according to the usual arrangement rule.

<System control procedure>
The processing contents respectively executed by the CPU 45 of the karaoke apparatus 10 and the CPU (not shown) of the server 58 when the user plays the electric guitar 4 while reproducing the karaoke music as described above are shown in FIG. The flowchart will be described.

  In FIG. 7, this flow is started when, for example, the user presses the power button of the control device 20 of the karaoke device 10 to turn on the power of the control device 20. In conjunction with the rising of the power supply of the control device 20, the power supply of the amplifier 16, the musical instrument connection board 8 and the monitors 13, 14 is started up. Note that, as described above, in this example, a case where the user performs with the electric guitar 4 connected to the musical instrument connection board 8 will be described. The server 58 is always on. Further, in the figure, in order to avoid the complexity of illustration, the contents related to the music data are omitted, and each term is abbreviated.

  In FIG. 7, first, in step SK10, the CPU 45 accepts a reservation operation for a piece of music that the user himself wants to play minus one. Specifically, an operation in which the user inputs the song name or song selection number of the song that the user wants to play minus one with the remote controller 30 is accepted.

  Next, the process proceeds to step SK13, where the CPU 45 of the karaoke apparatus outputs to the monitors 13 and 14 a display for prompting the user to photograph the appearance of the musical instrument playing the minus one on the monitor 25, and the appearance image of the musical instrument via the camera 25. Accepts shooting.

  Next, the process proceeds to step SK17, and the CPU 45 of the karaoke apparatus specifies the type of musical instrument photographed by the camera 25 by the musical instrument image recognition process described in FIG. 5 as a minus one performance part.

  Next, the process proceeds to step SK20, and the CPU 45 of the karaoke apparatus requests a collaboration image request signal including the reserved music received in step SK10 and the minus one performance part information specified in step SK17 (see A in FIG. 8 described later). ) To the server 58.

  Next, proceeding to step SS10, the CPU of the server 58 that has received the request signal is moving image data corresponding to the minus one designation of the request signal, that is, the playing image of the designated minus one playing part corresponding to the designated music piece. The collaboration image from which the image is missing is acquired from the mass storage device 58a. Then, the acquired moving image data of the collaboration image and the separately acquired music data are returned to the karaoke apparatus 10 of the request source (request signal transmission source) as a response signal (see B in FIG. 8 described later).

  Next, moving to step SK30, the CPU 45 of the karaoke apparatus downloads the moving image data and music data of the collaboration image, which is a response signal from the server 58.

  Next, the process proceeds to step SK40, and the CPU 45 of the karaoke apparatus determines whether minus one performance is designated in step SK10. In particular, when the minus one performance is not designated or when the vocal is designated as the minus one performance part, the determination is not satisfied and the routine goes to Step SK50.

  In step SK50, the CPU 45 of the karaoke apparatus sets so that the main vocal is designated as minus one in the same manner as in a normal karaoke performance. Then, the process proceeds to step SK70.

  On the other hand, if it is determined in step SK40 that a minus one performance part is designated, the determination is satisfied, and the routine goes to step SK60.

  In step SK60, the CPU 45 of the karaoke apparatus performs setting so that the musical instrument or chorus designated by the OFF operation is designated as minus one. Then, the process proceeds to step SK70.

  In step SK70, the CPU 45 of the karaoke apparatus synchronously reproduces the moving image data and the music data of the collaboration image downloaded in step SK30, and at the placement location corresponding to the song / instrument set as minus one in the collaboration image. Is displayed. It also records camera images.

  Next, the process proceeds to step SK80, and the CPU 45 of the karaoke apparatus transmits the recorded data (see D in FIG. 8 described later) to the server 58. Specifically, the main contents of the recording data to be transmitted are the moving image data of the playing image recorded by the camera 25 this time, the musical instrument information (corresponding to the part type information) which is the performance part played this time minus one. This is identification information (moving image ID) of moving image data of a collaboration image.

  Next, moving to step SS20, the CPU of the server 58 that has received the above data collaborates from the musical instrument of the performance part that was originally synthesized in the current collaboration image and the musical instrument of the performance part that was performed this time minus one. Re-determine where to place each performance part in the image. This re-determination of the arrangement location is performed according to the basic arrangement rule shown in FIG.

  Next, the process proceeds to step SS30, where the collaboration image is re-synthesized from the moving image data of the original collaboration image, the recording data of the currently-playing image recorded this time, and the information on the arrangement location re-determined in step SS20. Then, the collaboration image is recorded in the mass storage device 58a in association with the song title or performance part. Then, this flow ends.

  In the above, the procedure of step SK17 corresponds to the part type image designating means described in each claim, the procedure of step SS30 corresponds to the co-star image synthesizing means described in each claim, and the procedure of step SK70 is described in each claim. This corresponds to the co-star image display means, and the procedure of step SK80 corresponds to the in-performance image transmitting means described in each claim.

  FIG. 8 shows the data format of each piece of information sent and received and recorded in the above flow and the relationship between them. In addition, in order to avoid the complexity of illustration, description of music data (MIDI data) is omitted. A in FIG. 8 indicates a data format of a request signal transmitted from the karaoke apparatus 10 to the server 58. This request signal includes a user ID that is user identification information, a reserved song title (or song identification information), and musical instrument information of a performance part that is the subject of the minus one performance. Thereby, the server 58 can confirm the reserved music and the minus one performance part.

  B in FIG. 8 indicates a data format of a response signal downloaded from the server 58 by the karaoke apparatus 10. This response signal includes the user ID, the song name of the reserved music, the moving image ID and moving image data of the acquired collaboration image, and the musical instrument information of the playing image synthesized with the collaboration image. . Thereby, the karaoke apparatus 10 can appropriately fit and display the playing image of the minus one performance photographed by the reproduction of the collaboration image and the camera 25.

  FIG. 8C shows the data format of various moving image data recorded in the mass storage device 58a of the server 58. The recorded moving image data is roughly divided into two types, one is the moving image data of the already-combined collaboration image, and the other is the moving image data of the individual performance images received so far.

  One collaboration image includes the name of the corresponding music piece, the moving image ID and moving image data of the collaboration image, and the musical instrument information of the playing image synthesized with the collaboration image. The response signal B is obtained corresponding to the currently reserved music and the minus one performance part in the moving image data of the collaboration image. Note that the musical instrument information of the playing image included in the moving image data may be replaced with musical instrument information of a performance part capable of performing minus one performance instead.

  The other playing image includes the song name of the corresponding music piece, moving image data of the playing image recorded by the other user in the minus one performance, and instrument information of the performance part at that time. By recording the performance images individually in this way, it is possible to synthesize collaboration images in various combinations for the same music.

  FIG. 8D shows a data format of recorded data transmitted from the karaoke apparatus 10 to the server 58 after the collaboration performance (minus one performance). This recorded data includes at least the name of the music piece that has been collaborated, the video ID of the collaborative image used, the video data shot by the camera 25 during the minus one performance, and the instrument information of the performance part at that time. It is. The user ID included in the recorded data is used to specify the poster of the captured video data (for example, when a member is displayed as a musical instrument player in the synthesized image). This is not necessary for the synthesis of the collaboration image determined in accordance with the type).

  The server 58 that has received the recording data acquires the moving image data of the original collaboration image from the large-capacity storage device 58a based on the moving image ID included therein, and the musical instrument of the playing image synthesized with the collaboration image. Information can be identified. In addition, a performance image recorded in the large-capacity storage device 58a corresponding to the same music can be acquired. The server 58 corresponds to the same musical composition based on the specified musical instrument information, the acquired in-playing image, the song name included in the recording data, the current recording data, and the musical instrument information of minus one performance. A new collaboration image can be recombined according to the basic form of the six arrangement rules. A separate moving image ID is assigned to the collaboration image to form new moving image data, which is recorded in the mass storage device 58a.

  As described above, in the present embodiment, the playing image extracted from the large-capacity storage device 58a in accordance with the procedure of step SS30 is arranged in accordance with a customary arrangement rule according to the reserved music genre, organization form, and the like. Synthesize collaborative images arranged in a grid. The collaboration image synthesized in this way and displayed on the monitor 13 or 14 does not feel uncomfortable not only to the performer of minus one performance but also to other users who are present. As a result, a natural arrangement on the musical instrument performance in the collaboration image can be realized.

  In addition, when the processing capability of the karaoke apparatus 10 is high, the collaboration image synthesis process may be performed on the karaoke apparatus 10 side. In this case, an individual playing image may be downloaded from the server 58, and the collaboration image may be synthesized and displayed in real time before or during the minus one performance. Alternatively, a collaboration image may be downloaded from the server 58 and a new collaboration image may be re-synthesized based on the collaboration image and the recorded data after the minus one performance.

  Moreover, although the case where only one performance part (only a guitar) is performed in the karaoke apparatus 10 is shown in the above embodiment, a plurality of performance parts may be collaborated (minus one performance) at the same time. In this case, positions corresponding to the plurality of performance parts are blank in the collaboration image, and the images during performance photographed by the plurality of cameras 25 are respectively fitted and displayed.

  In the present embodiment, the part type of the minus one performance part is automatically specified by the instrument image recognition process (see FIG. 5 above) from the appearance image of the instrument photographed separately in the procedure of step SK17. Thereby, it is possible to easily specify the part type of the minus one performance part without requiring any manual operation or performance by the user (minus one player himself or another user in the same seat). In the present embodiment, the musical instrument image recognition process described above is performed on the karaoke apparatus 10 side, but the musical instrument image recognition process may be performed on the server 58 side. In this case, the request signal (A in FIG. 8) transmitted from the karaoke apparatus 10 to the server 58 in the procedure of step SK20 includes data of the appearance image of the instrument photographed by the camera 25 instead of the instrument information data. Will be.

  In addition, during the minus one performance, it is possible to synthesize a collaborative image and display the image during the performance of the performer of the minus one performance part while simultaneously reflecting the photographed performance image in real time. Alternatively, a large processing load is imposed on the communication network 15. On the other hand, in the present embodiment, in particular, the performance image captured during the current minus one performance in the procedure of step SK70 is displayed in the existing collaboration image, and after the completion, the next time for the same music in the procedure of step SS30. By re-synthesizing the collaboration image so as to be reflected at the subsequent minus one performance, the processing load on the karaoke apparatus 10 and the communication network 15 can be greatly reduced. Thereby, the synthesis | combination and display of a collaboration image are realizable using the karaoke apparatus 10 and the communication network 15 of simple structure.

  In the present embodiment, in particular, in the procedure of step SS30, the extracted in-performance image is arranged and synthesized at each specified position in the display area of the collaboration image. Thereby, the arrangement | positioning relationship of each part of the extracted image during performance can be visually recognized clearly. Further, since the collaborative images can be synthesized simply by arranging the extracted images during performance at respective specified positions according to a predetermined arrangement rule, the processing load on the server 58 in the procedure of step SS30 can be reduced. As for the playing image of the part that has not been recorded in the large-capacity storage device 58a of the server 58 and could not be extracted, the name of the part type and the image of the corresponding instrument may be arranged at the specified position. .

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and technical idea of the present invention. Hereinafter, such modifications will be described.

<When changing the position and size according to the number and type of performance parts>
In the above embodiment, the arrangement and size of each performance image in the collaboration image are fixed, but the present invention is not limited to this. For example, as shown in FIG. 9, the position and size thereof may be changed as appropriate according to the performance part type and number of images being played.

  As an example shown in the figure, when only one performance part (only vocal) is displayed, as shown in FIG. 9A, only the in-performance image of the one performance part is enlarged to the same size as the entire display area. And display. On the other hand, when displaying with two performance parts (adding a drum), as shown in FIG. 9B, the display area of the two performance parts is enlarged to a size that divides the entire display area into two parts. indicate. At this time, in the case of the vocal and drum in the example shown in the figure, the vocal is arranged on the left and the drum is arranged on the right. Further, although not particularly illustrated, the arrangement may be reversed left and right, or may be arranged in two vertically divided parts with the vocals above and the drums below.

  What is important here is that the drum is not always placed in front of the vocal by convention, and such an uncomfortable arrangement, that is, the upper and lower divided arrangement with the drum on and the vocal below is not performed. Although not specifically shown, when only two performance parts other than vocals, for example, a bass and a drum are displayed, the bass is not conventionally positioned on the right and bottom of the drum. For this reason, the arrangement is either left and right divided into two with the base left and the drum right, or two vertically divided with the base up and the drum down.

  If three performance parts, vocal, drum, and bass are displayed, the bass is located to the left of the vocal and the drum is located under the vocal, as shown in FIG. 9C. In this arrangement, the display area is divided into four parts, top, bottom, left, and right, so that the performance image of each performance part can be displayed larger than the basic form of FIG. In this way, when multiple performance parts are displayed, the display area is displayed in an arrangement and size that maximizes the display area while maintaining only the positional relationship between the two performance parts in any combination. To do.

  When four performance parts, vocal, drum, bass, and guitar are displayed, as shown in Fig. 9 (d), the bass is located under the vocal, the guitar is located to the right of the vocal, and the right of the vocal The drum is located below. This is the basic arrangement for a four-piece collaboration of a combination of vocals, drums, bass, and guitar. The fact that the guitar is located beside the vocal is a result of considering that it is a prominent performance part compared to the bass.

  As described above, in the procedure of step SS30, the extracted performance images are arranged and synthesized so as to change the position and size in the display area of the collaboration image according to the number and part type information. At the time of synthesis, the size and arrangement pattern may be determined at that time so as to consider the arrangement relationship between the performance parts in accordance with the number and type of the extracted images during performance, or in advance in a table or the like. Arrangement patterns corresponding to all combinations of performance parts may be stored and determined in accordance therewith.

  This makes it possible to synthesize a collaboration image that maximizes the entire display area of the collaboration image using only the extracted part-playing images while maintaining the positional relationship between the individual performance parts. The visibility of the part is improved.

  In addition to those already described above, the methods according to the above-described embodiments and modifications may be used in appropriate combination.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

DESCRIPTION OF SYMBOLS 1 Karaoke system 4 Electric guitar 10 Karaoke apparatus 13, 14 Monitor 25 Camera (part image photographing means)
58 server 58a mass storage device (part image recording means)

Claims (4)

Part type image designation means for automatically designating a part type of a minus one performance part to be subjected to a minus one performance for a predetermined musical piece by image recognition from an external image of a separately photographed instrument,
Part image photographing means for photographing an image during performance of the music of the performer of the minus one performance part;
A karaoke device having
A server having a part image recording unit that records the performance image captured by the part image capturing unit of the karaoke apparatus in association with song identification information of the predetermined music piece and part type information of the minus one performance part; ,
A karaoke system comprising:
In any one of the karaoke apparatus and the server,
When performing a minus one performance by specifying a song and a minus one performance part, it has already been recorded corresponding to the song identification information of the designated song and the part type information of parts other than the designated minus one performance part. A performance image synthesizing means for synthesizing the co-starring images obtained by extracting the performance images being performed from the part image recording means and arranging the extracted performance images in a grid pattern according to a predetermined arrangement rule;
The karaoke device
A co-star image display means for displaying the co-star image synthesized by the co-star image synthesis means ;
The co-starring image synthesizing means is
With reference to a table storing arrangement patterns in which the extracted performance images are associated with combinations between performance parts according to the number and type of the performance images, the number and type of the images are determined according to the part type information. A karaoke system characterized in that the co-star images are arranged and synthesized so as to change the position and size in the display area .   2. The karaoke system according to claim 1, wherein the co-starring image synthesizing unit synthesizes the performance image captured during the current minus one performance by reflecting the same musical piece during the subsequent minus one performance.   3. The karaoke system according to claim 1, wherein the co-starring image synthesizing unit synthesizes the extracted in-playing performance images by arranging them at respective specified positions in the display area of the co-starring image. Part type image designation means for automatically designating a part type of a minus one performance part to be subjected to a minus one performance for a predetermined musical piece by image recognition from an external image of a separately photographed instrument,
Part image photographing means for photographing an image during performance of the performer of the minus one performance part;
During the performance of transmitting to the external server so as to record the performance image captured by the part image capturing means in association with the song identification information of the predetermined music and the part type information of the minus one performance part A karaoke apparatus having image transmission means,
When the minus one performance music and the minus one performance part are designated, the server corresponds to the song identification information of the designated music and the part type information of the part other than the designated minus one performance part Corresponding table that has already recorded performance images and stores the extracted performance images corresponding to combinations between performance parts according to the number and type of performance images. see, received while changing the position and size in the display area of the played image depending on their number and the part type information is synthesized to place in a grid pattern, the played image from the server A karaoke apparatus comprising co-starring image display means for display.

Images