JP6835182B2 - Electronic musical instruments, control methods for electronic musical instruments, and programs - Google Patents

Description

The present invention relates to an electronic musical instrument that reproduces a singing voice in response to an operation of an operator such as a keyboard, a control method for the electronic musical instrument, and a program.

Conventionally, there is known an electronic musical instrument that outputs a singing voice synthesized by a voice-synthesized synthesis method in which elements of recorded voice are connected and processed (for example, Patent Document 1).

JP-A-9-050287

However, this method, which can be said to be an extension of the PCM (Pulse Code Modulation) method, requires a long recording work at the time of development, and is complicated to smoothly connect the elements of the recorded voice. It was necessary to perform various calculation processes and make adjustments to make the singing voice natural.

Therefore, an object of the present invention is to mount a learned model that learns the singing voice of a certain singer, so that the electronic singer can sing well at a pitch specified by the operation of each operator by the performer. It is to provide musical instruments.

In the electronic musical instrument of the embodiment, the musical instrument sound type data corresponding to the set tone color, and the musical instrument sound form data corresponding to each of the plurality of pitch data acquired according to the user's operation of playing the chord, is obtained. Acoustic features output by the trained model by inputting lyrics data stored in the memory before the performance into the trained model that is acquired according to the performance operation of the chord and learned the singing voice of a certain singer. A chord generated by acquiring quantity data according to a performance operation of the chord, synthesizing the acquired acoustic feature data indicated by the acquired acoustic feature data with the acquired instrument sound type data, and synthesizing the sum. The inferred singing voice data of is output even if the user does not sing.

According to the present invention, by mounting a learned model that has learned the singing voice of a certain singer, an electronic musical instrument that the certain singer sings well at a pitch specified by the operation of each operator by the performer can be obtained. Can be provided.

It is a figure which shows the appearance example of one Embodiment of an electronic keyboard instrument. It is a block diagram which shows the hardware configuration example of one Embodiment of the control system of an electronic keyboard instrument. It is a block diagram which shows the structural example of the voice learning part and the voice synthesis part. It is explanatory drawing of the 1st Embodiment of the statistical speech synthesis processing. It is explanatory drawing of the 2nd Embodiment of the statistical speech synthesis processing. It is a figure which shows the data structure example of this embodiment. It is a main flowchart which shows the control processing example of the electronic musical instrument in this embodiment. It is a flowchart which shows the detailed example of the initialization process, the tempo change process, and the song start process. It is a flowchart which shows the detailed example of a switch process. It is a flowchart which shows the detailed example of the automatic performance interrupt processing. It is a flowchart which shows the detailed example of a song play process.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an external example of an embodiment 100 of an electronic keyboard instrument. The electronic keyboard instrument 100 includes a keyboard 101 composed of a plurality of keys as a performance operator, and a first switch panel 102 for instructing various settings such as volume designation, song playback tempo setting, song playback start, and accompaniment playback. , A second switch panel 103 for selecting songs and accompaniment songs, selecting tones, and the like, and an LCD 104 (Liquid Keyboard Display) for displaying lyrics, scores, and various setting information during song playback. Further, although not particularly shown, the electronic keyboard instrument 100 is provided with a speaker for emitting a musical sound generated by the performance on a back surface portion, a side surface portion, a back surface portion, or the like.

FIG. 2 is a diagram showing a hardware configuration example of an embodiment of the control system 200 of the electronic keyboard instrument 100 of FIG. In FIG. 2, the control system 200 includes a CPU (central processing unit) 201, a ROM (read-only memory) 202, a RAM (random access memory) 203, a sound source LSI (large-scale integrated circuit) 204, a speech synthesis LSI 205, and FIG. The key scanner 101 to which the keyboard 101, the first switch panel 102, and the second switch panel 103 are connected, and the LCD controller 208 to which the LCD 104 of FIG. 1 is connected are connected to the system bus 209, respectively. Further, a timer 210 for controlling the sequence of automatic performance is connected to the CPU 201. Further, the musical sound output data 218 (musical instrument sound type data) and the singing voice output data 217 output from the sound source LSI 204 and the voice synthesis LSI 205, respectively, are the analog musical sound output signal and the analog singing voice output signal by the D / A converters 211 and 212, respectively. Is converted to. The analog music output signal and the analog singing voice output signal are mixed by the mixer 213, and after the mixed signal is amplified by the amplifier 214, they are output from a speaker or an output terminal (not shown). Further, the output of the sound source LSI 204 is input to the speech synthesis LSI 205. The sound source LSI 204 and the voice synthesis LSI 205 may be integrated into one LSI.

The CPU 201 executes the control operation of the electronic keyboard instrument 100 of FIG. 1 by executing the control program stored in the ROM 202 while using the RAM 203 as the work memory. In addition to the control program and various fixed data, the ROM 202 stores song data including lyrics data and accompaniment data.

The timer 210 used in the present embodiment is mounted on the CPU 201, and counts the progress of the automatic performance of the electronic keyboard instrument 100, for example.

The sound source LSI 204 reads the musical tone type data from, for example, a waveform ROM (not shown) in accordance with the sound generation control instruction from the CPU 201, and outputs the music to the D / A converter 211. The sound source LSI 204 has the ability to oscillate up to 256 voices at the same time.

When the voice synthesis LSI 205 is given the text data of the lyrics and the information on the pitch as the singing voice data 215 from the CPU 201, the voice synthesis LSI 205 synthesizes the voice data of the corresponding singing voice and outputs it to the D / A converter 212.

The musical sound output data of a predetermined sounding channel (multiple channels are possible) output from the sound source LSI 204 is input to the speech synthesis LSI 205 as the musical sound output data 220 for the vocal sound source.

The key scanner 206 constantly scans the key press / release state of the key 101 of FIG. 1, the switch operation state of the first switch panel 102, and the second switch panel 103, and interrupts the CPU 201. Communicate change.

The LCD controller 609 is an IC (integrated circuit) that controls the display state of the LCD 505.

FIG. 3 is a block diagram showing a configuration example of a voice synthesis unit, a sound effect addition unit, and a voice learning unit in the present embodiment. Here, the voice synthesis unit 302 and the sound effect addition unit 320 are built into the electronic keyboard instrument 100 as one function executed by the voice synthesis LSI 205 of FIG.

The voice synthesis unit 302 obtains singing voice data 215 including lyrics and pitch information instructed by the CPU 201 via the key scanner 206 of FIG. 2 based on the key pressed on the keyboard 101 of FIG. 1 by the song reproduction process described later. By inputting, the singing voice voice output data 321 is synthesized and output. At this time, the processor of the speech synthesis unit 302 refers to the trained acoustic model set in the acoustic model unit 306 in response to the operation of any one of the plurality of keys (operators) on the keyboard 101. Then, the singing voice data 215 including the lyrics information and the pitch information associated with any of the keys is input, and the spectrum information 318 output by the acoustic model unit 306 and the sound source LSI 204 are output in response to the input. Based on the musical sound output data 220 for the vocal sound source, the sound source information use inference singing voice data output process for outputting the singing voice voice output data 321 (first sound source information use inference singing voice data) inferring the singing voice of the singer is executed.

By inputting the singing voice data 215 including the effect information, the sound effect adding unit 320 can generate a vibrating effect, a tremolo effect, a wow effect, etc. with respect to the singing voice output data 321 output by the voice synthesizing unit 302. A sound effect is added, and the final singing voice voice output data 217 (see FIG. 2) is output.

As shown in FIG. 3, for example, the voice learning unit 301 may be implemented as a function executed by a server computer 300 existing outside the electronic keyboard instrument 100 of FIG. 1. Alternatively, although not shown in FIG. 3, the voice learning unit 301 is built into the electronic keyboard instrument 100 as a function executed by the voice synthesis LSI 205 if the processing capacity of the voice synthesis LSI 205 in FIG. 2 is sufficient. May be good.

The speech learning unit 301 and the speech synthesis unit 302 of FIG. 2 are implemented based on, for example, the technique of "statistical speech synthesis based on deep learning" described in Non-Patent Document 1 below.

(Non-Patent Document 1)
Yoshi Hashimoto, Shinji Takagi, "Statistical Speech Synthesis Based on Deep Learning," Journal of the Acoustical Society of Japan, Vol. 73, No. 1 (2017), pp. 55-62

As shown in FIG. 3, for example, the voice learning unit 301 of FIG. 2, which is a function executed by the external server computer 300, includes a learning text analysis unit 303, a learning acoustic feature extraction unit 304, and a model learning unit 305. Including.

In the voice learning unit 301, as the learning singing voice voice data 312, for example, a voice recorded by a singer singing a plurality of singing songs of an appropriate genre is used. Further, as the learning singing voice data 311, the lyrics text of each song is prepared.

The learning text analysis unit 303 inputs the learning singing voice data 311 including the lyrics text and analyzes the data. As a result, the learning text analysis unit 303 estimates and outputs the learning language feature sequence 313, which is a discrete numerical sequence expressing phonemes, pitches, etc. corresponding to the learning singing voice data 311.

The learning acoustic feature amount extraction unit 304 is for learning, which is acquired through a microphone or the like by a certain singer singing the lyrics text corresponding to the learning singing voice data 311 in accordance with the input of the learning singing voice data 311. Singing voice voice data 312 is input and analyzed. As a result, the learning acoustic feature amount extraction unit 304 extracts and outputs the learning acoustic feature amount series 314 representing the voice features corresponding to the learning singing voice voice data 312.

The model learning unit 305 uses the following equation (1) to obtain a learning language feature sequence 313 (this is used).
And the acoustic model (put this)
From), the learning acoustic feature series 314 (this is
Probability of generating (put this)
Acoustic model that maximizes
Is estimated by machine learning. That is, the relationship between the language feature series that is text and the acoustic feature series that is voice is expressed by a statistical model called an acoustic model.

here,
Indicates an operation to calculate the arguments listed below it, which gives the maximum value for the function listed on the right.

The model learning unit 305 is an acoustic model calculated as a result of performing machine learning according to equation (1).
The model parameter expressing the above is output as the learning result 315.

As shown in FIG. 3, for example, the learning result 315 (model parameter) is stored in the ROM 202 of the control system of the electronic keyboard instrument 100 of FIG. 2 at the time of factory shipment of the electronic keyboard instrument 100 of FIG. 1, and is stored in the ROM 202 of the electronic keyboard instrument 100 of FIG. When the power of 100 is turned on, the ROM 202 in FIG. 2 may be loaded into the acoustic model unit 306 described later in the voice synthesis LSI 205. Alternatively, as shown in FIG. 3, for example, the learning result 315 can be obtained by operating the second switch panel 103 of the electronic keyboard instrument 100, such as the Internet or a USB (Universal Serial Bus) cable (not shown). It may be downloaded from the network of the above to the acoustic model unit 306 described later in the speech synthesis LSI 205 via the network interface 219.

The speech synthesis unit 302, which is a function executed by the speech synthesis LSI 205, includes a text analysis unit 307, an acoustic model unit 306, and a vocal model unit 308. The voice synthesis unit 302 synthesizes the singing voice output data 321 corresponding to the singing voice data 215 including the lyrics text by predicting it using a statistical model called an acoustic model set in the acoustic model unit 306. Execute the process.

The text analysis unit 307 inputs singing voice data 215 including information on the phonemes, pitches, etc. of the lyrics designated by the CPU 201 of FIG. 2 as a result of the performer's performance in accordance with the automatic performance, and analyzes the data. As a result, the text analysis unit 307 analyzes and outputs the language feature sequence 316 expressing the phonemes, part of speech, words, etc. corresponding to the singing voice data 215.

By inputting the language feature sequence 316, the acoustic model unit 306 estimates and outputs the corresponding acoustic feature sequence 317. That is, the acoustic model unit 306 uses the following equation (2) to input the language feature quantity series 316 (again) from the text analysis unit 307.
), And the acoustic model set as the learning result 315 by machine learning in the model learning unit 305.
Based on and, the acoustic feature series 317 (again)
Probability of generating (put this)
Estimated value of acoustic feature series 317 that maximizes
To estimate.

The vocalization model unit 308 generates the singing voice output data 321 corresponding to the singing voice data 215 including the lyrics text designated by the CPU 201 by inputting the acoustic feature amount series 317. The singing voice voice output data 321 is converted into the final singing voice voice output data 217 by adding a sound effect by the sound effect addition unit 320 described later, and the mixer 213 and the amplifier 214 are converted from the D / A converter 212 of FIG. The sound is output from a speaker (not shown).

The acoustic features represented by the learning acoustic feature series 314 and the acoustic feature series 317 include spectral information that models the human vocal tract and sound source information that models the human vocal cords. As the spectrum information, for example, a mer cepstrum, a line spectrum pair (Line Spectral Pairs: LSP), or the like can be adopted. As the sound source information, a fundamental frequency (F0) indicating the pitch frequency of human voice and a power value can be adopted. The vocalization model unit 308 includes a synthetic filter unit 310. The output of the musical sound output data 220 for the vocal sound source of the predetermined sound source channel (several channels is possible) of the sound source LSI 204 of FIG. 2 is input to the synthetic filter unit 310. The synthetic filter unit 310 is a portion that models the human vocal tract, forms a digital filter that models the vocal tract based on a series of spectral information 318 sequentially input from the acoustic model unit 306, and inputs from the sound source LSI 204. The vocal tract voice output data 321 of the digital signal is generated and output using the vocal tract sound output data 220 for the vocal tract sound source of the predetermined sounding channel (multiple channels are possible) as the excitation source signal. The musical sound output data 220 for a vocal sound source input from the sound source LSI 204 is polyphonic for a predetermined sounding channel.

As described above, the musical sound output data 220 for a vocal sound source generated and output by the sound source LSI 204 based on the performance on the performer's keyboard 101 (FIG. 1) operates based on the spectrum information 318 input from the acoustic model unit 306. The singing voice audio output data 321 is output from the synthetic filter unit 310 by inputting to the synthetic filter unit 310. The singing voice voice output data 321 generated and output in this way uses the musical instrument sound generated by the sound source LSI 204 as the sound source signal. For this reason, the fidelity is slightly lost compared to the singing voice of the singer, but the atmosphere of the instrument sound set by the sound source LSI204 remains well, and the voice quality of the singing voice of the singer remains well, which is an effective singing voice. It is possible to output the output data 321. Further, in the vocoder mode, since polyphonic operation is possible, it is possible to produce an effect in which a plurality of singing voices are harmonized.

The sound source LSI 204 operates so as to supply the output of a plurality of predetermined sound channels to the voice synthesis LSI 205 as the musical sound output data 220 for the vocal sound source, and at the same time, output the output of the other channels as the normal musical sound output data 218. You may. As a result, the accompaniment sound can be pronounced with a normal musical instrument sound, or the musical instrument sound of the melody line can be pronounced, and at the same time, the singing voice of the melody can be uttered from the voice synthesis LSI 205.

The musical sound output data 220 for a vocal sound source input to the synthetic filter unit 310 in the vocoder mode may be any signal, but due to its nature as a sound source signal, it contains many harmonic components and lasts for a long time, for example, a brass sound. , String sounds, instrument sounds such as organ sounds are preferred. Of course, even if an instrument sound that does not comply with such a standard at all, for example, an instrument sound such as an animal bark, is used for a large effect, a very interesting effect can be obtained. As a specific example, for example, the data obtained by sampling the barking of a pet dog is input to the synthetic filter unit 310 as a musical instrument sound. Then, the speaker is made to sound based on the singing voice output data 217 output from the synthetic filter unit 310 and the sound effect addition unit 320. This has the very interesting effect of making it sound like your dog is singing the lyrics.

The sampling frequency for the learning singing voice data 312 is, for example, 16 KHz (kilohertz). Further, when, for example, a mel cepstrum parameter obtained by a mel cepstrum analysis process is adopted as the spectrum parameter included in the learning acoustic feature quantity series 314 and the acoustic feature quantity series 317, the update frame period is, for example, 5 msec (milliseconds). ). Further, in the case of the mer cepstrum analysis process, the analysis window length is 25 msec, the window function is the Blackman window, and the analysis order is 24th order.

The singing voice voice output data 321 output from the voice synthesis unit 302 is further added with sound effects such as a vibrato effect, a tremolo effect, or a wow effect by the sound effect addition unit 320 in the voice synthesis LSI 205.

The vibrato effect is an effect of periodically swinging the pitch with a predetermined swing width (depth) when the sound is stretched in singing. As the configuration of the sound effect addition unit 320 for adding the vibrato effect, for example, the technique described in Patent Document 2 or 3 below can be adopted.
<Patent Document 2>
Japanese Unexamined Patent Publication No. 06-167976 <Patent Document 3>
Japanese Unexamined Patent Publication No. 07-199931

The tremolo effect is the effect of playing the same or multiple notes in small steps. As the configuration of the sound effect addition unit 320 for adding the tremolo effect, for example, the technique described in Patent Document 4 below can be adopted.
<Patent Document 4>
Japanese Unexamined Patent Publication No. 07-028471

The wah effect is to obtain the effect of talking as "wah wah" by moving the frequency at which the gain of the bandpass filter peaks. As the configuration of the sound effect addition unit 320 for adding the wah effect, for example, the technique described in Patent Document 5 below can be adopted.
<Patent Document 5>
Japanese Unexamined Patent Publication No. 05-006173

The performer is in a state of continuing to output the singing voice output data 321 (pressing the first key) by the first key (first operator) on the keyboard 101 (FIG. 1) for instructing the singing voice. When the second key (second operator) on the keyboard 101 is repeatedly pressed in the state), the sound effect addition unit 320 receives the first switch panel of the vibrato effect, the tremolo effect, or the wow effect. The sound effect selected in advance in 102 (FIG. 1) can be added.

In this case, the performer further presses the second key repeatedly with respect to the pitch of the first key designated as the singing voice, so that the pitch difference between the second key and the first key becomes the desired pitch difference. By designating to, the degree of pitch effect in the sound effect addition unit 320 can be changed. For example, if the pitch difference between the 2nd key and the 1st key is one octave apart, the maximum value of the depth of the sound effect is set, and as the pitch difference becomes smaller, the degree of the sound effect increases. It can be changed to be weaker.

The second key on the keyboard 101 to be repeatedly hit may be a white key, but if it is a black key, for example, it is less likely to interfere with the performance operation of the first key for specifying the pitch of the singing voice. ..

As described above, in the present embodiment, the singing voice voice output data 321 output from the voice synthesizing unit 302 is further added with various sound effects by the sound effect adding unit 320, and the final singing voice voice output data. It is possible to generate 217.
If the key press on the second key is not detected for the set time (for example, several hundred milliseconds), the addition of the sound effect ends.

As another embodiment, such a sound effect can be produced even if the second key is pressed once while the first key is pressed, that is, even if the second key is not repeatedly pressed as described above. It may be added. In this case as well, the depth of such a sound effect may be changed according to the pitch difference between the first key and the second key. Further, the sound effect may be added while the second key is pressed, and the addition of the sound effect may be terminated in response to the detection of the release of the key to the second key.

As another embodiment, such an acoustic effect may be added even if the first key is released after the second key is pressed while the first key is pressed. Further, such a pitch effect may be added by detecting a "trill" in which the first key and the second key are repeatedly pressed.

In this specification, the playing method to which these acoustic effects are added may be referred to as "so-called legato playing method" for convenience.

Next, a first embodiment of the statistical speech synthesis process including the speech learning unit 301 and the speech synthesis unit 302 of FIG. 3 will be described. In the first embodiment of the statistical speech synthesis processing, as the acoustic model represented by the learning result 315 (model parameter) set in the acoustic model unit 306, the above-mentioned Non-Patent Document 1 and the following Non-Patent Document 2 The described HMM (Hidden Markov Model: Hidden Markov Model) is used.

(Non-Patent Document 2)
Shinji Sakou, Keijiro Saino, Yoshihiko Minamikaku, Keiichi Tokuda, Tadashi Kitamura "Singing Voice Synthesis System that Can Automatically Learn Voice Quality and Singing Style" Information Processing Society of Japan Research Report Music Information Science (MUS) 2008 (12 (2008-MUS-074) )), Pp. 39-44, 2008-02-08

In the first embodiment of the statistical speech synthesis process, when the user utters lyrics along a certain melody, the vocal cord vibration and the characteristic parameters of the vocal tract characteristics of the singing voice are uttered while changing with time. The lyrics are learned by the HMM acoustic model. More specifically, the HMM acoustic model is a phoneme-based model of the spectrum, fundamental frequency, and their time structure obtained from the singing voice data for learning.

First, the processing of the voice learning unit 301 of FIG. 3 in which the HMM acoustic model is adopted will be described. The model learning unit 305 in the voice learning unit 301 includes a learning language feature amount series 313 output by the learning text analysis unit 303 and the learning acoustic feature amount series 314 output by the learning acoustic feature amount extraction unit 304. By inputting, the HMM acoustic model having the maximum likelihood is learned based on the above-mentioned equation (1). The likelihood function of the HMM acoustic model is shown by Eq. (3) below.

here,
Is the amount of acoustic features in frame t, T is the number of frames,
Is the state series of the HMM acoustic model,
Represents the state number of the HMM acoustic model in frame t. Also,
State
From state
Represents the state transition probability to
Is the mean vector
, Covariance matrix
Normal distribution and state
Represents the output probability distribution of. The learning of the HMM acoustic model based on the likelihood maximization standard is efficiently performed by using the Expectation Maximization (EM) algorithm.

The spectral parameters of the singing voice can be modeled by continuous HMM. On the other hand, since the log fundamental frequency (F0) is a variable-dimensional time-series signal that takes a continuous value in the voiced section and has no value in the unvoiced section, it cannot be directly modeled by a normal continuous HMM or discrete HMM. Therefore, MSD-HMM (Multi-Space probability Distribution HMM), which is an HMM based on a probability distribution on multiple spaces corresponding to variable dimensions, is used, and Melkepstram is used as a spectral parameter with a multidimensional Gaussian distribution and a logarithmic fundamental frequency (F0). Simultaneously model voice sound as a Gaussian distribution in 1-dimensional space and unvoiced sound as a Gaussian distribution in 0-dimensional space.

Further, it is known that the characteristics of phonemes constituting a singing voice fluctuate under the influence of various factors even if the phonemes have the same acoustic characteristics. For example, the spectrum of phonemes and the logarithmic fundamental frequency (F0), which are basic phoneme units, differ depending on the singing style and tempo, the lyrics before and after, the pitch, and the like. Factors that affect such acoustic features are called contexts. In the statistical speech synthesis processing of the first embodiment, an HMM acoustic model (context-dependent model) in consideration of context can be adopted in order to accurately model the acoustic features of speech. Specifically, the learning text analysis unit 303 considers not only the phonemes and pitches for each frame, but also the phonemes immediately before and after, and the vibrato and accents immediately before and after the current position, and the language features for learning. The series 313 may be output. In addition, decision tree-based context clustering may be used to improve the efficiency of context combinations. This is a method of clustering HMM acoustic models for each combination of similar contexts by dividing a set of HMM acoustic models into a tree structure using a binary tree. Each node of the tree has a question that divides the context such as "Is the previous phoneme / a /?", And each leaf node has a learning result 315 (model parameter) corresponding to a specific HMM acoustic model. ). Any combination of contexts can reach any leaf node by following the tree along the question in the node, and the learning result 315 (model parameter) corresponding to that leaf node can be selected. By selecting an appropriate decision tree structure, it is possible to estimate an HMM acoustic model (context-dependent model) with high accuracy and high generalization performance.

FIG. 4 is an explanatory diagram of the HMM decision tree in the first embodiment of the statistical speech synthesis process. For each context-dependent phoneme, each state of the phoneme is associated with, for example, an HMM consisting of the three states 401 of # 1, # 2, and # 3 shown in FIG. 4 (a). The arrows that input and output for each state indicate the state transition. For example, state 401 (# 1) is, for example, a state that models the vicinity of the start of the phoneme. Further, the state 401 (# 2) is, for example, a state in which the vicinity of the center of the phoneme is modeled. Further, state 401 (# 3) is, for example, a state that models the vicinity of the end of the phoneme.

Further, depending on the phoneme length, the length at which each of the states 401 of # 1 to # 3 shown by the HMM of FIG. 4 (a) continues is determined by the state continuation length model of FIG. 4 (b). The model learning unit 305 of FIG. 3 is derived from the learning language feature quantity series 313 corresponding to the context of a large number of phonemes related to the state continuation length extracted from the learning singing voice data 311 of FIG. 3 by the learning text analysis unit 303 of FIG. , The state continuation length determination tree 402 for determining the state continuation length is generated by learning, and is set in the acoustic model unit 306 in the speech synthesis unit 302 as the learning result 315.

Further, the model learning unit 305 of FIG. 3 is, for example, a learning acoustic feature corresponding to a large number of phonemes related to the merkepstram parameter extracted from the learning singing voice data 312 of FIG. 3 by the learning acoustic feature amount extraction unit 304 of FIG. From the quantity series 314, a merkepstram parameter determination tree 403 for determining the merkepstram parameter is generated by learning, and the learning result 315 is set in the acoustic model unit 306 in the speech synthesis unit 302.

Further, the model learning unit 305 of FIG. 3 is, for example, a learning corresponding to a large number of fundamental frequencies (F0) extracted from the learning singing voice data 312 of FIG. 3 by the learning acoustic feature amount extraction unit 304 of FIG. A log fundamental frequency determination tree 404 for determining the log fundamental frequency (F0) is generated by learning from the sound feature sequence 314 for learning, and is set in the acoustic model unit 306 in the speech synthesis unit 302 as the learning result 315. As described above, the voiced section and the unvoiced section of the log fundamental frequency (F0) are modeled as 1-dimensional and 0-dimensional Gaussian distributions by MSD-HMM corresponding to the variable dimension, respectively, and the log fundamental frequency determination tree. 404 is generated.

In addition, the model learning unit 305 of FIG. 3 is a learning language feature quantity sequence corresponding to the context of a large number of phonemes related to the state continuation length extracted from the learning singing voice data 311 of FIG. 3 by the learning text analysis unit 303 of FIG. From 313, a determination tree for determining the context such as pitch vibrates and accents may be generated by learning, and the learning result 315 may be set in the acoustic model unit 306 in the speech synthesis unit 302.

Next, the processing of the voice synthesis unit 302 of FIG. 3 in which the HMM acoustic model is adopted will be described. The acoustic model unit 306 inputs the language feature sequence 316 related to the phonemes, pitches, and other contexts of the lyrics output by the text analysis unit 307, and the decision trees 402 and 403 illustrated in FIG. 4 for each context. , 404 and the like are connected, and the acoustic feature sequence 317 (spectrum information 318 and sound source information 319) that maximizes the output probability is predicted from each connected HMM.

At this time, the acoustic model unit 306 uses the language feature quantity series 316 (=) input from the text analysis unit 307 according to the above equation (2).
) And the acoustic model (=) set as the learning result 315 by machine learning in the model learning unit 305.
) And the acoustic feature series 317 (=)
) Is generated (=)
) Is the estimated value (=) of the acoustic feature series 317 that maximizes
) Is estimated. Here, the above-mentioned equation (2) is a state sequence estimated by the state continuation length model of FIG. 4 (b).
By using, it is approximated by the following equation (4).

here,
And
When
Each state
The mean vector and covariance matrix in. Language feature series
The average vector and the covariance matrix are calculated by tracing each decision tree set in the acoustic model unit 306 using. Estimated value of acoustic feature series 317 from Eq. (4) (=
) Is the average vector (=
), But
Is a discontinuous series that changes stepwise at the part where the state changes. When the synthetic filter unit 310 synthesizes the singing voice output data 321 from such a discontinuous acoustic feature amount series 317, the synthetic voice becomes low quality from the viewpoint of naturalness. Therefore, in the first embodiment of the statistical speech synthesis processing, the model learning unit 305 may adopt an algorithm for generating the learning result 315 (model parameter) in consideration of the dynamic features. Static features
And dynamic features
From the acoustic feature series in frame t (=
) Is configured, the acoustic feature series (=) at each time
) Is represented by the following equation (5).

here,

Is a static feature series
Acoustic features series including dynamic features from
Is a matrix for finding. The model learning unit 305 solves the above-mentioned equation (4) as shown in the following equation (6) with the above-mentioned equation (5) as a constraint.

here,
Is a static feature series that maximizes the output probability while constraining the dynamic features. By considering the dynamic features, the discontinuity of the state boundary can be solved, the acoustic feature sequence 317 that changes smoothly can be obtained, and the synthetic filter unit 310 generates high-quality singing voice output data 321. It becomes possible.

Here, the phoneme boundaries of the singing voice data often do not match the notes boundaries defined by the score. Such temporal fluctuations can be said to be essential from the viewpoint of musical expression. Therefore, in the first embodiment of the statistical speech synthesis processing adopting the above-mentioned HMM acoustic model, in the vocalization of the singing voice, the time affected by various factors such as the difference in the phonation, the pitch and the rhythm, etc. Assuming that there is a bias, a technique for modeling the deviation between the timing of vocalization and the score in the training data may be adopted. Specifically, as a note-based deviation model, the deviation between the singing voice and the score seen in note units is represented by a one-dimensional Gaussian distribution, and it depends on the context in the same way as other spectral parameters and log fundamental frequency (F0). It may be treated as an HMM acoustic model. In singing voice synthesis using the HMM acoustic model including the context of "deviation", the time boundary represented by the score is first determined, and then the simultaneous probability of both the deviation model for each note and the phoneme state continuation length model is maximized. By doing so, it becomes possible to determine the time structure in consideration of the fluctuation of the notes in the training data.

Next, a second embodiment of the statistical speech synthesis process including the speech learning unit 301 and the speech synthesis unit 302 of FIG. 3 will be described. In the first embodiment of the statistical speech synthesis process, the acoustic model unit 306 is implemented by a deep neural network (DNP) in order to predict the acoustic feature sequence 317 from the language feature sequence 316. Correspondingly, the model learning unit 305 in the speech learning unit 301 learns the model parameters representing the non-linear conversion function of each neuron in the DNN from the language feature quantity to the acoustic feature quantity, and learns the model parameters. As 315, it is output to the DNN of the acoustic model unit 306 in the voice synthesis unit 302.

Usually, the acoustic feature amount is calculated in units of a frame having a width of, for example, 5.1 msec (milliseconds), and the language feature amount is calculated in units of phonemes. Therefore, the acoustic feature amount and the language feature amount have different time units. In the first embodiment of the statistical speech synthesis processing adopting the HMM acoustic model, the correspondence between the acoustic features and the language features is expressed by the state sequence of the HMM, and the model learning unit 305 describes the acoustic features and the language features. The correspondence between the above was automatically learned based on the learning singing voice data 311 and the learning singing voice data 312 of FIG. On the other hand, in the second embodiment of the statistical speech synthesis processing using DNN, the DNN set in the acoustic model unit 306 is the input language feature sequence 316 and the output acoustic feature sequence 317. Since it is a model representing a one-to-one correspondence between, it is not possible to train DNN using input / output data pairs having different time units. Therefore, in the second embodiment of the statistical speech synthesis processing, the correspondence between the acoustic feature series in the frame unit and the language feature series in the phoneme unit is set in advance, and the acoustic feature quantity and the language feature quantity in the frame unit are set in advance. A pair is generated.

FIG. 5 is an operation explanatory diagram of the speech synthesis LSI 205 showing the above-mentioned correspondence. For example, the singing phoneme string "/ k /" "/ i" which is a linguistic feature sequence corresponding to the lyrics character strings "ki", "ra", and "ki" (Fig. 5 (a)) of the nursery rhyme "Kirakira Hoshi". When / ”,“ / r / ”,“ / a / ”,“ / k / ”, and“ / i / ”(FIG. 5 (b)) are obtained, these language feature series are frame-based phoneme features. The quantity series (FIG. 5 (c)) is associated with a one-to-many relationship (relationship between (b) and (c) in FIG. 5). Since the language feature amount is used as an input to the DNN in the acoustic model unit 306, it needs to be expressed as numerical data. Therefore, as a language feature series, "Is the phoneme immediately before" / a / "? Binary data (0 or 1) for contextual questions such as "How many phonemes are contained in the current word?", Or numerical data obtained by concatenating consecutive answers are prepared. ..

The model learning unit 305 in the speech learning unit 301 of FIG. 3 in the second embodiment of the statistical speech synthesis processing is shown in FIG. 5 (b) in frame units as shown by the broken line arrow group 501 of FIG. The phoneme sequence of the corresponding learning language feature sequence 313 and the pair of the learning acoustic feature sequence 314 corresponding to FIG. 5C are sequentially given to the DNN of the acoustic model unit 306 for learning. The DNN in the acoustic model unit 306 includes a neuron group composed of an input layer, one or more intermediate layers, and an output layer, as shown as a group of gray circles in FIG.

On the other hand, at the time of speech synthesis, the phoneme sequence of the language feature sequence 316 corresponding to FIG. 5B is input to the DNN of the acoustic model unit 306 in the frame unit. As a result, the DNN of the acoustic model unit 306 outputs the acoustic feature amount series 317 in the frame unit as shown by the thick solid line arrow group 502 in FIG. Therefore, also in the vocalization model unit 308, the sound source information 319 and the spectrum information 318 included in the acoustic feature amount series 317 are given to the sound source generation unit 309 and the synthesis filter unit 310, respectively, in the above-mentioned frame unit, and the voice synthesis is executed. Will be done.

As a result, the vocalization model unit 308 outputs singing voice voice output data 321 of, for example, 225 samples (samples) for each frame, as shown by the thick solid line arrow group 503 in FIG. Since the frame has a time width of 5.1 msec, one sample is "5.1 msec ÷ 225 ≈ 0.0227 msec", and therefore the sampling frequency of the singing voice output data 321 is 1 / 0.0227 ≈ 44 kHz (kilohertz). Is.

The DNN learning is performed by the square error minimization standard calculated by the following equation (7) using the pair of the acoustic feature amount and the language feature amount in each frame.

here,
When
Are the acoustic features and language features in the t-th frame t, respectively.
Is the DNN model parameter of the acoustic model unit 306,
Is a non-linear transformation function represented by DNN. The DNN model parameters can be efficiently estimated by the backpropagation method. Considering the correspondence with the processing of the model learning unit 305 in the statistical speech synthesis represented by the above-mentioned equation (1), the learning of DNN can be expressed as the following equation (8).

Here, the following equation (9) holds.

As in the above equations (8) and (9), the relationship between the acoustic features and the language features is a normal distribution with the output of DNN as the average vector.
Can be represented by. In the second embodiment of the statistical speech synthesis process using DNN, a language feature sequence is usually used.
Covariance matrix independent of, i.e. the same covariance matrix in all frames
Is used. Also, the covariance matrix
When is an identity matrix, Eq. (8) shows a learning process equivalent to Eq. (7).

As described with reference to FIG. 5, the DNN of the acoustic model unit 306 estimates the acoustic feature sequence 317 independently for each frame. Therefore, the obtained acoustic feature sequence 317 includes discontinuities that deteriorate the quality of the synthesized speech. Therefore, in the present embodiment, for example, the quality of the synthesized speech can be improved by using the parameter generation algorithm using the dynamic features as in the case of the first embodiment of the statistical speech synthesis processing. it can.

The operation of the embodiment of the electronic keyboard instrument 100 of FIGS. 1 and 2 using the statistical speech synthesis processing described with reference to FIGS. 3 to 5 will be described in detail below. FIG. 6 is a diagram showing a data configuration example of song data read from ROM 202 to RAM 203 in FIG. 2 in the present embodiment. This data structure example conforms to the standard MIDI file format, which is one of the MIDI (Musical Instrument Digital Interface) file formats. This song data is composed of data blocks called chunks. Specifically, the song data consists of a header chunk at the beginning of the file, a track chunk 1 in which the lyrics data for the following lyrics part is stored, and a track chunk 2 in which the performance data for the accompaniment part is stored. It is composed.

The header chunk consists of four values: ChunkID, ChunkSize, FormatType, NumberOfTrack, and TimeDivision. The Chunk ID is a 4-byte ASCII code "4D 54 68 64" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MThd" indicating that it is a header chunk. The ChunkSize is 4-byte data indicating the data length of the FormatType, NumberOfTrack, and TimeDivision parts excluding the ChunkID and the ChunkSize in the header chunk, and the data length is 6 bytes: "00 00 00 06" (numbers are hexadecimal numbers). It is fixed to. In the case of this embodiment, the Format Type is 2-byte data “00 01” (numbers are hexadecimal numbers), which means format 1 using a plurality of tracks. In the case of the present embodiment, the NumberOfTrack is 2-byte data "00 02" (numbers are hexadecimal numbers) indicating that two tracks corresponding to the lyrics part and the accompaniment part are used. The Time Division is data indicating a time base value indicating a resolution per quarter note, and in the case of the present embodiment, it is 2-byte data "01 E0" (number is a hexadecimal number) indicating 480 in decimal notation.

Track chunks 1 and 2 are ChunkID, ChunkSize, and DataTime_1 [i] and Event_1 [i] (in the case of track chunk 1 / lyrics part) or DeltaTime_2 [i] and Event_2 [i] (track chunk 2 / accompaniment part, respectively). Case) is composed of a performance data set (0 ≦ i ≦ L: track chunk 1 / lyrics part, 0 ≦ i ≦ M: track chunk 2 / accompaniment part). The Chunk ID is a 4-byte ASCII code "4D 54 72 6B" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MTrk" indicating that it is a track chunk. The ChunkSize is 4-byte data indicating the data length of the portion of each track chunk excluding the ChunkID and the ChunkSize.

DeltaTime_1 [i] is 1 to 4 byte variable length data indicating the waiting time (relative time) from the execution time of Event_1 [i-1] immediately before that. Similarly, DeltaTime_2 [i] is 1 to 4 byte variable length data indicating the waiting time (relative time) from the execution time of Event_2 [i-1] immediately before that. Event_1 [i] is a meta event (timing information) that indicates the utterance timing and pitch of the lyrics in the track chunk 1 / lyrics part. Event_2 [i] is a MIDI event instructing note-on or note-off, or a meta event (timing information) instructing time signature in the track chunk 2 / accompaniment part. For each performance data set DeltaTime_1 [i] and Event_1 [i] for the track chunk 1 / lyrics part, after waiting for DeltaTime_1 [i] from the execution time of Event_1 [i-1] immediately before that, Event_1 [i] Is executed, the vocalization progress of the lyrics is realized. On the other hand, for the track chunk 2 / accompaniment part, in each performance data set DeltaTime_2 [i] and Event_2 [i], after waiting for the execution time of Event_2 [i-1] immediately before that, Event_2 [i] By executing i], the progress of automatic accompaniment is realized.

FIG. 7 is a main flowchart showing an example of control processing of an electronic musical instrument according to the present embodiment. This control process is, for example, an operation in which the CPU 201 of FIG. 2 executes a control process program loaded from the ROM 202 into the RAM 203.

The CPU 201 first executes the initialization process (step S701), and then repeatedly executes a series of processes from steps S702 to S708.

In this iterative process, the CPU 201 first executes the switch process (step S702). Here, the CPU 201 executes a process corresponding to the switch operation of the first switch panel 102 or the second switch panel 103 of FIG. 1 based on the interrupt from the key scanner 206 of FIG.

Next, the CPU 201 executes a keyboard process for determining and processing whether or not any key of the key 101 of FIG. 1 has been operated based on the interrupt from the key scanner 206 of FIG. 2 (step S703). .. Here, the CPU 201 outputs music sound control data 216 instructing the sound source LSI 204 of FIG. 2 to start or stop sounding in response to an operation of pressing or releasing any key by the performer.

Next, the CPU 201 processes data to be displayed on the LCD 104 of FIG. 1, and executes a display process of displaying the data on the LCD 104 via the LCD controller 208 of FIG. 2 (step S704). The data displayed on the LCD 104 includes, for example, the lyrics corresponding to the singing voice output data 217 to be played, the score of the melody corresponding to the lyrics, and various setting information.

Next, the CPU 201 executes the song playback process (step S705). In this process, the CPU 201 executes the control process described with reference to FIG. 5 based on the performance of the performer, generates singing voice data 215, and outputs the singing voice data 215 to the speech synthesis LSI 205.

Subsequently, the CPU 201 executes sound source processing (step S706). In the sound source processing, the CPU 201 executes control processing such as envelope control of the musical sound being sounded in the sound source LSI 204.

Subsequently, the CPU 201 executes the speech synthesis process (step S707). In the speech synthesis process, the CPU 201 controls the execution of speech synthesis by the speech synthesis LSI 205.

Finally, the CPU 201 determines whether or not the performer has pressed a power-off switch (not shown) to power off (step S708). If the determination in step S708 is NO, the CPU 201 returns to the process in step S702. If the determination in step S708 is YES, the CPU 201 ends the control process shown in the flowchart of FIG. 7, and turns off the power of the electronic keyboard instrument 100.

8 (a), (b), and (c) are the initialization process of step S701 of FIG. 7, the tempo change process of step S902 of FIG. 9, which will be described later in the switch process of step S702 of FIG. 7, and the same. It is a flowchart which shows the detailed example of the song start processing of step S906 of FIG.

First, in FIG. 8A showing a detailed example of the initialization process in step S701 of FIG. 7, the CPU 201 executes the ticktime initialization process. In the present embodiment, the progress of the lyrics and the automatic accompaniment proceed in units of time called TickTime. The timebase value specified as the TimeDivision value in the header chunk of the song data of FIG. 6 indicates the resolution of the quarter note, and if this value is, for example, 480, the quarter note has a time length of 480TickTime. Further, the waiting time DeltaTime_1 [i] value and the DeltaTime_2 [i] value in the track chunk of the song data of FIG. 6 are also counted by the time unit of TickTime. Here, how many seconds 1 Tick Time actually becomes depends on the tempo specified for the song data. If the tempo value is Tempo [beat / minute] and the time base value is Time Division, the number of seconds of Tick Time is calculated by the following equation.

TickTime [seconds] = 60 / Tempo / TimeDivision (10)

Therefore, in the initialization process exemplified in the flowchart of FIG. 8A, the CPU 201 first calculates the TickTime [seconds] by the arithmetic processing corresponding to the above equation (10) (step S801). It is assumed that a predetermined value, for example, 60 [beats / second] is stored in the ROM 202 of FIG. 2 in the initial state of the tempo value Tempo. Alternatively, the tempo value at the time of the previous end may be stored in the non-volatile memory.

Next, the CPU 201 sets a timer interrupt according to the TickTime [seconds] calculated in step S801 with respect to the timer 210 of FIG. 2 (step S802). As a result, every time the TickTime [seconds] elapses in the timer 210, an interrupt for lyrics progression and automatic accompaniment (hereinafter referred to as “automatic performance interrupt”) is generated for the CPU 201. Therefore, in the automatic performance interrupt process (FIG. 10 described later) executed by the CPU 201 based on this automatic performance interrupt, the control process for advancing the lyrics progress and the automatic accompaniment is executed for each TickTime.

Subsequently, the CPU 201 executes other initialization processing such as initialization of the RAM 203 of FIG. 2 (step S803). After that, the CPU 201 ends the initialization process of step S701 of FIG. 7 exemplified by the flowchart of FIG. 8A.

The flowcharts of FIGS. 8B and 8C will be described later. FIG. 9 is a flowchart showing a detailed example of the switch processing in step S702 of FIG.

First, the CPU 201 determines whether or not the tempo of the lyrics progression and the automatic accompaniment has been changed by the tempo change switch in the first switch panel 102 of FIG. 1 (step S901). If the determination is YES, the CPU 201 executes the tempo change process (step S902). Details of this process will be described later with reference to FIG. 8 (b). If the determination in step S901 is NO, the CPU 201 skips the process in step S902.

Next, the CPU 201 determines whether or not any song song has been selected on the second switch panel 103 of FIG. 1 (step S903). If the determination is YES, the CPU 201 executes the song song reading process (step S904). This process is a process of reading the music data having the data structure described in FIG. 6 from the ROM 202 of FIG. 2 into the RAM 203. The song song reading process may be performed not during the performance or before the start of the performance. From then on, data access to track chunks 1 or 2 in the data structure illustrated in FIG. 6 is performed on the song data read into RAM 203. If the determination in step S903 is NO, the CPU 201 skips the process in step S904.

Subsequently, the CPU 201 determines whether or not the song start switch has been operated on the first switch panel 102 of FIG. 1 (step S905). If the determination is YES, the CPU 201 executes the song start process (step S906). Details of this process will be described later with reference to FIG. 8 (c). If the determination in step S905 is NO, the CPU 201 skips the process in step S906.

Subsequently, the CPU 201 determines whether or not the effect selection switch has been operated on the first switch panel 102 of FIG. 1 (step S907). If the determination is YES, the CPU 201 executes the effect selection process (step S908). Here, as described above, when the sound effect addition unit 320 of FIG. 3 adds a sound effect to the uttered voice of the singing voice output data 321, which of the vibrato effect, the tremolo effect, and the wow effect is used. Is to be added by the first switch panel 102, which allows the performer to select. As a result of this selection, the CPU 201 sets any of the above sound effects selected by the performer in the sound effect addition unit 320 in the speech synthesis LSI 205. If the determination in step S907 is NO, the CPU 201 skips the process in step S908.

Depending on the setting, a plurality of effects may be added at the same time.

Finally, the CPU 201 determines whether or not other switches have been operated on the first switch panel 102 or the second switch panel 103 of FIG. 1, and executes a process corresponding to each switch operation (step S909). .. In this process, at least brass sound, string sound, and organ sound are used as the musical instrument sound of the musical sound output data 220 for the vocal sound source supplied by the performer from the sound source LSI 204 of FIG. 2 or 3 to the vocal model unit 308 in the voice synthesis LSI 205. , And a tone on the second switch panel 103 that allows one of the brass, string, organ, and animal sounds to be selected from a plurality of instrument sounds, including any of the animal sounds. Includes processing for selection switches (selection controls).

After that, the CPU 201 ends the switch process in step S702 of FIG. 7, which is illustrated in the flowchart of FIG. This process includes, for example, a switch operation of selecting a tone color of the musical tone output data 220 for a vocal sound source and selecting a predetermined sound channel of the musical tone output data 220 for a vocal sound source.

FIG. 8B is a flowchart showing a detailed example of the tempo change process in step S902 of FIG. As mentioned above, when the tempo value is changed, the TickTime [seconds] is also changed. In the flowchart of FIG. 8B, the CPU 201 executes the control process related to the change of the TickTime [seconds].

First, in the same manner as in the case of step S801 of FIG. 8A, which was executed in the initialization process of step S701 of FIG. 7, the CPU 201 is subjected to the arithmetic processing corresponding to the above-described equation (10) to TickTime [seconds]. Is calculated (step S811). It is assumed that the tempo value Tempo is stored in the RAM 203 or the like after being changed by the tempo change switch in the first switch panel 102 of FIG.

Next, the CPU 201 applies the TickTime [calculated in step S811 to the timer 210 of FIG. 2 in the same manner as in the case of step S802 of FIG. 8A executed in the initialization process of step S701 of FIG. Seconds] sets the timer interrupt (step S812). After that, the CPU 201 ends the tempo change process of step S902 of FIG. 9 exemplified by the flowchart of FIG. 8 (b).

FIG. 8C is a flowchart showing a detailed example of the song start process of step S906 of FIG.

First, in the progress of the automatic performance, the CPU 201 sets the values of the variables DeltaT_1 (track chunk 1) and DeltaT_2 (track chunk 2) on the RAM 203 for counting the relative time from the occurrence time of the immediately preceding event in units of TickTime. Is initially set to 0 for both. Next, the CPU 201 specifies the values of i of each of the performance data sets DeltaTime_1 [i] and Event_1 [i] (1 ≦ i ≦ L-1) in the track chunk 1 of the song data illustrated in FIG. Each of the variable AutoIndex_1 on the RAM 203 and the variable AutoIndex_1 on the RAM 203 for designating i of each of the performance data sets DeltaTime_2 [i] and Event_2 [i] (1 ≦ i ≦ M-1) in the track chunk 2. Both values are initially set to 0 (above, step S821). As a result, in the example of FIG. 6, as an initial state, first, the first performance data set DeltaTime_1 [0] and Event_1 [0] in the track chunk 1, and the first performance data set DeltaTime_2 [0] in the track chunk 2 Event_1 [0] is referenced respectively.

Next, the CPU 201 initially sets the value of the variable SongIndex on the RAM 203 that indicates the current song position to 0 (step S822).

Further, the CPU 201 initializes the value of the variable SongStart on the RAM 203 indicating whether the lyrics and accompaniment progress (= 1) or not (= 0) to 1 (progress) (step S823).

After that, the CPU 201 determines whether or not the performer has set the first switch panel 102 of FIG. 1 to reproduce the accompaniment in accordance with the reproduction of the lyrics (step S824).

If the determination in step S824 is YES, the CPU 201 sets the value of the variable Bansou on the RAM 203 to 1 (with accompaniment) (step S825). On the contrary, if the determination in step S824 is NO, the CPU 201 sets the value of the variable Bansou to 0 (no accompaniment) (step S826). After the process of step S825 or S826, the CPU 201 ends the song start process of step S906 of FIG. 9 shown in the flowchart of FIG. 8 (c).

FIG. 13 shows an automatic performance interrupt process executed based on an interrupt generated every TickTime [seconds] in the timer 210 of FIG. 2 (see step S802 of FIG. 8A or step S812 of FIG. 8B). It is a flowchart which shows the detailed example of. The following processing is executed on the performance data sets of the track chunks 1 and 2 of the song data exemplified in FIG.

First, the CPU 201 executes a series of processes (steps S1001 to S1006) corresponding to the track chunk 1. First, the CPU 201 determines whether or not the SongStart value is 1, that is, whether or not the progress of the lyrics and accompaniment is instructed (step S1001).

When the CPU 201 determines that the progress of the lyrics and the accompaniment is not instructed (the determination in step S1001 is NO), the CPU 201 does not proceed with the lyrics and the accompaniment, and is illustrated in the flowchart of FIG. The automatic performance interrupt processing ends as it is.

When the CPU 201 determines that the progress of the lyrics and the accompaniment is instructed (the determination in step S1001 is YES), the DeltaT_1 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 1 is set. It is determined whether or not the waiting time DeltaTime_1 [AutoIndex_1] of the performance data set to be executed to be executed, which is indicated by the AutoIndex_1 value, is matched (step S1002).

If the determination in step S1002 is NO, the CPU 201 increments the DeltaT_1 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chuck 1, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1003). After that, the CPU 201 shifts to step S1007, which will be described later.

When the determination in step S1002 becomes YES, the CPU 201 executes the event event [AutoIndex_1] of the performance data set indicated by the AutoIndex_1 value with respect to the track chuck 1 (step S1004). This event is a song event that includes lyrics data.

Subsequently, the CPU 201 stores the AutoIndex_1 value indicating the position of the next song event to be executed in the track chunk 1 in the variable SongIndex on the RAM 203 (step S1004).

Further, the CPU 201 increments the AutoIndex_1 value for referencing the performance data set in the track chunk 1 by +1 (step S1005).

Further, the CPU 201 resets the DeltaT_1 value indicating the relative time from the occurrence time of the song event referred to this time with respect to the track chunk 1 to 0 (step S1006). After that, the CPU 201 shifts to the process of step S1007.

Next, the CPU 201 executes a series of processes (steps S1007 to S1013) corresponding to the track chunk 2. First, the CPU 201 determines whether or not the DeltaT_2 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 2 matches the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set to be executed, which is indicated by the AutoIndex_2 value. Is determined (step S1007).

If the determination in step S1007 is NO, the CPU 201 increments the DeltaT_2 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chuck 2, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1008). After that, the CPU 201 ends the automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1007 is YES, the CPU 201 determines whether or not the value of the variable Bansou on the RAM 203 instructing accompaniment reproduction is 1 (with accompaniment) (step S1009) (step 8 (c)). (See S824 to S826).

If the determination in step S1009 is YES, the CPU 201 executes the event Event_2 [AutoIndex_2] relating to the accompaniment related to the track chuck 2 indicated by the AutoIndex_2 value (step S1010). If the event Event_2 [AutoIndex_2] executed here is, for example, a note-on event, an accompaniment musical tone sounding command is issued to the sound source LSI 204 of FIG. 2 according to the key number and velocity specified by the note-on event. publish. On the other hand, if the event Event_2 [AutoIndex_2] is, for example, a note-off event, a mute command for the accompaniment musical tone being sounded to the sound source LSI 204 of FIG. 2 is issued according to the key number and velocity specified by the note-off event. publish.

On the other hand, if the determination in step S1009 is NO, the CPU 201 skips step S1010, so that the event Event_2 [AutoIndex_2] related to this accompaniment is not executed, and the progress is synchronized with the lyrics, so that the next step S1011 Proceed to the process of, and execute only the control process that advances the event.

After step S1010 or when the determination in step S1009 is NO, the CPU 201 increments the AutoIndex_2 value for referencing the performance data set for accompaniment data on track chunk 2 by +1 (step S1011).

Further, the CPU 201 resets the DeltaT_2 value indicating the relative time from the occurrence time of the event executed this time with respect to the track chunk 2 to 0 (step S1012).

Then, the CPU 201 is whether or not the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set on the next executed track chunk 2 indicated by the AutoIndex_2 value is 0, that is, an event executed at the same time as this event. Whether or not it is determined (step S1013).

If the determination in step S1013 is NO, the CPU 201 ends the current automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1013 is YES, the CPU 201 returns to step S1009 and repeats the control process relating to the event Event_2 [AutoIndex_2] of the performance data set to be executed next on the track chunk 2 indicated by the AutoIndex_2 value. The CPU 201 repeatedly executes the processes of steps S1009 to S1013 as many times as the number of times it is executed simultaneously this time. The above processing sequence is executed when a plurality of note-on events are sounded at the same timing, such as a chord.

FIG. 11 is a flowchart showing a detailed example of the song reproduction process of step S705 of FIG.

First, the CPU 201 determines in step S1004 in the automatic performance interrupt process of FIG. 10 whether or not a value is set in the variable SongIndex on the RAM 203 and is no longer a Null value (step S1101). This SongIndex value indicates whether or not the current timing is the reproduction timing of the singing voice.

When the determination in step S1101 becomes YES, that is, when the current time is the timing for playing the song, the CPU 201 detects a new key pressed by the performer on the keyboard 101 in FIG. 1 by the keyboard processing in step S703 in FIG. It is determined whether or not (step S1102).

If the determination in step S1102 is YES, the CPU 201 sets the pitch specified by the key press by the performer in a register (not particularly shown) or a variable on the RAM 203 as the vocal pitch (step S1103).

Next, the CPU 201 causes the CPU 201 to produce a musical tone with the tone color of the musical tone preset in step S909 of FIG. 9 and a predetermined musical tone channel with the vocal pitch set based on the key press set in step S1103. Note-on data for this is generated, and the sound source LSI 204 is instructed to perform musical tone sounding processing (step S1105). The sound source LSI 204 generates a musical sound signal for a predetermined sound channel of a predetermined tone color specified by the CPU 201, and uses the vocoder mode switch 320 in the voice synthesis LSI 205 as the musical sound output data 220 for the vocal sound source to the synthesis filter unit 310. Let me enter.

Subsequently, the CPU 201 reads the lyrics character string from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203. The CPU 201 generates singing voice data 215 for uttering the singing voice output data 321 corresponding to the read lyrics character string at the utterance sound pitch set based on the key press set in step S1103, and produces the voice. Instruct the synthesis LSI 205 to perform vocalization processing (step S1105). The voice synthesis LSI 205 plays the lyrics designated as song data from the RAM 203 by executing the first embodiment or the second embodiment of the statistical voice synthesis processing described with reference to FIGS. 3 to 5. The singing voice voice output data 321 that the person sings in real time corresponding to the pitch of the key pressed on the keyboard 101 is synthesized and output.

As a result, the musical sound output data 220 for the vocal sound source generated and output by the sound source LSI 204 based on the performance on the player's keyboard 101 (FIG. 1) operates based on the spectrum information 318 input from the acoustic model unit 306. It is input to the filter unit 310, and the singing voice audio output data 321 is output from the synthetic filter unit 310 in a polyphonic operation.

On the other hand, if it is determined by the determination in step S1101 that the current time is the timing for playing the song, and the determination in step S1102 is NO, that is, it is determined that no new key press has been detected at the present time, the CPU 201 determines. Variables on RAM203 Read pitch data from song event Event_1 [SongIndex] on track chunk 1 of song data on RAM203 indicated by SongIndex, and use this pitch as the vocal pitch in a register not shown or a variable on RAM203. (Step S1104).

After that, the CPU 201 instructs the voice synthesis LSI 205 to perform the utterance processing of the singing voice output data 321 by executing the processing after step S1105 described above (steps S1105 and S1106). In the speech synthesis LSI 205, the performer presses any key on the keyboard 101 by executing the first embodiment or the second embodiment of the statistical speech synthesis process described with reference to FIGS. 3 to 5. Even if the key is not used, the lyrics designated as song data from RAM 203 are combined and output as singing voice voice output data 321 that sings corresponding to the pitch designated as song data by default.

After the process of step S1105, the CPU 201 stores the reproduced song position indicated by the variable SongIndex on the RAM 203 in the variable SongIndex_pre on the RAM 203 (step S1107).

Further, the CPU 201 clears the value of the variable SongIndex to the Null value, and sets the timing after that to a state other than the timing of song playback (step S1108). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

When the determination in step S1101 described above is NO, that is, when the current time is not the timing of song playback, the CPU 201 performs the keyboard processing in step S703 in FIG. 7 to add an effect on the keyboard 101 in FIG. It is determined whether or not the so-called legato playing method is detected (step S1109). As described above, this legato playing method is, for example, a playing method in which the other second key is repeatedly hit while the first key for song reproduction is pressed in step S1102. In this case, when the CPU 201 detects the pressing of the second key in step S11080, the CPU 201 determines that the legato playing method is being executed when the repetition rate of the pressing is equal to or higher than a predetermined speed.

If the determination in step S1109 is NO, the CPU 201 as it is ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

If the determination in step S1109 is YES, the CPU 201 determines the pitch of the vocalized sound set in step S1103 and the pitch of the key repeatedly hit on the keyboard 101 of FIG. 1 by the "so-called legato playing method". The pitch difference is calculated (step S1110).

Subsequently, the CPU 201 sets the effect amount according to the pitch difference calculated in step S1109 to the sound effect addition unit 320 (FIG. 3) in the voice synthesis LSI 205 of FIG. 2 (step S1111). As a result, the sound effect addition unit 320 performs the addition processing of the sound effect selected in step S908 of FIG. 9 on the singing voice voice output data 321 output from the synthesis filter unit 310 in the voice synthesis unit 302. It is executed with the amount of effect, and the final singing voice voice output data 217 (FIGS. 2 and 3) is output.

By the processing of steps S1110 and S1111 described above, it is possible to add sound effects such as a vibrato effect, a tremolo effect, or a wow effect to the singing voice output data 321 output from the voice synthesis unit 302. Singing voice expression is realized.

After the process of step S1109, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

In the first embodiment of the statistical speech synthesis processing using the HMM acoustic model described with reference to FIGS. 3 and 4, it is possible to reproduce a subtle musical expression such as a specific singer or singing style, and the connection is made. It is possible to realize smooth singing voice sound quality without distortion. Furthermore, by converting the learning result 315 (model parameter), it becomes possible to adapt to another singer and express various voice qualities and emotions. Furthermore, by being able to automatically learn all the model parameters in the HMM acoustic model from the learning singing voice data 311 and the learning singing voice data 312, the characteristics of a specific singer are acquired as the HMM acoustic model, and those characteristics are acquired at the time of synthesis. It is possible to automatically construct a singing voice synthesis system that reproduces. The fundamental frequency and length of the singing voice follow the melody and tempo of the score, and the time structure of the pitch and rhythm can be uniquely determined from the score, but the singing voice synthesized from it is monotonous and mechanical. It becomes a thing and lacks the charm as a singing voice. The actual singing voice is not limited to the standardized one according to the score, but each singer has his or her own style due to changes in the pitch of the voice and their temporal structure in addition to the voice quality. In the first embodiment of the statistical speech synthesis process that employs the HMM acoustic model, the time-series changes of the spectral information and the pitch information in the singing voice can be modeled based on the context, and by further considering the musical score information. , It is possible to reproduce the singing voice closer to the actual singing voice. Further, in the HMM acoustic model adopted in the first embodiment of the statistical speech synthesis processing, when uttering lyrics along a certain melody, the acoustic feature sequence of the singing voice in the vibration of the vocal cords of the singer and the vocal tract characteristics It corresponds to the generation model of what kind of time change the voice is uttered. Further, in the first embodiment of the statistical speech synthesis process, by using an HMM acoustic model including the context of "deviation" between the note and the singing voice, the singing tends to change in a complicated manner depending on the vocal characteristics of the singer. Singing voice synthesis that can accurately reproduce the method is realized. By fusing the technique of the first embodiment of the statistical voice synthesis processing adopting such an HMM acoustic model with, for example, the technique of real-time performance by the electronic keyboard instrument 100, a conventional electron by a element piece synthesis method or the like can be used. It is possible to accurately reflect the singing method and voice quality of the model singer, which was not possible with musical instruments, and the performance of the singing voice as if the singer was actually singing, the keyboard performance of the electronic keyboard instrument 100, etc. It will be possible to realize it according to.

In the second embodiment of the statistical speech synthesis process adopting the DNN acoustic model described with reference to FIGS. 3 and 5, the statistical speech synthesis process is expressed as an expression of the relationship between the language feature sequence and the acoustic feature sequence. The context-dependent HMM acoustic model based on the decision tree in the first embodiment is replaced by DNN. This makes it possible to express the relationship between the linguistic feature series and the acoustic feature series by a complicated nonlinear conversion function that is difficult to express with a decision tree. Further, in the context-dependent HMM acoustic model based on the decision tree, the corresponding learning data is also classified based on the decision tree, so that the learning data assigned to the HMM acoustic model depending on each context is reduced. On the other hand, in the DNN acoustic model, a single DNN is learned from the entire training data, so that the training data can be used efficiently. Therefore, the DNN acoustic model can predict the acoustic features with higher accuracy than the HMM acoustic model, and the naturalness of the synthesized speech can be significantly improved. Further, in the DNN acoustic model, it becomes possible to use a language feature sequence related to a frame. That is, in the DNN acoustic model, the temporal correspondence between the acoustic feature series and the language feature series is determined in advance, so that it was difficult to consider in the HMM acoustic model, "the current number of continuous phoneme frames". It is possible to use language features related to frames such as "the position in the phoneme of the current frame". This makes it possible to model more detailed features by using the linguistic features related to the frame, and it is possible to improve the naturalness of the synthetic speech. By fusing the technique of the second embodiment of the statistical voice synthesis processing adopting such a DNN acoustic model with the technique of real-time performance by the electronic keyboard instrument 100, for example, the performance of the singing voice based on the keyboard performance or the like can be performed. , It becomes possible to get closer to the singing method and voice quality of the model singer more naturally.

In the embodiment described above, by adopting the technique of statistical voice synthesis processing as the voice synthesis method, it is possible to realize a much smaller memory capacity as compared with the conventional element piece synthesis method. For example, an electronic musical instrument of the elemental composition method requires a memory having a storage capacity of several hundred megabytes for audio elemental data, but in the present embodiment, the model parameter of the learning result 315 of FIG. 3 is used. Only a memory with a storage capacity of only a few megabytes is required for storage. Therefore, it is possible to realize a lower-priced electronic musical instrument, and it is possible to have a wider user group use the high-quality singing voice performance system.

Further, in the conventional element data method, since the element data needs to be manually adjusted, it takes a huge amount of time (yearly) and labor to create the data for singing voice performance. However, according to this embodiment. Creating the model parameters of the training result 315 for the HMM acoustic model or the DNN acoustic model requires only a fraction of the creation time and effort because data adjustment is rarely required. This also makes it possible to realize a lower-priced electronic musical instrument. In addition, a general user learns his / her own voice, family voice, celebrity voice, etc. by using the learning function built in the server computer 300 or the voice synthesis LSI 205 that can be used as a cloud service, and models it. It is also possible to play a singing voice with an electronic musical instrument as voice. In this case as well, it is possible to realize a singing voice performance that is much more natural and has higher sound quality than before as a lower-priced electronic musical instrument.

In this embodiment, in particular, since the musical sound output data 220 for the vocal sound source of the musical instrument sound generated by the sound source LSI 204 is used as the sound source signal, the atmosphere of the musical instrument sound set by the sound source LSI 204 remains well, and the voice quality of the singer's singing voice is also improved. It becomes a well-remaining singing voice, and it becomes possible to output effective singing voice sound output data 217. Further, since the polyphonic operation is possible, it is possible to produce an effect in which a plurality of singing voices are harmonized. This makes it possible to provide an electronic musical instrument that sings well with a singing voice corresponding to the singing voice of the singer learned based on each pitch designated by the performer.

Although the embodiment described above is an embodiment of the present invention for an electronic keyboard instrument, the present invention can also be applied to other electronic musical instruments such as electronic stringed instruments.

Further, the speech synthesis method that can be adopted as the vocal model unit 308 of FIG. 3 is not limited to the cepstrum speech synthesis method, and various speech synthesis methods including the LSP speech synthesis method can be adopted.

Further, in the embodiment described above, the speech synthesis method of the first embodiment of the statistical speech synthesis processing using the HMM acoustic model or the second embodiment after a long time using the DNN acoustic model has been described. The present invention is not limited to this, and any speech synthesis method may be adopted as long as it is a technique using statistical speech synthesis processing such as an acoustic model combining HMM and DNN.

In the embodiment described above, the lyrics information is given as song data, but text data obtained by voice recognition of the content sung by the performer in real time may be given as lyrics information in real time.

Regarding the above embodiments, the following additional notes will be further disclosed.
(Appendix 1)
Multiple controls with which each pitch information is associated,
A learned acoustic model trained by machine learning processing using learning score data including learning lyrics information and learning pitch information, and learning singing voice data of a singer, and singing lyrics information. A memory that stores a learned acoustic model that outputs spectral information that models the voice path of the singer by inputting pitch information, and
With the processor
Including
The processor is associated with the lyrics information and the one of the controls with respect to the learned acoustic model in response to an operation on any of the plurality of controls. The pitch information is input, and the spectrum information output by the trained acoustic model in response to the input and the instrument sound type data corresponding to the pitch information associated with any of the controls. First instrument sound use inferred singing voice data output processing for inferring the singer's singing voice based on
An electronic musical instrument that runs.
(Appendix 2)
In the electronic musical instrument described in Appendix 1,
One of the brass sound, the string sound, the organ sound, and the animal bark from among a plurality of instrument sounds including at least one of the brass sound, the string sound, the organ sound, and the animal bark. A selection operator that lets you select a sound,
Have,
The musical instrument sound use inferred singing voice data output process outputs the first musical instrument sound use inferred singing voice data based on the musical instrument sound wave shape data corresponding to the musical instrument sound selected by the selection operator.
(Appendix 3)
In the electronic musical instrument described in Appendix 1 or Appendix 2,
The memory stores song data having melody data and accompaniment data to be automatically played.
The melody data includes each pitch information, each timing information for outputting a sound corresponding to each pitch information, and each lyrics information associated with each pitch information.
The processor
Accompaniment data automatic performance processing that causes the sounding unit to pronounce based on the accompaniment data,
Instead of inputting the pitch information associated with any of the manipulated controls into the trained acoustic model, the pitch information included in the melody data is input and the melody data is input. Input processing to input the lyrics information included in (* Please include in the specification that the input timing may be before the start of the performance or not during the performance)
And run
The musical instrument sound use inference singing voice data output process is performed when any of the plurality of controls is not operated by the performer according to the timing indicated by the timing information included in the melody data. The singer is based on the spectrum information output from the trained acoustic model according to the processing and the musical instrument sound type data corresponding to the pitch information included in the melody data input by the input processing. The second musical instrument sound use inferred singing voice data inferred from the above singing voice is output in accordance with the timing indicated by the timing information included in the melody data.
(Appendix 4)
In the electronic musical instrument described in any of Appendix 1 to Appendix 3,
The plurality of controls include a first operator as any of the operated operators and a second operator that satisfies the conditions set from the viewpoint of the first operator.
When the first musical instrument sound use inferred singing voice data is output by the musical instrument sound use inferred singing voice data output process, the second operator repeatedly operates while the operation on the first operator continues. When this is done, effect processing that applies at least one of the effects of vibrato, tremolo, and wow wow to the inferred singing voice data using the first musical instrument sound.
To execute.
(Appendix 5)
In the electronic musical instrument described in Appendix 4,
In the effect processing, the first pitch indicated by the pitch information associated with the first operator and the second pitch indicated by the pitch information associated with the second operator. The degree of the effect is changed according to the difference in pitch.
(Appendix 6)
In the electronic musical instrument described in Appendix 4 or 5,
The second operator is a black key.
(Appendix 7)
Multiple controls with which each pitch information is associated,
A learned acoustic model trained by machine learning processing using learning score data including learning lyrics information and learning pitch information, and learning singing voice data of a singer, and singing lyrics information. A memory that stores a learned acoustic model that outputs spectral information that models the voice path of the singer by inputting pitch information, and
To electronic musical instrument computers, including
In response to an operation on any of the plurality of controls, the lyrics information and the pitch information associated with the one of the controls are assigned to the trained acoustic model. , And based on the spectrum information output by the trained acoustic model in response to the input and the instrument sound type data corresponding to the pitch information associated with any of the controls. First instrument sound use inferred singing voice data output processing for inferring the singer's singing voice
How to run.
(Appendix 8)
A learned acoustic model trained by machine learning processing using learning score data including learning lyrics information and learning pitch information, and learning singing voice data of a singer, and singing lyrics information. A memory that stores a learned acoustic model that outputs spectral information that models the voice path of the singer by inputting pitch information, and
To electronic musical instrument computers, including
In response to an operation on any of the plurality of controls, the lyrics information and the pitch information associated with the one of the controls are assigned to the trained acoustic model. , And based on the spectrum information output by the trained acoustic model in response to the input and the instrument sound type data corresponding to the pitch information associated with any of the controls. First instrument sound use inferred singing voice data output processing for inferring the singer's singing voice
A program that executes.

100 Electronic keyboard instrument 101 Keyboard 102 First switch panel 103 Second switch panel 104 LCD
200 Control system 201 CPU
202 ROM
203 RAM
204 Sound source LSI
205 Speech synthesis LSI
206 Key Scanner 208 LCD Controller 209 System Bus 210 Timer 211, 212 D / A Converter 213 Mixer 214 Amplifier 215 Singing Voice Data 216 Sound Effect Control Data 217, 321 Singing Voice Output Data 218 Music Sound Output Data 219 Network Interface 220 Music Sound Output Data for Voice Sound Source 300 Server computer 301 Speech learning unit 302 Speech synthesis unit 303 Learning text analysis unit 304 Learning acoustic feature amount extraction 305 Model learning unit 306 Acoustic model unit 307 Text analysis unit 308 Speech model unit 309 Sound effect generation unit 310 Synthesis filter unit 311 Learning Singing voice data 312 Learning singing voice data 313 Learning language feature series 314 Learning acoustic feature series 315 Learning result 316 Language information quantity series 317 Acoustic feature series 318 Spectral information 319 Sound effect information 320 Sound effect addition part

Claims (7)

Musical instrument sound wave data corresponding to the set timbre, and musical instrument sound wave data corresponding to each of the plurality of pitch data acquired according to the performance operation of the chord by the user, according to the performance operation of the chord. Acquired,
By inputting the lyrics data stored in the memory before the performance into the trained model that has learned the singing voice of a certain singer, the acoustic feature data output by the trained model can be obtained according to the performance operation of the chord. Get and
The acoustic feature amount indicated by the acquired acoustic feature amount data is synthesized with the acquired musical instrument sound wave type data.
The inferred singing voice data of the chord generated by synthesizing is output even if the user does not sing.
Electronic musical instrument. The inferred singing voice data of the chord is generated by synthesizing the acoustic feature amount indicated by the acoustic feature amount data with the instrument sound wave data oscillated from each of the plurality of first sound wave channels as the excitation source signal.
The electronic musical instrument according to claim 1. The second sounding channels other than the plurality of first sounding channels are used for the output of accompaniment data.
The acoustic features are not combined with the output from the second sounding channel.
The electronic musical instrument according to claim 2. The inferred singing voice data of the chord is output at the set first tempo, and
The tempo is changed from the first tempo to the second tempo according to the tempo change instruction operation by the user.
The inferred singing voice data of the chord is output at the changed second tempo.
The electronic musical instrument according to any one of claims 1 to 3. The electronic musical instrument according to any one of claims 1 to 4, wherein the set tone color includes at least one of a brass sound, a string sound, and an organ sound. On the computer of electronic musical instruments
Musical instrument sound wave data corresponding to the set timbre, and musical instrument sound wave data corresponding to each of the plurality of pitch data acquired according to the performance operation of the chord by the user, according to the performance operation of the chord. Get it,
By inputting the lyrics data stored in the memory before the performance into the trained model that has learned the singing voice of a certain singer, the acoustic feature data output by the trained model can be obtained according to the performance operation of the chord. And get it
The acquired acoustic feature data is combined with the acoustic feature data indicated by the acquired acoustic feature data.
Inferred singing voice data of chords generated by synthesis is output without the user singing.
How to control an electronic musical instrument. On the computer of electronic musical instruments
Musical instrument sound wave data corresponding to the set timbre, and musical instrument sound wave data corresponding to each of the plurality of pitch data acquired according to the performance operation of the chord by the user, according to the performance operation of the chord. Get it,
By inputting the lyrics data stored in the memory before the performance into the trained model that has learned the singing voice of a certain singer, the acoustic feature data output by the trained model can be obtained according to the performance operation of the chord. And get it
The acquired acoustic feature data is combined with the acoustic feature data indicated by the acquired acoustic feature data.
Inferred singing voice data of chords generated by synthesis is output without the user singing.
program.