JP6872197B2 - Acoustic signal generation model learning device, acoustic signal generator, method, and program - Google Patents

Description

The present invention relates to an acoustic signal generation model learning device, an acoustic signal generation device, a method, and a program, and relates to an acoustic signal generation model learning device, an acoustic signal generation device, a method, and a program for generating an acoustic signal.

In acoustic signal processing, there are many situations where processing (including emphasis processing) for the amplitude spectrogram obtained by short-time Fourier transform (STFT) is effective. Moreover, in recent years, the effectiveness of the approach of generating a signal through the synthesis of an amplitude spectrogram has been shown also in speech synthesis. Since the amplitude spectrogram does not contain phase information, it is necessary to reconstruct the phase spectrogram in order to reconstruct the acoustic signal from the amplitude spectrogram. Conventionally, a method based on signal processing by Griffin-Lim et al. (Non-Patent Document 1) (hereinafter, Griffin-Lim method) has been widely used as a phase reconstruction method for STFT amplitude spectrograms. This method is based on the fact that the complex spectrogram obtained by STFT is a redundant representation of the time domain signal, and uses the amplitude spectrogram as an input.

Step 0: Initialize the phase spectrogram.
Step 1: Construct a complex spectrogram from the input amplitude spectrogram and the phase spectrogram obtained in the previous stage, and calculate the time domain signal by inverse STFT.
Step 2: Perform STFT on the time domain signal in Step 1 above and recalculate the complex spectrogram.
Step 3: The declination component of the complex spectrogram in Step 2 above is used as the estimated value of the new phase spectrogram, and the process returns to Step 1 above.
The acoustic signal can be reconstructed from the amplitude spectrogram by the iterative algorithm.

Daniel Griffin and Jae Lim: "Signal estimation from modi_ed short-time Fourier transform," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236-243, 1984.

It is known that this method does not always reconstruct a high-quality acoustic signal with a small number of iterations, and it often takes a large number of iterations to obtain a high-quality acoustic signal. Therefore, there is a drawback that it is not suitable for real-time processing.

The present invention has been made to solve the above problems, and an object of the present invention is to provide an acoustic signal generation model learning device, an acoustic signal generation device, a method, and a program capable of generating an acoustic signal with high quality. And.

The acoustic signal generation model learning apparatus according to the first invention for achieving the above object is an acoustic signal generation model learning apparatus that learns a generator which is a neural network for generating an acoustic signal, and is a teacher sound. Based on the pair of the time region signal or complex spectrogram of the signal and the amplitude spectrogram of the teacher acoustic signal, the generator that takes the amplitude spectrogram as an input and outputs the time region signal or complex spectrogram of the acoustic signal is used as the teacher acoustic signal. Learning to reduce the learning criterion indicating the magnitude of the error between the time region signal or complex spectrogram of the above and the time region signal or complex spectrogram output by the generator by inputting the amplitude spectrogram of the teacher acoustic signal. It is composed of parts.

The acoustic signal generation model learning device according to the second invention is an acoustic signal generation model learning device that learns a generator that is a neural network for generating acoustic signals, and is time from the spectrogram of the teacher acoustic signal. Based on the input generator that constitutes the region signal or complex spectrogram, and the pair of the time region signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal, the time region signal or complex spectrogram is input and the acoustic signal is used. The generator that outputs the time region signal or complex spectrogram of the teacher acoustic signal is composed of the time region signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal by the input generator. Is included as an input, and includes a learning unit that learns so as to reduce the learning standard indicating the magnitude of the error from the time region signal or the complex spectrogram output by the generator.

The acoustic signal generator according to the third invention uses a generator, which is a neural network for generating acoustic signals, to input an amplitude spectrogram and output a time region signal or a complex spectrogram of the acoustic signal. The generator includes, based on a pair of a time region signal or complex spectrogram of the teacher acoustic signal and an amplitude spectrogram of the teacher acoustic signal, the time region signal or complex spectrogram of the teacher acoustic signal and the teacher acoustic signal. The time region signal of the acoustic signal, which is input to the amplitude spectrogram and has been trained in advance so as to reduce the learning criterion indicating the magnitude of the error from the time region signal or the complex spectrogram output by the generator with the amplitude spectrogram as the input. Or the generator that outputs a complex spectrogram.

The acoustic signal generator according to the fourth invention uses an input generator that constitutes a time region signal or a complex spectrogram from the input amplitude spectrogram, and a generator that is a neural network for generating the acoustic signal. The generator includes a time region signal or a complex spectrogram composed of the input generation unit and outputs a time region signal or a complex spectrogram of an acoustic signal, and the generator is a time region signal of a teacher acoustic signal or a time region signal or a complex spectrogram. Based on the pair of the complex spectrogram and the amplitude spectrogram of the teacher acoustic signal, the time region signal or complex spectrogram composed of the time region signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal is input. The time region signal or complex spectrogram of the acoustic signal is input and the time region signal or complex is learned so as to reduce the learning criterion indicating the magnitude of the error from the time region signal or complex spectrogram output by the generator. The generator that outputs the spectrogram.

The acoustic signal generation model learning method according to the fifth invention is an acoustic signal generation model learning method in an acoustic signal generation model learning apparatus that learns a generator which is a neural network for generating an acoustic signal, and the learning unit The generator that takes an amplitude spectrogram as an input and outputs a time region signal or a complex spectrogram of an acoustic signal based on a pair of a time region signal or a complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. To reduce the learning criterion that represents the magnitude of the error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input. learn.

The acoustic signal generation model learning method according to the sixth invention is an acoustic signal generation model learning method in an acoustic signal generation model learning apparatus that learns a generator that is a neural network for generating an acoustic signal, and is an input generation unit. Constructs a time region signal or complex spectrogram from the amplitude spectrogram of the teacher acoustic signal, and the learning unit bases the time region signal or complex spectrogram of the teacher acoustic signal on the amplitude spectrogram of the teacher acoustic signal. The generator that takes the time region signal or complex spectrogram as input and outputs the time region signal or complex spectrogram of the acoustic signal is the time region signal or complex spectrogram of the teacher acoustic signal, and the teacher acoustic signal by the input generator. Learning is performed so as to reduce the learning criterion indicating the magnitude of the error from the time region signal or complex spectrogram output by the generator by inputting the time region signal or complex spectrogram composed of the amplitude spectrogram.

The program according to the seventh invention is a program for operating a computer as each part of the above-mentioned acoustic signal generation model learning device or acoustic signal generation device.

As described above, according to the acoustic signal generation model learning device, method, and program of the present invention, it is possible to obtain an effect that a generator that generates an acoustic signal with high quality can be learned.

Further, according to the acoustic signal generator and the program of the present invention, the effect that the acoustic signal can be generated with high quality can be obtained.

It is a figure which shows the phase reconstruction flow in 1st Embodiment. It is a figure which shows the learning method in 1st Embodiment. It is the schematic which shows the structure of the acoustic signal generation model learning apparatus which concerns on 1st Embodiment. It is the schematic which shows the structure of the acoustic signal generation apparatus which concerns on 1st Embodiment. It is a figure which shows the phase reconstruction flow in 2nd Embodiment. It is a figure which shows the learning method in 2nd Embodiment. It is a figure which shows the phase reconstruction flow in 3rd Embodiment. It is the schematic which shows the structure of the acoustic signal generation model learning apparatus which concerns on 3rd Embodiment. It is the schematic which shows the structure of the acoustic signal generation apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the content of the learning processing routine in the acoustic signal generation model learning apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the content of the acoustic signal generation processing routine in the acoustic signal generation apparatus which concerns on 3rd Embodiment. It is a figure which shows the generation flow in this experiment. It is a figure which shows the structure of the neural network in this experiment. It is a figure which shows the experimental result of the method of 3rd Embodiment and the conventional method. It is a figure which shows the experimental result of the method of 3rd Embodiment and the conventional method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The technique proposed in the present invention is an acoustic signal generation technique for the purpose of generating an acoustic signal from an amplitude spectrogram.

<Outline of Embodiment of the present invention>
The features of the embodiments of the present invention are as follows.

Using a pair of the time region signal (or complex spectrogram) and its amplitude spectrogram as training data, the process of reconstructing the time region signal (or complex spectrogram) from the amplitude spectrogram is modeled by a neural network (generator), and the signal reconstruction (generator) Or phase reconstruction) The process itself is determined by learning.

The error between the teacher signal (or teacher complex spectrogram) and the signal (or complex spectrogram) generated by the generator is used as the learning criterion.

An example of how to measure the error is as follows.

(Method 1) The sum of the errors of the teacher signal (or teacher complex spectrogram) and the signal generated by the generator (or complex spectrogram) for each element is used as the learning criterion.

(Method 2) The learning standard is the error measured between the output values of the intermediate layer of the classifier that distinguishes between the signal generated by the generator and the signal of the actual data.

The expected effects are as follows.

The acoustic signal can be generated faster than the Griffin-Lim method. In addition, it is possible to generate a higher quality acoustic signal than the Griffin-Lim method.

[First Embodiment]
<Proposed method 1>
The existing method makes it possible to reconstruct the phase spectrogram from only the amplitude spectrogram, using the fact that the complex spectrogram is a redundant representation of the time domain signal. On the other hand, in the method according to the embodiment of the present invention, the conversion function from the amplitude spectrogram to the time domain signal or the complex spectrogram is modeled by a neural network (NN), and the amplitude spectrogram to the time domain signal or the complex spectrogram is modeled. Neural network parameters are determined by learning using pairs as training data. The flow of the generation process is shown in FIG. By using the trained neural network, it is possible to obtain a time domain signal or a complex spectrogram for the input amplitude spectrogram.

<Learning method 1>
First, as training data, prepare an acoustic signal such as voice or music, or its complex spectrogram x obtained by STFT or wavelet transform. For this data, performs STFT and wavelet transform, to obtain an amplitude spectrogram (amplitude component of the complex spectrogram) f _a. Taking this amplitude spectrogram f _a as an input, the neural network is trained so that x is reconstructed. Specifically, when the output of the neural network input amplitude spectrogram f _a and ~ x, to optimize the parameters of the neural network so as to reduce some distance index x and ~ x. An example of a distance index is a least squares error. When the least squares error is used as a distance index, the objective function L2 is expressed by the following equation.

FIG. 2 shows a learning processing flow.

<Learning method supplement 1>
Unless the generator is specifically constrained, the amplitude spectrogram calculated from the time domain signal or complex spectrogram output by the generator does not always exactly match the input amplitude spectrogram. Therefore, in order to force the amplitude spectrogram calculated from the time region signal or complex spectrogram output by the generator to match the input amplitude spectrogram, the amplitude spectrogram of the time region signal or complex spectrogram is input to the final layer of the generator. A layer corresponding to the operation to be replaced with the amplitude spectrogram may be added.

<Learning method supplement 2>
Since the Fourier transform of a real number signal has an even function in the real part and an odd function in the imaginary part, its symmetry can be used depending on the assumed time-frequency analysis method. For example, if the frame length of the STFT is 1024 points, the number of frequency bins of the obtained complex spectrogram will be 1024 including negative frequencies, but if symmetry is used, it corresponds to the frequency from 0 to the Nyquist frequency 513. Only point information is sufficient to construct a time domain signal. Therefore, when the output of the generator is a complex spectrogram, the complex spectrogram corresponding to some frequency bins may be used as the output.

<Generation method>
It is possible to obtain a reconstructed signal or reconstructed complex spectrogram ~ x by inputting arbitrary amplitude spectrogram series f _a neural network learning is completed.

<Generation method supplement 1>
The amplitude spectrogram calculated from the time domain signal or complex spectrogram output by the generator does not always exactly match the input amplitude spectrogram. Therefore, a process of recalculating the time domain signal after replacing the amplitude spectrogram of the time domain signal or the complex spectrogram output from the generator with the input amplitude spectrogram may be added.

<Generation method supplement 2>
Similar to the learning method Supplement 2, the symmetry can be used by the assumed time-frequency analysis method.

<System configuration>
<Configuration of acoustic signal generation model learning device>
Next, the present invention is applied to an acoustic signal generation model learning device that learns a generator that is a neural network for generating an acoustic signal, and an acoustic signal generator that generates an acoustic signal using the learned generator. An embodiment of the present invention will be described by taking the above case as an example.

As shown in FIG. 3, the acoustic signal generation model learning device 100 according to the first embodiment is a computer including a CPU, a RAM, and a ROM storing a program for executing a learning processing routine described later. It is composed of, and is functionally configured as shown below.

As shown in FIG. 3, the acoustic signal generation model learning device 100 includes an input unit 10, a calculation unit 20, and an output unit 50.

The input unit 10 receives a pair of a time domain signal or a complex spectrogram of the teacher acoustic signal and an amplitude spectrogram of the teacher acoustic signal.

The calculation unit 20 includes a learning unit 24.

The learning unit 24 uses a generator that takes an amplitude spectrogram as an input and outputs a time region signal or a complex spectrogram of the acoustic signal based on the pair of the time region signal or the complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. , Learning to reduce the learning criteria that represent the magnitude of the error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input. To do.

The learning criterion represents, for example, the sum of the elements of the time domain signal or complex spectrogram of the teacher acoustic signal and the time domain signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input.

The parameters of the generator learned by the learning unit 24 are output by the output unit 50.

<Configuration of acoustic signal generator>
As shown in FIG. 4, the acoustic signal generation device 150 according to the first embodiment is a computer including a CPU, a RAM, and a ROM storing a program for executing a sound signal generation processing routine described later. It is composed of, and is functionally configured as shown below.

As shown in FIG. 4, the acoustic signal generation device 150 includes an input unit 60, a calculation unit 70, and an output unit 90.

The input unit 60 receives the amplitude spectrogram.

The calculation unit 70 includes a generation unit 76.

The generation unit 76 uses a generator, which is a neural network for generating an acoustic signal, learned by the acoustic signal generation model learning device 100, from the input amplitude spectrogram, to a time region signal or a complex spectrogram of the acoustic signal. Is generated and output by the output unit 90.

<Operation of acoustic signal generation model learning device>
Next, the operation of the acoustic signal generation model learning device 100 according to the first embodiment will be described. First, when a time domain signal or a complex spectrogram of the teacher acoustic signal and a pair of the amplitude spectrogram of the teacher acoustic signal are input to the acoustic signal generation model learning device 100, the learning processing routine is performed in the acoustic signal generation model learning device 100. Will be executed. In the learning processing routine, the learning unit 24 learns the generator based on the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. Then, the learned generator parameters are output by the output unit 50.

<Action of acoustic signal generator>
Next, the operation of the acoustic signal generation device 150 according to the first embodiment will be described. First, the parameters of the generator learned by the acoustic signal generation model learning device 100 are input to the acoustic signal generation device 150. Further, when the amplitude spectrogram is input to the acoustic signal generation device 150, the acoustic signal generation device 150 executes the acoustic signal generation processing routine. In the acoustic signal generation processing routine, the generation unit 76 generates a time domain signal or a complex spectrogram of the acoustic signal from the input amplitude spectrogram by using the generator, and outputs the time domain signal or the complex spectrogram by the output unit 90.

As described above, according to the acoustic signal generation model learning apparatus according to the first embodiment, the amplitude spectrogram is based on the time region signal or the complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. Is output by the generator as a neural network that outputs the time region signal or complex spectrogram of the acoustic signal, and the time region signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal as inputs. By learning to reduce the learning criteria that represent the magnitude of the error from the time region signal or the complex spectrogram, it is possible to learn a generator that produces an acoustic signal with high quality.

Further, according to the acoustic signal generator according to the first embodiment, the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as inputs. Higher acoustic signals by using a generator that takes an amplitude spectrogram as input and outputs a time region signal or complex spectrogram of the acoustic signal, which has been trained in advance to reduce the learning criteria that represent the magnitude of the error from the spectrogram. Can be produced to quality.

[Second Embodiment]
Next, the second embodiment will be described. Since the configurations of the acoustic signal generation model learning device and the acoustic signal generation device according to the second embodiment are the same as those of the first embodiment, the same reference numerals are given and the description thereof will be omitted.

<Proposed method 2>
In the above proposed method 1, the sum of the errors for each element between the time domain signals or the complex spectrogram is used as the learning criterion, but when such a criterion is used, it fits the target time domain signal or the complex spectrogram on average. The parameters of the generator G will be learned to output what it does. It is possible that random components will be removed from the generator G learned in this way, resulting in the generation of only over-smoothed time domain signals or complex spectrograms. Many real-world signals contain a significant amount of random components, and the removal of random components during signal (or phase) reconstruction can limit the audible quality of the reconstructed signal. is there. Therefore, in the proposed method 2, a learning standard different from that of the proposed method 1 is used in order to prevent the loss of the minute random components originally inherent in the real-world signal in the process of signal (or phase) reconstruction by the generator G. To learn the generator G. Specifically, a neural network (hereinafter referred to as discriminator D) that discriminates whether the signal is a signal generated by the generator G or a signal of actual data is introduced, and measurement is performed between the output values of the intermediate layer of the discriminator D. The error is the learning criterion. The processing flow of the proposed method 2 is shown in FIG.

<Learning method>
FIG. 6 shows the processing flow of the learning method. First, as training data, a vectorized version x of an acoustic signal such as voice or music or its complex spectrogram obtained by STFT or wavelet transform is prepared. This data is subjected to STFT or wavelet transform to obtain _a vectorized amplitude spectrogram (amplitude component of a complex spectrogram). Taking this amplitude spectrogram f _a and the random source component z as inputs, we train the generator G so that x is reconstructed. Here, the random source component z is a random number vector generated according to a certain distribution (for example, uniform distribution). Furthermore, we introduce a classifier D that discriminates between two classes (for example, outputs 0 or 1) whether it is a signal generated by the generator G or a signal of actual data, and this is also learned together with the generator G. .. Assuming that the output of the generator G inputting the amplitude spectrogram f _a and the random source component z is ~ x, the error measured between the output values in the intermediate layer of the classifier D when x and ~ x are input is small. Learn the parameters of generator G so that it does. On the other hand, the classifier D learns using some discriminant score as a learning criterion so as to correctly discriminate whether the signal is a signal generated by the generator G or a signal of actual data.

First, consider the criteria such as equation (2). In equation (2), G stands for Generator and D stands for Discriminator. This criterion represents a discriminant score called cross-entropy, which indicates whether discriminator D is able to correctly discriminate whether the input time domain signal or complex spectrogram is real data or generated from generator G. Therefore, increasing this criterion with respect to D means that discriminator D can correctly discriminate whether the input time domain signal or complex spectrogram is real data or generated from generator G. To do.

Generator G, on the other hand, wants to learn that the reconstructed time domain signal or complex spectrogram is (incorrectly) identified by discriminator D as real data. This is achieved by reducing the above criteria with respect to G. In addition, we want the output ~ x of the generator to be as close as possible to the target signal x. The middle layer of discriminator D, which has been learned to some extent, should be a feature space that makes it as easy as possible to distinguish (separate) ~ x and x, so ~ x and x should be selected in such a space. It is expected that by getting as close as possible, it will be possible to obtain ~ x that makes the difference from x more indistinguishable. Therefore, in addition to Eq. (2), the error of each output value in the intermediate layer of the classifier D of ~ x and x.

As a learning criterion for generator G. However, D _h (・) represents the output value of the h layer of the classifier D. Therefore, D ₀ (x) = x. C (α, β) is a function that expresses the distance between the vectors of α and β, for example, the p-norm.

And so on. Also, w _h is a non-negative weight constant. Therefore, when considering the distance of only the output value of the hth layer, w _h ′ = 0 (h ′ ≠ h) may be set.

を、生成器G に関しては

をそれぞれ学習規準として識別器D と生成器G を競争させながら学習する。ただし、λは第二項の重みを表す定数で、0≦λ≦1 の範囲で適当に定めるものとする。 From the above, regarding the classifier D

For generator G

Are to be learned while competing the classifier D and the generator G, respectively, as learning criteria. However, λ is a constant representing the weight of the second term, and shall be appropriately determined within the range of 0 ≤ λ ≤ 1.

The methodology for learning generators by competing discriminators and generators in this way is called Generative Adversarial Network (GAN). The present embodiment is new in that this methodology is introduced in a form suitable for the signal (or phase) reconstruction problem as described above. The criteria shown in equation (2) are those proposed in Non-Patent Document 2. In recent years, research on GAN has been wide-ranging, and various improvements have been made, including Non-Patent Document 3 and Non-Patent Document 4.

[Non-Patent Document 2] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozairy, Aaron Courville and Yoshua Bengio: “Generative Adversarial Nets,” arXiv: 1406.2661, 2014.

[Non-Patent Document 3] MAO, Xudong, et al .: “Least squares generative adversarial networks,” arXiv preprint ArXiv: 1611.04076, 2016.

[Non-Patent Document 4] Martin Arjovsky, Soumith Chintala, and Leon Bottou: “Wasserstein GAN,” arXiv preprint arXiv: 1701.07875, 2017.

As the optimization function to be designed in this learning method, it is not necessary to limit it to the equation (2), and those proposed in Non-Patent Document 3 and Non-Patent Document 4 may be used.

<Learning method supplement 1>
The discriminator D may also input the voice feature sequence f'as in Eq. (6). Examples of the voice feature series f'include vocal cord sound source information and amplitude spectrogram. In the latter case, f ′ = f _a .

<Learning method supplement 2>
When learning this neural network, the generator G may be pre-learned by using the proposed method 1.

<Learning method supplement 3>
In a complex spectrogram, if the phase difference between frames of each frequency is the same, even if the phase spectrum (initial phase) of the first frame is arbitrary, the corresponding time domain signals are perceived as the same by human hearing. It is known to be done. Therefore, if the differences that cannot be perceived by humans are regarded as the same, there are many time domain signals or complex spectrograms that are correct for each input amplitude spectrogram due to the arbitraryness of the initial phase. Therefore, a large number of time domain signals or complex spectrograms in which the initial phase is changed for each input amplitude spectrogram may be prepared for the target teacher data.

<Learning method supplement 4>
Unless the generator is specifically constrained, the amplitude spectrogram calculated from the time domain signal or complex spectrogram output by the generator does not always exactly match the input amplitude spectrogram. Therefore, in order to force the amplitude spectrogram calculated from the time region signal or complex spectrogram output by the generator to match the input amplitude spectrogram, the amplitude spectrogram of the time region signal or complex spectrogram is input to the final layer of the generator. A layer corresponding to the operation to be replaced with the amplitude spectrogram may be added.

<Learning method supplement 5>
Since the Fourier transform of a real number signal has an even function in the real part and an odd function in the imaginary part, its symmetry can be used depending on the assumed time-frequency analysis method. For example, if the frame length of the STFT is 1024 points, the number of frequency bins of the obtained complex spectrogram will be 1024 including negative frequencies, but if symmetry is used, it corresponds to the frequency from 0 to the Nyquist frequency 513. Only point information is sufficient to construct a time domain signal. Therefore, when the output of the generator is a complex spectrogram, the complex spectrogram corresponding to some frequency bins may be used as the output.

<Learning method supplement 6>
A new complex spectrogram is constructed by adding arbitrary values to the phase components of the complex spectrogram obtained by performing the Fourier transform on the time domain signal. It is known that even if this new complex spectrogram is converted into a time domain signal by the inverse Fourier transform, it is perceived by human hearing as the same as the original time domain signal. Utilizing this property, a large number of teacher data (time domain signals or complex spectrograms) corresponding to the input amplitude spectrogram may be prepared as in Supplement 3 of the learning method.

<Generation method>
It is possible to obtain a reconstructed signal or reconstructed complex spectrogram ~ x by inputting arbitrary amplitude spectrogram series f _a neural network learning is completed.

<Generation method supplement 1>
The amplitude spectrogram calculated from the time domain signal or complex spectrogram output by the generator does not always exactly match the input amplitude spectrogram. Therefore, a process of recalculating the time domain signal after replacing the amplitude spectrogram of the time domain signal or the complex spectrogram output from the generator with the input amplitude spectrogram may be added.

<Generation method supplement 2>
Similar to the learning method Supplement 5, the symmetry can be used depending on the assumed time-frequency analysis method.

<System configuration>
<Configuration of acoustic signal generation model learning device>
The learning unit 24 of the acoustic signal generation model learning apparatus 100 according to the second embodiment inputs an amplitude spectrogram based on a pair of a time region signal or a complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. Then, the generator, which is a neural network that outputs a time region signal or a complex spectrogram of an acoustic signal, and the discriminator, which is a neural network for discriminating whether or not the data is real data, are reduced in learning criteria. learn.

Specifically, the learning criterion is output by the generator with the output value of the intermediate layer when the time region signal or complex spectrogram of the teacher acoustic signal is input to the classifier and the amplitude spectrogram of the teacher acoustic signal as inputs. Represents the error from the output value of the intermediate layer when a time region signal or complex spectrogram is input to the classifier.

The parameters of the generator learned by the learning unit 24 are output by the output unit 50.

<Configuration of acoustic signal generator>
The generation unit 76 of the acoustic signal generation device 150 according to the second embodiment is input using a generator which is a neural network for generating an acoustic signal learned by the acoustic signal generation model learning device 100. A time region signal or a complex spectrogram of an acoustic signal is generated from the amplitude spectrogram, and is output by the output unit 90.

Since the other configurations and operations of the acoustic signal generation model learning device 100 and the acoustic signal generation device 150 according to the second embodiment are the same as those of the first embodiment, the description thereof will be omitted.

As described above, according to the acoustic signal generation model learning apparatus according to the second embodiment, the amplitude spectrogram is based on the time region signal or the complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. A generator that is a neural network that outputs a time region signal or complex spectrogram of an acoustic signal, and a classifier that is a neural network for discriminating whether or not it is actual data, in the time region of a teacher acoustic signal. When the output value of the intermediate layer when the signal or complex spectrogram is input to the discriminator and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as the input are input to the discriminator. By learning so as to reduce the learning criterion representing the error from the output value of the intermediate layer, it is possible to learn a generator that generates an acoustic signal with high quality.

Further, according to the acoustic signal generator according to the second embodiment, the output value of the intermediate layer when the time region signal or the complex spectrogram of the teacher acoustic signal is input to the classifier and the amplitude spectrogram of the teacher acoustic signal. Input an amplitude spectrogram pre-learned to reduce the learning criteria that represent the error between the time region signal or complex spectrogram output by the generator as input and the output value of the middle layer when input to the classifier. By using a generator that outputs a time region signal or a complex spectrogram of the acoustic signal, the acoustic signal can be generated with high quality.

[Third Embodiment]
Next, a third embodiment will be described. The parts having the same configuration as that of the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

<Proposed method 3>
In the proposed methods 1 and 2, the amplitude spectrogram and the random source component were used as the inputs of the generator, but in the proposed method 3, a time domain signal or a complex spectrogram constructed by using the input amplitude spectrogram and the phase spectrogram generated by some method is generated. It is the input of the vessel. The input time domain signal or complex spectrogram may consist of an input amplitude spectrogram and a randomly generated phase spectrogram, or may be obtained by applying the Griffin-Lim method. However, when the Griffin-Lim method is used, if the number of algorithm iterations is increased, it takes a long time to calculate the input of the generator. Therefore, in that case, for example, the number of repetitions is set to about 5. FIG. 7 shows the processing flow of the proposed method 3.

<Learning method>
Except for the fact that instead of taking the amplitude spectrogram and the random source component as the generator input, the time domain signal or complex spectrogram constructed using the input amplitude spectrogram and the phase spectrogram generated in some way is taken as the generator input. , The same as the learning method and learning method supplements 1 to 6 of the proposed method 2 described in the second embodiment.

<Supplementary learning method>
When randomly generating a phase spectrogram to be input to the generator, the phase spectrum of a certain frame may be fixed. Considering the phase component of an acoustic signal, what kind of phase should be held at a certain moment is not absolutely determined, but what kind of phase should be held from the relative relationship with the phases of the preceding and following signals. Is decided. Using this property, when random sampling of the phase, for example, if only the phase component of the first frame is fixed, the generator will reconstruct the phase of the second and subsequent frames relative to the phase component of the first frame. There is a possibility that learning will proceed efficiently because it is only necessary to learn the specific relationships.

<Generation method>
Except for the fact that instead of taking the amplitude spectrogram and the random source component as the generator input, the time domain signal or complex spectrogram constructed using the input amplitude spectrogram and the phase spectrogram generated in some way is taken as the generator input. , The generation method of the proposed method 2 described in the second embodiment and the generation method supplements 1 and 2 are the same.

<System configuration>
<Configuration of acoustic signal generation model learning device>
As shown in FIG. 8, the acoustic signal generation model learning device 100 according to the third embodiment includes an input unit 10, a calculation unit 320, and an output unit 50.

The input unit 10 receives a pair of a time domain signal or a complex spectrogram of the teacher acoustic signal and an amplitude spectrogram of the teacher acoustic signal.

The calculation unit 320 includes an input generation unit 322 and a learning unit 324.

The input generation unit 322 constitutes a time domain signal or a complex spectrogram from the amplitude spectrogram of the teacher acoustic signal. For example, the input amplitude spectrogram and the phase spectrogram generated in some way are used to construct a time domain signal or a complex spectrogram by the Griffin-Lim method.

The learning unit 324 takes the time region signal or the complex spectrogram as an input and outputs the time region signal or the complex spectrogram of the acoustic signal based on the pair of the time region signal or the complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. The generator and the discriminator, which is a neural network for discriminating whether or not the data is actual data, are trained so as to reduce the learning standard.

The learning criterion is, for example, a time region signal composed of an output value of the intermediate layer when a time region signal or a complex spectrogram of the teacher acoustic signal is input to the classifier and an amplitude spectrogram of the teacher acoustic signal by the input generator 322. Alternatively, it represents the error from the output value of the intermediate layer when the time region signal or the complex spectrogram output by the generator with the complex spectrogram as the input is input to the discriminator.

The parameters of the generator learned by the learning unit 24 are output by the output unit 50.

<Configuration of acoustic signal generator>
As shown in FIG. 9, the acoustic signal generation device 350 according to the third embodiment includes an input unit 60, a calculation unit 370, and an output unit 90.

The input unit 60 receives the amplitude spectrogram.

The calculation unit 370 includes an input generation unit 372 and a generation unit 376.

The input generation unit 372 constructs a time domain signal or a complex spectrogram from the input amplitude spectrogram.

The generation unit 376 generates a time domain signal or a complex spectrogram of the acoustic signal from the time domain signal or the complex spectrogram configured by the input generation unit 372 by using the generator learned by the acoustic signal generation model learning device 300. , Output by the output unit 90.

<Operation of acoustic signal generation model learning device>
Next, the operation of the acoustic signal generation model learning device 300 according to the third embodiment will be described. First, when a time region signal or a complex spectrogram of the teacher acoustic signal and a pair of the amplitude spectrogram of the teacher acoustic signal are input to the acoustic signal generation model learning device 300, the acoustic signal generation model learning device 300 shows in FIG. The learning processing routine is executed.

First, in step S300, the input generation unit 322 constructs a time domain signal or a complex spectrogram from the amplitude spectrogram of the teacher acoustic signal.

Then, in step S302, the learning unit 324 causes the generator and the discriminator to have a smaller learning criterion based on the pair of the time domain signal or the complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal. learn. Then, the learned generator parameters are output by the output unit 50.

<Action of acoustic signal generator>
Next, the operation of the acoustic signal generation device 350 according to the third embodiment will be described. First, the parameters of the generator learned by the acoustic signal generation model learning device 300 are input to the acoustic signal generation device 350. Further, when the amplitude spectrogram is input to the acoustic signal generation device 350, the acoustic signal generation device 150 executes the acoustic signal generation processing routine shown in FIG.

First, in step S310, the input generation unit 372 constructs a time domain signal or a complex spectrogram from the input amplitude spectrogram.

In step S312, the generation unit 376 uses the generator learned by the acoustic signal generation model learning device 300 to obtain the time domain signal or complex spectrogram of the acoustic signal from the time domain signal or complex spectrogram configured in step S310. Is generated and output by the output unit 90.

As described above, according to the acoustic signal generation model learning apparatus according to the third embodiment, the time region is based on the time region signal of the teacher acoustic signal or the pair of the complex spectrogram and the amplitude spectrogram of the teacher acoustic signal. A teacher sound is a generator that is a neural network that takes a signal or a complex spectrogram as an input and outputs a time region signal or a complex spectrogram of an acoustic signal, and a classifier that is a neural network for discriminating whether or not it is real data. The output value of the intermediate layer when the time region signal or complex spectrogram composed of the time region signal or complex spectrogram of the signal is input to the classifier, and the time region signal or complex spectrogram composed of the amplitude spectrogram of the teacher acoustic signal. An acoustic signal by learning a time region signal or complex spectrogram output by the generator with the complex spectrogram as input to reduce the learning criteria that represent the error from the output value of the middle layer when input to the classifier. You can learn the generator that produces high quality.

Further, according to the acoustic signal generator according to the third embodiment, the intermediate layer when the time region signal or the complex spectrogram composed of the time region signal or the complex spectrogram of the teacher acoustic signal is input to the classifier. Output value of, and the output of the intermediate layer when the time region signal or complex spectrogram output by the generator with the time region signal or complex spectrogram composed of the amplitude spectrogram of the teacher acoustic signal is input to the discriminator. By using a generator trained in advance so as to reduce the learning criterion representing the error from the value, it is possible to generate an acoustic signal with high quality.

<Experiment>
In order to show the effectiveness of the proposed method 3 described in the third embodiment of the present invention, an experiment was conducted using one realization method.

<Experimental settings (with respect to quality)>
<Overview of neural network>
Non-Patent Document 5 was used as a data set for experiments. There are noise-containing audio data and noise-free audio data in this data set, but in this experiment, only noise-free audio data was used. Since the training data and the test data were already divided in the dataset, they were used as they were for training and testing. The sampling frequency of the audio signal was set to 16 kHz. However, during learning, the audio data was separated every second with a duplication of 0.5 seconds.

[Non-Patent Document 5] Cassia Valentini-Botinhao: “Noisy speech database for training speech enhancement algorithms and TTS models,” [dataset]. University of Edinburgh. School of Informatics. Center for Speech Technology Research (CSTR). Http: // dx.doi.org/10.7488/ds/1356.

In the proposed method 3, a complex spectrogram or an acoustic signal constructed by using _{an amplitude spectrogram fa and a} phase component z given by a random number is used as an input of the generator (see FIG. 7). Of these, the complex spectrogram x'was used in this experiment. The output of the generator is a complex spectrogram ^ x. However, the symmetry of the complex spectrogram described in Supplement 5 of the learning method of the second embodiment was used. An acoustic signal x was used as an input for the classifier (see FIG. 6). Here, since the classifier inputs the acoustic signal and the generator outputs the complex spectrogram, the inverse STFT is applied to the complex spectrogram ^ x output by the generator to convert it to the acoustic signal ~ x before identification. I gave it to the vessel. In other words, the discriminator aims to classify the true data x and ~ x obtained by converting the output of the generator by inverse STFT based on the authenticity as an acoustic signal, and the generator seems to be genuine when converted to an acoustic signal. The purpose is to generate a complex spectrogram.

<Data preprocessing>
Here, the preprocessing performed when learning the neural network will be described. First, STFT was applied to the speech data and converted into a complex spectrogram. At this time, the window width of the Fourier transform was 1024 points, the shift width was 512 points, and the window function was a Blackman window. Next, the amplitude spectrogram was extracted by taking the absolute value of the complex spectrogram. In addition, for the complex spectrogram converted from the voice data, the average and variance of each dimension in the frequency direction in the time series direction are obtained. This mean and variance are used during training and generation.

<Learning method>
Next, consider the construction of a complex spectrogram to be input to the generator. The Griffin-Lim method with 5 iterations was applied to the amplitude spectrogram obtained as described above to construct a complex spectrogram to be input to the generator. However, the construction of this complex spectrogram was done in parallel with the learning. That is, every time a complex spectrogram was input to the generator, the complex spectrogram was constructed from the amplitude spectrogram by the Griffin-Lim method. At this time, the initial values of the phase components used in the Griffin-Lim method were randomly sampled from a uniform distribution each time.

In addition, when the constructed complex spectrogram is input to the generator, the complex spectrogram is normalized using the mean and variance obtained in the data preprocessing. The normalization applied here refers to a method in which the mean of the data is 0 and the variance is 1. The phase components were randomly sampled when constructing the complex spectrogram to be input to the generator, but the phase of the first frame was fixed by applying the learning method supplement described in the third embodiment above.

In order to fix the phase of the first frame, the phase obtained from the true audio signal was used.

For the output of the generator, the learning method supplement 4 described in the second embodiment was applied. That is, a process was applied in which the amplitude component of the complex spectrogram output by the generator was replaced with the amplitude component of the complex spectrogram input to the generator. Since the complex spectrogram input to the generator is normalized, the process of restoring the scale was applied to the complex spectrogram output by the generator.

As for the true audio signal to be input to the classifier, a large number of audio signals having different phases were constructed and given by applying the learning method supplement 6 described in the second embodiment. Also, when giving an audio signal to the classifier, it was divided by 16384.

としては、非特許文献３で提案されたものを用いた。また、識別器の中間層におけるターゲットの信号x と生成器が生成した信号~x の距離に関する学習規準

について、距離尺度はL2 ノルムを用いて、重み定数w_hはw_h≠0＝1、w₀= 0とした。また、上記第２の実施の形態で説明した学習方法補足１にあるように、識別器には真のデータないし再構成されたデータと共に、音響特徴量系列f′を与えた。音響特徴量系列としては、振幅スペクトログラムf_a を用いた。 Objective function that optimizes while competing generator and classifier

As, the one proposed in Non-Patent Document 3 was used. Also, a learning criterion for the distance between the target signal x and the generator-generated signal ~ x in the middle layer of the classifier.

As for the distance scale, the L2 norm was used, and the weight constant w _h was set to w _{h ≠ 0} = 1 and w ₀ = 0. Further, as described in the learning method supplement 1 described in the second embodiment, the discriminator is given the acoustic feature sequence f'along with the true data or the reconstructed data. The acoustic feature sequence, using amplitude spectrogram f _a.

<Generation method>
The generation method in this experiment is shown in FIG. The complex spectrogram input to the neural network is normalized using the mean / variance obtained in the preprocessing, and the output is processed to restore the scale. In addition, the phase of the first frame was fixed at the time of learning, but all the phase components were randomly sampled at the time of the test.

<Network structure>
FIG. 13 shows the neural network constructed in this experiment. This time, a neural network was constructed with reference to Non-Patent Document 6.

[Non-Patent Document 6] Christian Ledig et al .: “Photo-realistic single image super-resolution using a generative adversarial network,” arXiv preprint arXiv: 1609.04802, 2016.

When inputting the complex spectrogram to the generator, the real part and the imaginary part were separated into two channels. The layer shown by the upward slash in the figure represents a convolutional neural network (CNN). The symbols written above the CNN (eg 2D k11 s1 c64) represent the hyperparameters of the CNN. D indicates the dimension of the convolution operation. When a two-dimensional convolution operation is performed, it is shown as 2D. k represents the size of the kernel. For example, in 2D and k5, the kernel size is 5 ² = 25. When 1D and k32, the kernel size is 32 ¹ = 32. s indicates the stride width of the kernel. s1 indicates a stride width of 1. c represents the number of kernels. c64 indicates that 64 kernels were used for the convolution operation. Next, the layer of dots shows the activation function. All generators used PReLU and all classifiers used Leaky ReLU. The layer indicated by the downward slash indicates the linear combination, and the number above indicates the dimension of the output. Layers without a symbol have the same settings as the previous layer. The structure of the neural network used in this experiment is only an example.

<Evaluation method>
In this experiment, the AB test was used as the evaluation method. A pair of voice data reconstructed by the Griffin-Lim method with 400 iterations and the proposed method 3 was created and evaluated by 5 subjects. We also gave 10 pairs of voices to each subject and asked each pair which one had the higher quality. At this time, the test data of Non-Patent Document 5 was used, but only the data having a length of 2-5 seconds was evaluated.

<Evaluation result>
FIG. 14 shows the results of a subjective evaluation experiment on quality. In this experiment, out of 50 pairs (10 pairs / person x 5 people), the voice signal whose phase was restored by the proposed method was better than the Griffin-Lim method in which the number of iterations was 400 for about 76% of the pairs. , It was evaluated that it sounds natural without any discomfort. Therefore, it was shown that the proposed method 3 can restore the phase so that a natural sound can be obtained, rather than the Griffin-Lim method.

<Experimental setting (speed of phase restoration processing)>
<Overview of neural network>
Regarding the implementation of the neural network, the settings were the same as in the quality experiment.

<Evaluation method>
In this experiment, the processing speeds of the proposed method 3 and the non-patent document 1 when phase restoration is performed are compared. A comparison was made for each of the following two situations.

・ 1000 samples of test data divided into 1 second are given, and the time required for phase restoration per sample ・ All samples are given without dividing the test data, and the time required for phase restoration per sample

In addition, the proposed method 3 measured the processing time when using the GPU and when using only the CPU. The CPU model of the computer used in the experiment is "Intel (R) Core (TM) i7-6850K CPU @ 3.60GHz", the number of cores is 12, and the clock frequency is 3799.968 [MHz]. The GPU used was "GeForce GTX 1080" manufactured by NVIDIA (R).

<Evaluation result>
First, we compare the time required for phase restoration for a 1-second audio signal. Table 1 shows the average processing time and standard deviation. The number in parentheses is the standard deviation. The processing speed is the fastest when the proposed method 3 is executed on the GPU, and the phase restoration processing is performed about 10 times faster than when the existing method is executed on the CPU.

Next, the test data was given to each method without being separated, and the processing time was measured for each sample.

The results are shown in FIG. The processing speed was the fastest when the proposed method 3 was executed on the GPU, as in the case where the given voice was divided into 1 second.

As described above, by using the method of the embodiment of the present invention, the process of reconstructing a time domain signal (or complex spectrogram) from an amplitude spectrogram can be modeled by a neural network (generator). become. This allows the acoustic signal to be reconstructed from the amplitude spectrogram faster and with higher quality than the Griffin-Lim method. In addition, the method using GAN takes into consideration the randomness of real-world signals, and a natural acoustic signal can be obtained.

The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, in the above-described embodiment, the acoustic signal generation model learning device and the acoustic signal generation device are configured as separate devices, but they may be configured as one device.

Further, the above-mentioned acoustic signal generation model learning device and acoustic signal generation device have a computer system inside, but if the "computer system" is using the WWW system, the homepage providing environment ( Alternatively, the display environment) shall also be included.

Further, although described as an embodiment in which the program is pre-installed in the specification of the present application, it is also possible to provide the program by storing it in a computer-readable recording medium.

10, 60 Input unit 20, 70, 320, 370 Calculation unit 24, 324 Learning unit 50, 90 Output unit 76, 376 Generation unit 100, 300 Acoustic signal generation model Learning device 150, 350 Acoustic signal generator 322, 372 Input generation Department

Claims (8)

An acoustic signal generation model learning device that learns a generator that is a neural network for generating acoustic signals.
Based on the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal.
The generator that takes an amplitude spectrogram as an input and outputs a time domain signal or a complex spectrogram of an acoustic signal.
To reduce the learning criterion that represents the magnitude of the error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input. look including a learning section for learning,
The learning criteria are the output value of the intermediate layer when the time region signal or the complex spectrogram of the teacher acoustic signal is input to the classifier, which is a neural network for discriminating whether or not it is real data, and the teacher acoustic. An acoustic signal generation model learning device that represents an error between a time region signal or a complex spectrogram output by the generator with a signal amplitude spectrogram as an input and an output value of an intermediate layer when the spectrogram is input to the classifier. An acoustic signal generation model learning device that learns a generator that is a neural network for generating acoustic signals.
From the amplitude spectrogram of the teacher acoustic signal, the input generator that composes the time domain signal or complex spectrogram,
Based on the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal.
The generator that takes a time domain signal or a complex spectrogram as an input and outputs a time domain signal or a complex spectrogram of an acoustic signal.
The time domain signal or complex spectrogram composed of the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal by the input generator is input and the time domain signal or complex is output by the generator. An acoustic signal generation model learning device that includes a learning unit that learns to reduce the learning criteria that represent the magnitude of the error from the spectrogram. It contains a generator that takes an amplitude spectrogram as an input and outputs a time domain signal or a complex spectrogram of the acoustic signal using a generator that is a neural network for generating an acoustic signal.
The generator
Based on the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal.
To reduce the learning criterion that represents the magnitude of the error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input. Pre-learned in
An input amplitude spectrogram, Ri said generator der to output a time domain signal or complex spectrogram of the acoustic signal,
The learning criteria are the output value of the intermediate layer when the time region signal or the complex spectrogram of the teacher acoustic signal is input to the classifier, which is a neural network for discriminating whether or not it is real data, and the teacher acoustic. An acoustic signal generator that represents an error between a time region signal or a complex spectrogram output by the generator with a signal amplitude spectrogram as an input and an output value of an intermediate layer when the spectrogram is input to the classifier. An input generator that composes a time domain signal or complex spectrogram from the input amplitude spectrogram,
Using a generator that is a neural network for generating acoustic signals, a generator that takes the time region signal or complex spectrogram configured by the input generator as an input and outputs the time region signal or complex spectrogram of the acoustic signal. Including and
The generator
Based on the time domain signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal.
The error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the time region signal or complex spectrogram composed of the amplitude spectrogram of the teacher acoustic signal as input. Learned to reduce the size learning criteria,
An acoustic signal generator that is a generator that takes a time domain signal or a complex spectrogram as an input and outputs a time domain signal or a complex spectrogram of an acoustic signal. It is an acoustic signal generation model learning method in an acoustic signal generation model learning device that learns a generator which is a neural network for generating an acoustic signal.
The learning unit is based on a pair of a time domain signal or a complex spectrogram of the teacher acoustic signal and an amplitude spectrogram of the teacher acoustic signal.
The generator that takes an amplitude spectrogram as an input and outputs a time domain signal or a complex spectrogram of an acoustic signal.
To reduce the learning criterion that represents the magnitude of the error between the time region signal or complex spectrogram of the teacher acoustic signal and the time region signal or complex spectrogram output by the generator with the amplitude spectrogram of the teacher acoustic signal as input. Including learning to
The learning criteria are the output value of the intermediate layer when the time region signal or complex spectrogram of the teacher acoustic signal is input to the classifier, which is a neural network for discriminating whether or not it is real data, and the teacher acoustic. A method for learning an acoustic signal generation model that represents an error between a time region signal or a complex spectrogram output by the generator with a signal amplitude spectrogram as an input and an output value of an intermediate layer when the spectrogram is input to the classifier. It is an acoustic signal generation model learning method in an acoustic signal generation model learning device that learns a generator which is a neural network for generating an acoustic signal.
The input generator constructs a time domain signal or a complex spectrogram from the amplitude spectrogram of the teacher acoustic signal.
The learning unit is based on a pair of a time domain signal or a complex spectrogram of the teacher acoustic signal and an amplitude spectrogram of the teacher acoustic signal.
The generator that takes a time domain signal or a complex spectrogram as an input and outputs a time domain signal or a complex spectrogram of an acoustic signal.
The time region signal or complex spectrogram composed of the time region signal or complex spectrogram of the teacher acoustic signal and the amplitude spectrogram of the teacher acoustic signal by the input generator is input and the time region signal or complex is output by the generator. An acoustic signal generation model learning method that learns to reduce the learning criteria that represent the magnitude of the error from the spectrogram. Program for causing a computer to function as each section of the sound signal generator model training equipment according to claim 1 or claim 2. 請Motomeko 3 or program for causing a computer to function as each section of the audio signal generating apparatus according to claim 4.

Images