JP6409417B2 - Sound processor - Google Patents

Description

  The present invention relates to a technique for processing an acoustic signal.

  Various techniques for changing the pitch of sound such as voice and music have been proposed. For example, Patent Literature 1 discloses a technique for changing the pitch of a sound produced by a user. Further, for example, Patent Document 2 discloses a configuration in which voice quality conversion is performed by decomposing a singing voice into a harmonic component and a non-harmonic component.

JP 2005-025234 A JP 2000-010600 A

  As a configuration for generating a sound having a desired pitch (hereinafter referred to as “target pitch”) with a timbre equivalent to a specific sound (hereinafter referred to as “target sound”) among the recorded acoustic signals, for example, from the acoustic signal A configuration (pitch shift → morphing) in which the pitch of the extracted target sound is changed to the target pitch and the timbre approaches the target sound after the change can be assumed. However, for example, when the acoustic signal includes a plurality of acoustic components including the target sound, it is difficult to extract only the target sound to be processed with high accuracy, and acoustic components other than the target sound are inevitably targeted. Can accompany sound. In the above situation, there is a problem that acoustic components other than the target sound accompanying the target sound become obvious due to the change in the pitch, and further become apparent due to the conversion of the timbre. In view of the above circumstances, an object of the present invention is to suppress deterioration in sound quality when changing the pitch of a specific sound among acoustic signals.

  In order to solve the above problems, the sound processing apparatus of the present invention includes a first reference signal representing a first reference sound having a tone different from the target sound and a pitch equivalent to the target sound, and a pitch of the target sound. Reference sound acquisition means for acquiring a second reference signal representing a second reference sound having a target tone pitch different from that of the first reference sound, and a target signal representing the target sound and a first reference signal By using the analysis processing means for generating a conversion filter for bringing the first reference sound close to the timbre of the target sound, and applying the conversion filter to the second reference signal, the target pitch can be obtained with a timbre approximating the target sound. And a sound processing means for generating a converted signal representing the sound. In the above aspect, the conversion filter for bringing the first reference sound having the same pitch as the target sound closer to the timbre of the target sound is generated according to the target signal and the first reference signal, and the second reference of the target pitch is obtained. A conversion signal is generated by applying a conversion filter to the second reference signal representing the sound. That is, it is not necessary in principle to convert the pitch of the target sound. Therefore, there is an advantage that it is possible to prevent a decrease in sound quality due to a change in the pitch of the target sound.

  In a preferred aspect of the present invention, the reference sound acquisition means adjusts the pitch of one of the target signal and the first reference signal to the other pitch. In the above aspect, since the conversion filter is generated after adjusting the target signal and the second reference signal to the same pitch, the conversion filter is used in a state where the pitch is different between the target signal and the first reference signal. Compared with the case where it produces | generates, there exists an advantage that the conversion filter which can convert a reference sound into the timbre of an object sound with high precision can be produced | generated.

  For example, in a configuration including a component extraction unit that generates a target signal by suppressing sound other than the target sound from the acoustic signal, a residual component other than the target sound may accompany the target signal. Therefore, in the configuration in which the target signal is changed to the pitch of the first reference signal, there is a possibility that the residual component becomes obvious due to the change in the pitch. Therefore, a configuration in which the reference sound acquisition unit adjusts the first reference signal to a pitch equivalent to that of the target signal is suitable.

  The configuration of the present invention is suitably employed for a configuration that changes the pitch of a specific sound of an acoustic signal. Specifically, the pitch analysis means for analyzing the time series of the pitch of the acoustic signal, the target sound whose pitch should be changed in the time series of the pitch analyzed by the pitch analysis means, and the target pitch after the change The instruction accepting means for accepting the instruction from the user, the reference sound obtaining means for obtaining the reference signal representing the reference sound generated by the external sound source, and the reference sound of the reference signal obtained by the reference sound obtaining means for the timbre of the target sound In a sound processing apparatus comprising: a timbre conversion unit that generates a conversion signal of a target pitch that is close to the sound signal; and a mixing processing unit that mixes the separated signal generated by the component extraction unit and the conversion signal generated by the timbre conversion unit. The above-described embodiments can be used for the timbre conversion means.

  The sound processing apparatus according to each of the above aspects is realized by hardware (electronic circuit) such as a DSP (Digital Signal Processor) dedicated to processing of an acoustic signal, or a general-purpose operation such as a CPU (Central Processing Unit). This is also realized by cooperation between the processing device and the program. The program of the present invention can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. For example, the program of the present invention can be provided in the form of distribution via a communication network and installed in a computer.

1 is a configuration diagram of a sound processing apparatus according to a first embodiment of the present invention. It is explanatory drawing of the process (nonnegative matrix factorization) which produces | generates a pitch series. It is a schematic diagram of a pitch transition image. It is a flowchart of an acoustic edit process. It is a block diagram of an acoustic processing part. It is a flowchart of a timbre conversion process. It is explanatory drawing of a timbre conversion process. It is explanatory drawing of the pronunciation range in 2nd Embodiment. It is a flowchart of the timbre conversion process in 3rd Embodiment.

<First Embodiment>
FIG. 1 is a configuration diagram of a sound processing apparatus 100 according to the first embodiment of the present invention. As illustrated in FIG. 1, the sound processing device 100 includes a computer processing device 10, a storage device 12, a display device 14, an input device 16, a signal supply device 22, a sound source device 24, and a sound emission device 26. It is realized with. For example, a portable information processing device such as a mobile phone or a smartphone, or a portable or stationary information processing device such as a personal computer can be used as the sound processing device 100.

  The signal supply device 22 outputs an acoustic signal X representing an acoustic time waveform. The acoustic signal X of the first embodiment is a signal recorded in an acoustic space having a specific acoustic characteristic such as a live house or a concert hall, for example, and the singing sound of a song and the performance sound of an instrument (hereinafter referred to as “target instrument”). To express the mixed sound waveform. Note that an acoustic signal X including performance sounds of musical instruments other than the target musical instrument can also be processed. A playback device that acquires and outputs the acoustic signal X from a portable or built-in recording medium, and a communication device that receives and outputs the acoustic signal X from a communication network can be used as the signal supply device 22. The acoustic processing device 100 according to the first embodiment changes a specific portion of the performance sound of the target instrument in the acoustic signal X output from the signal supply device 22 (for example, a location where the performer has failed to perform the target instrument). This is a signal processing device that generates an acoustic signal Z.

  The display device 14 (for example, a liquid crystal display panel) displays an image instructed from the arithmetic processing device 10. The input device 16 is an operation device operated by a user for various instructions to the sound processing device 100, and includes, for example, a plurality of operators operated by the user. A touch panel configured integrally with the display device 14 can also be used as the input device 16. The sound emitting device 26 (for example, a speaker or headphones) emits sound according to the acoustic signal Z generated by the arithmetic processing device 10.

  The sound source device 24 is an external sound source that generates an acoustic signal (hereinafter referred to as “reference signal”) R representing the performance sound of the target musical instrument. The sound source device 24 of the first embodiment can generate a reference signal R having an arbitrary pitch. For example, a known sound source such as a PCM (Pulse Code Modulation) sound source can be arbitrarily employed as the sound source device 24. Further, the function of the sound source device 24 can be realized by the arithmetic processing device 10 executing the program stored in the storage device 12.

  The storage device 12 stores a program executed by the arithmetic processing device 10 and various data used by the arithmetic processing device 10. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media is arbitrarily employed as the storage device 12. The arithmetic processing device 10 executes a program stored in the storage device 12 to generate a plurality of functions (sound source separation unit 32, pitch analysis unit 34, display control unit) for generating the acoustic signal Z from the acoustic signal X. 36, an instruction receiving unit 38, a component extracting unit 40, an acoustic processing unit 42, and a mixing processing unit 44). A configuration in which each function of the arithmetic processing device 10 is distributed to a plurality of devices or a configuration in which a dedicated electronic circuit realizes a part of the function of the arithmetic processing device 10 may be employed.

  The sound source separation unit 32 generates an acoustic signal XA and an acoustic signal XB from the acoustic signal X output from the signal supply device 22. The acoustic signal XA is a signal in which the singing sound is emphasized in the acoustic signal X (ideally, a signal from which the performance sound of the target instrument is removed), and the acoustic signal XB is a performance of the target instrument in the acoustic signal X. This is a signal in which the sound is emphasized (ideally, a signal from which the singing sound has been removed). A known technique can be arbitrarily employed for generating the acoustic signal XA and the acoustic signal XB. For example, a sound source separation process that separates a singing sound and a performance sound using a difference in position where the sound images of the singing sound and the performance sound are localized is preferably used for generating the acoustic signal XA and the acoustic signal XB.

  The pitch analysis unit 34 analyzes a time series (hereinafter referred to as “pitch sequence”) S of pitches in the acoustic signal XB after separation by the sound source separation unit 32. The pitch series S can be rephrased as a temporal transition of the pitch of the performance sound of the target musical instrument. The pitch analysis unit 34 of the first embodiment generates a pitch series S by non-negative matrix factorization (NMF) for the acoustic signal XB.

  FIG. 2 is an explanatory diagram of non-negative matrix factorization in the first embodiment. As illustrated in FIG. 2, the pitch analysis unit 34 decomposes the observation matrix W representing the acoustic signal XB into a base matrix B and a coefficient matrix G. The observation matrix W is a non-negative matrix of M rows and N columns in which the intensity spectra of N frames obtained by dividing the acoustic signal XB on the time axis are arranged in time series. The intensity spectrum of any one frame is a series of intensity (amplitude and power) at each of M frequencies on the frequency axis. As understood from the above description, the observation matrix W represents a spectrogram of the acoustic signal XB.

  The base matrix B expresses the acoustic characteristics of the performance sound of the target musical instrument. As illustrated in FIG. 2, the base matrix B of the first embodiment arranges K base vectors b [1] to b [K] corresponding to performance sounds of different pitches of the target musical instrument in the horizontal direction. This is a non-negative matrix of M rows and K columns. An arbitrary one basis vector b [k] (k = 1 to K) represents the performance sound of the kth pitch among the K types of pitches (for example, 88 notes of piano) that can be generated by the target musical instrument. It corresponds to an intensity spectrum and is a series of intensities at each of M frequencies on the frequency axis. The base matrix B is generated by analyzing the performance sound of the target instrument and stored in the storage device 12 in advance. The pitch analysis unit 34 of the first embodiment is a coefficient matrix by supervised non-negative matrix factorization (Supervised NMF) of the acoustic signal XB using the base matrix B stored in the storage device 12 as teacher information (prior information). G is generated.

  As illustrated in FIG. 2, the coefficient matrix G includes K rows in which K coefficient vectors g [1] to g [K] corresponding to different base vectors b [k] of the base matrix B are arranged in the vertical direction. It is a non-negative matrix with N columns. The coefficient vector g [k] in the k-th row of the coefficient matrix G is configured with N coefficients a [k, 1] to a [k, N] corresponding to different frames on the time axis. Any one coefficient a [k, n] (n = 1 to N) of the coefficient vector g [k] means a weight value for the base vector b [k] of the base matrix B. Specifically, the N coefficients a [k, 1] to a [k, N] constituting the coefficient vector g [k] are set to the base vector b [k] among the K pitches of the target musical instrument. This corresponds to the time series of the intensity (activity) of the acoustic component of the corresponding kth pitch. That is, in the nth frame having a large coefficient a [k, n], the acoustic component of the kth pitch of the target musical instrument is dominant. In consideration of the above tendency, the pitch analysis unit 34 of the first embodiment calculates the coefficient matrix G as the pitch series S. Specifically, the pitch analysis unit 34 sequentially calculates the coefficient matrix G by repeating the arithmetic processing for updating the coefficient matrix G so that the matrix product of the base matrix B and the coefficient matrix G approaches the observation matrix W. The coefficient matrix G at the time when the predetermined convergence condition is satisfied (for example, when the predetermined value of the update calculation is reached) is determined as the pitch series S. Each coefficient a [k, n] (initial value) of the coefficient matrix G applied to the first calculation process is set to, for example, a random number.

  The display control unit 36 in FIG. 1 causes the display device 14 to display the pitch transition image 142 in FIG. 3 representing the pitch series S analyzed by the pitch analysis unit 34. As illustrated in FIG. 3, the pitch transition image 142 is a piano roll-like image in which the pitch series S is drawn on the coordinate plane in which the time axis (horizontal axis) and the pitch axis (vertical axis) are set. is there. Each point on the time axis corresponds to each of the N frames, and each point on the pitch axis corresponds to each of the K pitches. The points corresponding to the nth frame on the time axis and the kth pitch on the pitch axis depend on the magnitude of the coefficient a [k, n] of the pitch sequence S (coefficient matrix G). It is displayed in a mode (for example, gradation or color). That is, the pitch and sound generation period of each sound (single note for each note) included in the sound signal XB are represented by the pitch transition image 142. Accordingly, the user can intuitively grasp the time series of the performance sound of the target musical instrument (the sound generation period and sound intensity) of the target musical instrument by visually recognizing the pitch transition image 142.

  The instruction receiving unit 38 in FIG. 1 receives an instruction from the user for the input device 16. The instruction receiving unit 38 according to the first embodiment can arbitrarily change the pitch from the pitch sequence S analyzed by the pitch analysis unit 34 (the pitch transition image 142 displayed on the display device 14 by the display control unit 36). The instruction of the performance sound (hereinafter referred to as “target sound”) T is received from the user. As illustrated in FIG. 3, for example, the user appropriately operates the input device 16 while visually recognizing the pitch transition image 142, so that the pitch among the plurality of performance sounds expressed by the pitch transition image 142 is displayed. It is possible to select a target sound T that is desired to be changed and to specify a pitch P (hereinafter referred to as “target pitch”) P after the change of the target sound T. The instruction receiving unit 38 receives an instruction of the target sound T and an instruction of the target pitch P for the pitch transition image 142 from the user. It is also possible for the instruction receiving unit 38 to receive an instruction for a plurality of different target sounds T and an instruction for a target pitch P for each target sound T.

  The component extraction unit 40 in FIG. 1 generates a separation signal YA and a target signal YB from the acoustic signal XB in which the performance sound of the target instrument is emphasized. The separated signal YA is an acoustic signal obtained by suppressing (ideally removing) the target sound T instructed by the user in the acoustic signal XB, and the target signal YB is an acoustic signal that emphasizes the target sound T in the acoustic signal XB. A signal (ideally an acoustic signal from which performance sounds other than the target sound T have been removed). A known technique can be arbitrarily employed to generate the separated signal YA and the target signal YB. For example, sound source separation processing (separation of the target sound T) in the frequency domain using a Wiener filter or the like is preferable. .

  The acoustic processing unit 42 generates an acoustic signal (hereinafter referred to as “conversion signal”) YC representing the performance sound of the target pitch P by the target musical instrument. Specifically, the acoustic processing unit 42 generates the converted signal YC of the target pitch P by processing the reference signal R generated by the sound source device 24. As illustrated in FIG. 1, the acoustic processing unit 42 of the first embodiment includes a reference sound acquisition unit 52 and a timbre conversion unit 54. The reference sound acquisition unit 52 acquires the reference signal R generated by the sound source device 24.

  If the target sound T of the sound signal XB is replaced with a reference sound of the target pitch P generated by the sound source device 24, it is possible to form an acoustic signal Z in which the target sound T is changed to the target pitch P in form. It is. However, since the acoustic signal XB has acoustic characteristics inherent to the recording environment (for example, an acoustic space such as a live house), the target sound T of the acoustic signal XB is simply replaced with the reference sound generated by the sound source device 24. Then, the acoustic characteristics are significantly different between the existing performance sound of the acoustic signal XB and the replacement performance sound (reference sound). Therefore, there is a possibility that the listener of the reproduced sound perceives a sense of incongruity. Considering the above circumstances, the timbre conversion unit 54 of the first embodiment has a target pitch P obtained by bringing the timbre of the reference signal R acquired by the reference sound acquisition unit 52 close to the timbre of the target sound T of the acoustic signal XB. A conversion signal YC is generated. The specific content of the process of converting the timbre of the reference signal R into the timbre of the target sound T (hereinafter referred to as “timbre conversion process”) will be described later.

  1 includes a sound signal XA of the singing sound generated by the sound source separation unit 32, a separation signal YA other than the target sound T generated by the component extraction unit 40, and an acoustic processing unit 42 (tone conversion unit 54). The acoustic signal Z is generated by mixing (for example, a weighted sum) with the conversion signal YC generated by (1). That is, an acoustic signal Z in which the pitch of the target sound T of the target musical instrument in the acoustic signal X is changed to the target pitch P is generated.

  The mixing processing unit 44 of the first embodiment executes various types of acoustic processing before and after mixing the acoustic signal XA, the separated signal YA, and the converted signal YC. For example, adjustment processing (equalizing) for adjusting the frequency characteristics of each signal is executed. Note that the degree of reverberation may be different between the acoustic signal XA and the separated signal YA and the converted signal YC. Therefore, the reverberation feeling is unified by sequentially executing the reverberation suppression process for suppressing the reverberation component from each signal before mixing and the reverberation providing process for adding an appropriate reverberation component to the mixed acoustic signal Z. An acoustic signal Z can be generated. The reproduced sound of the acoustic signal Z generated by the mixing processing unit 44 is emitted from the sound emitting device 26. As can be understood from the above description, the reproduced sound obtained by changing the pitch of the target sound T indicated by the user to the target pitch P among the sounds represented by the acoustic signal X is emitted from the sound emitting device 26.

  FIG. 4 is a flowchart of an operation (hereinafter referred to as “acoustic editing process”) in which the arithmetic processing device 10 generates the acoustic signal Z from the acoustic signal X. The sound editing process is started in response to an instruction (acoustic process start instruction) from the user to the input device 16.

  When the sound editing process is started, the sound source separation unit 32 generates the sound signal XA of the singing sound and the sound signal XB of the performance sound of the target instrument from the sound signal X output from the signal supply device 22 (SA1). The pitch analysis unit 34 performs a non-negative matrix factorization using the base matrix B stored in the storage device 12 as teacher information on the observation matrix W of the acoustic signal XB, thereby generating a pitch sequence S (coefficient matrix G ) Is generated (SA2), and the display control unit 36 causes the display device 14 to display a pitch transition image 142 representing the pitch series S (SA3).

  When the instruction receiving unit 38 receives an instruction for the target sound T and the target pitch P for the pitch transition image 142 from the user (SA4: YES), the component extracting unit 40 uses the acoustic signal XB generated by the sound source separating unit 32. A separation signal YA other than the target sound T and a target signal YB of the target sound T are generated (SA5). The acoustic processing unit 42 generates the conversion signal YC by timbre conversion processing (morphing) that brings the reference signal R generated by the sound source device 24 close to the timbre of the target sound T (SA6). The mixing processing unit 44 generates the acoustic signal Z by mixing the acoustic signal XA, the separated signal YA, and the converted signal YC (SA7).

<Acoustic processing part 42>
FIG. 5 is a specific configuration diagram of the acoustic processing unit 42. As illustrated in FIG. 5, the timbre conversion unit 54 of the acoustic processing unit 42 in the first embodiment includes an analysis processing unit 62 and an acoustic processing unit 64. FIG. 6 is a flowchart of the timbre conversion process SA6 executed by the acoustic processing unit 42 (reference sound acquisition unit 52, timbre conversion unit 54) of the first embodiment, and FIG. 7 is an explanatory diagram of the timbre conversion process SA6.

  When the tone color conversion process SA6 is started, the reference sound acquisition unit 52 specifies the pitch of the target sound T of the target signal YB (SB1), and generates a reference signal R1 representing the reference sound Q1 having the same pitch as the target sound T. Obtained from the sound source device 24 (SB2). As described above, the tone color of the reference sound Q1 is different from the target sound T of the acoustic signal XB. As illustrated in FIGS. 5 and 7, the analysis processing unit 62 uses the target signal YB generated by the component extraction unit 40 and the reference signal R1 acquired by the reference sound acquisition unit 52 in step SB2 to convert the conversion filter H. Is generated (SB3). The conversion filter H is a filter for bringing the timbre of the reference sound Q1 generated by the sound source device 24 closer to the timbre of the target sound T.

  Specifically, the analysis processing unit 62 sets the conversion filter H for each pair of frames (for example, frames whose acoustic feature values are similar to each other) corresponding to each other between the target signal YB and the reference signal R1. Generate. A known technique such as dynamic programming is arbitrarily employed for analyzing the correspondence of each frame between the target signal YB and the reference signal R1. The conversion filter H of the first embodiment is a series of adjustment values (gains) h corresponding to each of a plurality of bands (hereinafter referred to as “analysis bands”) set on the frequency axis. Each analysis band is simply set to an equal bandwidth, but the bandwidth of each analysis band can be set to a logarithmic relationship so that the tendency of human auditory characteristics is reflected. The adjustment value h of an arbitrary analysis band of the conversion filter H is calculated as, for example, the relative ratio (h = VY / VR) of the intensity VY of the target signal YB to the intensity VR of the reference signal R1. The intensity VR of the reference signal R1 is the sum of the intensity over a plurality of frequencies in the analysis band in the intensity spectrum of the reference signal R1, and the intensity VY of the target signal YB is in the analysis band of the intensity spectrum of the target signal YB. The sum of intensities over multiple frequencies. A configuration in which each adjustment value h is adjusted so that the average of the plurality of adjustment values h constituting the conversion filter H becomes zero (zero average) may be employed.

  When the analysis processing unit 62 generates the conversion filter H in the procedure exemplified above, the reference sound acquisition unit 52 represents the reference signal R2 representing the reference sound Q2 having the target pitch P (pitch different from the target sound T). Is acquired from the sound source device 24 (SB4). The tone of the reference sound Q2 is equivalent to the reference sound Q1. As illustrated in FIGS. 5 and 7, the acoustic processing unit 64 generates the converted signal YC by applying the conversion filter H generated by the analysis processing unit 62 in step SB3 to the reference signal R2 (SB5). Specifically, the acoustic processing unit 64 multiplies each analysis band obtained by dividing the intensity spectrum of each frame of the reference signal R2 on the frequency axis by each adjustment value h of the conversion filter H. As described above, the conversion filter H acts to bring the timbre of the reference sound Q1 closer to the timbre of the target sound T. Therefore, by applying the conversion filter H to the reference signal R2, the target timbre approximates the target sound T. A conversion signal YC representing the sound of the pitch P is generated. The above is the specific content of the timbre conversion process SA6.

  As understood from the above description, in the first embodiment, the converted signal YC of the target pitch P generated by processing the reference signal R acquired from the sound source device 24 is converted into the separated signal YA after the suppression of the target sound T. Since they are mixed, it is possible to suppress a decrease in sound quality of the acoustic signal Z as compared with the configuration in which the target signal YB of the target sound T is converted into the target pitch P. The target signal YB generated by the component extraction unit 40 is ideally composed only of the target sound, but actually includes sound other than the target sound (hereinafter referred to as “residual component”). In the configuration in which the pitch of the target signal YB is converted to the target pitch P, the residual component becomes particularly apparent due to the change in the pitch. On the other hand, in the first embodiment in which the converted signal YC of the target pitch P generated from the reference signal R is mixed with the separated signal YA, it is not necessary to change the pitch of the target signal YB. Even when the accuracy is low (when a residual component is included in the target signal YB), there is an advantage that a high-quality sound signal Z can be generated. On the other hand, in the configuration in which the reference signal R generated independently of the acoustic signal XB is simply mixed with the separated signal YA, the auditory discomfort due to the difference in timbre between the two becomes a problem, but the first embodiment Then, since the reference sound of the reference signal R is converted into the timbre of the target sound T, it is possible to eliminate the auditory discomfort caused by the difference between the timbre of the acoustic signal XB and the timbre of the reference sound.

  By the way, as a structure which produces | generates the sound of the target pitch P by the timbre equivalent to the target sound T, for example, the structure which changes the pitch of the target sound T to the target pitch P and makes the timbre approach the target sound T after the change. (Pitch shift → morphing) can be assumed. However, as described above, there is a problem that the residual component that has become apparent due to the change in the pitch of the target sound T becomes more apparent due to the conversion of the timbre. For the above circumstances, in the first embodiment, a conversion filter H for making the reference sound Q1 having the same pitch as the target sound T close to the timbre of the target sound T is generated from the target signal YB and the reference signal R1, By applying the reference signal R2 of the reference sound Q2 having the target pitch P to the conversion filter H, the conversion signal YC is generated. That is, it is not necessary in principle to convert the pitch of the target sound T. Therefore, according to the first embodiment, there is an advantage that it is possible to prevent a decrease in sound quality due to a change in the pitch of the target sound T.

Second Embodiment
A second embodiment of the present invention will be described. In addition, about the element which an effect | action and function are the same as that of 1st Embodiment in each structure illustrated below, the code | symbol used by description of 1st Embodiment is diverted, and each detailed description is abbreviate | omitted suitably.

  In the coefficient matrix G (pitch series S) generated by the pitch analysis unit 34, ideally, only the coefficient a [k, n] corresponding to the actual performance sound of the target musical instrument is set to a significant numerical value. However, in reality, for example, the pitch coefficient a [k, n] having a specific relationship (for example, a pitch of 5 degrees) with the performance sound of the target musical instrument is actually played. There is a possibility that it will be a significant number even if it is not. That is, a significant numerical coefficient a [k, n] may exist outside the pitch range in which the actual pitch of the performance sound of the target instrument in the acoustic signal XB is distributed. The user appropriately operates the input device 16 so that the sound of the acoustic signal XB (the performance sound of the target instrument) in the pitch transition image 142 displayed on the display device 14 is displayed as illustrated in FIG. It is possible to designate a range A (hereinafter referred to as “sound generation range”) A on the time axis and the pitch axis that is presumed to exist. For example, assuming a keyboard instrument (for example, a piano) as the target instrument, the high pitch range played with the right hand of the performer and the low pitch range played with the left hand are designated as the pronunciation range A. Is done. The instruction receiving unit 38 of the second embodiment receives an instruction for the pronunciation range A described above from the user.

  The pitch analysis unit 34 according to the second embodiment reanalyzes the pitch series S in consideration of the pronunciation range A received by the instruction receiving unit 38. Specifically, the pitch analysis unit 34 sets each coefficient a [k, n] outside the sound generation range A instructed by the user to zero and the sound generation range A as illustrated in FIG. The pitch sequence S is obtained by non-negative matrix factorization using a matrix in which each coefficient a [k, n] inside is set to a significant numerical value λ other than zero as an initial value (initial matrix) of the coefficient matrix G. Calculate. The numerical value λ is set to a random number, for example. The display control unit 36 causes the display device 14 to display a pitch transition image 142 representing the pitch series S reanalyzed by the pitch analysis unit 34. The process of generating the acoustic signal Z in response to an instruction from the user with respect to the pitch transition image 142 is the same as in the first embodiment.

  In the second embodiment, the same effect as in the first embodiment is realized. In the second embodiment, the pitch sequence S is generated by non-negative matrix factorization using a matrix in which each coefficient a [k, n] outside the sound generation range A is set to zero as an initial value of the coefficient matrix G. Generated. That is, the pitch series S is updated so that the sound generation range A instructed by the user is reflected. Therefore, there is an advantage that the pitch sequence S can be generated with higher accuracy than the configuration in which the instruction of the sound generation range A is not reflected in the pitch sequence S.

<Third Embodiment>
FIG. 9 is a flowchart of the timbre conversion process SA6 executed by the acoustic processing unit 42 (reference sound acquisition unit 52, timbre conversion unit 54) of the third embodiment. In the first embodiment, the generation of the conversion filter H corresponding to the target signal YB and the reference signal R1 is exemplified on the assumption that the target sound T and the reference sound Q1 have the same pitch. For example, the pitch of the target sound T and the reference sound Q1 may be different due to the tuning of the target musical instrument or the state of tuning of the acoustic signal XB. Considering the above circumstances, as illustrated in FIG. 9, the reference sound acquisition unit 52 of the third embodiment performs a process (SB10) of adjusting the target sound T and the reference sound Q1 to equivalent pitches. It is executed between the acquisition of the reference signal R1 (SB2) and the generation of the conversion filter H (SB3). Specifically, the reference sound acquisition unit 52 of the third embodiment adjusts the reference sound Q1 to the pitch of the target sound T by processing the reference signal R1 of the reference sound Q1. A known technique (pitch shift) is arbitrarily employed to change the pitch of the reference signal R1. The analysis processing unit 62 uses the adjusted reference signal R1 and the target signal YB of the target sound T to generate the conversion filter H by the same method as in the first embodiment (SB3).

  In the third embodiment, the same effect as in the first embodiment is realized. In the third embodiment, since the conversion filter H is generated after adjusting the target sound T and the reference sound Q1 to the same pitch, the target sound T and the reference sound Q1 have different pitches. Compared with the case where the conversion filter H is generated, there is an advantage that the conversion filter H that can convert the reference sound Q1 (and thus the reference sound Q2) into the timbre of the target sound T with high accuracy can be generated. In the above description, the configuration in which the reference sound Q1 is adjusted to the pitch of the target sound T is exemplified, but the target sound T can also be adjusted to a pitch equivalent to the reference sound Q1. However, as described above, the target sound T includes a residual component other than the target sound, and the residual component may become obvious due to a change in the pitch of the target sound T. Considering the above circumstances, a configuration in which the reference sound Q1 of the reference signal R1 is adjusted to the pitch of the target sound T is particularly suitable.

<Modification>
Each of the above forms can be variously modified. Specific modifications are exemplified below. Two or more aspects arbitrarily selected from the following examples can be appropriately combined.

(1) In each of the above embodiments, the pitch sequence S generated by the non-negative matrix factorization for the acoustic signal XB is not limited to the above examples. For example, a well-known analysis technique such as automatic music transcription can be used to generate the pitch sequence S. In the second embodiment, a provisional pitch sequence S is generated by a method other than non-negative matrix factorization, and each of the pitch transition images 142 of the pitch sequence S corresponding to the outside of the pronunciation range A. It is also possible to re-analyze the definite pitch sequence S by executing non-negative matrix factorization of the observation matrix W using the coefficient matrix G in which the coefficient a [k, n] is set to zero as an initial value. . That is, the method for generating the provisional pitch sequence S before the sounding range A is instructed may be different from the method for generating the definite pitch sequence S reflecting the sounding range A. It is also possible to repeat the instruction of the sound generation range A and the reanalysis of the pitch series S a plurality of times.

(2) In each of the above-described embodiments, the coefficient matrix G is calculated by non-negative matrix factorization of the observation matrix W using K basis matrices B corresponding to performance sounds of different pitches of the target musical instrument. The content of the non-negative matrix factorization performed on the matrix W can be changed as appropriate. For example, a basis matrix B composed of K basis vectors (hereinafter referred to as “provisional basis vectors”) in which each element is initialized with a random number is sequentially converted together with a coefficient matrix G by non-negative matrix factorization repetitive calculation. The structure to update is also employ | adopted.

  It is also possible to use a base matrix B in which a base vector prepared in advance for a performance sound of the target musical instrument and an arbitrary provisional base vector are mixed for non-negative matrix factorization. In the configuration in which the base vector of the target instrument and an arbitrary provisional base vector are mixed in the base matrix B, for example, the performance sound of an instrument other than the target instrument (hereinafter referred to as “other instrument”) in addition to the target instrument is included in the acoustic signal XB. When included, the basis matrix B is sequentially updated so that the performance sound of the other musical instrument is reflected in the provisional basis vector. Therefore, there is an advantage that the pitch series S of the target musical instrument can be specified with high accuracy even when the performance sound of the other musical instrument is included in the acoustic signal XB. When the second embodiment is applied to the above configuration in which the base vector of the target instrument and an arbitrary provisional base vector are mixed in the base matrix B, each base of the target instrument in the initial coefficient matrix G is used. A configuration in which only the coefficient a [k, n] outside the sounding range A is set to zero for only the coefficient vector g [k] corresponding to the vector (each coefficient vector g [k] corresponding to each provisional base vector A configuration in which the coefficient a [k, n] is not zero) is preferable. In addition, for example, the constraint conditions exemplified in Japanese Patent Application Laid-Open No. 2013-033196 can be applied to the non-negative matrix factorization of the observation matrix W.

(3) In a configuration in which the sound source device 24 can generate a reference signal R for a performance sound of a plurality of types of musical instruments (the same type but different types of musical instruments can be distinguished from different types), the user selects among the multiple types of musical instruments It is also possible for the reference sound acquisition unit 52 to acquire the reference signal R of the performance sound of the musical instrument (the musical instrument whose acoustic characteristics are estimated to be approximated from the reproduced sound of the acoustic signal X).

(4) In the second embodiment, the configuration in which the user instructs the sound generation range A is exemplified, but the method of setting the sound generation range A is not limited to the above illustration. For example, the distribution of each note on the time axis and the pitch axis by referring to music data (for example, time series data compliant with the MIDI standard) that specifies the musical performance of the music of the acoustic signal X (time series of notes) The range can be specified, and the pitch analysis unit 34 can set the range as the sound generation range A. Further, if it is assumed that the coefficient a [k, n] where the performance sound actually exists is set to a relatively large value, the coefficient matrix G (pitch series S) exceeds the threshold value. It is also possible to set the range in which the coefficient a [k, n] is distributed as the sound generation range A. In the second embodiment, the sound generation range A is defined by both the range on the pitch axis and the range on the time axis. However, the range on the pitch axis (the entire range on the time axis) is set as the sound generation range A. And a configuration in which a range on the time axis (a whole range on the pitch axis) is set as the sound generation range A may be employed.

(5) In each of the above-described embodiments, the case where the pitch of the target sound is changed is illustrated for convenience. However, the sound generation period (start point and end point) of the target sound can be changed together with the pitch. For example, a configuration in which the reference signal R2 acquired by the reference sound acquisition unit 52 is applied to the reference filter R after the timbre conversion unit 54 (conversion processing unit 64) expands or contracts to the target duration, or a conversion filter H for the reference signal R2 A configuration in which the timbre conversion unit 54 (conversion processing unit 64) expands and contracts the conversion signal YC generated by applying the above to the target continuation length may be employed.

(6) When the target sound T and the target pitch P are tentatively indicated in the pitch transition image 142, the conversion signal YC can be generated and emitted from the sound emitting device 26. According to the above configuration, there is an advantage that the user can audition the change result of the target sound T in advance.

(7) In the third embodiment, the configuration in which one of the target signal YB and the reference signal R1 is adjusted to the other pitch is exemplified, but the pitches of the target signal YB and the reference signal R1 are set to a plurality of preset pitches. A configuration for changing (quantizing) the pitch to the closest pitch among the pitches may be employed. Further, if there is a minute pitch fluctuation (swing) in the target sound T of the target signal YB or the reference sound Q1 of the reference signal R1, the conversion filter H is suppressed after suppressing the minute fluctuation (ideally removed). Can also be generated. For example, since the acoustic signal XB of the singing sound generated by speech synthesis can be accompanied by minute fluctuations such as vibrato, a configuration that suppresses minute fluctuations in the pitch from the target signal YB is particularly suitable. It is also possible to generate the conversion filter H after removing residual components and noise components from the target signal YB.

(8) In each of the above-described embodiments, the reference sound acquisition unit 52 acquires the reference signal R generated by the sound source device 24. However, the reference signal R generated by the sound source device 24 is stored in the storage device 12 in advance. In addition, a configuration in which the reference sound acquisition unit 52 acquires the reference signal R from the storage device 12 may be employed. It is also possible to generate the basis matrix B (each basis vector b [k]) by converting the reference signal R of each pitch generated by the sound source device 24 into the frequency domain.

(9) In each of the above-described embodiments, the acoustic signal X is separated into the singing sound acoustic signal XA and the performance sound acoustic signal XB of the target instrument, but the configuration for separating the singing sound acoustic signal XA may be omitted. For example, in the configuration for processing the acoustic signal X that does not include the singing sound, the sound source separation unit 32 is omitted, and the mixing processing unit 44 generates the acoustic signal Z by mixing the separated signal YA and the converted signal YC.

(10) The sound processing apparatus 100 can also be realized by a server device that communicates with a terminal device such as a mobile phone. For example, the acoustic processing device 100 generates an acoustic signal Z from the acoustic signal X received from the terminal device and transmits the acoustic signal Z to the terminal device.

DESCRIPTION OF SYMBOLS 100 ... Sound processing device, 10 ... Arithmetic processing device, 12 ... Memory | storage device, 14 ... Display device, 16 ... Input device, 22 ... Signal supply device, 24 ... Sound source device, 26 ... Sound emission Device 32. Sound source separation unit 34. Pitch analysis unit 36 ... Display control unit 38 ... Instruction receiving unit 40 ... Component extraction unit 42 ... Acoustic processing unit 44 ... Mixing process , 52... Reference sound acquisition unit, 54... Timbre conversion unit, 62.

Claims (3)

A first reference signal representing a first reference sound having a tone different from the target sound and having a pitch equivalent to the target sound, and a target pitch different from the pitch of the target sound, equivalent to the first reference sound Reference sound acquisition means for acquiring a second reference signal representing the second reference sound of the timbre of
Analysis processing means for generating a conversion filter for bringing the first reference sound closer to the timbre of the target sound using the target signal representing the target sound and the first reference signal;
An acoustic processing device comprising: an acoustic processing unit configured to apply the conversion filter to the second reference signal to generate a converted signal representing the sound of the target pitch with a tone color approximate to the target sound. The sound processing apparatus according to claim 1, wherein the reference sound acquisition unit adjusts the pitch of one of the target signal and the first reference signal to the other pitch. Comprising component extraction means for generating the target signal by suppressing sound other than the target sound from the acoustic signal;
The sound processing device according to claim 2, wherein the reference sound acquisition unit adjusts the first reference signal to a pitch equivalent to that of the target signal.

Images