CN118280384A - Multi-sound source signal separation system and method based on microphone array - Google Patents

Abstract

The present invention discloses a multi-source signal separation system and method based on a microphone array, including: a frequency-divided MUSIC sound source localization module, used to identify the number and orientation of target sound sources; an active target sound source identification module, used to identify the start and end time of active segments in audio signals, and the target sound sources therein; a frequency-divided beamforming module, for optimizing the steering vector and beamforming the target sound sources in the active segments in frequency divisions to obtain directionally enhanced audio; and a deep learning noise reduction module, for performing noise reduction processing on the directionally enhanced audio to obtain a separation result. The present invention achieves high-quality directional enhancement by optimizing the steering vector in frequency divisions and performing beamforming, and further noise reduction is performed using a neural network to improve the separation effect. It successfully combines the advantages of signal processing and deep learning in the task of multi-source separation, and effectively improves the quality, flexibility and adaptability of multi-source separation.

Description

A multi-sound source signal separation system and method based on microphone array

Technical Field

The present invention belongs to the field of intelligent perception and edge computing technology, and specifically relates to a multi-sound source signal separation system and method based on a microphone array.

Background technique

With the development of audio data processing technology, audio signals have new uses in natural language processing, human-computer interaction and other aspects. For example, the automatic transcription function from audio to text can be completed through voice transcription programs, as well as the voice control function of smart devices such as smartphones and smart speakers. In these emerging audio applications, there are often certain requirements for the quality of audio. They all require clear, high signal-to-noise ratio audio signals to ensure the robustness and effectiveness of the application. However, the actual usage scenarios are not all noise-free or low-noise single-source environments. In many cases, there are multiple sound sources in the environment. Therefore, in order to ensure the stability and effectiveness of these applications in the actual environment, it is necessary to separate multiple sound sources from the multi-source environment through some means. At present, there have been many works studying the multi-source separation task, but these methods all have certain problems.

At present, the mainstream multi-source separation algorithms include:

1. Methods based on signal processing, such as independent component analysis, non-negative matrix decomposition, beamforming, etc.; the advantage of such methods lies in flexibility. For example, beamforming can separate any number of sound sources, but the defects are also obvious. They often require the estimation of the statistical characteristics of some parameters, which leads to such methods not being ideal when dealing with the problem of multiple sound source separation in actual scenarios; and such processing methods are inevitably affected by the multipath effect, and the effect is poor in non-open environments.

2. End-to-end methods based on deep learning, such as Wave U-Net, Dual-path RNN, etc.; the advantage of this type of method is that it is effective in dealing with simple separation problems such as two low-noise sound sources, but the problem is also obvious. They often have poor flexibility and can only separate two or three fixed sound sources. It is difficult to deal with more sound sources.

Therefore, based on the above considerations, it is necessary to propose a new sound source separation system and method that can overcome the virtual sound source problem caused by the multipath effect, and still have good robustness and separation quality even in a small environment where the multipath effect is obvious; at the same time, it should be flexible enough to still have good robustness and separation quality when the number of sound sources in the scene increases.

Summary of the invention

In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a multi-source signal separation system and method based on a microphone array to solve the problem that the existing multi-source separation method is difficult to solve the multipath effect, difficult to cope with the change in the number of sound sources, and can only process a fixed number of sound sources. The method of the present invention separates the sound signal of the target sound source located in the direction specified by the user by processing the collected multi-channel audio.

To achieve the above object, the technical solution adopted by the present invention is as follows:

A multi-sound source signal separation system based on a microphone array of the present invention comprises: a frequency-divided MUSIC sound source localization module, an active target sound source identification module, a frequency-divided beamforming module and a deep learning noise reduction module;

The frequency-division MUSIC sound source localization module uses the frequency-division MUSIC algorithm to process the multi-channel audio data collected by the microphone array, obtains the azimuth of the target sound source compared to the microphone array, and realizes the localization of the target sound source;

The active target sound source identification module uses the audio collected in the reference channel to identify the start and end time of the active segment with strong sound energy, and further identifies the direction of the target sound source in the active segment;

The frequency division beamforming module uses the identified active segments to perform frequency division steering vector optimization and beamforming for each target sound source in the segment to obtain directionally enhanced audio.

The deep learning denoising module uses a deep learning algorithm to perform deep learning denoising on the directionally enhanced audio, and the denoised audio is used as the final separation result.

Furthermore, the microphone array is composed of more than one microphone arranged in a fixed geometric structure.

Furthermore, the number of target sound sources that need to be separated does not exceed the number of microphone arrays.

Furthermore, the directionally enhanced audio refers to audio in which the signal-to-noise of the sound source in the enhanced direction is higher than that in the original audio.

A method for separating multiple sound source signals based on a microphone array of the present invention is based on the above system and comprises the following steps:

1) Identify and locate the target sound source and obtain the azimuth angle of the target sound source relative to the microphone array;

2) Identify the start time and end time of the active segment in the collected audio, and further identify the direction of the target sound source in the active segment;

3) optimizing the steering vector used for beamforming each target sound source in the active segment in frequency bands and performing beamforming in frequency bands using the optimized steering vectors to obtain directionally enhanced audio;

4) Perform noise reduction processing on the directionally enhanced audio, and the processing result is used as the final separation result.

Furthermore, the step 1) specifically includes:

11) Collect an audio clip in which all target sound sources sound simultaneously;

12) Use the frequency division MUSIC algorithm to process the audio data collected in step 11) to calculate the number N _s of target sound sources in the scene and the azimuth angle A _s of each target sound source relative to the microphone array. The value of n is determined by the calculated number of target sound sources, 1≤n≤N _s .

Furthermore, the specific method of step 12) is:

121) selecting a microphone in the microphone array that is close to the center of the array geometric layout as a reference microphone;

122) Calculate the steering vector matrix Vec _steering , the size of the matrix is (FR, AR), where FR represents the frequency resolution set by the user, and AR represents the angle resolution set by the user; Vec _steering [freq, angle] represents the steering vector on the frequency band freq when the sound source is at angle angle; it is calculated by the following formula:

Among them, j is the imaginary number symbol, Avg _freq represents the average value of the frequency band freq, It indicates the distance difference between the i-th microphone and the reference microphone in the direction perpendicular to the azimuth angle of the sound source when the sound source is at the angle component angle, 1≤i≤N _m ;

123) performing a short-time Fourier transform operation on the collected audio clip to obtain a multidimensional matrix Arr _fft , which stores the energy changes over time in different frequency bands in each channel of the audio clip;

124) extracting the matrix describing the change of energy of all channels over time from the multidimensional matrix Arr _fft in frequency bands, and recording the matrix corresponding to the frequency band f as Arr _f ; calculating the eigenvalues and eigenvectors of the covariance matrix of Arr _f , and calculating the total MUSIC spectrum of f on the frequency band in combination with the steering vector matrix; selecting subsets corresponding to two mutually orthogonal groups of channels in the matrix Arr _f , respectively calculating the eigenvalues and eigenvectors of the covariance matrices of the two subsets, and calculating the MUSIC spectra corresponding to the two groups of orthogonal channels on the frequency band f in combination with the steering vector matrix; accumulating the total MUSIC spectra on all frequency bands and the MUSIC spectra corresponding to the two groups of orthogonal channels, and obtaining the final total MUSIC spectrum and the MUSIC spectrum corresponding to the two groups of orthogonal channels;

125) Count all the peaks in the total MUSIC spectrum calculated in step 124). For each peak, if there is a peak at the corresponding position of the MUSIC spectrum corresponding to the two sets of orthogonal channels, it means that the peak corresponds to a target sound source, and the position of the peak represents the azimuth of the sound source relative to the microphone array.

Furthermore, the step 2) specifically includes:

21) Read a segment of data from the microphone array and use it to identify the start and end times of the active segment;

22) After identifying the start time and end time of the active segment, further identify the number of target sound sources N _as and the corresponding azimuth set in the active segment

Furthermore, the step 21) specifically includes:

211) taking an unprocessed data frame of length l bytes from the taken audio segment and calculating the average value of the amplitude of the sampling unit in the data frame. If the average value exceeds k times the average intensity _An of the ambient noise, the data frame is an active frame, otherwise the data frame is an inactive frame; wherein l and k are parameters set by the user, and l can divide the size of the read audio segment;

212) An active segment consists of continuous active frames or inactive frames, and the first frame and the last frame are active frames; after an active frame is detected, the construction of an active segment begins; the active segment consists of the first active frame and the last active frame within the time range from the start time T _b of the first active frame identified to the end time T _b + T _s , as well as all active frames and inactive frames between the two; T _s is the value set by the user representing the maximum time of the active segment.

Furthermore, the step 22) specifically includes:

221) selecting a microphone in the microphone array that is close to the center of the array geometric layout as a reference microphone;

222) using the active segment to calculate the total MUSIC spectrum and the MUSIC spectrum corresponding to the two sets of orthogonal channels, the calculation method is the same as step 122) to step 124);

223) Count all the peaks in the total MUSIC spectrum in step 222), for each peak, if there are peaks at corresponding positions in the MUSIC spectrum corresponding to the two sets of orthogonal channels, it means that the peak corresponds to a sound source, and the position of the peak corresponds to the azimuth of the sound source relative to the microphone array; take the intersection of the azimuths of all sound sources and the target sound source _azimuth set As, and obtain the azimuth set of the target sound source that sounds in the active segment Where _Nas is the size of the intersection, indicating the number of target sound sources emitting the sound.

Furthermore, the step 3) specifically includes:

31) optimizing the steering vector used in beamforming for each target sound source identified in step 2) in frequency bands to weaken the enhancement of other sound sources by beamforming;

32) Using the steering vector obtained in step 31), the spectrum diagram of each frequency band is beamformed, and the beamforming result is regularized to achieve smoothness; the beamforming result is adjusted and combined according to the weight, and then the calculated combination result is inverse Fourier transformed to obtain directionally enhanced audio.

Furthermore, the specific method of step 31) is:

311) calculating the steering vector matrix Vec _steering in the same way as step 122);

312) When calculating beamforming for a certain sound target sound source, the remaining sound sources are regarded as interference sound sources. If there is no interference sound source, the steering vector corresponding to the azimuth angle of the target sound source is directly used for beamforming; otherwise, the steering vector of beamforming is adjusted in frequency bands;

When beamforming is performed on the azimuth angle θ in the frequency band f, the enhancement coefficient of the interfering sound source at the azimuth angle θ _{infer is} The calculation method is as follows:

When there are multiple interference sound sources, the enhancement coefficient of each interference sound source is calculated and summed up as the total interference coefficient, recorded as

313) For each target sound source θ _target , solve the optimization problem under certain constraints The angle adjustment difference Δθ on the frequency band f is obtained. Then, the steering vector is indexed in Vec _steering through the frequency band f and the angle θ _target + Δθ as the steering vector for beamforming the target sound source θ _target on the frequency band f.

Furthermore, the step 32) specifically includes:

321) Calculate the spectrogram group Spec _raw of the active segment, Spec _raw consists of the spectrograms of each channel,

322) beamforming the spectrum graph corresponding to each frequency band; specifically: extracting the corresponding part of the frequency band f from the spectrum graph group Using the steering vector obtained in step 313) as the weight, Perform weighted summation to obtain the beamforming result of the sound source at the azimuth angle θ _target in the frequency band f; then divide the result by the enhancement coefficient in the target direction Normalize and get the normalized beamforming result

323) The normalized beamforming results of all frequency bands are spliced and combined according to the weights to obtain the final beamforming result Spec _bf (θ _target ); the weight corresponding to each frequency band depends on the frequency band f itself and the target angle θ _target ; specifically: using the data of all channels, the MUSIC spectrum corresponding to the frequency band f is calculated, and the calculation process is the same as step 12); then the spectrum value corresponding to the target angle θ _target on the MUSIC spectrum corresponding to the frequency band f is taken as the weight;

324) Perform inverse Fourier transform on Spec _bf (θ _{tar get} ) to obtain audio with directionality enhancement in the direction of θ _{tar get} .

Furthermore, the step 4) specifically includes:

41) taking the beamforming result of the active sound source segment and the original unseparated audio data collected by any channel, performing Fourier transform and modulo operation, and splicing the result of the modulo operation into a dual-channel image;

42) Using the deep learning model to calculate the dual-channel image in step 41), the amplitude part Amp of the denoised audio frequency graph is obtained;

43) Using the beamforming result obtained in step 32), phase filling is performed on the amplitude part of the time-frequency diagram obtained in step 42) to obtain the final separation result.

Furthermore, the deep model used in the step 42) is Dual-Path GAN, which consists of a generator and a discriminator. In the generator, the dual-channel image is first encoded by an encoder composed of dense convolution modules to obtain an intermediate result; then two groups of four consecutive conformer modules are used to process the intermediate result and the transposition of the intermediate result respectively, and the processing results will be spliced together as the input of the decoder; the decoder uses dense convolution to decode its input, and uses a convolution module to reduce the dimension of the output to obtain the amplitude part Amp of the audio time-frequency diagram after denoising; the discriminator is used in the training process of the model, and the discriminator splices the amplitude map Amp generated by the generator and the amplitude map Amp _bf of the beamforming result into a dual-channel image as input, and generates a judgment result through a continuous convolution module to optimize the model;

The dense convolution module is a module composed of a Dense Net and a convolution module. Like the convolution module, the dense convolution module performs a convolution operation on the input, concatenates the convolution result with the input of all the dense convolution modules in front of it, and forms the output of the dense convolution module.

Furthermore, the specific method of step 43) is:

431) Performing Fourier transform on the beamforming result to obtain a spectrum diagram; then taking the phase information of the spectrum diagram to obtain a phase diagram Phase, the size of the phase diagram is the same as the size of the spectrum diagram;

432) Multiply the elements at corresponding positions of Amp and Phase to obtain a time-frequency diagram of the denoised audio;

433) Perform inverse Fourier transform on the time-frequency diagram of the denoised audio to obtain a denoised result.

Beneficial effects of the present invention:

1. The present invention realizes multi-sound source separation based on beamforming, has no special requirements on the geometric layout of the microphone array, and has low restrictions on the number of separated sound sources. Any microphone array can be used to separate an indefinite number of sound sources, thereby improving the flexibility of multi-sound source separation.

2. The present invention improves the effect of beamforming directional separation through frequency division beamforming, and uses Dual-PathGAN to enhance the result of directional separation, effectively improving the quality of multi-source separation results, and improving the adaptability of multi-source separation, and can maintain separation quality in complex multipath environments;

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a structural block diagram of the system of the present invention.

FIG. 2 is a flow chart of the method of the present invention.

FIG3 is a structural diagram of the deep model of the present invention, which is Dual-Path GAN.

Detailed ways

In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

1 , a multi-sound source signal separation system based on a microphone array of the present invention includes: a frequency-divided MUSIC sound source localization module, an active target sound source identification module, a frequency-divided beamforming module, and a deep learning noise reduction module;

The frequency-division MUSIC sound source localization module uses the frequency-division MUSIC algorithm to process the multi-channel audio data collected by the microphone array, obtains the azimuth of the target sound source compared to the microphone array, and realizes the localization of the target sound source;

The active target sound source identification module uses the audio collected in the reference channel to identify the start and end time of the active segment with strong sound energy, and further identifies the direction of the target sound source in the active segment;

The frequency division beamforming module uses the identified active segments to perform frequency division steering vector optimization and beamforming for each target sound source in the segment to obtain directionally enhanced audio.

The deep learning denoising module uses a deep learning algorithm to perform deep learning denoising on the directionally enhanced audio, and the denoised audio is used as the final separation result.

The microphone array is composed of more than one microphone arranged in a fixed geometric structure.

The number of target sound sources that need to be separated does not exceed the number of microphone arrays.

The directionally enhanced audio refers to audio in which the signal-to-noise ratio of the sound source in the enhanced direction is higher than that in the original audio.

2, a method for separating multiple sound source signals based on a microphone array according to the present invention is based on the above system, and the steps are as follows:

1) Identify and locate the target sound source and obtain the azimuth angle of the target sound source relative to the microphone array; specifically including:

11) Collect an audio clip in which all target sound sources sound simultaneously;

12) Use the frequency division MUSIC algorithm to process the audio data collected in step 11) to calculate the number N _s of target sound sources in the scene and the azimuth angle A _s of each target sound source relative to the microphone array. The value of n is determined by the calculated number of target sound sources, 1≤n≤N _s .

Wherein, the specific method of step 12) is:

121) selecting a microphone in the microphone array that is close to the center of the array geometric layout as a reference microphone;

122) Calculate the steering vector matrix Vec _steering , the size of the matrix is (FR, AR), where FR represents the frequency resolution set by the user, and AR represents the angle resolution set by the user; Vec _steering [freq, angle] represents the steering vector on the frequency band freq when the sound source is at angle angle; it is calculated by the following formula:

Among them, j is the imaginary number symbol, Avg _freq represents the average value of the frequency band freq, It indicates the distance difference between the i-th microphone and the reference microphone in the direction perpendicular to the azimuth angle of the sound source when the sound source is at the angle component angle, 1≤i≤N _m ;

123) performing a short-time Fourier transform operation on the collected audio clip to obtain a multidimensional matrix Arr _fft , which stores the energy changes over time in different frequency bands in each channel of the audio clip;

124) extracting the matrix describing the change of energy of all channels over time from the multidimensional matrix Arr _fft in frequency bands, and recording the matrix corresponding to the frequency band f as Arr _f ; calculating the eigenvalues and eigenvectors of the covariance matrix of Arr _f , and calculating the total MUSIC spectrum of f on the frequency band in combination with the steering vector matrix; selecting subsets corresponding to two mutually orthogonal groups of channels in the matrix Arr _f , respectively calculating the eigenvalues and eigenvectors of the covariance matrices of the two subsets, and calculating the MUSIC spectra corresponding to the two groups of orthogonal channels on the frequency band f in combination with the steering vector matrix; accumulating the total MUSIC spectra on all frequency bands and the MUSIC spectra corresponding to the two groups of orthogonal channels, and obtaining the final total MUSIC spectrum and the MUSIC spectrum corresponding to the two groups of orthogonal channels;

125) Count all the peaks in the total MUSIC spectrum calculated in step 124). For each peak, if there is a peak at the corresponding position of the MUSIC spectrum corresponding to the two sets of orthogonal channels, it means that the peak corresponds to a target sound source, and the position of the peak represents the azimuth of the sound source relative to the microphone array.

2) Identify the start time and end time of the active segment in the collected audio, and further identify the direction of the target sound source in the active segment; specifically including:

21) Read a segment of data from the microphone array and use it to identify the start and end times of the active segment;

22) After identifying the start time and end time of the active segment, further identify the number of target sound sources N _as and the corresponding azimuth set in the active segment

Wherein, the step 21) specifically includes:

211) taking an unprocessed data frame of length l bytes from the taken audio segment and calculating the average value of the amplitude of the sampling unit in the data frame. If the average value exceeds k times the average intensity _An of the ambient noise, the data frame is an active frame, otherwise the data frame is an inactive frame; wherein l and k are parameters set by the user, and l can divide the size of the read audio segment;

212) An active segment consists of continuous active frames or inactive frames, and the first frame and the last frame are active frames; after an active frame is detected, the construction of an active segment begins; the active segment consists of the first active frame and the last active frame within the time range from the start time T _b of the first active frame identified to the end time T _b + T _s , as well as all active frames and inactive frames between the two; T _s is the value set by the user representing the maximum time of the active segment.

Wherein, the step 22) specifically includes:

221) selecting a microphone in the microphone array that is close to the center of the array geometric layout as a reference microphone;

222) using the active segment to calculate the total MUSIC spectrum and the MUSIC spectrum corresponding to the two sets of orthogonal channels, the calculation method is the same as step 122) to step 124);

223) Count all the peaks in the total MUSIC spectrum in step 222), for each peak, if there are peaks at the corresponding positions in the MUSIC spectrum corresponding to the two sets of orthogonal channels, it means that the peak corresponds to a sound source, and the position of the peak corresponds to the azimuth of the sound source relative to the microphone array; take the intersection of the azimuths of all sound sources and the target sound source azimuth set _As , and obtain the azimuth set of the target sound source that sounds in the active segment Where _Nas is the size of the intersection, indicating the number of target sound sources emitting the sound.

3) optimizing the steering vector used for beamforming each target sound source in the active segment in frequency bands and performing beamforming in frequency bands using the optimized steering vectors to obtain directionally enhanced audio; specifically including:

31) optimizing the steering vector used in beamforming for each target sound source identified in step 2) in frequency bands to weaken the enhancement of other sound sources by beamforming;

32) Using the steering vector obtained in step 31), the spectrum diagram of each frequency band is beamformed, and the beamforming result is regularized to achieve smoothness; the beamforming result is adjusted and combined according to the weight, and then the calculated combination result is inverse Fourier transformed to obtain directionally enhanced audio.

Wherein, the specific method of step 31) is:

311) calculating the steering vector matrix Vec _steering in the same way as step 122);

312) When calculating beamforming for a certain sound target sound source, the remaining sound sources are regarded as interference sound sources. If there is no interference sound source, the steering vector corresponding to the azimuth angle of the target sound source is directly used for beamforming; otherwise, the steering vector of beamforming is adjusted in frequency bands (in order to avoid beamforming enhancing the interference sound source);

When beamforming is performed on the azimuth angle θ in the frequency band f, the enhancement coefficient of the interfering sound source at the azimuth angle θ _{infer is} The calculation method is as follows:

When there are multiple interference sound sources, the enhancement coefficient of each interference sound source is calculated and summed up as the total interference coefficient, recorded as

313) For each target sound source θ _target , under certain constraints (such as ) to solve the optimization problem The angle adjustment difference Δθ on the frequency band f is obtained. Then, the steering vector is indexed in Vec _steering through the frequency band f and the angle θ _target + Δθ as the steering vector for beamforming the target sound source θ _target on the frequency band f.

Wherein, the step 32) specifically includes:

321) Calculate the spectrogram group Spec _raw of the active segment, Spec _raw is composed of the spectrograms of each channel,

322) beamforming the spectrum graph corresponding to each frequency band; specifically: extracting the corresponding part of the frequency band f from the spectrum graph group Using the steering vector obtained in step 313) as the weight, Perform weighted summation to obtain the beamforming result of the sound source at the azimuth angle θ _target in the frequency band f; then divide the result by the enhancement coefficient in the target direction Normalize and get the normalized beamforming result

323) The normalized beamforming results of all frequency bands are spliced and combined according to the weights to obtain the final beamforming result Spec _bf (θ _target ); the weight corresponding to each frequency band depends on the frequency band f itself and the target angle θ _target ; specifically: using the data of all channels, the MUSIC spectrum corresponding to the frequency band f is calculated, and the calculation process is the same as step 12); then the spectrum value corresponding to the target angle θ _target on the MUSIC spectrum corresponding to the frequency band f is taken as the weight;

324) performs an inverse Fourier transform on Spec _bf (θ _target ) to obtain audio with directionality enhancement in the θ _target direction.

4) Performing noise reduction processing on the directional enhanced audio, and the processing result is used as the final separation result; specifically including:

41) taking the beamforming result of the active sound source segment and the original unseparated audio data collected by any channel, performing Fourier transform and modulo operation, and splicing the result of the modulo operation into a dual-channel image;

42) Using the deep learning model to calculate the dual-channel image in step 41), the amplitude part Amp of the denoised audio frequency graph is obtained;

43) Using the beamforming result obtained in step 32), phase filling is performed on the amplitude part of the time-frequency diagram obtained in step 42) to obtain the final separation result.

As shown in FIG3 , the deep model used in step 42) is Dual-Path GAN, which consists of a generator and a discriminator. In the generator, the dual-channel image is first encoded by an encoder composed of dense convolution modules to obtain an intermediate result; then two groups of four consecutive conformer modules are used to process the intermediate result and the transposition of the intermediate result respectively, and the processing results will be spliced together as the input of the decoder; the decoder uses dense convolution to decode its input, and uses a convolution module to reduce the dimension of the output to obtain the amplitude part Amp of the audio time-frequency diagram after denoising; the discriminator is used in the training process of the model, and the discriminator splices the amplitude map Amp generated by the generator and the amplitude map Amp _bf of the beamforming result into a dual-channel image as input, and generates a judgment result through continuous convolution modules to optimize the model;

The dense convolution module is a module composed of a Dense Net and a convolution module. Like the convolution module, the dense convolution module performs a convolution operation on the input, concatenates the convolution result with the input of all the dense convolution modules in front of it, and forms the output of the dense convolution module.

Wherein, the specific method of step 43) is:

431) Performing Fourier transform on the beamforming result to obtain a spectrum diagram; then taking the phase information of the spectrum diagram to obtain a phase diagram Phase, the size of the phase diagram is the same as the size of the spectrum diagram;

432) multiplying the elements at corresponding positions of Amp and Phase to obtain a time-frequency diagram of the denoised audio;

433) Perform inverse Fourier transform on the time-frequency diagram of the denoised audio to obtain a denoised result.

The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims (10)

1. A microphone array-based multi-sound source signal separation system, comprising: the system comprises a frequency division MUSIC sound source positioning module, an active target sound source identification module, a frequency division beam forming module and a deep learning noise reduction module;

The frequency division MUSIC sound source positioning module processes multichannel audio data acquired by the microphone array by utilizing a frequency division MUSIC algorithm, acquires azimuth angles of the target sound source compared with the microphone array, and realizes positioning of the target sound source;

the active target sound source identification module is used for identifying the starting time and the ending time of an active fragment with strong sound energy by utilizing the audio acquired in the reference sound channel, and further identifying the azimuth of a target sound source sounding in the active fragment;

The frequency division beam forming module is used for carrying out frequency division guiding vector optimization and beam forming on each target sound source sounding in the frequency division beam forming module by utilizing the identified active fragments so as to obtain directional enhanced audio;

and the deep learning noise reduction module is used for carrying out deep learning noise reduction on the obtained directional enhanced audio by using a deep learning algorithm, and the noise reduced audio is used as a final separation result.

2. The microphone array-based multi-source signal separation system of claim 1 wherein the directionally-enhanced audio refers to audio in which the signal-to-noise of the source sound in the enhanced direction is higher than in the original audio.

3. A method of separating multiple sound source signals based on a microphone array, based on the system of any of claims 1-2, characterized by the steps of:

1) Identifying and positioning a target sound source, and acquiring the azimuth angle of the target sound source compared with a microphone array;

2) Identifying the starting time and the ending time of an active segment in the acquired audio, and further identifying the azimuth of a target sound source sounding in the active segment;

3) Optimizing a steering vector used when carrying out beam forming on each sounding target sound source in the active segment in a frequency division manner, and carrying out beam forming in the frequency division manner by utilizing the optimized steering vector to obtain directional enhanced audio;

4) And carrying out noise reduction treatment on the audio with enhanced orientation, wherein the treatment result is used as a final separation result.

4. The method for separating multiple sound sources based on microphone array according to claim 3, wherein the step 1) specifically comprises:

11 Collecting a section of audio frequency fragments which are simultaneously sounded by all target sound sources;

12 Processing the audio data acquired in step 11) using a frequency division MUSIC algorithm, calculating the number N _s of target sound sources in the scene and the azimuth angle a _s of each target sound source compared to the microphone array, Wherein the value of N is determined by the calculated number of target sound sources, and N is more than or equal to 1 and less than or equal to N _s.

5. The microphone array-based multi-sound source signal separation method according to claim 4, wherein the specific method of step 12) is as follows:

121 Selecting a microphone in the microphone array, which is close to the geometric layout center of the array, as a reference microphone;

122A vector matrix Vec _steering, the matrix being of size (FR, AR), where FR represents the frequency resolution set by the user and AR represents the angular resolution set by the user; vec _steering [ freq, angle ] represents the steering vector over the frequency band freq when the sound source is at an angle; calculated by the following formula:

Where j is an imaginary symbol, avg _freq represents the average value of the frequency band freq, Indicating that when the sound source is at an angle component angle, the i-th microphone has a distance difference of 1.ltoreq.i.ltoreq.N _m in a direction perpendicular to the prescribed azimuth angle of the sound source as compared with the reference microphone;

123 Performing a short-time fourier transform operation on the acquired audio segment to obtain a multi-dimensional matrix Arr _fft in which the time-dependent changes of energy in different frequency bands in each channel in the audio segment are stored;

124 Extracting a matrix describing the change of energy of all channels along time in a frequency band manner from a multidimensional matrix Arr _fft, and marking a matrix corresponding to a frequency band f as Arr _f; calculating eigenvalues and eigenvectors of covariance matrix of Arr _f, and calculating total MUSIC spectrum of f on the frequency band by combining with the vector matrix; selecting subsets corresponding to two groups of mutually orthogonal channels in a matrix Arr _f, respectively calculating eigenvalues and eigenvectors of covariance matrixes of the two subsets, and calculating MUSIC spectrums corresponding to the two groups of orthogonal channels on a frequency band f by combining a guide vector matrix; accumulating the total MUSIC spectrum on all frequency bands and the MUSIC spectrums corresponding to the two groups of orthogonal channels respectively to obtain the total MUSIC spectrum which is finally used and the MUSIC spectrums corresponding to the two groups of orthogonal channels;

125 Statistics of all peaks in the total MUSIC spectrum calculated in step 124), for each peak, if there is a peak at the corresponding position of the MUSIC spectrum corresponding to the two sets of orthogonal channels at the same time, it indicates that the peak corresponds to a target sound source, and the position of the peak represents the azimuth angle of the sound source compared to the microphone array.

6. The method for separating multiple sound sources based on microphone array according to claim 3, wherein the step 2) specifically comprises:

21 Reading a piece of data from the microphone array and using it to identify a start time and an end time of the active segment;

22 After identifying the start time and the end time of the active segment, further identifying the number N _as of target sound sources sounding in the active segment and the set of azimuth angles corresponding to the number N _as

7. The microphone array-based multi-sound source signal separation method according to claim 6, wherein the step 21) specifically includes:

211 Taking an unprocessed data frame with the length of l bytes from the taken audio fragment and calculating the average value of the amplitude of a sampling unit in the data frame, wherein if the average value exceeds more than k times of the average intensity A _n of environmental noise, the data frame is an active frame, otherwise, the data frame is an inactive frame; wherein l and k are parameters set by a user, and l can divide the size of the read audio fragment;

212 The active segment is composed of continuous active frames or inactive frames, and the first frame and the last frame are active frames; after an active frame is detected, starting to construct an active segment; the active segment is composed of a first active frame and a last active frame and all active frames and inactive frames therebetween in a time range from a start time T _b of the identified first active frame to an end time T _b+T_s; t _s is a value set by the user that represents the longest time of the active segment.

8. The method for separating multiple sound sources based on microphone array according to claim 3, wherein the step 3) specifically comprises:

31 Optimizing steering vectors used in beamforming each of the sound-emitting target sound sources identified in step 2) in a frequency-division manner to attenuate enhancement of other sound-emitting sound sources by beamforming;

32 Carrying out beam forming on the spectrogram of each frequency band by utilizing the guide vector obtained in the step 31), and regularizing the beam forming result to realize smoothing; and adjusting and combining the beam forming result according to the weight, and then performing inverse Fourier transform on the calculated combined result to obtain the directional enhanced audio.

9. The microphone array-based multi-sound source signal separation method according to claim 8, wherein the step 4) specifically includes:

41 Taking the beam forming result of the active sound source segment, carrying out Fourier transform on the original unseparated audio data acquired by any sound channel, carrying out mode taking operation, and splicing the result of the mode taking operation into a double-channel image;

42 Calculating the two-channel image in the step 41) by using a deep learning model to obtain an amplitude part Amp of the noise-reduced audio time-frequency diagram;

43 Using the beam forming result obtained in step 32), phase filling is carried out on the time-frequency diagram amplitude value part obtained in step 42), and a final separation result is obtained.

10. The microphone array-based multi-sound source signal separation method according to claim 9, wherein the depth model used in the step 42) is Dual-Path GAN, and the Dual-Path GAN is composed of a generator and a discriminator, and in the generator, the Dual-channel image is first encoded by Encoder composed of a close convolution module to obtain an intermediate result; then using two groups of four conformer modules to process the intermediate result and the transposition of the intermediate result respectively, and splicing the processed results together to be used as the input of a Decoder; the Decoder decodes the input by utilizing the dense convolution, and uses a convolution module to reduce the dimension of the output to obtain the amplitude part Amp of the noise-reduced audio time-frequency diagram; the discriminator is used for the training process of the model, the discriminator is used for splicing the amplitude diagram Amp generated by the generator and the amplitude diagram Amp _bf of the beam forming result into a double-channel picture as input, and the continuous convolution module is used for generating a judging result so as to optimize the model;

The Dense convolution module is formed by combining a Dense Net module and a convolution module, and as the convolution module, the Dense convolution module carries out convolution operation on the input, and the convolution result is spliced with the input of all the Dense convolution modules connected in series in front of the Dense convolution module to form the output of the Dense convolution module.