Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

2.1. Encoder

2.2. Separator

2.3. Decoder

3.1. Residual Conformer Block

• The branch of the ResCon block expands channels for information capture and enhancement. Furthermore, this branch uses depthwise separable convolution, which combines depth-wise and point-wise convolution to drastically reduce overall processing requirements when compared to a standard convolution network.
• The ResCon block uses a GLU (Gated Linear Unit) which operates as follows:

  $$\mathbf{Y}=(\mathbf{X}\ast \mathbf{W}+\mathbf{b})\otimes \sigma (\mathbf{X}\ast \mathbf{V}+\mathbf{c}),$$

  (5)

  where $\mathbf{X}$ is the input, $\mathbf{W}$, $\mathbf{V}$, $\mathbf{b}$, and $\mathbf{c}$ are learnable affine transformation parameters, and $\sigma$ is the sigmoid function. The GLU assists in learning which features of the input to pass through and which to block, allowing it to focus on relevant features for the given task. In addition, the number of channels can be adjusted using GLU’s affine transformations.
• The Swish function is represented as $y=x\sigma \left(\beta x\right)$, where $\beta$ is a learnable parameter and $\sigma$ is the sigmoid function. Swish has been demonstrated to outperform ReLU activation for deep neural network training. Unlike ReLU, Swish does not have the problem of terminating the learning process when the input is negative.
• The ResCon block’s residual connection branch preserves all input information, guaranteeing that the main branch only grasps information that will improve or sustain performance. Furthermore, the point-wise convolution applied to the residual connection aids in alternating the number of input channels with little computational effort.

3.2. Multi-View Attention Block

• Path Separation: The input travels through three routes in the MA block, each of which has a convolution layer that changes the channel size from C to $\lfloor C/3\rfloor$. We use chunking with a $50\%$ overlap ratio for both global and local attention paths, splitting $x\in {\mathbb{R}}^{\lfloor C/3\rfloor \times L}$ into $x\in {\mathbb{R}}^{\lfloor C/3\rfloor \times P\times S}$, where P and S stand for chunk size and number of chunks, respectively. This way, long sequential features are efficiently presented due to separating the global and local information. The three perspectives of attention are described as follows.
  • Channel Attention: Given the input $x_c\in {\mathbb{R}}^{\lfloor C/3\rfloor \times L}$ for the channel attention path, the average and max pooling processes are used to aggregate the signal information for each channel $i=1,2,...,\lfloor C/3\rfloor$. The pooling outputs, $x_c^{avg}$ and $x_c^{max}$, are then passed through a common densely connected network with one hidden layer having $\lfloor C/6\rfloor$ nodes, followed by the sigmoid activation function $\sigma$ to obtain the channel attention weight ${\alpha }_c\in {\mathbb{R}}^{\lfloor C/3\rfloor \times 1}$:

    $${\alpha }_c=\sigma \left(W_1\left(W_0\left(x_c^{avg}\right)\right)+W_1\left(W_0\left(x_c^{max}\right)\right)\right),$$

    (6)

    where $W_1\in {\mathbb{R}}^{\lfloor C/3\rfloor \times \lfloor C/6\rfloor }$ and $W_0\in {\mathbb{R}}^{\lfloor C/3\rfloor \times \lfloor C/6\rfloor }$ denote the weights of the used densely connected network. The channel attention weight is then broadcasted along the time dimension and multiplied with $x_c$ to produce the channel attention output $x_c^{\prime }\in {\mathbb{R}}^{\lfloor C/3\rfloor \times L}$.
  • Global Attention: The global attention is based on the self-attention of Transformer, in which the chunk-wise representation $x_g\in {\mathbb{R}}^{\lfloor C/3\rfloor \times P\times S}$ for the global attention input is taken into account. The corresponding output $x_g^{\prime }\in {\mathbb{R}}^{\lfloor C/3\rfloor \times P\times S}$ is determined by multi-head self-attention (MHA):

    $${\alpha }_g=softmax\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right),x_g^{\prime }=W\left({\alpha }_{g}V\right),$$

    (7)

    where the linear transformation with each weight, $W_{q,k,v}\in {\mathbb{R}}^{1\times S\times S}$, and $x_g\in {\mathbb{R}}^{\lfloor C/3\rfloor \times P\times S}$, represent Q, K, and $V\in {\mathbb{R}}^{\lfloor C/3\rfloor \times P\times S}$. Ultimately, a linear layer, $W\in {\mathbb{R}}^{1\times S\times S}$, is applied to ${\alpha }_{g}V$ in order to extract the global attention output.
  • Local Attention: The local sequential features in each chunk are represented by local attention. On the chunked input $x_L\in {\mathbb{R}}^{P\times \lfloor C/3\rfloor \times S}$, a depthwise convolution layer with a kernel size of $\lfloor S/2\rfloor -1$ is applied. Following the depthwise convolution layer, we concatenate the channel-wise average and max pooling to estimate the local attention weight ${\alpha }_l\in {\mathbb{R}}^{P\times 1\times S}$ as follows:

    $${\alpha }_l=\sigma \left(F\left([x_l^{avg};x_l^{max}]\right)\right),$$

    (8)

    where F is the convolution layer that halves the channel size. At last, we express the final result of the local attention path as:

    $$x_l^{\prime }=x_l\times {\alpha }_l.$$

    (9)
• Path Aggregation: Following the three attention paths, each output is concatenated and then passed through a convolution layer. A residual gate with sigmoid activation, hyperbolic tangent activation, and ReLU is used to process the resulting output further. Finally, a residual connection is made.

5.1. Data

5.2. Data Generation and Augmentation

5.3. Separation Network Configurations

5.4. Training Objectives

5.5. Evaluation Details

5.6. Programming