CN111128191B - Online end-to-end voice transcription method and system - Google Patents

Abstract

The invention provides a method and a system for online end-to-end voice transcription, wherein in one embodiment, acoustic features are extracted from an audio file; carrying out nonlinear transformation and down-sampling on the acoustic features and outputting a first feature sequence; partitioning the first characteristic sequence, sequentially inputting each block of characteristic sequence into an encoder and outputting a plurality of groups of second characteristic sequences; modeling the second characteristic sequence, outputting a plurality of groups of Chinese character sequences and scoring the plurality of groups of Chinese character sequences; and taking the Chinese character sequence with the highest score as a final transcription result. The encoder structure is improved to process the partitioned audio; by improving the structure of the decoder, the Chinese characters are output on the basis of audio truncation. So that text is transcribed while audio is being input.

Description

An online end-to-end speech transcription method and system

technical field

The invention relates to the technical field of speech transcription, and in particular to an online end-to-end speech transcription method and system.

Background technique

Speech transcription technology is an important technology for converting input audio into text, and it is also an important research content in the field of human-computer interaction.

Traditional speech transcription technology includes acoustic models, pronunciation dictionaries, and language models, and uses weighted finite state transition machines to build complex decoding networks to convert acoustic feature sequences into text sequences. The current emerging end-to-end language transcription technology uses a single neural network model to directly convert acoustic features into text sequences, which greatly simplifies the decoding process in the speech transcription process. However, the current high-performance end-to-end speech transcription must wait for the complete audio input before it can start converting into text sequences, which limits the application of end-to-end speech transcription technology to online tasks of real-time transcription.

Contents of the invention

In view of this, the embodiment of the present application provides an online end-to-end voice transcription method and system, which overcomes the problem that the existing end-to-end voice transcription technology cannot be applied to real-time transcription online tasks. The end-to-end speech transcription technology of the encoder and decoder structure enables the encoder and decoder to start converting into text sequences without relying on the complete audio.

In the first aspect, the application of the present invention provides an online end-to-end voice transcription method including:

Acquiring an audio file, and extracting acoustic features from the audio file;

performing nonlinear transformation and downsampling on the acoustic features and outputting a first feature sequence;

dividing the first feature sequence into blocks, sequentially inputting each block of feature sequences into the encoder and outputting multiple sets of second feature sequences;

Modeling the second feature sequence, outputting multiple sets of Chinese character sequences and scoring the multiple sets of Chinese character sequences;

The Chinese character sequence with the highest score is taken as the final transliteration result.

Optionally, the acquiring the audio file, and extracting the acoustic features from the audio file include:

The logarithmic mel spectrum features are extracted from the acquired audio files as frame-level acoustic features.

Optionally, the encoder is an online encoder based on a self-attention mechanism;

The encoder is composed of a stack of 12 identical sub-modules, and each sub-module is sequentially composed of a self-attention network, a residual network, a layer normalization network, a fully connected network, a residual network, and a layer normalization network stack.

Optionally, the processing the second feature sequence, outputting multiple groups of Chinese character sequences and scoring the multiple groups of Chinese character sequences includes:

Build an online decoder based on the self-attention mechanism, the decoder models the second feature sequence, and scores the output multiple groups of Chinese character sequences;

The decoder is composed of 6 identical sub-module stacks, wherein each sub-module is a layer of self-attention network, a layer of residual network, a layer of normalized network, a layer of truncated attention network, a layer of residual network , a layer-by-layer normalization network, a layer-by-layer fully connected network, a layer-by-layer residual network, and a layer-by-layer normalization network.

Optionally, the decoder models the second feature sequence, and scoring the output multiple groups of Chinese character sequences includes:

Pass multiple groups of second feature sequences through 6 submodules of the decoder in turn, and input the output features of the layer specification network of the last submodule into the Chinese character classifier;

The Chinese character classifier outputs multiple groups of Chinese characters and the corresponding scores of each group of Chinese characters;

Take the top ten Chinese characters and input them into the decoder to output the next Chinese character until the decoder outputs the terminator.

In the second aspect, the application of the present invention provides an online end-to-end speech transcription system including:

Acquisition unit: used to collect audio and extract acoustic features from the audio;

Processing unit: used to perform nonlinear transformation and down-sampling on the acoustic features extracted by the acquisition unit and output the first feature sequence; divide the first feature sequence into blocks, input each block of feature sequences into the encoder in turn and output multiple groups the second feature sequence;

The processing unit is also used to model the second feature sequence, output multiple sets of Chinese character sequences and score the multiple sets of Chinese character sequences;

Output unit: used to take the Chinese character sequence with the highest score among the Chinese character sequences output by the processing unit as the final transcription result, and output it.

Optionally, extracting acoustic features from the audio includes:

The logarithmic mel spectrum features are extracted from the acquired audio files as frame-level acoustic features.

Optionally, the encoder is an online encoder based on a self-attention mechanism;

The encoder is composed of a stack of 12 identical sub-modules, and each sub-module is sequentially composed of a self-attention network, a residual network, a layer normalization network, a fully connected network, a residual network, and a layer normalization network stack.

Optionally, the processing the second feature sequence, outputting multiple groups of Chinese character sequences and scoring the multiple groups of Chinese character sequences includes:

Build an online decoder based on the self-attention mechanism, the decoder models the second feature sequence, and scores the output multiple groups of Chinese character sequences;

The decoder is composed of 6 identical sub-module stacks, wherein each sub-module is a layer of self-attention network, a layer of residual network, a layer of normalized network, a layer of truncated attention network, a layer of residual network , a layer-by-layer normalization network, a layer-by-layer fully connected network, a layer-by-layer residual network, and a layer-by-layer normalization network.

Optionally, the decoder models the second feature sequence, and scoring the output multiple groups of Chinese character sequences includes:

Pass multiple groups of second feature sequences through 6 submodules of the decoder in turn, and input the output features of the layer norm network of the last submodule into the Chinese character classifier;

The Chinese character classifier outputs multiple groups of Chinese characters and the corresponding scores of each group of Chinese characters;

Take the top ten Chinese characters and input them into the decoder to output the next Chinese character until the decoder outputs the terminator.

An embodiment of the present application provides an online end-to-end speech transcription method and system. In one embodiment, logarithmic mel spectrum features are extracted from the audio as frame-level acoustic features; a front-end neural network is constructed to convert the logarithmic mel spectrum Non-linear transformation and downsampling of features; construction of an online encoder based on self-attention mechanism, modeling the output feature sequence of the front-end neural network, and output a new set of feature sequences; construction of online decoding based on self-attention mechanism The device models the feature sequence output by the encoder, and outputs multiple sets of Chinese character sequences; uses the beam search algorithm to search for the Chinese character sequence with the highest score, and uses it as the final transliteration result. By improving the structure of the encoder, let it process the divided audio; by improving the structure of the decoder, let it output Chinese characters on the basis of truncated audio. Enables transcription of text while inputting audio.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 is a schematic structural diagram of an online end-to-end speech transcription system for the application of the present invention;

Fig. 2 is a flow chart of applying for an online end-to-end voice transcription method according to the present invention;

Fig. 3 is a flow chart of the processing of the feature sequence input by the online encoder based on the self-attention mechanism;

Fig. 4 is a flow chart of the processing of the feature sequence input by the online decoder based on the self-attention mechanism.

Detailed ways

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Fig. 1 is a schematic structural diagram of an online end-to-end speech transcription method of the present invention. Referring to Fig. 1, an end-to-end far-field speech recognition system in the embodiment of the present application includes: an acquisition unit 101, a processing unit 102 and output unit 103.

The collection unit 101 is used to collect audio signals, and pass the collected audio signals through a high-pass filter for pre-emphasis to increase the high frequency part of the audio signals.

Framing the audio signal passed through the high-pass filter, each frame is 25 milliseconds, and the frame is shifted by 10 milliseconds. Windowing is performed on each frame, and the window function is a Hamming window. Then perform fast Fourier transform on each frame to obtain the frequency spectrum of each frame, and further obtain the energy spectrum of each frame. Further, calculate the energy passing through the Mel filter for the energy spectrum of each frame, and take the logarithm to obtain the logarithmic Mel spectrum, wherein the number of Mel filters is 80, so each frame gets an 80-dimensional pair Several mel spectral features.

The processing unit 102 includes: a first processing unit 1021 , a second processing unit 1022 and a third processing unit 1023 .

The first processing unit 1021 is used to construct a front-end neural network. Among them, the front-end neural network includes two layers of two-dimensional convolutional network, one layer of linear network and one layer of position encoding network. Among them, the convolution kernel size of the convolution network is 3, the step size is 2, and the number of convolution kernels is 256. After two layers of two-dimensional convolution layers, the length of the feature sequence becomes a quarter of the original length, and the linear layer The output features of the convolutional layer are projected to 256 dimensions, and the position encoding layer adds the output features of the linear layer and the 256-dimensional position features.

The logarithmic mel spectrum feature sequence extracted by the acquisition unit 101 is input into the front-end neural network for nonlinear transformation and down-sampling, and the sampling rate is one-fourth.

The second processing unit 1022 is used to construct an online encoder based on the self-attention mechanism. Concatenation network, residual network and layer normalization network stack composition.

Divide the feature sequences output from the front-end neural network into blocks, and then input each block of feature sequences into the online encoder in turn, and output multiple sets of new feature sequences. Among them, the feature sequence output from the online encoder is the same as the feature sequence input to the online encoder.

The third processing unit 1023 is used to build an online decoder based on the self-attention mechanism. The decoder is composed of 6 identical sub-modules stacked. Attention network, residual network, layer normalization network, fully connected network, residual network and layer normalization network stack.

Before inputting the output features of the online encoder to the online decoder, it is also necessary to input a starting symbol for the online decoder as the starting point of the online decoder. Add the word embedding feature of the start symbol and the position feature, and feed it into an online decoder based on the self-attention mechanism.

The output unit 103 is configured to take the Chinese character sequence with the highest score among the Chinese character sequences output by the processing unit 102 as the final transcription result, and output it.

In a possible embodiment, the output unit 103 uses a beam search algorithm to control the number of Chinese characters output by the third processing unit 1023 each time, sort them according to the scores from high to low, and input the top ten Chinese characters into the decoder for output. The next Chinese character until the decoder outputs a terminator. At the end, the Chinese character sequence with the highest score will be the final transliteration result.

Fig. 2 is a flowchart of an online end-to-end speech transcription method for the present invention. Referring to Fig. 2, an online end-to-end speech transcription method includes steps S201-step S205:

Step S201: Acquire audio files, and extract acoustic features from the audio files.

Extract log-mel spectral features from audio as frame-level acoustic features. Specifically, it includes: pre-emphasizing the acquired audio files, and enhancing high-frequency parts. That is, the speech signal in the audio file is passed through a high-pass filter:

H(z)＝1-0.97z ^-1 (1)

Framing and windowing are performed on the audio in the audio file, where each frame is 25 milliseconds, the frame shift is 10 milliseconds, and the window function is a Hamming window.

Perform fast Fourier transform on each frame to obtain the frequency spectrum of each frame, and further process the frequency spectrum of each frame to obtain the energy spectrum of each frame.

Calculate the energy spectrum of each frame through the energy of the Mel filter, and take the logarithm to obtain the logarithmic Mel spectrum, where the number of Mel filters is 80, so each frame obtains an 80-dimensional logarithmic Mel spectrum feature.

Step S202: Perform nonlinear transformation and down-sampling on the acoustic features and output a first feature sequence.

The extracted acoustic features are input into the front-end neural network, and the acoustic features are nonlinearly transformed and down-sampled by the front-end neural network to output the first feature sequence; wherein, the sampling rate is 1/4.

In a possible embodiment, the constructed front-end neural network includes two layers of two-dimensional convolutional network, one layer of linear network and one layer of position encoding network, wherein the convolutional network has a convolution kernel size of 3 and a step size of 2, The number of convolution kernels is 256. After two layers of two-dimensional convolutional layers, the length of the feature sequence becomes a quarter of the original length. The linear layer projects the output features of the convolutional layer to 256 dimensions, and the position encoding layer converts the linear layer The output features of and the 256-dimensional position features are added, and the overall calculation process is as follows:

Y=ReLU(Conv(ReLU(Conv(X))) (2)

z _i =Linear(y _i )+p _i (3)

Among them, Conv(·) represents the convolutional layer; ReLU(·) represents the activation function, and its expression is:

ReLU(x)=max(0,x) (4)

Linear(·) represents the linear layer, X, Y represent the logarithmic mel spectrum feature sequence and the output feature sequence of the two-layer two-dimensional convolutional layer, respectively, y _i , p _i , z _i represent the i-th two-dimensional two-dimensional The output features of the convolutional layer, the i-th position feature, and the output feature of the i-th front-end neural network, where the calculation formula for each dimension of the position feature is:

p _i,2k+1 = sin(i/10000 ^k/128 ) (5)

p _i,2k+2 = cos(i/10000 ^k/128 ) (6)

Step S203: Divide the first feature sequence into blocks, sequentially input each block of feature sequences into the encoder and output multiple sets of second feature sequences.

In a possible embodiment, each block feature sequence is input to an online encoder based on the self-attention mechanism, which is composed of 12 identical sub-modules stacked, and each sub-module is a layer of self-attention network, One layer of residual network, one layer of normalized network, one layer of fully connected network, one layer of residual network and one layer of normalized network, each layer of network output feature is 256 dimensions.

The first feature sequence is divided into blocks, and the length of each feature sequence is 64. Input each block of feature sequences into the encoder for processing, the processing flow chart is shown in Figure 3, including steps S2031-S2038:

Step S2031: Input each feature sequence into the self-attention network of the first module of the encoder, and the calculation is as follows:

SAN(Q,K,V)＝[head ₁ ,...,head ₄ ]W ^O (7)

Among them, W _i ^Q , W _i ^K , W _i ^V , and W ^O represent the parameter matrix, Q represents one of the feature sequences, and 64 future input features are spliced on the right side, and K represents one of the feature sequences, and 64 historical input features are spliced on the left side. Input features, 64 future input features are concatenated on the right, V and K are the same.

Step S2032: Add the input and output features of the self-attention network as the output features of the residual network.

Step S2033: Input the output features of the residual network into the layer normalization network, and the calculation is as follows:

Among them, the mean value μ and variance var are calculated for the input feature h of each frame, and the value of each dimension of h is regularized and linearly transformed through the model parameters w _i and b _i to output a new feature sequence

Step S2034: Input the output features of the layer normalization network into the fully connected network, the calculation formula of which is:

F(x)＝max(0,xW ₁ +b ₁ )W ₂ +b ₂ (12)

Step S2035: add the input and output features of the fully connected network as the output features of the residual network.

Step S2036: Input the output features of the residual network into the layer normalization network, and the calculation formula is the same as that in step S2033.

Step S2037: Determine whether the current module is the last sub-module, if the current module is the last sub-module, end the encoder calculation process; otherwise, execute step S2038.

Step S2038: Save the output features of the layer normalization network as the input of the self-attention network of the next sub-module, and execute step S2032.

The output features of the normalized network of the preservation layer will be used as the historical frame used for splicing in the next feature processing process, and the output features will be input into the next sub-module.

In a possible embodiment, the lengths of the first feature sequence input to the online encoder and the second feature sequence output by the online encoder are the same.

Step S204: Process the second feature sequence, output multiple sets of Chinese character sequences and score the multiple sets of Chinese character sequences.

The second feature sequence output by the online encoder is modeled by an online decoder. Among them, the online decoder based on the self-attention mechanism is composed of 6 identical sub-modules stacked, and each sub-module is a layer of self-attention network, a layer of residual network, a layer of normalized network, and a layer of truncated attention. network, a layer of residual network, a layer of normalized network, a layer of fully connected network, a layer of residual network and a layer of normalized network, each layer of network output feature is 256 dimensions.

The second feature sequence output by the online encoder is processed by the online decoder. Its processing flowchart is shown in Figure 4, including steps:

Step S2041: Input the feature sequence formed by adding the word embedding feature of the starting symbol and the position feature and the feature sequence output by the decoder into the self-attention network. The calculation formula in the self-attention network is the same as formula (7) and formula (8). Among them, where Q represents an input feature, K represents the feature sequence composed of the current and all historical input features, and V and K are the same.

Step S2042: Add the input and output features of the self-attention network as the output features of the residual network.

Step S2043: Input the output features of the residual network into the layer normalization network, and the calculation formula is the same as that in step S2033.

Step S2044: Input the i-th output feature q _i of the layer normalization network into the truncated attention network, the calculation formula is as follows, for each encoder output feature k _j , sequentially calculate j=1,2,...:

When a certain j satisfies p _ij >0.5, and j is greater than the previous truncation point, it is established as the truncation point of the current truncated attention network, and the output features are calculated:

Step S2045: add the input and output features of the truncated attention network as the output features of the residual network;

Step S2046: Input the output features of the residual network into the layer normalization network, and the calculation formula is the same as that in step S2033.

Step S2047: Input the output features of the layer normalization network into the fully connected network, and the calculation formula is the same as that in step S2034.

Step S2048: add the input and output features of the fully connected network as the output features of the residual network, and input the output features of the residual network into the layer normalization network. The calculation formula is the same as the calculation formula in step S2033;

Step S2049: Determine whether the current module is the last sub-module, if the current module is the last sub-module, execute step S20411; otherwise, execute step S20410.

Step S20410: Save the output features of the layer normalization network as the input of the self-attention network of the next sub-module, and execute step S2042.

Step S20411: Input the output features of the layer normalization network into the Chinese character classifier, and output multiple groups of Chinese characters and corresponding scores. For each Chinese character, add its word embedding and position features, and input it into the decoder to predict the next Chinese character.

Step S205: The Chinese character sequence with the highest score is taken as the final transcription result.

Use the beam search algorithm to control the number of Chinese characters output by the decoder each time, and sort them according to the score from high to low. Take the top ten Chinese characters and input them into the decoder to output the next Chinese character until the decoder outputs the terminator. . At the end, the Chinese character sequence with the highest score will be the final transliteration result.

Those skilled in the art should be aware that, in the above one or more examples, the functions described in the present invention may be implemented by hardware, software, firmware or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the protection scope of the present invention.

Claims (6)

1. An online end-to-end voice transcription method, comprising: Acquiring audio files, and extracting acoustic features from the audio files, including: Extract logarithmic mel spectrum features from the acquired audio files as frame-level acoustic features; performing nonlinear transformation and downsampling on the acoustic features and outputting a first feature sequence; dividing the first feature sequence into blocks, sequentially inputting each block of feature sequences into the encoder and outputting multiple sets of second feature sequences; Modeling the second feature sequence, outputting multiple sets of Chinese character sequences and scoring the multiple sets of Chinese character sequences; Wherein, the online decoder based on the self-attention mechanism is constructed, and the decoder models the second feature sequence, and scores the output multiple groups of Chinese character sequences; The decoder is composed of 6 identical sub-module stacks, wherein each sub-module consists of a layer of self-attention network, a layer of residual network, a layer of normalized network, a layer of truncated attention network, a layer of residual network , a layer-by-layer normalization network, a layer-by-layer fully connected network, a layer-by-layer residual network and a layer-by-layer normalization network; The truncated attention network processes the output features in the second feature sequence one by one, determines a truncation point, truncates the second feature sequence into multiple subsequences to calculate output features, and sends the output features to other The sub-modules are processed to obtain the final output features; Input the final output features into the Chinese character classifier for scoring; The Chinese character sequence with the highest score is taken as the final transliteration result. 2. The method according to claim 1, wherein the encoder is an online encoder based on a self-attention mechanism; The encoder is composed of a stack of 12 identical sub-modules, and each sub-module is sequentially composed of a self-attention network, a residual network, a layer normalization network, a fully connected network, a residual network, and a layer normalization network stack. 3. The method according to claim 1, wherein the decoder models the second feature sequence, and scoring the output multiple groups of Chinese character sequences includes: Pass multiple groups of second feature sequences through 6 submodules of the decoder in turn, and input the output features of the layer norm network of the last submodule into the Chinese character classifier; The Chinese character classifier outputs multiple groups of Chinese characters and the corresponding scores of each group of Chinese characters; Take the top ten Chinese characters and input them into the decoder to output the next Chinese character until the decoder outputs the terminator. 4. An online end-to-end speech transcription system, comprising: Acquisition unit: used to collect audio and extract acoustic features from the audio, including: Extract logarithmic mel spectrum features from the acquired audio files as frame-level acoustic features; Processing unit: used to perform nonlinear transformation and down-sampling on the acoustic features extracted by the acquisition unit and output the first feature sequence; divide the first feature sequence into blocks, input each block of feature sequences into the encoder in turn and output multiple groups the second feature sequence; The processing unit is also used to model the second feature sequence, output multiple sets of Chinese character sequences and score the multiple sets of Chinese character sequences; Wherein, the processing unit constructs an online decoder based on a self-attention mechanism, and the decoder models the second feature sequence, and scores the output multiple groups of Chinese character sequences; The decoder is composed of 5 identical sub-module stacks, wherein each sub-module consists of a layer of self-attention network, a layer of residual network, a layer of normalized network, a layer of truncated attention network, a layer of residual network , a layer-by-layer normalization network, a layer-by-layer fully connected network, a layer-by-layer residual network and a layer-by-layer normalization network; The truncated attention network processes the output features in the second feature sequence one by one, determines a truncation point, truncates the second feature sequence into multiple subsequences to calculate output features, and sends the output features to other The sub-modules are processed to obtain the final output features; Input the final output features into the Chinese character classifier for scoring; Output unit: used to take the Chinese character sequence with the highest score among the Chinese character sequences output by the processing unit as the final transcription result, and output it. 5. The system according to claim 4, wherein the encoder is an online encoder based on a self-attention mechanism; The encoder is composed of a stack of 12 identical sub-modules, and each sub-module is sequentially composed of a self-attention network, a residual network, a layer normalization network, a fully connected network, a residual network, and a layer normalization network stack. 6. The system according to claim 4, wherein the decoder models the second feature sequence, and scoring the output multiple groups of Chinese character sequences includes: Pass multiple groups of second feature sequences through 6 submodules of the decoder in turn, and input the output features of the layer specification network of the last submodule into the Chinese character classifier; The Chinese character classifier outputs multiple groups of Chinese characters and the corresponding scores of each group of Chinese characters; Take the top ten Chinese characters and input them into the decoder to output the next Chinese character until the decoder outputs the terminator.

Images