CN111612084A - An Optimization Method for Deep Non-negative Matrix Factorization Networks with Classifiers - Google Patents

Abstract

The invention relates to an optimization method of a deep non-negative matrix factorization (NMF) network with a classifier, and belongs to the technical field of artificial intelligence. The invention includes model construction and parameter optimization. In the aspect of model construction, multiple NMF layers and classification layers are cascaded to form a deep network, that is, the decomposition result of the previous NMF layer is used as the input of the next NMF layer, and a mapping function is used between different NMF layers. connect. In terms of parameter optimization, the unsupervised layer-by-layer pre-training of the deep NMF network is performed based on the multiplicative iterative rule, and the supervised global optimization is based on the BP algorithm to optimize the weight parameters of each NMF layer and the Softmax classification layer as a whole. The test data is analyzed by using a deep NMF network optimized by training, and the classification output results are obtained. The invention is suitable for applications related to classification and identification tasks such as state monitoring and diagnosis.

Description

An Optimization Method for Deep Non-negative Matrix Factorization Networks with Classifiers

technical field

The invention belongs to the technical field of artificial intelligence, and relates to an optimization method for a deep non-negative matrix decomposition network with a classifier.

Background technique

Non-negative matrix factorization is a relatively novel matrix factorization idea, which requires all components after decomposition to be non-negative values (requiring purely additive description), and can achieve nonlinear dimension reduction and sparse feature representation. It has been successfully applied in many fields such as image processing, computer vision, pattern recognition, biomedicine, and signal processing. However, the feature expression ability of traditional single-layer network is limited. In order to further improve the classification or regression ability of the network, deep learning has become an important research direction in the field of machine learning in recent years. By constructing multiple abstract layers with self-learning ability, deep network can The input data is extracted step by step to achieve a higher-level feature representation of the data. At present, deep networks have achieved remarkable results in many classification and recognition tasks.

In this context, the combination of non-negative matrix factorization and deep learning is expected to take into account the advantages of both. By constructing a deep non-negative matrix factorization network to improve the feature extraction performance of single-layer NMF, NMF is purely additively described as a deep network. Provides an intuitive and understandable hierarchical feature learning process. Deep NMF Network (Guo Z, Zhang S. Sparse DeepNonnegative Matrix Factorization [J]. Big Data Mining and Analytics, 2020, 3(1): 13-28. SongHA, Kim B, Xuan T L, et al, Hierarchical feature extraction by multi-layernon-negative matrix factorization network for classification task [J]. Neurocomputing, 2015, 165: 63-74. Cichocki A, Zdunek R. Multilayer non negative matrix factorization, Electron. Lett. 2006, 42(16): 947–948.) began to gain attention. However, the existing deep NMF network only trains and optimizes the NMF feature extraction layer, and does not realize the unified model global optimization of the NMF feature extraction layer and the classification layer. In response to this problem, the present invention proposes a multi-layer non-negative matrix factorization network with a classifier with a deep learning structure, and designs a model parameter optimization method combining unsupervised layer-by-layer pre-training and supervised global optimization.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide an optimization method for a deep non-negative matrix factorization network with a classifier. By analyzing the basic structure of non-negative matrix factorization and deep network, the cascade idea is used to construct a multi-layer non-negative matrix factorization network with classifiers with deep learning structure. The global optimization of the model with the unified feature extraction layer and classification layer of the deep model.

To achieve the above object, the present invention provides the following technical solutions:

An optimization method for a deep non-negative matrix factorization network with a classifier, the method includes the following steps:

S1: Input the original data, the training set is {X _k ,G _k }, where k=1,2,...,n, n is the number of training samples, preprocess the input data of the model, record the preprocessed data as

S2: Set the number of NMF layers in the deep network to L, and the low-dimensional feature space dimensions of each NMF layer are r ⁽¹⁾ , r ⁽²⁾ ,..., r ^(L) respectively, for multiple NMF layers and classification layers Concatenation is performed to construct a deep NMF network with a classifier. The decomposition result of the previous NMF layer is used as the input of the next NMF layer, and the different NMF layers are connected by a mapping function;

S3: For the deep NMF network constructed in step S2, perform unsupervised pre-training on each layer of NMF based on the multiplicative iterative rule;

S4: For the pre-trained deep NMF network obtained in step S3, perform supervised global optimization on the connection weight parameters of each NMF layer and the Softmax classification layer based on the BP algorithm;

S5: According to the obtained training and optimized deep NMF network in step S4, analyze the input test data samples to obtain corresponding classification output results.

Optionally, in the step S1, a short-time Fourier transform time-frequency analysis method is used to preprocess the input data to calculate the time-frequency amplitude spectrum.

In the step S2, Softmax is used as the classification layer function to construct a deep NMF network with a classifier.

Optionally, in the step S3, unsupervised pre-training is performed on the deep NMF network based on the multiplicative iterative rule:

S31: Input the data X ^(i-1) into the i-th layer NMF network;

为N维向量数据集合，则基向量矩阵

低维特征矩阵

r为低维特征空间维数，一般情况下，r比N和n小很多，且满足((N+n)r＜Nn；S32: set the algorithm termination threshold e and the maximum number of iterations t _max , initialize the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ ; set

is an N-dimensional vector data set, then the basis vector matrix

low-dimensional feature matrix

r is the dimension of the low-dimensional feature space. In general, r is much smaller than N and n, and satisfies ((N+n)r<Nn;

S33: Update the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative update rule is defined as:

S34: Calculate the objective function value C of the NMF layer in the pre-training stage, and C is defined as

S35: Compare the objective function values C ^(t) and C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, cycle from step S33 to step S35;

S36: Perform the mapping process on the low-dimensional feature H ⁽ⁱ⁾ to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is:

f(x)=1/(1+exp(-x))

S37: Repeat steps S31 to S37 until i>L to complete the unsupervised layer-by-layer pre-training of each NMF layer.

Optionally, in the step S4, supervised global optimization is performed on the deep NMF network based on the BP algorithm:

S41: Input the labeled data X ^(i-1) into the i-th layer NMF network;

S42: Set the algorithm termination threshold e and the maximum number of iterations t _max , and initialize the low-dimensional feature matrix H ⁽ⁱ⁾ ;

S43: fix the basis vector matrix W ⁽ⁱ⁾ , iteratively update the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative rule is:

S44: Calculate the objective function value C of the NMF layer in the supervised global optimization stage, where C is defined as

S45: Compare the objective function values C ^(t) and C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, loop steps S43 to S45;

S46: Calculate the cost function of the i-th layer NMF network

为权重约束项，α为平衡权重约束项的系数，f(w_ij)定义为：in,

is the weight constraint term, α is the coefficient of the balance weight constraint term, f(w _ij ) is defined as:

S47: Perform the mapping process on the low-dimensional feature H ⁽ⁱ⁾ to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is:

f(x)=1/(1+exp(-x))

S48: Repeat steps S41 to S47 until i>L;

S49: Input the output X ^(L) of the L-th layer NMF into the Softmax classifier, and calculate the cost function value of the classification layer. The formula is as follows:

为Softmax错误分类代价函数，K为类别数，y_r为样本类标签；in,

is the Softmax misclassification cost function, K is the number of categories, and y _r is the sample class label;

S410: Calculate the overall cost function value of the deep NMF network with the classifier, the formula is as follows:

Among them, W _DN includes each NMF layer and Softmax classification layer weight parameters.

S411: Treat all layers of the deep NMF network with a classifier as a model, and based on the gradient descent algorithm, after multiple iterations, it is hoped that the overall cost function value of the network is minimized. Calculate the output of each layer, the reconstruction error of each layer, correct the corresponding parameters according to the error, and optimize the weight parameters of each NMF layer and Softmax classification layer.

The beneficial effects of the invention are: the NMF algorithm has the characteristics of fast convergence speed and small storage space for left and right non-negative matrices, it can reduce the dimension of high-dimensional data matrix, and the decomposed low-dimensional matrix has natural sparsity and robustness It is suitable for processing large-scale data. The deep NMF network improves the feature extraction performance of single-layer NMF, while providing an intuitive and understandable hierarchical feature learning process for the deep network.

Other advantages, objects, and features of the present invention will be set forth in the description that follows, and will be apparent to those skilled in the art based on a study of the following, to the extent that is taught in the practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the following description.

Description of drawings

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be preferably described in detail below with reference to the accompanying drawings, wherein:

1 is a schematic diagram of the deep non-negative matrix decomposition network structure with a classifier according to the present invention;

FIG. 2 is a flow chart of the classification and recognition based on the deep non-negative matrix factorization network of the present invention.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments are only used to illustrate the basic idea of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

Among them, the accompanying drawings are only used for exemplary description, and represent only schematic diagrams, not physical drawings, and should not be construed as limitations of the present invention; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings will be omitted, The enlargement or reduction does not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the accompanying drawings may be omitted.

The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms “upper”, “lower”, “left” and “right” , "front", "rear" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must be It has a specific orientation, is constructed and operated in a specific orientation, so the terms describing the positional relationship in the accompanying drawings are only used for exemplary illustration, and should not be construed as a limitation of the present invention. situation to understand the specific meaning of the above terms.

Please refer to Figure 1 to Figure 2, an optimization method of a deep NMF network with a classifier, which performs unsupervised layer-by-layer pre-training of the deep NMF network based on multiplicative iterative rules, which specifically includes the following steps:

1) Input the data X ^(i-1) into the i-th layer NMF network;

为N维向量数据集合，则基向量矩阵

低维特征矩阵

r为低维特征空间维数，一般情况下，r比N和n小很多，且满足((N+n)r＜Nn；2) Set the algorithm termination threshold e and the maximum number of iterations t _max , initialize the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ ; set

is an N-dimensional vector data set, then the basis vector matrix

low-dimensional feature matrix

r is the dimension of low-dimensional feature space. In general, r is much smaller than N and n, and satisfies ((N+n)r<Nn;

3) Update the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative update rule is defined as:

4) Calculate the objective function value C of the NMF layer in the pre-training stage, and C is defined as

5) Compare the objective function value C ^(t) with C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, loop step 3) to step 5);

6) The low-dimensional feature H ⁽ⁱ⁾ is mapped to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is:

f(x)=1/(1+exp(-x))

7) Repeat steps S31 to S37 until i>L to complete the unsupervised layer-by-layer pre-training of each NMF layer.

The supervised global optimization of the deep NMF network based on the BP algorithm, the specific steps are as follows:

1) Input the labeled data X ^(i-1) into the i-th layer NMF network;

2) Set the algorithm termination threshold e and the maximum number of iterations t _max , and initialize the low-dimensional feature matrix H ⁽ⁱ⁾ ;

3) Fix the basis vector matrix W ⁽ⁱ⁾ , and iteratively update the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative rules are:

4) S44: Calculate the objective function value C of the NMF layer in the supervised global optimization stage, where C is defined as

5) Compare the objective function value C ^(t) with C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, cycle from step 3) to step 5);

6) Calculate the cost function of the i-th layer NMF network

为权重约束项，α为平衡权重约束项的系数，f(w_ij)定义为in,

is the weight constraint term, α is the coefficient of the balance weight constraint term, f(w _ij ) is defined as

7) The low-dimensional feature H ⁽ⁱ⁾ is mapped to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is:

f(x)=1/(1+exp(-x))

8) Repeat step 1) to step 7) until i>L;

9) Input the output X ^(L) of the L-th layer of NMF into the Softmax classifier, and calculate the cost function value of the classification layer. The formula is as follows:

为Softmax错误分类代价函数，K为类别数，y_r为样本类标签；in,

is the Softmax misclassification cost function, K is the number of categories, and y _r is the sample class label;

10) Calculate the overall cost function value of the deep NMF network with the classifier, the formula is as follows:

Among them, W _DN includes each NMF layer and Softmax classification layer weight parameters;

11) Considering all layers of the deep NMF network with classifiers as a model, based on the gradient descent algorithm, after many iterations, the overall cost function value of the network is expected to be minimized. Calculate the output of each layer, the reconstruction error of each layer, correct the corresponding parameters according to the error, and further optimize the weight parameters of each NMF layer and Softmax classification layer.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should all be included in the scope of the claims of the present invention.

Claims (5)

1. an optimization method of a deep non-negative matrix decomposition network with a classifier, the non-negative matrix decomposition network with a deep classifier comprises an input layer, an NMF layer 1, an NMF layer 2, ..., an NMF layer L and a classification layer, Wherein, a mapping function is used to connect different NMF layers, and it is characterized in that: the method includes the following steps: S1: Input the original data, the training set is {X _k ,G _k }, where k=1,2,...,n, n is the number of training samples, preprocess the input data of the model, record the preprocessed data as

S2: Set the number of NMF layers in the deep network to L, and the low-dimensional feature space dimensions of each NMF layer are r ⁽¹⁾ , r ⁽²⁾ ,..., r ^(L) respectively, for multiple NMF layers and classification layers Concatenation is performed to construct a deep NMF network with a classifier. The decomposition result of the previous NMF layer is used as the input of the next NMF layer, and the different NMF layers are connected by a mapping function; S3: For the deep NMF network constructed in step S2, perform unsupervised pre-training on each layer of NMF based on the multiplicative iterative rule; S4: For the pre-trained deep NMF network obtained in step S3, perform supervised global optimization on the connection weight parameters of each NMF layer and the Softmax classification layer based on the BP algorithm; S5: According to the obtained training and optimized deep NMF network in step S4, analyze the input test data samples to obtain corresponding classification output results. 2. the optimization method of a kind of deep non-negative matrix decomposition network with classifier according to claim 1, is characterized in that: in described step S1, adopt short-time Fourier transform time-frequency analysis method to calculate input data The time-frequency amplitude spectrum is preprocessed. 3. The optimization method of a deep non-negative matrix factorization network with a classifier according to claim 1, characterized in that: in the step S2, Softmax is used as a classification layer function to construct a deep NMF network with a classifier. 4. The optimization method of a deep non-negative matrix factorization network with a classifier according to claim 1, wherein in the step S3, an unsupervised pre-training step is performed on the deep NMF network based on the multiplicative iterative rule as follows: S31: Input the data X ^(i-1) into the i-th layer NMF network; S32: set the algorithm termination threshold e and the maximum number of iterations t _max , initialize the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ ; set

is an N-dimensional vector data set, then the basis vector matrix

low-dimensional feature matrix

r is the dimension of low-dimensional feature space. In general, r is much smaller than N and n, and satisfies ((N+n)r<Nn; S33: Update the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative update rule is defined as:

S34: Calculate the objective function value C of the NMF layer in the pre-training stage, and C is defined as

S35: Compare the objective function values C ^(t) and C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the basis vector matrix W ⁽ⁱ⁾ and the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, cycle from step S33 to step S35; S36: Perform the mapping process on the low-dimensional feature H ⁽ⁱ⁾ to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is: f(x)=1/(1+exp(-x)) S37: Repeat steps S31 to S37 until i>L to complete the unsupervised layer-by-layer pre-training of each NMF layer. 5. the optimization method of a kind of deep non-negative matrix decomposition network with classifier according to claim 1, is characterized in that: in described step S4, based on BP algorithm, deep NMF network is carried out supervised global optimization, supervised The steps of global optimization are as follows: S41: Input the labeled data X ^(i-1) into the i-th layer NMF network; S42: Set the algorithm termination threshold e and the maximum number of iterations t _max , and initialize the low-dimensional feature matrix H ⁽ⁱ⁾ ; S43: fix the basis vector matrix W ⁽ⁱ⁾ , iteratively update the low-dimensional feature matrix H ⁽ⁱ⁾ , the iterative rule is:

S44: Calculate the objective function value C of the NMF layer in the supervised global optimization stage, where C is defined as:

S45: Compare the objective function values C ^(t) and C ^(t+1) , if ||C ^(t+1) -C ^(t) ||<e is established or the maximum number of iterations t _max is reached, the algorithm terminates, and Obtain the low-dimensional feature H ⁽ⁱ⁾ of the i-th layer NMF network, otherwise, loop steps S43 to S45; S46: Calculate the cost function of the i-th layer NMF network

in,

is the weight constraint term, α is the coefficient of the balance weight constraint term, f(w _ij ) is defined as:

S47: Perform the mapping process on the low-dimensional feature H ⁽ⁱ⁾ to obtain the input data X ⁽ⁱ⁾ of the i+1th layer NMF, and the nonlinear mapping based on the sigmoid function is: f(x)=1/(1+exp(-x)) S48: Repeat steps S41 to S47 until i>L; S49: Input the output X ^(L) of the L-th layer NMF into the Softmax classifier, and calculate the cost function value of the classification layer. The formula is as follows:

in,

is the Softmax misclassification cost function, K is the number of categories, and y _r is the sample class label; S410: Calculate the overall cost function value of the deep NMF network with the classifier, the formula is as follows:

Among them, W _DN includes each NMF layer and Softmax classification layer weight parameters; S411: Treat all layers of the deep NMF network with a classifier as a model, based on the gradient descent algorithm, after several iterations to minimize the overall cost function value of the network; calculate the output of each layer, the reconstruction error of each layer, according to the error Correct the corresponding parameters, and optimize the weight parameters of each NMF layer and Softmax classification layer.

Images