CN110503060B - A spectral signal denoising method and system thereof - Google Patents

Abstract

The invention discloses a spectral signal denoising method and a system thereof, wherein a one-dimensional discrete spectral signal is subjected to singular value decomposition, and then a signal component obtained by reconstructing each singular value is subjected to fast Fourier transform to obtain each The frequency value corresponding to the maximum amplitude in the fast Fourier transform result of the singular value component signal, and then perform a first-order lag difference on the frequency value of the corresponding component signal according to the singular value decreasing method to obtain the main frequency difference spectrum, and finally set the difference threshold selection. The first position in the main frequency difference spectrum not less than the difference threshold is regarded as the effective order of the singular value, the remaining singular values are set to zero, the reconstruction matrix is obtained by the inverse process of singular value decomposition, and the denoised signal is obtained through inversion. In the present invention, the singular value order reconstruction signal can be accurately selected by the above method, and the denoising effect can be improved.

Description

A spectral signal denoising method and system thereof

technical field

The invention belongs to the technical field of spectral signal denoising, and in particular relates to a spectral signal denoising method and a system thereof.

Background technique

In the quantitative analysis of UV-visible spectrum, random noises are generated due to the light source, detector, electronic components, dark current and external environmental interference inside the miniature spectrometer, which have a huge impact on the accuracy of quantitative spectral analysis results. Therefore, the preprocessing of the spectral data is an essential step before establishing the concentration model. A reasonable preprocessing method can retain the useful chemical information of the sample to be tested in the spectral signal and filter out the noise information and redundant information, so as to improve the robustness and prediction ability of the model. At present, there are many methods for spectral signal data and processing, mainly including wavelet transform, empirical mode decomposition and local curve fitting. Further research, at present, mostly relies on experiments and researchers' experience to determine.

The singular value denoising algorithm does not need to be very dependent on the performance of the wavelet filter like the time domain average, nor does it need a standard input signal like the adaptive filter, it can also achieve noise reduction for the frequency conversion signal, so it is also used. In signal de-noising, if there are too many singular values retained, even if the signal is preprocessed, there will still be a lot of noise, so that the signal-to-noise ratio is still small; if the retained singular values are too small, although the noise is removed, Some detailed features of the original signal will be lost, and even lead to serious distortion of the original signal, affecting the signal reconstruction accuracy. Therefore, how to accurately select the singular value order needs to be considered and perfected. The existing singular value difference spectrum method can effectively select the number of reconstructed singular values to a certain extent, but when the frequency components of the real signal are complex, after the signal is decomposed by singular values, since each singular value corresponds to a component signal, the difference The maximum spectral peak of the spectrum can only represent the huge difference between the two component signals, but the maximum difference is not always the boundary between the signal and the noise. Therefore, the singular value difference spectrum method that uses the singular value to directly differentiate It is greatly disturbed by the trend term of the real signal, which will violate the original selection method, and the accuracy of the obtained singular value order also needs to be improved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a spectral signal denoising method and a system thereof, which can automatically and accurately select the singular value order, thereby improving signal reconstruction accuracy and signal-to-noise ratio.

A spectral signal denoising method provided by the present invention includes the following steps:

S1: Perform singular value decomposition on the spectral signal to be denoised to obtain orthogonal matrices U, V ^T and singular value matrix Σ;

Λ为r个奇异值σ_i作为对角线元素构成的对角矩阵，r为Hankel矩阵的秩，0表示零矩阵；Among them, the singular value matrix

Λ is a diagonal matrix composed of r singular values σ _i as diagonal elements, r is the rank of the Hankel matrix, and 0 represents a zero matrix;

S2: Execute separately for r singular values: retain a single singular value σ _i in the diagonal matrix Λ and set the remaining singular values to zero to obtain a new diagonal matrix Λ ^′ _i , and then invert and reconstruct respectively to obtain new spectral signal components ;

S3: Perform fast Fourier transform on the r new spectral signal components in step S2, respectively obtain the frequency value corresponding to the maximum amplitude from the transform result, and then perform first-order lag on the corresponding frequency value according to the singular value decreasing method Difference to obtain the main frequency difference spectrum;

b _i =f _i+1 -f _i

In the formula, b _i is the i-th main frequency difference value in the main frequency difference spectrum, f _i and f _i+1 are the i-th singular value σ _i and the i+1-th singular value arranged in descending order of singular values, respectively. The frequency value corresponding to the maximum amplitude in the transformation result of the new spectral signal component corresponding to the singular value σ _i+1 ;

S4: Identify the first main frequency difference value b _j that is not less than the preset difference threshold in the main frequency difference spectrum, retain the first j singular values according to the singular value decreasing method, set the remaining singular values to zero to obtain a new diagonal matrix, and then The denoised spectral signal is obtained by inversion and reconstruction.

Further preferably, the execution process of step S1 is as follows:

Reconstruct the spectral signal to be denoised into a Hankel matrix;

Then, perform singular value decomposition on the Hankel matrix to obtain orthogonal matrices U, V ^T and singular value matrix Σ.

Λ＝diag(σ₁，σ₂，…，σ_r)，σ₁≥σ₂≥…≥σ_r，σ_i为矩阵H_m×n的奇异值。Among them, the singular value matrix

Λ=diag(σ ₁ , σ ₂ ,...,σ _r ), σ ₁ ≥σ ₂ ≥...≥σ _r , σ _i is a singular value of the matrix H _m×n .

Further preferably, the spectral signal to be denoised is reconstructed into a Hankel matrix whose size is m×n, m≤n, and the Hankel matrix is as follows:

In the formula, m is the embedding dimension, and m+n-1=N.

Further preferably, the preset differential threshold is in the range of [30, 50].

The present invention also provides a system based on the above method, comprising a singular value decomposition module, a reconstruction module, and a main frequency difference spectrum acquisition module;

Among them, the singular value decomposition module is used to perform singular value decomposition on the spectral signal to be denoised to obtain orthogonal matrices U, V ^T and singular value matrix Σ;

The reconstruction module is used for updating the new diagonal matrix A ^′ _i and inverting and reconstructing respectively based on the r singular values to obtain new spectral signal components;

The main frequency difference spectrum acquisition module is used to perform fast Fourier transformation on the r new spectral signal components respectively, and obtain the frequency value corresponding to the maximum amplitude from the transformation result, and then according to the singular value decreasing method. The frequency value is subjected to the first-order lag difference to obtain the main frequency difference spectrum;

The reconstruction module is used to identify the first main frequency difference value b _j that is not less than the preset difference threshold in the main frequency difference spectrum, retain the first j singular values according to the singular value decreasing method, and set the remaining singular values to zero to obtain The new diagonal matrix is inverted and reconstructed to obtain the denoised spectral signal.

beneficial effect

The present invention finds that after the signal is decomposed by singular value, the component signal of singular value with larger value represents the real signal, and the component signal of singular value with smaller value represents noise. At the same time, it is found that the frequency of the noise signal is higher than that of the real signal, that is The difference between the two is obvious in the frequency domain. After the fast Fourier transform of the boundary between the singular value component signal corresponding to the real signal and the singular value component signal corresponding to the noise signal, the dominant frequency will have a big mutation. Therefore, the present invention After performing fast Fourier transform on the spectral signal components of each singular value inversion, the main frequency difference spectrum is obtained by performing the difference in the frequency domain according to the decreasing singular value, and then the singular value can be automatically and accurately selected by using the characteristics of sudden change At the same time, the method has no special requirements for signal stationarity and noise characteristics. Compared with the singular value difference spectrum algorithm for selecting the singular value order, the present invention has stronger universality for signal preprocessing, does not require manual intervention in the data preprocessing process and can automatically select the singular value order, which can achieve spectral signal Online preprocessing requirements, and can greatly improve the signal reconstruction accuracy and signal-to-noise ratio.

Description of drawings

Fig. 1 is the ultraviolet-visible spectral signal of the 480nm to 800nm wavelength band of the embodiment of the present invention;

Fig. 2 is the first 50 singular value main frequency difference spectrum of the embodiment of the present invention;

FIG. 3 is the noise reduction effect on the measured spectral data in the embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the embodiments.

The present invention provides a spectral signal denoising method and system, which use singular value denoising algorithm to denoise the original spectrum, which can accurately select the singular value order and improve the denoising effect. With zinc hydrometallurgy as the background, mixed standard solutions with copper, cobalt, and nickel-iron ion concentrations of 2 mg/L, 1.8 mg/L, 1.6 mg/L, and 2 mg/L were prepared in the experiment. The sampling integration time of the micro-spectrometer is 9ms, and the sampling integration interval is 500ms. Among them, as shown in Figure 1, the ultraviolet-visible spectral signal in the wavelength range of 480nm to 800nm is collected as the spectral signal to be denoised. The execution process of the method in this embodiment is as follows:

Step 1: Reconstruct the spectral signal to be de-noised X=(x ₁ , x ₂ , . . . , x _N ) into a Hankel matrix. The size of the reconstructed Hankel matrix in this embodiment is m×n, m≤n, and is as follows:

Wherein, H _m×n is the Hankel matrix, 1<n<, m is the embedding dimension, and m+n−1=N is satisfied.

Step 2: Perform singular value decomposition on the Hankel matrix to obtain H _m×n =UΣV ^T .

为m×n维矩阵，Λ＝diag(σ₁，σ₂，…，σ_r),并且σ₁≥σ₂≥…≥σ_r，σ_i为矩阵H_m×n的奇异值，r为Hankel矩阵的秩，0为零矩阵。其他可行的实施例中，可以对上述矩阵中元素的位置进行适应性调整Among them, U=(u ₁ , u ₂ ,..., _um )∈R ^m×m , V=(v ₁ , v ₂ ,..., v _n )∈R ^n×n are orthogonal matrices, and u ₁ represents The first column of the orthogonal matrix U, v ₁ represents the first column of the orthogonal matrix V,

is an m×n-dimensional matrix, Λ=diag(σ ₁ , σ ₂ ,…,σ _r ), and σ ₁ ≥σ ₂ ≥…≥σ _r , σ _i is the singular value of the matrix H _m×n , r is Hankel The rank of the matrix, 0 is a zero matrix. In other feasible embodiments, the positions of the elements in the above-mentioned matrix can be adaptively adjusted

Step 3: Execute separately for r singular values: retain a single singular value σ _i in the diagonal matrix Λ and set the remaining singular values to zero to obtain a new diagonal matrix Λ ^′ _i , and then invert and reconstruct respectively to obtain a new spectral signal weight.

i＝1,2，3，…,r。For example: keep a single singular value σ _i in Λ=diag(σ ₁ ,σ ₂ ,...,σ _r ), and set all singular values except σ _i to zero, that is, Λ ^′ _i =diag(0, …,σ _i ,…,0),

i=1, 2, 3, ..., r.

After obtaining r new diagonal matrices Λ ^′ _i , the orthogonal matrices U and V ^T are used to calculate the new Hankel matrix H′ _i (H′ _i = UΣ ^′ _i V ^T ) in turn, and the inverse method is performed according to the construction method of the Hankel matrix. Then, r spectral signal components X _i (i=1, 2, 3, . . . , r) can be obtained.

Step 4: Perform fast Fourier transform on the r new spectral signal components respectively, and obtain the frequency value corresponding to the maximum amplitude from the transform result, and then perform first-order lag difference on the corresponding frequency value according to the singular value decreasing method to obtain Main frequency difference spectrum B, where, B=[b ₁ , b ₂ ,..., _br-1 ];

b _i =f _i+1 -f _i

In the formula, b _i is the i-th main frequency difference value in the main frequency difference spectrum, f _i and f _i+1 are the i-th singular value σ _i and the i+1-th singular value arranged in descending order of singular values, respectively. The frequency value corresponding to the maximum amplitude in the transformation result of the new spectral signal component corresponding to the singular value σ _i+1 .

Step 5: Identify the first main frequency difference value b _j that is not less than the preset difference threshold in the main frequency difference spectrum, retain the first j singular values according to the singular value decreasing method, and set the remaining singular values to zero to obtain a new diagonal matrix, Then invert and reconstruct to obtain the denoised spectral signal.

In this embodiment, Λ=diag(σ ₁ , σ ₂ ,...,σ _r ), σ ₁ ≥σ ₂ ≥...≥σ _r , if it is identified that b _j has the above situation, the first j in the diagonal matrix Λ One singular value is retained, and the remaining singular values are set to zero; then the Hankel matrix is reconstructed and the denoised spectral signal is obtained by inversion.

Since the main frequencies of the singular value component signal of the real signal and the singular value component signal of the noise signal are very different, the frequency will have a large mutation. Therefore, the present invention finds through research that a difference threshold is set to identify the mutation. As shown in FIG. 2 , in this embodiment, the preset difference threshold is 50, and the first position that is not less than the difference threshold is identified as the twelfth point, that is, the effective order of singular values is 12, and the first 12 One singular value is retained, the rest are set to zero, the reconstruction matrix is obtained by the inverse process of singular value decomposition, and the denoised signal is obtained through inversion.

Based on the above method, the system provided by the present invention includes: a singular value decomposition module, a reconstruction module, and a main frequency difference spectrum acquisition module;

Among them, the singular value decomposition module is used to perform singular value decomposition on the spectral signal to be denoised to obtain orthogonal matrices U, V ^T and singular value matrix Σ;

The reconstruction module is used to update the new diagonal matrix Λ ^′ _i based on the r singular values and perform inversion and reconstruction respectively to obtain new spectral signal components;

The main frequency difference spectrum acquisition module is used to perform fast Fourier transformation on the r new spectral signal components respectively, and obtain the frequency value corresponding to the maximum amplitude from the transformation result, and then according to the singular value decreasing method to the corresponding frequency value Perform the first-order lag difference to obtain the main frequency difference spectrum;

The reconstruction module is used to identify the first main frequency difference value b _j that is not less than the preset difference threshold in the main frequency difference spectrum, retain the first j singular values according to the singular value decreasing method, and set the remaining singular values to zero to obtain a new pair Angle matrix, and then inversion and reconstruction to obtain the denoised spectral signal.

It should be understood that the present invention can automatically and accurately select the order of the singular value considering the frequency domain characteristics of the singular value component signal, has no special requirements for signal stability and noise characteristics, and can meet the requirements of online preprocessing of spectral signals.

It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, so the present invention is not limited to the examples described in the specific implementation manner, and all the examples obtained by those skilled in the art according to the technical solutions of the present invention Other embodiments that do not depart from the spirit and scope of the present invention, whether modified or replaced, also belong to the protection scope of the present invention.

Claims (5)

1. A spectral signal denoising method is characterized in that: the method comprises the following steps:

s1: reconstructing the spectral signals to be denoised into a Hankel matrix, and performing singular value decomposition on the Hankel matrix to obtain an orthogonal matrix U, V^TAnd a singular value matrix Σ;

wherein the singular value matrix

Λ is r singular values σ_iR is the rank of Hankel matrix, and 0 represents zero matrix;

s2: performing for r singular values respectively: preserving a single singular value sigma in the diagonal matrix lambda_iAnd setting the residual singular values to zero to obtain a new diagonal matrix Lambda'_iReuse of orthogonal matrix U, V^TSequentially calculating a new Hankel matrix and carrying out inversion reconstruction to obtain a new spectrum signal component;

s3: respectively carrying out fast Fourier transform on the r new spectral signal components in the step S2, respectively obtaining frequency values corresponding to the maximum amplitude from the transform results, and carrying out first-order lag difference on the corresponding frequency values according to a singular value decreasing method to obtain a main frequency difference spectrum;

b_i＝f_i+1-f_i

in the formula, b_iIs the ith main frequency differential value, f in the main frequency differential spectrum_i、f_i+1Respectively, i-th singular value σ arranged in descending order according to singular value_iI +1 th singular value σ_i+1The frequency value corresponding to the maximum amplitude in the transformation result of the corresponding new spectral signal component;

s4: identifying a first main frequency difference value b not less than a preset difference threshold value in a main frequency difference spectrum_jAnd reserving the first j singular values according to a singular value decreasing mode, setting the rest singular values to zero to obtain a new diagonal matrix, and performing inversion reconstruction to obtain the denoised spectral signal.

2. The method of claim 1, wherein: the step S1 is performed as follows:

reconstructing the spectral signal to be denoised into a Hankel matrix;

then, singular value decomposition is carried out on the Hankel matrix to obtain an orthogonal matrix U, V^TAnd a singular value matrix Σ;

wherein the singular value matrix

Λ＝diag(σ₁,σ₂,…,σ_r)，σ₁≥σ₂≥…≥σ_r，σ_iAre the singular values of the Hankel matrix.

3. The method of claim 2, wherein: reconstructing the spectral signal to be denoised into a Hankel matrix with the size of mxn, wherein m is less than or equal to n, and the Hankel matrix is as follows:

in the formula, H_m×nRepresenting a Hankel matrix, m is the embedding dimension, and m + N-1 ═ N.

4. The method of claim 1, wherein: the preset differential threshold range is [30,50 ].

5. A system based on the method of any one of claims 1-4, characterized by: the system comprises a singular value decomposition module, a reconstruction module and a main frequency difference spectrum acquisition module;

the singular value decomposition module is used for reconstructing the spectral signal to be denoised into a Hankel matrix, and then performing singular value decomposition on the Hankel matrix to obtain an orthogonal matrix U, V^TAnd a singular value matrix Σ;

the reconstruction module is used for respectively carrying out new diagonal matrix Lambda 'on the diagonal matrix Lambda in the singular value matrix Sigma based on r singular values'_iUpdating and inverting reconstruction to obtain a new spectrum signal component;

the main frequency difference spectrum acquisition module is used for respectively carrying out fast Fourier transform on r new spectral signal components, respectively acquiring frequency values corresponding to the maximum amplitude from the transform results, and then carrying out first-order lag difference on the corresponding frequency values according to a singular value decreasing method to obtain a main frequency difference spectrum;

the reconstruction module is used for identifying a first main frequency difference value b which is not less than a preset difference threshold value in a main frequency difference spectrum_jReserving the first j singular values according to a singular value decreasing mode, setting the rest singular values to zero to obtain a new diagonal matrix, and then utilizing an orthogonal matrix U, V^TAnd sequentially calculating a new Hankel matrix and carrying out inversion reconstruction to obtain the denoised spectral signal.

Images