Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Service Function Chaining to Support Ultra-Low Latency Communication in NFV
Next Article in Special Issue
Practical and Malicious Multiparty Private Set Intersection for Small Sets
Previous Article in Journal
Thermal and Optical Analysis of Quantum-Dot-Converted White LEDs in Harsh Environments
Previous Article in Special Issue
Vertical Federated Unlearning on the Logistic Regression Model
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 12
Issue 18
10.3390/electronics12183845
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Fast and Accurate SNN Model Strengthening for Industrial Applications

by
Deming Zhou
1,
Weitong Chen
1,
Kongyang Chen
2,3,* and
Bing Mi
4
1
School of Mathematics and Information Science, Guangzhou University, Guangzhou 510006, China
2
Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou 510006, China
3
Pazhou Lab, Guangzhou 510330, China
4
School of Public Finance and Taxation, Guangdong University of Finance and Economics, Guangzhou 510320, China
*
Author to whom correspondence should be addressed.
Electronics 2023, 12(18), 3845; https://doi.org/10.3390/electronics12183845
Submission received: 30 July 2023 / Revised: 1 September 2023 / Accepted: 5 September 2023 / Published: 11 September 2023
(This article belongs to the Special Issue Security and Privacy Evaluation of Machine Learning in Networks)
Browse Figures
Figure 1
<p>The comparison of the untrained data and the malicious data.</p> "> Figure 2
<p>Data distribution in the NEU-CLS-64 dataset.</p> "> Figure 3
<p>Model strengthening accuracy.</p> "> Figure 4
<p>Model strengthening performance.</p> "> Figure 5
<p>The original images under different labels.</p> ">
Versions Notes

Abstract

:
In spiking neural networks (SNN), there are emerging security threats, such as adversarial samples and poisoned data samples, which reduce the global model performance. Therefore, it is an important issue to eliminate the impact of malicious data samples on the whole model. In SNNs, a naive solution is to delete all malicious data samples and retrain the entire dataset. In the era of large models, this is impractical due to the huge computational complexity. To address this problem, we present a novel SNN model strengthening method to support fast and accurate removal of malicious data from a trained model. Specifically, we use untrained data that has the same distribution as the training data. We can infer that the untrained data has no effect on the initial model, and the malicious data should have no effect on the final refined model. Thus, we can use the model output of the untrained data with respect to the initial model to guide the final refined model. In this way, we present a stochastic gradient descent method to iteratively determine the final model. We perform a comprehensive performance evaluation on two industrial steel surface datasets. Experimental results show that our model strengthening method can provide accurate malicious data elimination, with speeds 11.7 × to 27.2 × faster speeds than the baseline method.

1. Introduction

Given the emerging security and privacy issues, malicious users or data will have a strong impact on the global performance of models [1,2]. For example, in the model training process, malicious data samples will severely reduce the global model performance, in terms of worse classification accuracy or model generalization ability. Especially in the area of spiking neural networks (SNN), little attention has been paid to these security threats, leading to a huge risk potential, such as injecting a backdoor into the spiking neural network [3].
To deal with this problem, we generally need to strengthen the model by eliminating the effects of malicious data samples. One of the simplest solutions is to remove the malicious data samples from the training data and then retrain the training data. Of course, it will take a long time to refine the original model. In the current era of large models, it becomes even impractical due to the huge computational cost and long time.
In this paper, we propose a novel SNN model strengthening method to support fast and accurate removal of malicious data samples from a trained model. To this end, our goal is to improve the initial model by removing the influence of malicious data samples that make up a small part of the training data. To achieve this, our basic idea is to introduce untrained data that has almost the same data distribution as the training data. We can infer that the untrained data has no effect on the initial model, and the malicious data should have no effect on the final refined model. Thus, we can use the model output of the untrained data with respect to the initial model to guide the final refined model. In this way, we present a stochastic gradient descent method to iteratively determine the final model.
Specifically, to make the final refined model accurate, we can further optimize the iteration objective into two conditions. The first one is the fast convergence constraint to speed up the model refinement process towards the direction of a smaller distance in the tuple between (initial model, untrained data) and (final model, malicious data). The second one is the model accuracy constraint to achieve high classification accuracy for benign data samples. We balance these two constraints for fast model strengthening and good global model accuracy. We perform a comprehensive performance evaluation on two industrial steel surface datasets. Experimental results show that our model strengthening method can provide accurate malicious data removal, with speeds 11.7 × to 27.2 × faster speeds than the baseline method.
The remainder of this paper is organized as follows. Section 2 gives the related work. Section 3 introduces our method. Section 4 presents the experimental results. Finally, Section 5 concludes this paper.

2. Related Work

Security threats and model strengthening: There are emerging security threats in deep learning [1,2], such as the well-known backdoor attacks [4,5,6], membership inference attacks [7], adversarial sample attacks [8,9], data reconstruction attacks [10,11,12], etc., leading to serious information leakage or degraded system performance [13]. Previous solutions have focused on how to detect these security threats to notify the trainer. In recent years, researchers have also paid attention to the process of model strengthening. This involves looking at how to eliminate the degradation in system performance after a threat has been detected. For example, if we have already found that there are malicious data samples in the training data, how can we achieve a correct model? A simple and naive solution is to delete the malicious data samples from the training data and retrain the model, which is usually quite time-consuming due to the huge computational load. There are several solutions to address this problem in an efficient way [14,15,16,17]. For example, Cao [18] divided the training data into several groups, removed the group with malicious data samples, and updated the whole model accordingly. A similar solution is also presented in SISA [19]. Reference [20] further combined the idea of SISA and membership inference attacks to continuously change the output distributions of malicious data during model retraining. Chen [21] proposed a GAN-based model strengthening method for deep neural networks. Liu [22] focused on federated learning. The author removed the malicious data samples from the training data and retrained the model in the direction of the previous training with a larger iteration step. Brophy [23] discussed a random forest algorithm to support model retraining with a subtree. Nguyen [24] presented a Markov chain Monte Carlo algorithm for extracting data samples to estimate the posterior belief of model parameters. However, these solutions have high resource consumption due to the malicious data distribution and historical training information, making them difficult to use in a generic case.
Spiking Neural Networks (SNN): The SNN is one of the third generation of neural networks [25] with brain-inspired features. Due to the difficult training process in SNN, there have been many research efforts to provide efficient training of SNN models. For example, Deng [26] bridged the SNN and the DNN by transferring the model weights of DNNs to SNNs with almost no loss of accuracy. Diehl [27] and Dong [28] borrowed the idea of Spike Timing-Dependent Plasticity (STDP) from neural science and achieved significant performance on the MNIST vision dataset and TIDIGITS digital speech dataset with unsupervised learning. Wu [29,30] further combined the BP learning rule with the spatio-temporal features in model training to support biologically plausible interpretability and high model accuracy. Hunsberger [31] integrated spiking LIF neurons into deep neural networks, allowing well-trained static networks to be transformed into dynamic spiking networks. Recently, researchers have studied the security threats in SNNs, such as backdoor attacks [3]. However, little attention has been paid to model strengthening in SNNs. Therefore, we aim to investigate this issue, especially for industrial datasets.

3. Our Model Strengthening Method

In the area of model strengthening, we want to efficiently remove the influence of malicious data from the trained model. Let M i n i t i a l be an SNN model trained on original dataset D i n i t i a l , where D m a l i c i o u s is a subset in D i n i t i a l . To remove the influence of D m a l i c i o u s from M i n i t i a l , the simplest solution is to retrain a new model with D i n i t i a l − D m a l i c i o u s , which is the so-called retraining method. Obviously, it is not time efficient due to its large computational complexity.

3.1. Basic Idea

To support accurate and fast model strengthening, we propose that the strengthened model M f i n a l could be derived from the given M i n i t i a l to reduce computation. Inspired by the original training procedure, our idea can be described as follows.
As shown in Figure 1a, M i n i t i a l is trained on D i n i t i a l , and the untrained data (e.g.,  D u n t r a i n e d ) have little or no influence on D i n i t i a l . As in Figure 1b, after model strengthening, the malicious data should be removed from the final model M f i n a l . This means that D m a l i c i o u s has little or no impact on D f i n a l . In practice, these two items should be almost equivalent. In other words, after removing malicious data, the model output of untrained data with respect to the initial model (i.e., M i n i t i a l ( D u n t r a i n e d ) ) should be almost equivalent to the model output of malicious data with respect to the final model (i.e., M f i n a l ( D m a l i c i o u s ) ) , satisfying M i n i t i a l ( D u n t r a i n e d ) ≈ M f i n a l ( D m a l i c i o u s ) .
Similar to the generative adversarial network (GAN), we can use the element M i n i t i a l ( D u n t r a i n e d ) ) to guide the training of M f i n a l so that their outputs similar after several iterations. Thus, with suitable untrained data D u n t r a i n e d , M f i n a l can be set to M i n i t i a l at the beginning, and we then iterate in the direction of removing malicious data as the differences in M i n i t i a l ( D u n t r a i n e d ) and M f i n a l ( D m a l i c i o u s ) become smaller. Here, for simplicity, we can use the KL distance to measure their differences.

3.2. Model Strengthening Method

Inspired by Figure 1, we propose a fast model strengthening method. Given an initial model M i n i t i a l trained on D i n i t i a l , our goal is to eliminate the influence of D m a l i c i o u s and determine a final model M f i n a l from D i n i t i a l − D m a l i c i o u s .
Our goals can also be summarized as two constraints. The first one is the fast convergence constraint to speed up the model strengthening process towards a smaller distance between M i n i t i a l ( D u n t r a i n e d ) and M f i n a l ( D m a l i c i o u s ) . The second one is the model accuracy constraint, which states that the model accuracy should not be significantly affected after removing the malicious data, resulting in a high model accuracy M f i n a l ( D i n i t i a l − D m a l i c i o u s ) .
As listed in Algorithm 1, our model strengthening method can be described as follows. In Step 1, we construct untrained data D u n t r a i n e d with the same class categories as D t r a i n , where D u n t r a i n e d ∩ D t r a i n = ∅ . In Step 2, we initialize the final model with the initial model as M f i n a l = M i n i t i a l . In Step 3, we compute the model output of untrained data with respect to the initial model as M i n i t i a l ( D u n t r a i n e d ) . In Step 4, we compute the model output of malicious data with respect to the final model as M f i n a l ( D m a l i c i o u s ) . In Step 5, we compute the difference between these two model outputs as p = M i n i t i a l ( D u n t r a i n e d ) − M f i n a l ( D m a l i c i o u s ) . In Step 6, we compute the model accuracy of the remaining data with respect to the final model as q = M f i n a l ( D i n i t i a l − D m a l i c i o u s ) . In Step 7, we compute the tradeoff between the output difference and the model accuracy: λ p + ( 1 − λ ) q , where λ is a tradeoff parameter. Finally, we iterate from Step 3∼7 by a stochastic gradient descent method until convergence to determine the final model D f i n a l .
Algorithm 1 Our Model Strengthening Method.
Input: 
An initial SNN model M i n i t i a l trained on D i n i t i a l , and malicious data D m a l i c i o u s , where D m a l i c i o u s ⊂ D i n i t i a l .
Output: 
An SNN model M f i n a l trained on D i n i t i a l − D m a l i c i o u s .
1:
Construct untrained data D u n t r a i n e d with the same class categories of D t r a i n , where D u n t r a i n e d ∩ D t r a i n = ∅ .
2:
Initialize the final model: M f i n a l = M i n i t i a l .
3:
Compute the model output of untrained data with respect to the initial model: M i n i t i a l ( D u n t r a i n e d ) .
4:
Compute the model output of malicious data with respect to the final model: M f i n a l ( D m a l i c i o u s ) .
5:
Compute the difference between the two outputs: p = M i n i t i a l ( D u n t r a i n e d ) − M f i n a l ( D m a l i c i o u s ) .
6:
Compute the model accuracy: q = M f i n a l ( D i n i t i a l − D m a l i c i o u s ) .
7:
Compute the tradeoff between the output difference and the model accuracy: λ p + ( 1 − λ ) q , where λ is a tradeoff parameter.
8:
Iterate from Step 3 ∼ 7 by a stochastic gradient descent method until convergence to determine the final model D f i n a l .

4. Experiments

To illustrate the model strengthening performance of our method, we perform experiments on a real dataset, specifically the well-known NEU-CLS dataset for industrial steel surface detection. In our experiments, we run PyTorch codes on a single Tesla V100 GPU, manufactured by NVIDIA Inc., Taiwan, China.

4.1. Experiment Settings

As a baseline, we first evaluate the accuracy of the steel surface detection model with the initial dataset. In the following experiments, we use an SNN model with two convolutional layers and two fully connected layers. This model has approximately 0.3 million parameters. We also use a longer simulation time-step to ensure that all neurons are activated in the first training epoch. We use two typical industrial steel surface datasets, the NEU-CLS-64 dataset and the NEU-CLS-200 dataset.
In the NEU-CLS-64 dataset, the data distribution is unbalanced. As shown in Figure 2, the data sizes of the classes ‘cr’, ‘gg’, ‘in’, ‘pa’, ‘ps’, ‘rp’, ‘rs’, ‘sc’, and ‘sp’ are 962, 247, 605, 914, 644, 151, 1298, 617, and 342, respectively. The unbalanced data distribution leads to significant misclassification, especially for the classes with a smaller data size. To deal with this problem, we generally use an oversampling method [32,33] or an undersampling method [34,35]. The oversampling method involves deleting data samples of larger classes to balance the dataset, which leads to a serious loss of useful information and weakens the generalization capacity of the model. Therefore, we turn to the oversampling method, by randomly selecting and copying data samples from smaller classes to balance the whole dataset. Since the data oversampling method introduces duplicate samples, we also use data augmentation methods (e.g., flipping up, flipping down, flipping left, flipping right, and 90-degree rotation) to avoid potential data overfitting problems and improve the generalization capacity of the SNN. In this way, we construct the training data, the unlearned data, and the test data with the data sizes of 10,384, 1298, and 1445, respectively, as listed in Table 1. The initial SNN model with a four-layer CNN architecture achieves 92% classification accuracy on the test data.
For the NEU-CLS-200 dataset with an image size of 200 × 200, its data samples are balanced between different classes. So we skip the data oversampling process and use data augmentation directly. The corresponding data sizes of the training data, the unlearned data, and the test data are 1200, 240, and 360, respectively, as shown in Table 1. The initial SNN model achieves a classification accuracy of 87% on the test data.
For the SNN model strengthening, the learning rate is set to 5 × 10 − 4 , the mini-batch size is set to 32, and the tradeoff parameter λ is set to 0.01. For the NEU-CLS-64 dataset, the input data size is 32 × 32 , and we consider data samples with the label ‘1’ as the malicious data that should be removed from the initial model. For the NEU-CLS-200 dataset, its input data size is 32 × 32 , where data samples labeled ‘2’ should be removed. These parameters are also listed in Table 2.

4.2. Model Strengthening Accuracy

After model strengthening, malicious samples should have little or no impact on the final model. As mentioned in the previous section, we assume that all malicious samples belong to the same class (e.g., class ‘1’ in the NEU-CLS-64 dataset and class ‘2’ in the NEU-CLS-200 dataset), which is so-called the malicious class. Other classes with benign data samples are called benign classes. Table 3 and Figure 3 show the classification accuracy of our model strengthening method. For the NEU-CLS-64 dataset, the initial model provides classification accuracy of 99.31% for the malicious class and 95.65% for the benign classes. In contrast, our method gives a classification accuracy of 6.39% for the malicious class and 83.17% for the benign classes. This indicates that the malicious class is almost removed from the final model without significantly reducing the classification accuracy for the benign classes. Compared to the retraining method, which retrains the whole model after deleting all malicious data from the training data, we still have comparable classification performance (84.35% vs. 83.17%), even with an extremely fast coverage time, as shown in the next section. From the NEU-CLS-200 dataset, we can easily infer similar results with the NEU-CLS-64 dataset. In summary, the experimental results confirm the model strengthening accuracy of our method.
To investigate the detailed information, we also evaluate the F1-score, precision, and recall on the test data, as shown in Figure 4. Compared to the initial model, our method is not able to indicate the correct label for the malicious class (i.e., class ‘1’ in the NEU-CLS-64 dataset and class ‘2’ in the NEU-CLS-200 dataset). Thus, the malicious class is identified as other classes, confirming that our method can remove malicious data samples for model strengthening. Furthermore, our method provides good classification performance for other classes, showing that it can produce excellent results for benign data samples.

4.3. Model Strengthening Time Cost

We also evaluate the time cost of our method compared to the baseline. Table 4 shows the model strengthening time cost for the two datasets. We can see that our method achieves speeds 11.7 × to 27.2 × faster than the retraining method, even with comparable classification performance (e.g., 84.35% vs. 83.17% for the NEU-CLS-64 dataset, as shown in Table 3).

4.4. Performance Tradeoff

As discussed in Section 3, there is a tradeoff between classification accuracy for benign samples and removal performance for malicious samples, and thus we use a parameter λ to balance them in our algorithm. In this section, we will further discuss the impact of the tradeoff by using different λ , such as 0, 0.001, 0.01, 0.1, and 1. With a larger λ , our method pays more attention to malicious data removal than benign data classification, and vice versa. Specifically, when λ = 0 , it means that malicious data are kept in the training data and we do not perform model strengthening. When λ = 1 , it means that we want to maximize the removal of malicious data and neglect the classification performance on benign data.
The experimental results are presented in Table 5 and Table 6. After model strengthening, both the malicious and benign classes achieve stable classification accuracy when the λ varies among 0.001, 0.01, and 0.1. For example, the classification accuracy for the NEU-CLS-64 dataset is 7.16% and 7.24% for λ = 0.01 and λ = 0.1 , respectively, demonstrating the robustness of our method. We can also find that a small λ results in a longer time for model convergence.
For the extreme point of λ = 0 , there is no malicious data removal, and thus both the malicious class and the benign classes have high classification accuracy due to initial model training. In fact, it is identical to the initial model. At the extreme point of λ = 1 , we overlook the classification accuracy on the benign data, so the classification accuracy for the benign classes becomes lower (e.g., 70.05% in the NEU-CLS-64 dataset).

4.5. Model Strengthening Accuracy vs. Model Architecture

In this section, we further study how to improve the accuracy of model strengthening with different model architectures. We complete our model strengthening method on the NEU-CLS-64 dataset and record the malicious data removal performance, as listed in Table 7. We can see that the label ‘ps’ produces the best result in terms of the highest classification accuracy for benign classes (i.e., 93.01%), and the label ‘rs’ performs the worst, with the worst classification accuracy for benign classes (i.e., −52.04), even with the longest coverage time. Moreover, for the labels ‘rp’ and ‘sc’, our model strengthening method cannot really remove the malicious data samples, where the malicious class decreases quite slowly.
To investigate why different classes have very different model strengthening performance, we examine the original images of these classes. As shown in Figure 5, the images of the label ‘ps’ are very similar to each other and quite different from the images of the other labels. This means that the class ‘ps’ has a small intra-class distance and a large inter-class distance, which makes it easier to classify. In contrast, class ‘rp’ and class ‘sc’ have large intra-class distances, which leads to worse model strengthening performance.
To improve the model strengthening performance, we would like to introduce a stronger network architecture to capture more feature vectors for the original data samples. Here, we modify the basic structure of the original four-layer CNN into an eight-layer CNN with five convolutional layers and three fully connected layers, whose total number of parameters is 10.28 million. With a deeper convolutional layer, it is more likely to use more features with more filters. Thus, we train an eight-layer SNN model, which achieves a classification accuracy of 96% on the test data. We then apply our model strengthening method, specifically for the ‘rp’ class and the ‘sc’ class. As shown in Table 8, these malicious data samples can be successfully removed, with 7.78% and 4.55% classification accuracy, although the benign classes suffer from misclassification (e.g., 53.22% for ‘rp’). However, it still shows an obvious improvement in model strengthening, and we believe that a better tradeoff could be achieved with a deeper SNN model.

4.6. Performance Comparison with Two Baselines

We also compare our method with two baseline methods. The first is the retraining method, which retrains the training data after removing the malicious data. The second is the SISA method, which divides the training data into several isolated subsets and removes the subset with malicious data samples. We perform the model strengthening on the NEU-CLS-64 dataset. The experimental results are shown in Table 9. We can conclude that both retraining and SISA produce no classification error for malicious data, generally because they actually remove all malicious data directly from the training data. Our method produces a benign classification comparable to retraining, and superior to SISA. In addition, the time cost of retraining, SISA, and our method are 37.24 min, 26.8 min, and 3.19 min, respectively, indicating 8.4 × to 11.7 × improvements. Therefore, our method can produce refined model accuracy, with much faster model convergence speeds.

5. Conclusions

To eliminate the impact of malicious data samples in SNNs, we present a novel SNN model strengthening method to support fast and accurate removal of malicious data from a trained model. By constructing untrained data with the same distribution as the training data, we can use the model performance of the untrained data relative to the initial model to guide the final refined model. We also present a stochastic gradient descent method to iteratively determine the final model. Experimental results on two industrial steel surface datasets show that our model strengthening method can successfully eliminate the malicious data samples with speeds 11.7 × to 27.2 × faster than the baseline method.

Author Contributions

Conceptualization, K.C.; methodology, D.Z. and K.C.; software, D.Z.; validation, D.Z., W.C. and B.M.; formal analysis, D.Z.; investigation, D.Z.; resources, D.Z.; data curation, D.Z. and W.C.; writing—original draft preparation, D.Z. and W.C.; writing—review and editing, B.M. and K.C.; visualization, D.Z.; supervision, K.C.; project administration, K.C.; funding acquisition, K.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Natural Science Foundation of China grant number 61802383, Research Project of Pazhou Lab for Excellent Young Scholars grant number PZL2021KF0024, Guangzhou Basic and Applied Basic Research Foundation grant numbers 202201010330 and 202201020162, Guangdong Philosophy and Social Science Planning Project grant number GD19YYJ02, and Guangdong Regional Joint Fund Project grant number 2022A1515110157.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Zhu, E.; Zhang, J.; Yan, J.; Chen, K.; Gao, C. N-gram MalGAN: Evading machine learning detection via feature n-gram. Digit. Commun. Netw. 2022, 8, 485–491. [Google Scholar] [CrossRef]
  2. Hou, R.; Ai, S.; Chen, Q.; Yan, H.; Huang, T.; Chen, K. Similarity-based integrity protection for deep learning systems. Inf. Sci. 2022, 601, 255–267. [Google Scholar] [CrossRef]
  3. Abad, G.; Ersoy, O.; Picek, S.; Urbieta, A. Sneaky Spikes: Uncovering Stealthy Backdoor Attacks in Spiking Neural Networks with Neuromorphic Data. arXiv 2023, arXiv:2302.06279. [Google Scholar]
  4. Chen, K.; Zhang, H.; Feng, X.; Zhang, X.; Mi, B.; Jin, Z. Backdoor Attacks against Distributed Swarm Learning. ISA Trans. 2023; online ahead of print. [Google Scholar] [CrossRef]
  5. Wang, Y.; Chen, K.; Tan, Y.; Huang, S.; Ma, W.; Li, Y. Stealthy and Flexible Trojan in Deep Learning Framework. IEEE Trans. Dependable Secur. Comput. 2023, 20, 1789–1798. [Google Scholar] [CrossRef]
  6. Li, Y.; Yan, H.; Huang, T.; Pan, Z.; Lai, J.; Zhang, X.; Chen, K.; Li, J. Model Architecture Level Privacy Leakage in Neural Networks. Sci. China Inf. Sci. 2022. [Google Scholar]
  7. Shokri, R.; Stronati, M.; Song, C.; Shmatikov, V. Membership inference attacks against machine learning models. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA, 22–26 May 2017; IEEE: New York, NY, USA, 2017; pp. 3–18. [Google Scholar]
  8. Xiao, Y.; Zhou, J.; Chen, K.; Liu, Z. Revisiting the Transferability of Adversarial Examples via Source-agnostic Adversarial Feature Inducing Method. Pattern Recognit. 2023, 144, 109828. [Google Scholar] [CrossRef]
  9. Liu, J.; Zhang, Q.; Mo, K.; Xiang, X.; Li, J.; Cheng, D.; Gao, R.; Liu, B.; Chen, K.; Wei, G. An efficient adversarial example generation algorithm based on an accelerated gradient iterative fast gradient. Comput. Stand. Interfaces 2022, 82, 103612. [Google Scholar] [CrossRef]
  10. Zhu, L.; Liu, Z.; Han, S. Deep leakage from gradients. Adv. Neural Inf. Process. Syst. 2019, 32, 17–31. [Google Scholar]
  11. Zhang, X.; Li, J.; Zhang, J.; Yan, J.; Zhu, E.; Chen, K. Data Reconstruction from Gradient Updates in Federated Learning. In Proceedings of the Machine Learning for Cyber Security—4th International Conference, ML4CS 2022, Guangzhou, China, 2–4 December 2022; Xu, Y., Yan, H., Teng, H., Cai, J., Li, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2022; Volume 13655, pp. 586–596, Part I. [Google Scholar]
  12. Zhang, X.; Zhou, X.; Chen, K. Data Leakage with Label Reconstruction in Distributed Learning Environments. In Proceedings of the Machine Learning for Cyber Security—4th International Conference, ML4CS, Guangzhou, China, 2–4 December 2022; Xu, Y., Yan, H., Teng, H., Cai, J., Li, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2022; Volume 13655, pp. 185–197, Part I. [Google Scholar]
  13. Chen, K.; Zhang, X.; Zhou, X.; Mi, B.; Xiao, Y.; Zhou, L.; Wu, Z.; Wu, L.; Wang, X. Privacy Preserving Federated Learning for Full Heterogeneity. ISA Trans. 2023; online ahead of print. [Google Scholar] [CrossRef]
  14. Regulation, P. General data protection regulation. Intouch 2018, 25, 1–5. [Google Scholar]
  15. Chen, K.; Wang, Y.; Huang, Y. Lightweight machine unlearning in neural network. arXiv 2021, arXiv:2111.05528. [Google Scholar]
  16. Regulation, P. Regulation (EU) 2016/679 of the European Parliament and of the Council. Regul. EU 2016, 679, 2016. [Google Scholar]
  17. Chen, K.; Huang, Y.; Wang, Y. Machine unlearning via GAN. arXiv 2021, arXiv:2111.11869. [Google Scholar]
  18. Cao, Y.; Yang, J. Towards making systems forget with machine unlearning. In Proceedings of the 2015 IEEE Symposium on Security and Privacy, San Jose, CA, USA, 17–21 May 2015; IEEE: New York, NY, USA, 2015; pp. 463–480. [Google Scholar]
  19. Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C.A.; Jia, H.; Travers, A.; Zhang, B.; Lie, D.; Papernot, N. Machine unlearning. In Proceedings of the 2021 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, 24–27 May 2021; IEEE: New York, NY, USA, 2021; pp. 141–159. [Google Scholar]
  20. Felps, D.L.; Schwickerath, A.D.; Williams, J.D.; Vuong, T.N.; Briggs, A.; Hunt, M.; Sakmar, E.; Saranchak, D.D.; Shumaker, T. Class Clown: Data Redaction in Machine Unlearning at Enterprise Scale. arXiv 2020, arXiv:2012.04699. [Google Scholar]
  21. Chen, K.; Huang, Y.; Wang, Y.; Zhang, X.; Mi, B.; Wang, Y. Privacy Preserving Machine Unlearning for Smart Cities. Ann. Telecommun. 2023; online ahead of print. [Google Scholar] [CrossRef]
  22. Liu, G.; Ma, X.; Yang, Y.; Wang, C.; Liu, J. Federated unlearning. arXiv 2020, arXiv:2012.13891. [Google Scholar]
  23. Brophy, J.; Lowd, D. Machine unlearning for random forests. In Proceedings of the International Conference on Machine Learning; PMLR: Baltimore, MA, USA, 2021; pp. 1092–1104. [Google Scholar]
  24. Nguyen, Q.P.; Oikawa, R.; Divakaran, D.M.; Chan, M.C.; Low, B.K.H. Markov chain monte carlo-based machine unlearning: Unlearning what needs to be forgotten. arXiv 2022, arXiv:2202.13585. [Google Scholar]
  25. Maass, W. Networks of spiking neurons: The third generation of neural network models. Neural Netw. 1997, 10, 1659–1671. [Google Scholar] [CrossRef]
  26. Deng, S.; Gu, S. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv 2021, arXiv:2103.00476. [Google Scholar]
  27. Diehl, P.U.; Cook, M. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci. 2015, 9, 99. [Google Scholar] [CrossRef] [PubMed]
  28. Dong, M.; Huang, X.; Xu, B. Unsupervised speech recognition through spike-timing-dependent plasticity in a convolutional spiking neural network. PloS ONE 2018, 13, e0204596. [Google Scholar] [CrossRef]
  29. Wu, Y.; Deng, L.; Li, G.; Zhu, J.; Shi, L. Spatio-temporal backpropagation for training high-performance spiking neural networks. Front. Neurosci. 2018, 12, 331. [Google Scholar] [CrossRef] [PubMed]
  30. Wu, Y.; Deng, L.; Li, G.; Zhu, J.; Xie, Y.; Shi, L. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; Volume 33, pp. 1311–1318. [Google Scholar]
  31. Hunsberger, E.; Eliasmith, C. Spiking deep networks with LIF neurons. arXiv 2015, arXiv:1510.08829. [Google Scholar]
  32. Chawla, N.; Bowyer, K.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res. 2002, 16, 321–357. [Google Scholar] [CrossRef]
  33. Han, H.; Wang, W.; Mao, B. Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning. In Proceedings of the International Conference on Intelligent Computing, Hefei, China., 23–26 August 2005. [Google Scholar]
  34. Kubát, M.; Matwin, S. Addressing the Curse of Imbalanced Training Sets: One-Sided Selection. In Proceedings of the International Conference on Machine Learning, Nashville, TN, USA, 8–12 July 1997. [Google Scholar]
  35. Batista, G.E.A.P.A.; Prati, R.C.; Monard, M.C. A study of the behavior of several methods for balancing machine learning training data. SIGKDD Explor. 2004, 6, 20–29. [Google Scholar] [CrossRef]
Electronics 12 03845 g001
Figure 1. The comparison of the untrained data and the malicious data.
Figure 1. The comparison of the untrained data and the malicious data.
Electronics 12 03845 g001
Electronics 12 03845 g002
Figure 2. Data distribution in the NEU-CLS-64 dataset.
Figure 2. Data distribution in the NEU-CLS-64 dataset.
Electronics 12 03845 g002
Electronics 12 03845 g003
Figure 3. Model strengthening accuracy.
Figure 3. Model strengthening accuracy.
Electronics 12 03845 g003
Electronics 12 03845 g004
Figure 4. Model strengthening performance.
Figure 4. Model strengthening performance.
Electronics 12 03845 g004
Electronics 12 03845 g005
Figure 5. The original images under different labels.
Figure 5. The original images under different labels.
Electronics 12 03845 g005
Table 1. Data partitioning for the untrained data, the training data, and the test data.
Table 1. Data partitioning for the untrained data, the training data, and the test data.
Untrained DataTraining DataTest Data
NEU-CLS-64129810,3841445
NEU-CLS-2002401200360
Table 2. Parameters for the SNN model strengthening.
Table 2. Parameters for the SNN model strengthening.
DatasetLearning RateMini-Batch Size λ Input Data SizeClass
NEU-CLS-64  5 × 10 − 4 320.01 64 × 64 1
NEU-CLS-200  5 × 10 − 4 320.01 128 × 128 2
Table 3. Model strengthening accuracy for different datasets.
Table 3. Model strengthening accuracy for different datasets.
DatasetClassesInitial ModelRetrainingOurs
NEU-CLS-64malicious class99.3106.39
benign classes95.6584.3583.17
NEU-CLS-200malicious class97.0807.5
benign classes89.582.5877.5
Table 4. Model strengthening time cost.
Table 4. Model strengthening time cost.
DatasetsRetraining (min)Ours (min)Speed Up
NEU-CLS-6437.243.19 11.7 ×
NEU-CLS-20013.610.50 27.2 ×
Table 5. Tradeoff in model strengthening for the NEU-CLS-64 dataset.
Table 5. Tradeoff in model strengthening for the NEU-CLS-64 dataset.
Malicious ClassBenign ClassesTime Cost (min)
λ = 0 99.3195.64∞
λ = 0.001 6.8685.323.08
λ = 0.01 7.1684.942.79
λ = 0.1 7.2484.912.86
λ = 1 7.5570.042.83
Table 6. Tradeoff in model strengthening for the NEU-CLS-200 dataset.
Table 6. Tradeoff in model strengthening for the NEU-CLS-200 dataset.
Malicious ClassBenign ClassesTime Cost (min)
λ = 0 96.6789.75∞
λ = 0.001 6.2577.670.76
λ = 0.01 7.0877.830.58
λ = 0.1 7.0877.750.58
λ = 1 6.6777.830.56
Table 7. Model strengthening accuracy for different labels.
Table 7. Model strengthening accuracy for different labels.
LabelClassInitial ModelOurTime Cost (min)
‘cr’malicious class99.776.242.2441
benign classes95.5884.07
‘gg’malicious class99.316.393.1923
benign classes95.6583.17
‘pa’malicious class99.777.74.1807
benign classes95.5859.37
‘ps’malicious class92.227.551.4699
benign classes95.6593.01
‘rs’malicious class99.236.787.4573
benign classes95.6552.04
‘sp’malicious class83.983.621.3547
benign classes97.5576.87
Table 8. Model strengthening accuracy with an eight-layer SNN model.
Table 8. Model strengthening accuracy with an eight-layer SNN model.
LabelClassInitial ModelOursTime Cost (min)
‘rp’malicious class1007.7827.67
benign classes99.4753.22
‘sc’malicious class1004.5542.49
benign classes99.4730.55
Table 9. Performance comparison with two baselines.
Table 9. Performance comparison with two baselines.
ClassesInitial ModelRetrainingSISAOurs
malicious class99.31006.39
benign classes95.6584.3578.7583.17
time cost/37.2426.83.19
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Zhou, D.; Chen, W.; Chen, K.; Mi, B. Fast and Accurate SNN Model Strengthening for Industrial Applications. Electronics 2023, 12, 3845. https://doi.org/10.3390/electronics12183845

AMA Style

Zhou D, Chen W, Chen K, Mi B. Fast and Accurate SNN Model Strengthening for Industrial Applications. Electronics. 2023; 12(18):3845. https://doi.org/10.3390/electronics12183845

Chicago/Turabian Style

Zhou, Deming, Weitong Chen, Kongyang Chen, and Bing Mi. 2023. "Fast and Accurate SNN Model Strengthening for Industrial Applications" Electronics 12, no. 18: 3845. https://doi.org/10.3390/electronics12183845

APA Style

Zhou, D., Chen, W., Chen, K., & Mi, B. (2023). Fast and Accurate SNN Model Strengthening for Industrial Applications. Electronics, 12(18), 3845. https://doi.org/10.3390/electronics12183845

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop