Abstract
Anomaly detection is a binary classification task, which is to judge whether the input image contains an anomaly or not and the difficulty is that only normal samples are available at training. Due to this unsupervised nature, the classic supervised classification methods will fail. Knowledge distillation-based methods for unsupervised anomaly detection have recently drawn attention as it has shown the outstanding performance. In this paper, we present a novel knowledge distillation-based approach for anomaly detection (RKDAD). We propose to use the “distillation” of the “FSP matrix” from adjacent layers of a teacher network, pre-trained on ImageNet, into a student network which has the same structure as the teacher network to solve the anomaly detection problem, we show that the “FSP matrix” are more discriminative features for normal and abnormal samples than other traditional features like the latent vectors in autoencoders. The “FSP matrix” is defined as the inner product between features from two layers and we detect anomalies using the discrepancy between teacher’s and student’s corresponding “FSP matrix”. To the best of our knowledge, it is the first work to use the relation-based knowledge distillation framework to solve the unsupervised anomaly detection task. We show that our method can achieve competitive results compared to the state-of-the-art methods on MNIST, F-MNIST and surpass the state-of-the-art results on the object images in MVTecAD.
This research was supported by NSFC (No. 61871074).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Shuang, M., Yudan, W., Guojun, W.: Automatic fabric defect detection with a multi-scale convolutional denoising autoencoder network model. Sensors 18(4), 1064 (2018)
Bergmann P., Fauser M., Sattlegger D., et al.: MVTec AD — a Comprehensive Real-World dataset for unsupervised anomaly detection. In: 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9592–9600 (2019)
Zhe L., et al.: Thoracic disease identification and localization with limited supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8290–8299 (2018)
Zhou, J.T., Du, J., Zhu, H., et al.: AnomalyNet: an anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. 14(10), 2537–2550 (2019)
Bergmann P., Fauser M., Sattlegger D., et al.: Uninformed students: student-teacher anomaly detection with discriminative latent embeddings. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4182–4191 (2020)
Salehi M., Sadjadi N., Baselizadeh S., et al.: Multiresolution knowledge distillation for anomaly detection. arXiv preprint arXiv: 2011.11108 (2020)
Abati D., Porrello A., Calderara S., Cucchiara R.: Latent space autoregression for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 481–490 (2019)
Gong D., Liu L., Le V., et al.: Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1705–1714 (2019)
Venkataramanan, S., Peng, K.-C., Singh, R.V., Mahalanobis, A.: Attention guided anomaly localization in images. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12362, pp. 485–503. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58520-4_29
Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 146–157. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_12
Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 622–637. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_39
Akay, S., Atapour-Abarghouei, A., Breckon, T.P.: Skip-GANomaly: skip connected and adversarially trained encoder-decoder anomaly detection. In: IEEE International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Yim, J., Joo, D., Bae, J., et al.: A Gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7130–7138 (2017)
Jia, D., Wei, D., Richard, S., Li-Jia, L., Kai, L., Fei-Fei L.: Imagenet: a large-scale hierarchical imagedatabase. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks For Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014)
Diederik, P.K., Jimmy, B.: Adam: a method forstochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
LeCun, Y., Cortes, C., et al. http://yann.lecun.com/exdb/mnist. Accessed 05 April 2021
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a Novel Image Dataset For Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747 (2017)
Salehi, M., Arya, A., Pajoum, B., et al.: Arae: Adversarially robust training of autoencoders improves novelty detection. arXiv preprint arXiv:2003.05669 (2020)
Chen, Y., Xiang, S.Z., Huang, T.S.: One-class svm for learning in image retrieval. In: Proceedings 2001 International Conference on Image Processing, pp. 34–37 (2001)
Ruff, L., Vandermeulen, R.A., et al.: Deep one-class classification. In: International Conference on Machine Learning, pp. 4393–4402 (2018)
Li, X., Kiringa, I., Yeap, T., et al.: Exploring deep anomaly detection methods based on capsule net. In: ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning, pp. 375–387 (2020)
Perera, P., Nallapati, R., Bing, X.: Ocgan: one-class novelty detection using gans with constrained latent representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2906 (2019)
Zong, B., Song, Q., Martin Renqiang, M., et al.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International Conference on Learning Representations (2018)
Shuangfei, Z., Yu, C., Weining, L., Zhongfei, Z.: Deep structured energy-based models for anomaly detection. In: International Conference on Machine Learning, pp. 1100–1109 (2016)
Sabokrou, M., Pourreza, M., Fayyaz, M., et al.: Avid: adversarial visual irregularity detection. In: Asian Conference on Computer Vision, pp. 488–505 (2018)
Bergmann, P., Lwe, S., Fauser, M., et al.: Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In: International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), pp. 372–180 (2019)
Dehaene, D., Frigo, O., Combrexelle, S., et al.: Iterative energy-based projection on a normal data manifold for anomaly localization. In: International Conference on Learning Representations (2020)
Golan, I., El-Yaniv, R.: Deep anomaly detection using geometric transformations. In: Advances in Neural Information Processing Systems, pp. 9758–9769 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Cheng, H., Yang, L., Liu, Z. (2021). Relation-Based Knowledge Distillation for Anomaly Detection. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13019. Springer, Cham. https://doi.org/10.1007/978-3-030-88004-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-88004-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88003-3
Online ISBN: 978-3-030-88004-0
eBook Packages: Computer ScienceComputer Science (R0)