Abstract
Despite the success of deep learning in various domains such as natural language processing, speech recognition, and computer vision, learning from a limited amount of samples and generalizing to unseen data still pose challenges. Notably, in the tasks of outlier detection and imbalanced dataset classification, the label of interest is either scarce or its distribution is skewed, causing aggravated generalization problems. In this work, we pursue the direction of multi-task learning, specifically the idea of using supervised autoencoders (SAE), which allows us to combine unsupervised and supervised objectives in an end to end fashion. We extend this approach by introducing an adversarial supervised objective to enrich the representations which are learned for the classification task. We conduct thorough experiments on a broad range of tasks, including outlier detection, novelty detection, and imbalanced classification, and study the efficacy of our method against standard baselines using autoencoders. Our work empirically shows that the SAE methods outperform one class autoencoders, adversarially trained autoencoders and multi layer perceptrons in terms of AUPR score comparison. Additionally, our analysis of the obtained representations suggests that the adversarial reconstruction loss functions enforce the encodings to separate into class-specific clusters, which was not observed for non-adversarial reconstruction loss functions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aggarwal, C.C.: Outlier Analysis (2017)
Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 622–637. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_39
Boyd, K., Eng, K.H., Page, C.D.: Area under the precision-recall curve: point estimates and confidence intervals. In: Blockeel, H., Kersting, K., Nijssen, S., Železný, F. (eds.) ECML PKDD 2013. LNCS (LNAI), vol. 8190, pp. 451–466. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40994-3_29
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3(1) (2011)
Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in CNNs. Neural Netw. 106, 249–259 (2018)
Cardie, C., Howe, N.: Improving minority class prediction using case-specific feature weights (1997)
Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)
Chalapathy, R., Menon, A.K., Chawla, S.: Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360 (2018)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 1–58 (2009)
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321321–357 (2002)
Chen, J., Sathe, S., Aggarwal, C., Turaga, D.: Outlier detection with autoencoder ensembles. In: Proceedings of the SIAM International Conference on Data Mining (2017)
Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: Proceedings of the International Symposium on Neural Networks (2017)
Dau, H.A., Ciesielski, V., Song, A.: Anomaly detection using replicator neural networks trained on examples of one class. In: Dick, G., Tang, K. (eds.) SEAL 2014. LNCS, vol. 8886, pp. 311–322. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-13563-2_27
Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006)
Divekar, A., Parekh, M., Savla, V., Mishra, R., Shirole, M.: Benchmarking datasets for anomaly-based network intrusion detection: KDD CUP 99 alternatives. In: Proceedings of 3rd International Conference on Computing, Communication and Security (ICCCS) (2018)
Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)
Gao, S., Zhang, Y., Jia, K., Lu, J., Zhang, Y.: Single sample face recognition via learning deep supervised autoencoders. IEEE Trans. Inf. Forensics Secur. 10(10), 2108–2118 (2015)
Gogoi, P., Borah, B., Bhattacharyya, D., Kalita, J.: Outlier identification using symmetric neighborhoods. Procedia Technol. 6, 239–246 (2012)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing Adversarial Examples. arXiv preprint arXiv:1412.6572 (2014)
Hautamaki, V., Karkkainen, I., Franti, P.: Outlier detection using k-nearest neighbour graph. In: Proceedings of the 17th International Conference on Pattern Recognition, vol. 3 (2004)
Hawkins, D.M.: Identification of Outliers. Springer, Dordrecht (1980). https://doi.org/10.1007/978-94-015-3994-4
Hawkins, S., He, H., Williams, G., Baxter, R.: Outlier detection using replicator neural networks. In: Kambayashi, Y., Winiwarter, W., Arikawa, M. (eds.) DaWaK 2002. LNCS, vol. 2454, pp. 170–180. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46145-0_17
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: Proceedings of International Conference on Learning Representations (2017)
Huang, C., Li, Y., Change Loy, C., Tang, X.: Learning deep representation for imbalanced classification. In: Proceedings of Conference on Computer Vision and Pattern Recognition (2016)
Ishii, Y., Takanashi, M.: Low-cost unsupervised outlier detection by autoencoders with robust estimation. J. Inf. Process. 27, 335–339 (2019)
Japkowicz, N., Myers, C., Gluck, M., et al.: A novelty detection approach to classification. In: Proceedings of the International Joint Conference on Artificial Intelligence (1995)
Joachims, T.: Text categorization with Support Vector Machines: learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0026683
Kannan, R., Woo, H., Aggarwal, C.C., Park, H.: Outlier detection for text data. In: Proceedings of the International Conference on Data Mining (2017)
Kawaguchi, K., Kaelbling, L.P., Bengio, Y.: Generalization in deep learning. arXiv preprint arXiv:1710.05468 (2017)
Kubat, M., Holte, R.C., Matwin, S.: Machine learning for the detection of oil spills in satellite radar images. Machine Learn. 30, 195–215 (1998)
Kukar, M., Kononenko, I., et al.: Cost-sensitive learning with neural networks. In: Proceedings of European Conference on Artificial Intelligence (1998)
Lawrence, S., Burns, I., Back, A., Tsoi, A.C., Giles, C.L.: Neural network classification and prior class probabilities. In: Neural Networks: Tricks of the Trade (1998)
Le, L., Patterson, A., White, M.: Supervised autoencoders: improving generalization performance with unsupervised regularizers. In: Proceedings of Neural Information Processing Systems
Le, L., Patterson, A., White, M.: Supervised autoencoders: improving generalization performance with unsupervised regularizers. In: Advances in Neural Information Processing Systems (2018)
Lee, H., Cho, S.: The novelty detection approach for different degrees of class imbalance. In: King, I., Wang, J., Chan, L.-W., Wang, D.L. (eds.) ICONIP 2006. LNCS, vol. 4233, pp. 21–30. Springer, Heidelberg (2006). https://doi.org/10.1007/11893257_3
Liu, T., Tao, D., Song, M., Maybank, S.J.: Algorithm-dependent generalization bounds for multi-task learning. IEEE Trans. Pattern Analy. Mach. Intell. 39(2), 227–241 (2016)
Lübbering, M., Ramamurthy, R., Gebauer, M., Bell, T., Sifa, R., Bauckhage, C.: From imbalanced classification to supervised outlier detection problems: adversarially trained auto encoders. In: Farkaš, I., Masulli, P., Wermter, S. (eds.) ICANN 2020. LNCS, vol. 12396, pp. 27–38. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61609-0_3
Mazurowski, M.A., Habas, P.A., Zurada, J.M., Lo, J.Y., Baker, J.A., Tourassi, G.D.: Training neural network classifiers for medical decision making: the effects of imbalanced datasets on classification performance. Neural Netw. 21, 427–436 (2008)
McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N.: Exploring generalization in deep learning. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 5947–5956 (2017)
Olszewski, D.: A probabilistic approach to fraud detection in telecommunications. Knowl.-Based Syst. 26, 246–258 (2012)
Panigrahi, S., Kundu, A., Sural, S., Majumdar, A.K.: Credit card fraud detection: a fusion approach using Dempster-Shafer theory and Bayesian learning. Inf. Fusion 10(4), 354–363 (2009)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
Perera, P., Nallapati, R., Xiang, B.: OCGAN: one-class novelty detection using GANs with constrained latent representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2906 (2019)
Ranzato, M., Szummer, M.: Semi-supervised learning of compact document representations with deep networks. In: Proceedings of International Conference on Machine learning (2008)
Sarvari, H., Domeniconi, C., Prenkaj, B., Stilo, G.: Unsupervised boosting-based autoencoder ensembles for outlier detection. arXiv preprint arXiv:1910.09754 (2019)
Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 146–157. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_12
Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Optimization, and Beyond, Regularization (2001)
Tavallaee, M., Bagheri, E., Lu, W., Ghorbani, A.A.: A detailed Analysis of the KDD CUP 99 Data Set. In: Proceedings of IEEE Symposium on Computational Intelligence for Security and Defense Applications (2009)
Vu, H.S., Ueta, D., Hashimoto, K., Maeno, K., Pranata, S., Shen, S.M.: Anomaly detection with adversarial dual autoencoders. arXiv preprint arXiv:1902.06924 (2019)
Wang, X., Du, Y., Lin, S., Cui, P., Yang, Y.: Self-adversarial variational autoencoder with gaussian anomaly prior distribution for anomaly detection. CoRR, abs/1903.00904 (2019)
Weston, J., Ratle, F., Mobahi, H., Collobert, R.: Deep learning via semi-supervised embedding. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 639–655. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_34
Yang, Y., Liu, X.: A re-examination of text categorization methods. In: Proceedings of International Conference on Research and development in Information Retrieval (1999)
Zeiler, M.D.: ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012)
Zenati, H., Foo, C.S., Lecouat, B., Manek, G., Chandrasekhar, V.R.: Efficient GAN-based anomaly detection. arXiv preprint arXiv:1802.06222 (2018)
Zhou, C., Paffenroth, R.C.: Anomaly detection with robust deep autoencoders. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining (2017)
Zou, J., Zhang, J., Jiang, P.: Credit Card Fraud Detection Using Autoencoder Neural Network. arXiv preprint arXiv:1908.11553 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lübbering, M., Gebauer, M., Ramamurthy, R., Sifa, R., Bauckhage, C. (2021). Supervised Autoencoder Variants for End to End Anomaly Detection. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_44
Download citation
DOI: https://doi.org/10.1007/978-3-030-68790-8_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68789-2
Online ISBN: 978-3-030-68790-8
eBook Packages: Computer ScienceComputer Science (R0)