Abstract
During the past decade, many anomaly detection approaches have been introduced in different fields such as network monitoring, fraud detection, and intrusion detection. However, they require understanding of data pattern and often need a long off-line period to build a model or network for the target data. Providing real-time and proactive anomaly detection for streaming time series without human intervention and domain knowledge is highly valuable since it greatly reduces human effort and enables appropriate countermeasures to be undertaken before a disastrous damage, failure, or other harmful event occurs. However, this issue has not been well studied yet. To address it, this paper proposes RePAD, which is a Real-time Proactive Anomaly Detection algorithm for streaming time series based on Long Short-Term Memory (LSTM). RePAD utilizes short-term historical data points to predict and determine whether or not the upcoming data point is a sign that an anomaly is likely to happen in the near future. By dynamically adjusting the detection threshold over time, RePAD is able to tolerate minor pattern change in time series and detect anomalies either proactively or on time. Experiments based on two time series datasets collected from the Numenta Anomaly Benchmark demonstrate that RePAD is able to proactively detect anomalies and provide early warnings in real time without human intervention and domain knowledge.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lavin, A., Ahmad, S.: Evaluating real-time anomaly detection algorithms – the Numenta anomaly benchmark. In: 14th International Conference on Machine Learning and Applications (IEEE ICMLA 2015) (2015)
Hochenbaum, J., Vallis, O.S., Kejariwal, A.: Automatic anomaly detection in the cloud via statistical learning. arXiv preprint arXiv:1704.07706 (2017)
Aggarwal, C.C., Yu, P.S.: Outlier detection with uncertain data. In: Proceedings of the 2008 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, pp. 483–493 (2008)
Xu, J., Shelton, C.R.: Intrusion detection using continuous time Bayesian networks. J. Artif. Intell. Res. 39, 745–774 (2010)
Fisher, W.D., Camp, T.K., Krzhizhanovskaya, V.V.: Crack detection in earth dam and levee passive seismic data using support vector machines. Procedia Comput. Sci. 80, 577–586 (2016)
Wu, J., Zeng, W., Yan, F.: Hierarchical temporal memory method for time-series-based anomaly detection. Neurocomputing 273, 535–546 (2018)
Staudemeyer, R.C.: Applying long short-term memory recurrent neural networks to intrusion detection. S. Afr. Comput. J. 56(1), 136–154 (2015)
Bontemps, L., McDermott, J., Le-Khac, N.A.: Collective anomaly detection based on long short-term memory recurrent neural networks. In: International Conference on Future Data and Security Engineering, pp. 141–152. Springer, Cham, November 2016
Lee, T.J., Gottschlich, J., Tatbul, N., Metcalf, E., Zdonik, S.: Greenhouse: a zero-positive machine learning system for time-series anomaly detection. arXiv preprint arXiv:1801.03168 (2018)
Ma, X., Tao, Z., Wang, Y., Yu, H., Wang, Y.: Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Transp. Res. Part C: Emerg. Technol. 54, 187–197 (2015)
Yu, R., Li, Y., Shahabi, C., Demiryurek, U., Liu, Y.: Deep learning: a generic approach for extreme condition traffic forecasting. In: Proceedings of the 2017 SIAM International Conference on Data Mining, pp. 777–785. Society for Industrial and Applied Mathematics (2017)
Zhao, Z., Chen, W., Wu, X., Chen, P.C., Liu, J.: LSTM network: a deep learning approach for short-term traffic forecast. IET Intell. Transp. Syst. 11(2), 68–75 (2017)
Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertainty Fuzziness Knowl.-Based Syst. 6(2), 107–116 (1998)
What is early stopping. https://deeplearning4j.org/docs/latest/deeplearning4j-nn-early-stopping. Accessed 04 June 2019
Twitter/AnomalyDetection. https://github.com/twitter/AnomalyDetection
Zou, N., Wang, J., Chang, G.L., Paracha, J.: Application of advanced traffic information systems: field test of a travel-time prediction system with widely spaced detectors. Transp. Res. Rec. 2129(1), 62–72 (2009)
Numenta, Inc.: NAB: Numenta Anomaly Benchmark. Numenta, Inc., Redwood City, CA (2015). https://github.com/numenta/NAB
detect_anoms.R. https://github.com/twitter/AnomalyDetection/blob/master/R/detect_anoms.R
Lin, J.-C., Lee, M.-C.: Performance evaluation of job schedulers under Hadoop YARN. Concurr. Comput.: Pract. Exp. (CCPE) 28(9), 2711–2728 (2016)
Lee, M.-C., Lin, J.-C., Yahyapour, R.: Hybrid job-driven scheduling for virtual MapReduce clusters. IEEE Trans. Parallel Distrib. Syst. (TPDS) 27(6), 1687–1699 (2016)
Simula Research Laboratory AS eX3 research cluster. https://www.ex3.simula.no
Lee, M.-C., Lin, J.-C.: DALC: distributed automatic LSTM customization for fine-grained traffic speed prediction. In: Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA 2020). Springer. https://arxiv.org/abs/2001.09821
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Acknowledgments
This work was supported by the project eX3 - Experimental Infrastructure for Exploration of Exascale Computing funded by the Research Council of Norway under contract 270053 and the scholarship under project number 80430060 supported by Norwegian University of Science and Technology.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Lee, MC., Lin, JC., Gran, E.G. (2020). RePAD: Real-Time Proactive Anomaly Detection for Time Series. In: Barolli, L., Amato, F., Moscato, F., Enokido, T., Takizawa, M. (eds) Advanced Information Networking and Applications. AINA 2020. Advances in Intelligent Systems and Computing, vol 1151. Springer, Cham. https://doi.org/10.1007/978-3-030-44041-1_110
Download citation
DOI: https://doi.org/10.1007/978-3-030-44041-1_110
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44040-4
Online ISBN: 978-3-030-44041-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)