Abstract
The emergence of the Internet of Things and stream processing forces large scale organizations to consider anomaly detection as a key component of their business. Using machine learning to solve such complex use cases is generally a cumbersome, costly, time-consuming and error-prone process. It involves many tasks from data cleansing, to dimension reduction, algorithm selection and fine tuning. It also requires the involvement of various experts such as statisticians, programmers and testers. With RAMSSES, we remove the burden of this pipeline and demonstrate that these tasks can be automated. Our system leverages on a Lambda architecture based on Apache Spark to analyze historical data, perform cleansing and deal with the curse of dimensionality. Then, it identifies the most interesting attributes and uses a continuous semantic query generator executed over streams. The sampled data are processed by self-selected machine learning methods to detect anomalies, an iterative process using end user annotations improves significantly the accuracy of the system. After a description of RAMSSES’s main components, the performance and relevancy of the system are demonstrated via a thorough evaluation over real-world and synthetic datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41, 15:1–15:58 (2009)
Nathan Marz, J.W.: Big Data Principles and best practices of scalable realtime data systems. Manning, 1st edn. (2015)
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST 2010, Washington, DC, USA, pp. 1–10. IEEE Computer Society (2010)
Vavilapalli, V.K.: Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC 2013, New York, NY, USA, pp. 5:1–5:16. ACM (2013)
Boutsidis, C., Mahoney, M.W., Drineas, P.: Unsupervised feature selection for principal components analysis. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2008, pp. 61–69. ACM (2008)
S. C. from the ITISE Conference, Time Series Analysis and Forecasting. Springer (2016)
Aggarwal, C.C., Reddy, C.K.: Data Clustering: Algorithms and Applications. Chapman & Hall/CRC, 1st edn. (2013)
Laptev, N., Amizadeh, S., Flint, I.: Generic and scalable framework for automated time-series anomaly detection. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1939–1947. ACM (2015)
Box, G.E.P., Jenkins, G.: Time Series Analysis. Forecasting and Control. Holden-Day, Incorporated (1990)
Trebuna, J.P., Fil’o, M.: The importance of normalization and standardization in the process of clustering. In: Proceedings of IEEE 12th International Symposium on Applied Machine Intelligence and Informatics, SAMI (2014)
Yazici, B., Asma, S.: A comparison of various tests of normality, vol. 77, pp. 175–183, February 2007
Goldstein, M.: Unsupervised anomaly detection benchmark (2015)
YAHOO, S5 - a labeled anomaly detection dataset (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Belabbess, B., Bairat, M., Lhez, J., Curé, O. (2018). Combining Machine Learning and Semantics for Anomaly Detection. In: Faron Zucker, C., Ghidini, C., Napoli, A., Toussaint, Y. (eds) Knowledge Engineering and Knowledge Management. EKAW 2018. Lecture Notes in Computer Science(), vol 11313. Springer, Cham. https://doi.org/10.1007/978-3-030-03667-6_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-03667-6_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03666-9
Online ISBN: 978-3-030-03667-6
eBook Packages: Computer ScienceComputer Science (R0)