Anomaly Detection

A method is presented for detecting changes to the distribution of a criminal or terrorist point process between two time periods using a non-model-based approach. By treating the criminal/terrorist point process as an intelligent site selection problem, changes to the process can signify changes in the behavior or activity level of the criminals/terrorists. The locations of past events and an associated vector of geographic, environmental, and socio-economic feature values are employed in the analysis. By modeling the locations of events in each time period as a marked point process, we can then detect differences in the intensity of each component process. A modified PRIM (patient rule induction method) is implemented to partition the high-dimensional feature space, which can include mixed variables, into the most likely change regions. Monte Carlo simulations are easily and quickly generated under random relabeling to test a scan statistic for significance. By detecting local regions of change, not only can it be determined if change has occurred in the study area, but the specific spatial regions where change occurs is also identified. An example is provided of breaking and entering crimes over two-time periods to demonstrate the use of this technique for detecting local regions of change. This methodology also applies to detecting regions of differences between two types of events such as in case–control data.

With the increasing number of computers being connected to the Internet, security of an information system has never been more urgent. Because no system can be absolutely secure, the timely and accurate detection of intrusions is necessary. This is the reason of an entire area of research, called Intrusion Detection Systems (IDS). Anomaly systems detect intrusions by searching for an abnormal system activity. But the main problem of anomaly detection IDS is that; it is very difficult to build, because of the difficulty in defining what is normal and what is abnormal. Neural network with its ability of learning has become one of the most promising techniques to solve this problem. This paper presents an overview of neural networks and their use in building anomaly intrusion systems.

An implemented system for on-line analysis of multiple distributed data streams is presented. The system is conceptually universal since it does not rely on any particular platform feature and uses format adaptors to translate data streams into its own standard format. The system is as powerful as possible (from a theoretical standpoint) but still efficient enough for on-line analysis thanks to its novel rule-based language (RUSSEL) which is specifically designed for efficient processing of sequential unstructured data streams. The generic concepts are applied to security audit trail analysis. The resulting system provides powerful network security monitoring and sophisticated tools for intrusion/anomaly detection. The rule-based and command languages are described as well as the distributed architecture and the implementation. Performance measurements are reported, showing the effectiveness of the approach

Deregulation, cyber-terrorism, and increased interdependency are making large complex critical infrastructures, such as the telecommunications and electricity networks, increasingly vulnerable. Solutions are needed that can provide a rapid automatic response to the known and unknown dangers that threaten them today. This paper outlines the work that is being done at Queen Mary, University of London on the design of an agent-based anomaly detection and repair system that will address this problem. This will build up a model of normality for the telecommunications management network, interact with existing protection mechanisms, diagnose problems and carry out self-healing. The layered safeguards that will be offered by this system will substantially increase the survivability of large complex critical infrastructures in the face of attacks, failures and accidents caused by insiders and outsiders.

This paper examines short-term price reactions after one-day abnormal price changes and whether they create exploitable profit opportunities in various financial markets. A t-test confirms the presence of overreactions and also suggests that there is an “inertia anomaly”, i.e. after an overreaction day prices tend to move in the same direction for some time.  A trading robot approach is then used to test two trading strategies aimed at exploiting the detected anomalies to make abnormal profits. The results suggest that a strategy based on counter-movements after overreactions does not generate profits in the FOREX and the commodity markets, but it is profitable in the case of the US stock market. By contrast, a strategy exploiting the “inertia anomaly” produces profits in the case of the FOREX and the commodity markets, but not in the case of the US stock market.

Image classification in the open-world must handle out-of-distribution (OOD) images. Systems should ideally reject OOD images, or they will map atop of known classes and reduce reliability. Using open-set classifiers that can reject OOD inputs can help. However, optimal accuracy of open-set classifiers depend on the frequency of OOD data. Thus, for either standard or open-set classifiers, it is important to be able to determine when the world changes and increasing OOD inputs will result in reduced system reliability. However, during operations, we cannot directly assess accuracy as there are no labels. Thus, the reliability assessment of these classifiers must be done by human operators, made more complex because networks are not 100% accurate, so some failures are to be expected. To automate this process, herein, we formalize the open-world recognition reliability problem and propose multiple automatic reliability assessment policies to address this new problem using only the distribution of reported scores/probability data. The distributional algorithms can be applied to both classic classifiers with SoftMax as well as the open-world Extreme Value Machine (EVM) to provide automated reliability assessment. We show that all of the new algorithms significantly outperform detection using the mean of SoftMax.

Masqueraders are users who take control of a machine and perform malicious activities such as data exfiltration or system misuse on behalf of legitimate users. In the literature, there are various approaches for detecting masqueraders by modeling legitimate users' behavior during their daily tasks and automatically determine whether they are doing something suspicious. Usually, these techniques model user behavior using features extracted from various sources, such as file system, network activities, system calls, etc. In this work, we propose a one-class anomaly detection approach that measures similarities between a history of a user and events recorded in a time-window of the user's session which is to be classified. The idea behind our solution is the application of a graph partitioning technique on weighted oriented graphs generated from such event sequences, while considering that strongly connected nodes have to belong into the same cluster. First, a history of vertex clusters is build per each user and then this history is compared to a new input by using a similarity function, which leads either to the acceptance or rejection of a new input. This makes our approach substantially different from existing general graph-based approaches that consider graphs as a single entity. The approach can be applied for different kinds of homogeneous event sequences, however successful application of the approach will be demonstrated on file system access events only. The linear time complexity of the approach was demonstrated in the experiments and the performance evaluation was done using two state-of-the-art datasets-WUIL and TWOS-both of them containing file system access logs of legitimate users and masquerade attackers; for WUIL dataset we achieved an average per-user AUC of 0.94, a TPR over 95%, and a FPR less than 10%, while for TWOS dataset we achieved an average per-user AUC of 0.851, a TPR over 91% and a FPR around 11%.

Concerns of cyber-security threats are increasingly becoming a part of everyday operations of cyber-physical systems, especially in the context of critical infrastructures. However, despite the tight integration of cyber and physical components in modern critical infrastructures, the monitoring of cyber and physical subsystems is still done separately. For successful health monitoring of such systems, a holistic approach is needed. This paper presents an approach for holistic health monitoring of cyber-physical systems based on cyber and physical anomaly detection and correlation. We provide a data-driven approach for the detection of cyber and physical anomalies based on machine learning. The benefits of the presented approach are: 1) integrated architecture that supports the acquisition and real-time analysis of both cyber and physical data; 2) a metric for holistic health monitoring that allows for differentiation between physical faults, cyber intrusion, and cyber-physical attacks. We present experimental analysis on a power-grid use case using the IEEE-33 bus model. The system was tested on several types of attacks such as network scan, Denial of Service (DOS), and malicious command injections. INDEX TERMS Anomaly detection, cyber-physical systems, system health monitoring, cyber and physical anomalies.

Outlier (or anomaly) detection is an important problem for many domains, including fraud detection, risk analysis, network intrusion and medical diagnosis, and the discovery of significant outliers is becoming an integral aspect of data mining. This paper presents CURIO, a novel algorithm that uses quantisation and implied distance metrics to provide a fast algorithm that is linear for the number of objects and only requires two sequential scans of disk resident datasets. CURIO includes a novel direct quantisation technique and the ...

In wastewater industry, real-time sensing of surface temperature variations on concrete sewer pipes is paramount in assessing the rate of microbial-induced corrosion. However, the sensing systems are prone to failures due to the aggressively corrosive environmental conditions inside sewer assets. Therefore, reliable sensing in such infrastructures is vital for water utilities to enact efficient wastewater management. In this context, this paper presents a sensor failure detection and faulty data accommodation (SFDFDA) approach that aids to digitally monitor the health conditions of the sewer monitoring sensors. The SFDFDA approach embraces seasonal autoregressive integrated moving average (SARIMA) model with a statistical hypothesis testing technique for enabling temporal forecasting of sensor variable. Then, it identifies and isolates anomalies in a continuous stream of sensor data whilst detecting early sensor failure. Finally, the SFDFDA approach provides reliable estimates of sensor data in the event of sensor failure or during the scheduled maintenance period of sewer monitoring systems. The SFDFDA approach was evaluated by using the surface temperature data sourced from the instrumented wastewater infrastructure and the results have demonstrated the effectiveness of the SFDFDA approach and its applicability to surface temperature monitoring sensor suites.

Подтверждена геомагнитная гипотеза полтергейста и мысль о том, что этот феномен может быть некой формой природных явлений, связанных с геофизическими факторами. Причём связь эта носит универсальный характер независимо от его географии. Выявление подобных взаимосвязей в каждом конкретном случае может служить маркером достоверности ПГ-процесса и дает ценную информацию психиатрам и психотерапевтам, оказывающим помощь жертвам полтергейста.


----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The geomagnetic poltergeist hypothesis and the idea that this phenomenon may be a form of natural phenomena associated with geophysical factors have been confirmed. Moreover, this connection is universal, regardless of its geography. The identification of such relationships in each specific case can serve as a marker of the reliability of the PG-process and provides valuable information to psychiatrists and psychotherapists who provide assistance to victims of poltergeist.

The rapid proliferation of wireless networks and mobile computing applications has changed the landscape of network security. Anomaly detection is a pattern recognition task whose goal is to report the occurrence of abnormal or unknown behavior in a given system being monitored. This paper presents an efficient rough set based anomaly detection method that can effectively identify a group of

Wireless network has an exponential increase in various aspects of the human community. Accordingly, transmitting a vast volume of sensitive and non-sensitive data over the network puts them at risk of being attacked. To avoid this, Intrusion Detection System (IDS) security is intended to detect threats and protect devices from attacks. IDS usually uses one of the following alternative approaches: signature-based, anomaly-based, or hybrid of the two. In spite of the IDS has been the focus of much research in recent years, there is still space for improvement. Based on the anomalybased approach, this paper proposes a modified algorithm called a Multi-layer Feature Selection and Reduction IDS (MFSR-IDS) for providing high-level protection against Denial-of-Service (DoS) and Probe attacks. The MFSR-IDS framework makes three major contributions. First, it reduces the feature dimensionality of the network dataset across three layers. Second, it has a fast and accurate detection system. Third, it provides a mathematical model of the framework under consideration. The MFSR-IDS algorithm selects optimal number of features from KDDCUP'99 dataset which used to train the predictive model based on different learning classifiers and ensemble methodology. The performance of MFSR-IDS is evaluated in terms of Detection Rate (DR), False Positive Rate (FPR), FScore, ROC area, Accuracy (Acc) and Processing time. The experiments indicate that, the proposed MFSR-IDS outperforms some existing IDS frameworks in terms of DR, FPR, Acc and Processing time in detecting DoS and Probe attacks.

Conditional Anomaly Detection ... However, in contrast to problems in supervised learning where studies of classification accuracy are the norm, little research has systematically addressed the issue of accuracy in general-purpose unsupervised anomaly detection meth-ods. ...

—Despite all attempts to prevent fraud, it continues to be a major threat to industry and government. In this paper, we present a fraud detection method which detects irregular frequency of transaction usage in an Enterprise Resource Planning (ERP) system. We discuss the design, development and empirical evaluation of outlier detection and distance measuring techniques to detect frequency-based anomalies within an individual user's profile, relative to other similar users. Primarily, we propose three automated techniques: a univariate method, called Boxplot which is based on the sample's median; and two multivariate methods which use Euclidean distance, for detecting transaction frequency anomalies within each transaction profile. The two multivariate approaches detect potentially fraudulent activities by identifying: (1) users where the Euclidean distance between their transaction-type set is above a certain threshold and (2) users/data points that lie far apart from other users/clusters or represent a small cluster size, using k-means clustering. The proposed methodology allows an auditor to investigate the transaction frequency anomalies and adjust the different parameters, such as the outlier threshold and the Euclidean distance threshold values to tune the number of alerts. The novelty of the proposed technique lies in its ability to automatically trigger alerts from transaction profiles, based on transaction usage performed over a period of time. Experiments were conducted using a real dataset obtained from the production client of a large organization using SAP R/3 (presently the most predominant ERP system), to run its business. The results of this empirical research demonstrate the effectiveness of the proposed approach.

IEC61850 is the mainstream of the development for substation automation. This paper presents a practical consideration and analysis for implementing a secure sampled measured value (SeSV) message in substation automation system. Due to the lack of security features in the standard, IEC Working Group 15 of Technical Committee 57 published IEC62351 on security for IEC61850 profiles. However, the use of authentication methods for SV based on IEC62351 standards are still not integrated, and computational capabilities and performance are not validated and tested with commercial grade equipment. Hence, this paper shows the performance of security feature enabled SeSV packets transmitted between protection and control devices by appending a message authentication code (MAC) to the extended IEC61850 packets. A prototype implementation on a low cost commodity embedded system has proved that the MAC-enabled SV message can fully secure the process bus communication in the digital substation with negligible time delay.

Network anomaly detection system enables to monitor computer network that behaves differently from the network protocol and it is many implemented in various domains. Yet, the problem arises where different application domains have different defining anomalies in their environment. These make a difficulty to choose the best algorithms that suit and fulfill the requirements of certain domains and it is not straightforward. Additionally, the issue of centralization that cause fatal destruction of network system when powerful malicious code injects in the system. Therefore, in this paper we want to conduct experiment using supervised Machine Learning (ML) for network anomaly detection system that low communication cost and network bandwidth minimized by using UNSW-NB15 dataset to compare their performance in term of their accuracy (effective) and processing time (efficient) for a classifier to build a model. Supervised machine learning taking account the important features by labelling it from the datasets. The best machine learning algorithm for network dataset is AODE with a comparable accuracy is 97.26% and time taken approximately 7 seconds. Also, distributed algorithm solves the issue of centralization with the accuracy and processing time still a considerable compared to a centralized algorithm even though a little drop of the accuracy and a bit longer time needed.

Firewalls are core elements in network security. However, managing firewall rules, especially for enterprise networks, has become complex and error-prone. Firewall filtering rules have to be carefully written and organized in order to correctly implement the security policy. In ...

In this study, we introduce a novel unsupervised countermeasure for smart grid power systems, based on generative adversarial networks (GANs). Given the pivotal role of smart grid systems (SGSs) in urban life, their security is of particular importance. In recent years, however, advances in the field of machine learning, have raised concerns about cyber attacks on these systems. Power systems, among the most important components of urban infrastructure, have, for example, been widely attacked by adversaries. Attackers disrupt power systems using false data injection attacks (FDIA), resulting in a breach of availability, integrity, or confidential principles of the system. Our model simulates possible attacks on power systems using multiple generators in a step-by-step interaction with a discriminator in the training phase. As a consequence, our system is robust to unseen attacks. Moreover, the proposed model considerably reduces the well-known mode collapse problem of GAN-based models. Our method is general and it can be potentially employed in a wide range of one of one-class classification tasks. The proposed model has low computational complexity and outperforms baseline systems about 14% and 41% in terms of accuracy on the highly imbalanced publicly available industrial control system (ICS) cyber attack power system dataset.

Our aim is to estimate the perspective-effected geometric distortion of a scene from a video feed. In contrast to most related previous work, in this task we are constrained to using low-level, spatio-temporally local motion features only. This particular challenge arises in many semi-automatic surveillance systems which alert a human operator to potential abnormalities in the scene. Low-level, spatio-temporally local motion features are sparse (and thus require comparatively little storage space) and sufficiently powerful in the context of video abnormality detection to reduce the need for human intervention by more than 100-fold. This paper introduces three significant contributions: (i) we describe a dense algorithm for perspective estimation which uses motion features to estimate the perspective distortion at each image locus and then polls all such local estimates to arrive at the globally best estimate, (ii) we also present an alternative coarse algorithm which subdivides the image frame into blocks, and uses motion features to derive block-specific motion characteristics and constrain the relationships between these characteristics, with the perspective estimate emerging as a result of a global optimization scheme, and (iii) we report the results of an evaluation using nine large sets acquired using existing close-circuit television (CCTV) cameras, not installed specifically for the purposes of this work. Our findings demonstrate that both of the proposed methods are successful, their accuracy matching that of human labelling using complete visual data (by the constraints of the setup unavailable to our algorithms).

Wireless (Wi-Fi) networks based on IEEE 802.11 1 family of standards have been spreading its coverage last years and this trend is expected to grow. Every day more and more people use this type of networks to access Internet, company or other types of networks. Today's wireless networks are vulnerable in many ways (rogue access points, hijacking sessions, eavesdropping, illegal use, unauthorized access, denial of service attacks, floods, stealing data and other types of misuse and attacks etc). People are worried about unknowingly exposing their computers to illegal access through the air, from an undefined location. On wired networks the intruder can access by wire, but in wireless environments the intruder can access the network from anywhere in the neighborhood. At the present time there are IDS's but mostly deployed on wired networks, and based on rules and signatures of already known and analyzed intrusions. These systems can't answer the demand in environments where new intrusions are occurring every day due their legacy IDS's limitations. Intrusion detection agent presented in this paper is part of WIDSwireless intrusion detection system. WIDS Agent is software installed on mobile computer device. It detects intrusions and attacks by analyzing traffic and making conclusions and denies it. It works as standalone module or coupled (in contribution) with WIDS Sensor and Server that are also part of this system. Position of application is on personal computer (PC), including Pocket PC (PPC), other PDA devices and similar computerized and mobile devices. This system has capabilities such as: self learning, autonomy and decision, self-decision and self defense including alerting. This is multidimensional system in development which is intended to cover most of wireless networks specific vulnerabilities and intrusion. It should work in real-time and defend user i.e. his computer or system against majority of intrusions nevertheless of fact if they are already known or new kind of attacks. System is integrated in clients and performs local data collection and filtering, works as local detection engine cooperating with neighboring WIDS agents (cooperative detection engine). It provides local response and/or global response against intrusion. Also, this system works in the closest relationship with firewall software and devices, antivirus software, network management and other security tools.

Given the anticipated increases in highway traffic, the scale and complexity of the traffic infrastructure will continue to grow progressively in time and in distributed geographical areas. To assure transportation efficiency, safety, and security in the presence of such growth, it is critical to identify an infrastructure development methodology that can adapt to expansions while assuring reliable operation for both centralized monitoring and distributed management. In this paper, a wireless sensor network design methodology is presented, aimed at providing effective distributed surveillance, anomaly detection, and coordinated response. The proposed methodology integrates state-of-the-art traffic sensors, with flexibly programmable controller devices that can integrate with the available traffic control equipments. The system methodology provides a paradigm in which sensors and controllers can be progressively incorporated and programmed to autonomously coordinate with peer sensors and a hierarchy of controllers to detect, notify, and react to anomalous events. Since the system can tolerate failure of parts of the system, as the network connectivity continues to increase, the proposed sensor network will have positive implications on evacuation plans during natural disasters or terrorist attacks. To illustrate the design methodology and usage, a simulated system along a freeway corridor in South Carolina was constructed in an integrated microscopic traffic and wireless sensor network simulation platform, in which distributed incident detection and response functions were implemented. The test results, including detection and false alarm rates and wireless communication latencies, are analyzed to identify insights of the system's operation and potential enhancement strategies.

The use of the Internet has increased in all areas in recent years. With the huge growth and use of the internet increasing, there have been an increase in the number of intrusions and hackers. The risk of intrusion in the network environment is serious. The basic concept of the intrusion detection system is highlighted in this report. Most of IDS' work is based on two approaches: the approach to anomaly and the approach to misuse. This paper provides a short assessment of intrusion detection taxonomy and literature. As a starting point for research in the field of intrusion detection, an extensive bibliography is provided. The purpose of this paper is to cover the analysis of various available intrusion detection tools and data mining techniques for detecting intrusions in network. A review on current trends in intrusion detection together with the study on technologies implemented by some researchers in this area is presented

Anomaly detection is used for identifying data that deviate from 'normal' data patterns. Its usage on classical data finds diverse applications in many important areas like fraud detection, medical diagnoses, data cleaning and surveillance. With the advent of quantum technologies, anomaly detection of quantum data, in the form of quantum states, may become an important component of quantum applications. Machine learning algorithms are playing pivotal roles in anomaly detection using classical data. Two widely-used algorithms are kernel principal component analysis and one-class support vector machine. We find corresponding quantum algorithms to detect anomalies in quantum states. We show that these two quantum algorithms can be performed using resources logarithmic in the dimensionality of quantum states. For pure quantum states, these resources can also be logarithmic in the number of quantum states used for training the machine learning algorithm. This makes these algorithms potentially applicable to big quantum data applications.