research-article

Modeling and Tracking of Transaction Flow Dynamics for Fault Detection in Complex Systems

Authors:

Kenji YoshihiraAuthors Info & Claims

IEEE Transactions on Dependable and Secure Computing, Volume 3, Issue 4

Pages 312 - 326

https://doi.org/10.1109/TDSC.2006.52

Published: 01 October 2006 Publication History

Abstract

With the prevalence of Internet services and the increase of their complexity, there is a growing need to improve their operational reliability and availability. While a large amount of monitoring data can be collected from systems for fault analysis, it is hard to correlate this data effectively across distributed systems and observation time. In this paper, we analyze the mass characteristics of user requests and propose a novel approach to model and track transaction flow dynamics for fault detection in complex information systems. We measure the flow intensity at multiple checkpoints inside the system and apply system identification methods to model transaction flow dynamics between these measurements. With the learned analytical models, a model-based fault detection and isolation method is applied to track the flow dynamics in real time for fault detection. We also propose an algorithm to automatically search and validate the dynamic relationship between randomly selected monitoring points. Our algorithm enables systems to have self-cognition capability for system management. Our approach is tested in a real system with a list of injected faults. Experimental results demonstrate the effectiveness of our approach and algorithms.

References

[1]

D. Patterson, “A Simple Way to Estimate the Cost of Downtime,” Proc. 16th System Administration Conf. (LISA '02), pp. 185-188, 2002.

Digital Library

[2]

D. Oppenheimer, A. Ganapathi, and D. Patterson, “Why Do Internet Services Fail, and What Can Be Done About It,” Proc. Fourth Usenix Symp. Internet Technologies and Systems (USITS '03), 2003.

Digital Library

[3]

M. Chen, A. Accardi, E. Kiciman, J. Lloyd, D. Patterson, A. Fox, and E. Brewer, “Path-Based Failure and Evolution Management,” Proc. First USENIX Symp. Networked Systems Design and Implementation (NSDI '04), Mar. 2004.

[7]

R. Isermann, “Model-Based Fault Detection and Diagnosis—Status and Applications,” Proc. 16th IFAC Symp. Automatic Control in Aerospace (ACA '04), June 2004.

Digital Library

[8]

J. Gertler, Fault Detection and Diagnosis in Engineering Systems. Marcel Dekker, 1998.

[12]

L. Ljung, System Identification - Theory for the User, second ed. Prentice Hall PTR, 1998.

[13]

R. Redner and H. Walker, “Mixture Densities, Maximum Likelihood and the Em Algorithm,” SIAM Rev., vol. 26, no. 2, pp. 195-239, 1984.

[14]

H. Akaike, “Information Theory and an Extension of the Maximum Likelihood Principle,” Proc. Second Int'l Symp. Information Theory, 1973.

Digital Library

[15]

J. Rissanen, “Prediction Minimum Description Length Principles,” Ann. Statistics, vol. 14, 1986.

[17]

M. Spiegel, Theory and Problems of Probability and Statistics. McGraw-Hill, 1992.

Digital Library

[18]

J. Han and M. Kamber, Data Mining: Concepts and Techniques. Morgan Kaufman, 2000.

Digital Library

[19]

Proc. Int'l Conf. Dependable Systems and Networks (DSN), pp. 644-653, 2005.

Digital Library

[20]

Ira Cohen, Steve Zhang, Moises Goldszmidt, Julie Symons, Terence Kelly, Armando Fox, Capturing, indexing, clustering, and retrieving system history, ACM SIGOPS Operating Systems Review, v.39 n.5, December 2005

[21]

N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge Univ. Press, 2000.

Digital Library

[22]

R. Yager, M. Fedrizzi, and J. Kacprzyk, Advances in the Dempster-Shafer Theory of Evidence. Wiley, 1994.

Digital Library

[24]

J. O'Madadhain, D. Fisher, S. White, and Y. Boey, “The Jung (Java Universal Network/Graph) Framework,” Technical Report UCI-ICS 03-17, Univ. of California at Irvine, School of Information and Computer Sciences,

Digital Library

[25]

J. Voas and G. Mcgraw, Software Fault Injection: Inoculating Programs against Errors. John Wiley & Sons, 1997.

[26]

B. Tate, Bitter Java. Manning Publications, 2002.

[27]

E. Kiciman and A. Fox, “Detecting Application-Level Failures in Component-Based Internet Services,” IEEE Trans. Neural Networks, vol. 16, no. 5, pp. 1027-1041, 2006.

[28]

IEEE Comm. Magazine, vol. 34, no. 5, pp. 82-90, May 1996.

[29]

G. Jiang, H. Chen, C. Ungureanu, and K. Yoshihira, “Multi-Resolution Abnormal Trace Detection Using Varied-Length ngrams and Automata,” Proc. Second IEEE Int'l Conf. Autonomic Computing (ICAC '05), June 2005.

[30]

M. Aguilera, J. Mogul, J. Wiener, P. Reynolds, and A. Muthitacharoen, “Performance Debugging for Distributed Systems of Black Boxes,” Proc. 19th ACM Symp. Operating Systems Principles (SOSP '03), pp. 74-89, 2003.

[31]

R. Isermann and P. Balle, “Trends in the Application of Model-Based Fault Detection and Diagnosis of Industrial Process,” Control Eng. Practice, vol. 5, no. 5, 1997.

[32]

R. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” Trans. ASME-J. Basic Eng., vol. 82, no. series D, pp. 35-45, 1960.

Cited By

Horak MChandrasekaran STobar G(2022)NLP Based Anomaly Detection for Categorical Time Series2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI)10.1109/IRI54793.2022.00019(27-34)Online publication date: 9-Aug-2022
https://dl.acm.org/doi/10.1109/IRI54793.2022.00019
Zhao GHassan SZou YTruong DCorbin T(2021)Predicting Performance Anomalies in Software Systems at Run-timeACM Transactions on Software Engineering and Methodology10.1145/344075730:3(1-33)Online publication date: 23-Apr-2021
https://dl.acm.org/doi/10.1145/3440757
He SHuang HYoo SYan WXue FWang TXu CHung CHong JBechini ASong E(2021)Flight data anomaly detection and diagnosis with variable association changeProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441916(346-354)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441916
Show More Cited By

Index Terms

Modeling and Tracking of Transaction Flow Dynamics for Fault Detection in Complex Systems

Recommendations

Online fault detection and tolerance for photovoltaic energy harvesting systems
ICCAD '12: Proceedings of the International Conference on Computer-Aided Design

Photovoltaic energy harvesting systems (PV systems) are subject to PV cell faults, which decrease the efficiency of PV systems and even shorten the PV system lifespan. Manual PV cell fault detection and elimination are expensive and nearly impossible ...
An architecture for fault detection and isolation based on fuzzy methods

Model-based fault detection and isolation (FDI) is an approach with increasing attention in the academic and industrial fields, due to economical and safety related matters. In FDI, the discrepancies between system outputs and model outputs are called ...
Fault detection and fault tolerance issues at CMOS level through AUED encoding
DFT '96: Proceedings of the 1996 Workshop on Defect and Fault-Tolerance in VLSI Systems

A new CMOS gate structure tolerating all single transistor stuck-on faults and a large set of multiple faults is presented. Such technique is aimed at guaranteeing fault tolerance for a multiple output gate and the fault tolerance property is achieved ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Dependable and Secure Computing

IEEE Transactions on Dependable and Secure Computing Volume 3, Issue 4

October 2006

133 pages

ISSN:1545-5971

Issue’s Table of Contents

Copyright © 2006.

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 01 October 2006

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Horak MChandrasekaran STobar G(2022)NLP Based Anomaly Detection for Categorical Time Series2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI)10.1109/IRI54793.2022.00019(27-34)Online publication date: 9-Aug-2022
https://dl.acm.org/doi/10.1109/IRI54793.2022.00019
Zhao GHassan SZou YTruong DCorbin T(2021)Predicting Performance Anomalies in Software Systems at Run-timeACM Transactions on Software Engineering and Methodology10.1145/344075730:3(1-33)Online publication date: 23-Apr-2021
https://dl.acm.org/doi/10.1145/3440757
He SHuang HYoo SYan WXue FWang TXu CHung CHong JBechini ASong E(2021)Flight data anomaly detection and diagnosis with variable association changeProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441916(346-354)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441916
Ma MXu JWang YChen PZhang ZWang P(2020)AutoMAP: Diagnose Your Microservice-based Web Applications AutomaticallyProceedings of The Web Conference 202010.1145/3366423.3380111(246-258)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380111
Wang RYing SLi MJia S(2020)HSACMA: a hierarchical scalable adaptive cloud monitoring architectureSoftware Quality Journal10.1007/s11219-020-09524-z28:3(1379-1410)Online publication date: 1-Sep-2020
https://dl.acm.org/doi/10.1007/s11219-020-09524-z
Wei TLee HJeng ALamba HFaloutsos C(2019) WebHound: a data-driven intrusion detection from real-world web access logsSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-018-03750-123:22(11947-11965)Online publication date: 1-Nov-2019
https://dl.acm.org/doi/10.1007/s00500-018-03750-1
Luo CChen ZTang LShrivastava ALi ZChen HYe JGuo YFarooq F(2018)TINETProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3219819.3220003(1890-1899)Online publication date: 19-Jul-2018
https://dl.acm.org/doi/10.1145/3219819.3220003
Wang TXu JZhang WGu ZZhong H(2018)Self-adaptive cloud monitoring with online anomaly detectionFuture Generation Computer Systems10.1016/j.future.2017.09.06780:C(89-101)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1016/j.future.2017.09.067
Park JIvanov RWeimer JPajic MSon SLee I(2017)Security of Cyber-Physical Systems in the Presence of Transient Sensor FaultsACM Transactions on Cyber-Physical Systems10.1145/30648091:3(1-23)Online publication date: 9-May-2017
https://dl.acm.org/doi/10.1145/3064809
Cheng WNi JZhang KChen HJiang GShi YZhang XWang W(2017)Ranking Causal Anomalies for System Fault Diagnosis via Temporal and Dynamical Analysis on Vanishing CorrelationsACM Transactions on Knowledge Discovery from Data10.1145/304694611:4(1-28)Online publication date: 29-Jun-2017
https://dl.acm.org/doi/10.1145/3046946
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents