Abstract
In modern computer systems, system event logs have always been the primary source for checking system status. As computer systems become more and more complex, the interaction between software and hardware increases frequently. The components will generate enormous log information, including running reports and fault information. The sheer quantity of data is a great challenge for analysis relying on the manual method. In this paper, we implement a management and analysis system of log information, which can assist system administrators to understand the real-time status of the entire system, classify logs into different fault types, and determine the root cause of the faults. In addition, we improve the existing fault correlation analysis method based on the results of system log classification. We apply the system in a cloud computing environment for evaluation. The results show that our system can classify fault logs automatically and effectively. With the proposed system, administrators can easily detect the root cause of faults.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Zawoad S, Dutta A K, Hasan R. SecLaaS: Secure logging-as-a-service for cloud forensics. In Proc. the 8th ACM SIGSAC Symposium on Information, Computer and Communications Security, May 2013, pp.219-230.
Rao X, Wang H, Shi D et al. Identifying faults in large-scale distributed systems by filtering noisy error logs. In Proc. the IEEE/IFIP International Conference on Dependable Systems and Networks Workshops, June 2011, pp.140-145.
Yuan D, Mai H, Xiong W et al. SherLog: Error diagnosis by connecting clues from run-time logs. ACM SIGARCH Computer Architecture News, 2010, 38(1): 143-154.
Fu Q, Lou J G, Wang Y et al. Execution anomaly detection in distributed systems through unstructured log analysis. In Proc. the 9th IEEE International Conference on Data Mining, December 2009, pp.149-158.
Xu W, Huang L, Fox A et al. Detecting large-scale system problems by mining console logs. In Proc. the 22nd ACM Symposium on Operating Systems Principles, October 2009, pp.117-132.
Hansen J P, Siewiorek D P. Models for time coalescence in event logs. In Proc. the 22nd IEEE International Symposium on Fault-Tolerant Computing, July 1992, pp.221-227.
Mi H B, Wang H M, Zhou Y F et al. Localizing root causes of performance anomalies in cloud computing systems by analyzing request trace logs. Science China (Information Sciences), 2012, 55(12): 2757-2773.
Prewett J E, James E. Listening to your cluster with LoGS. In Proc. the 5th LCI International Conference on Linux Clusters: The HPC Revolution, May 2004.
Jain S, Singh I, Chandra A et al. Extracting the textual and temporal structure of supercomputing logs. In Proc. the 16th IEEE International Conference on High Performance Computing, December 2009, pp.254-263.
Takada T, Koike H. Tudumi: Information visualization system for monitoring and auditing computer logs. In Proc. the International Conference on Information Visualization, July 2002, pp.570-576.
Vaarandi R. A data clustering algorithm for mining patterns from event logs. In Proc. the 3rd IEEE Workshop on IP Operations and Management, October 2003, pp.119-126.
Bellec J H, Kechadi T M. Cufres: Clustering using fuzzy representative events selection for the fault recognition problem in telecommunication networks. In Proc. the 1st ACM Ph.D. Workshop on Information and Knowledge Management, November 2007, pp.55-62.
Ganapathi A, Patterson D. Crash data collection: A windows case study. In Proc. the IEEE/IFIP International Conference on Dependable Systems and Networks, June 28-July 1, 2005, pp.280-285.
Pecchia A, Cotroneo D, Kalbarczyk Z et al. Improving log-based field failure data analysis of multi-node computing systems. In Proc. the IEEE/IFIP International Conference on Dependable Systems and Networks, June 2011, pp.97-108.
Stearley J, Oliner A J. Bad words: Finding faults in Spirit’s syslogs. In Proc. the IEEE International Symposium on Cluster Computing and the Grid, May 2008, pp.765-770.
Stearley J. Towards informatic analysis of Syslogs. In Proc. the IEEE International Conference on Cluster Computing, September 2004, pp.309-318.
Xu W, Huang L, Fox A et al. Mining console logs for large-scale system problem detection. In Proc. the IEEE Conference on Tackling Computer Systems Problems with Machine Learning Techniques, December 2008.
Salfner F, Tschirpke S. Error log processing for accurate failure prediction. In Proc. the 1st USENIX Workshop on Analysis of System Logs, December 2008, p.4.
Wen X, Zhang X, Zhu Y. Design of fault detection observer based on Hyper Basis Function. Tsinghua Science and Technology, 2015, 20(2): 200-204.
Park J, Yoo G, Lee E. Proactive self-healing system based on multi-agent technologies. In Proc. the ACIS International Conference on Software Engineering Research, Management and Applications, August 2005, pp.256-263.
Li T, Liang F, Ma S et al. An integrated framework on mining logs files for computing system management. In Proc. the 11th ACM International Conference on Knowledge Discovery in Data Mining, August 2005, pp.776-781.
Vaarandi R. A breadth-first algorithm for mining frequent patterns from event logs. In Proc. Intelligence in Communication Systems, November 2004, pp.293-308.
Oliner A, Stearley J. What supercomputers say: A study of five system logs. In Proc. the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, June 2007, pp.575-584.
Simache C, Kaâniche M, Sa¨ıdane A. Event log based dependability analysis of Windows NT and 2K systems. In Proc. the Pacific Rim International Symposium on Dependable Computing, December 2002, pp.311-315.
Ester M, Kriegel H P, Sander J et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proc. the 2nd ACM International Conference on Knowledge Discovery and Data Mining, August 1996, pp.226-231.
Levenshtein V I. Binary codes capable of correcting deletions, insertions and reversals. Journal of Soviet Physics Doklady, 1966, 10: 707-711.
Tsao M M, Siewiorek D P. Trend analysis on system error files. In Proc. the IEEE International Symposium on Fault-Tolerant Computing, June 1983, pp.116-119.
Fu S, Xu C Z. Exploring event correlation for failure prediction in coalitions of clusters. In Proc. the ACM/IEEE Conference on High Performance Networking and Computing, November 2007.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the National Basic Research 973 Program of China under Grant No. 2014CB340600, the National Natural Science Foundation of China under Grant No. 61272072, and the Program for New Century Excellent Talents in University of China under Grant No. NCET-13-0241.
Rights and permissions
About this article
Cite this article
Zou, DQ., Qin, H. & Jin, H. UiLog: Improving Log-Based Fault Diagnosis by Log Analysis. J. Comput. Sci. Technol. 31, 1038–1052 (2016). https://doi.org/10.1007/s11390-016-1678-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-016-1678-7