research-article

Tools and benchmarks for automated log parsing

Authors:

Michael R. LyuAuthors Info & Claims

ICSE-SEIP '19: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice

Pages 121 - 130

https://doi.org/10.1109/ICSE-SEIP.2019.00021

Published: 27 May 2019 Publication History

Abstract

Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.

References

[1]

G. Lee, J. J. Lin, C. Liu, A. Lorek, and D. V. Ryaboy, "The unified logging infrastructure for data analytics at Twitter," PVLDB, vol. 5, no. 12, pp. 1771--1780, 2012.

Digital Library

[2]

A. Oprea, Z. Li, T. Yen, S. H. Chin, and S. A. Alrwais, "Detection of early-stage enterprise infection by mining large-scale log data," in DSN, 2015, pp. 45--56.

Digital Library

[3]

M. Chow, D. Meisner, J. Flinn, D. Peek, and T. F. Wenisch, "The mystery machine: End-to-end performance analysis of large-scale internet services," in OSDI, 2014, pp. 217--231.

Digital Library

[4]

K. Nagaraj, C. E. Killian, and J. Neville, "Structured comparative analysis of systems logs to diagnose performance problems," in NSDI, 2012, pp. 353--366.

Digital Library

[5]

D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy, "Sherlog: error diagnosis by connecting clues from run-time logs," in ASPLOS, 2010, pp. 143--154.

Digital Library

[6]

X. Xu, L. Zhu, I. Weber, L. Bass, and D. Sun, "POD-Diagnosis: Error diagnosis of sporadic operations on cloud applications," in DSN, 2014, pp. 252--263.

Digital Library

[7]

A. J. Oliner, A. Ganapathi, and W. Xu, "Advances and challenges in log analysis," Commun. ACM, vol. 55, no. 2, pp. 55--61, 2012.

Digital Library

[8]

H. Mi, H. Wang, Y. Zhou, M. R. Lyu, and H. Cai, "Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems," IEEE Trans. Parallel Distrib. Syst., vol. 24, no. 6, pp. 1245--1255, 2013.

Digital Library

[9]

P. He, J. Zhu, S. He, J. Li, and M. R. Lyu, "An evaluation study on log parsing and its use in log mining," in DSN, 2016, pp. 654--661.

[10]

M. Du, F. Li, G. Zheng, and V. Srikumar, "Deeplog: Anomaly detection and diagnosis from system logs through deep learning," in CCS, 2017, pp. 1285--1298.

Digital Library

[11]

Q. Lin, H. Zhang, J. Lou, Y. Zhang, and X. Chen, "Log clustering based problem identification for online service systems," in ICSE, 2016.

[12]

S. He, J. Zhu, P. He, and M. R. Lyu, "Experience report: System log analysis for anomaly detection," in ISSRE, 2016, pp. 207--218.

[13]

Splunk. {Online}. Available: http://www.splunk.com

[14]

ELK. {Online}. Available: https://www.elastic.co/elk-stack

[15]

Logentries. {Online}. Available: https://logentries.com

[16]

A beginnerś guide to logstash grok. {Online}. Available: https://logz.io/blog/logstash-grok

[17]

W. Xu, "System problem detection by mining console logs," Ph.D. dissertation, University of California, Berkeley, 2010.

Digital Library

[18]

W. Xu, L. Huang, A. Fox, D. A. Patterson, and M. I. Jordan, "Detecting large-scale system problems by mining console logs," in SOSP, 2009, pp. 117--132.

Digital Library

[19]

M. Nagappan, K. Wu, and M. A. Vouk, "Efficiently extracting operational profiles from execution logs using suffix arrays," in ISSRE, 2009, pp. 41--50.

Digital Library

[20]

R. Vaarandi, "A data clustering algorithm for mining patterns from event logs," in IPOM, 2003.

[21]

R. Vaarandi and M. Pihelgas, "Logcluster - a data clustering and pattern mining algorithm for event logs," in CNSM, 2015, pp. 1--7.

Digital Library

[22]

A. Makanju, A. Zincir-Heywood, and E. Milios, "Clustering event logs using iterative partitioning," in KDD, 2009.

Digital Library

[23]

Q. Fu, J.-G. Lou, Y. Wang, and J. Li, "Execution anomaly detection in distributed systems through unstructured log analysis," in ICDM, 2009, pp. 149--158.

Digital Library

[24]

M. Du and F. Li, "Spell: Streaming parsing of system event logs," in ICDM, 2016, pp. 859--864.

[25]

P. He, J. Zhu, Z. Zheng, and M. R. Lyu, "Drain: An online log parsing approach with fixed depth tree," in ICWS, 2017, pp. 33--40.

[26]

K. Shima, "Length matters: Clustering system log messages using length of words," arXiv:1611.03213, 2016.

[27]

H. Hamooni, B. Debnath, J. Xu, H. Zhang, G. Jiang, and A. Mueen, "LogMine: fast pattern recognition for log analytics," in CIKM, 2016, pp. 1573--1582.

Digital Library

[28]

S. Messaoudi, A. Panichella, D. Bianculli, L. Briand, and R. Sasnauskas, "A search-based approach for accurate identification of log message formats," in ICPC, 2018.

[29]

Loggly: Cloud log management service. {Online}. Available: https://www.loggly.com

[30]

A. Oliner and J. Stearley, "What supercomputers say: A study of five system logs," in DSN, 2007.

Digital Library

[31]

Loghub: A collection of system log datasets for intelligent log analysis. {Online}. Available: https://github.com/logpai/loghub

[32]

Overview of logs-based metrics. {Online}. Available: https://cloud.google.com/logging/docs/logs-based-metrics

[33]

P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, "Using magpie for request extraction and workload modelling," in OSDI, 2004, pp. 259--272.

Digital Library

[34]

J. Lou, Q. Fu, S. Yang, Y. Xu, and J. Li, "Mining invariants from console logs for system problem detection," in ATC, 2010.

[35]

R. Ding, Q. Fu, J. G. Lou, Q. Lin, D. Zhang, and T. Xie, "Mining historical issue repositories to heal large-scale online service systems," in DSN, 2014, pp. 311--322.

[36]

M. Lim, J. Lou, H. Zhang, Q. Fu, A. B. J. Teoh, Q. Lin, R. Ding, and D. Zhang, "Identifying recurrent and unknown performance issues," in ICDM, 2014, pp. 320--329.

Digital Library

[37]

Automated root cause analysis for spark application failures. {Online}. Available: https://www.oreilly.com/ideas/automated-root-cause-analysis-for-spark-application-failures

[38]

Logz.io. {Online}. Available: https://logz.io

[39]

New automated log parsing. {Online}. Available: https://blog.rapid7.com/2016/03/03/new-automated-log-parsing

[40]

Log parsing - automated, easy to use, and efficient. {Online}. Available: https://logz.io/product/log-parsing

[41]

Automated parsing log types. {Online}. Available: https://www.loggly.com/docs/automated-parsing

[42]

M. Nagappan and M. A. Vouk, "Abstracting log lines to log event types for mining software system logs," in MSR, 2010, pp. 114--117.

[43]

L. Tang, T. Li, and C.-S. Perng, "LogSig: Generating system events from raw textual logs," in CIKM, 2011, pp. 785--794.

Digital Library

[44]

M. Mizutani, "Incremental mining of system log format," in SCC, 2013, pp. 595--602.

Digital Library

[45]

Z. M. Jiang, A. E. Hassan, P. Flora, and G. Hamann, "Abstracting execution logs to execution events for enterprise applications," in QSIC, 2008, pp. 181--186.

Digital Library

[46]

D. Yuan, S. Park, and Y. Zhou, "Characterizing logging practices in open-source software," in ICSE, 2012, pp. 102--112.

Digital Library

[47]

Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie, "Where do developers log? an empirical study on logging practices in industry," in ICSE, 2014, pp. 24--33.

Digital Library

[48]

J. Zhu, P. He, Q. Fu, H. Zhang, M. R. Lyu, and D. Zhang, "Learning to log: Helping developers make informed logging decisions," in ICSE, vol. 1, 2015, pp. 415--425.

Digital Library

[49]

X. Zhao, K. Rodrigues, Y. Luo, M. Stumm, D. Yuan, and Y. Zhou, "Log20: Fully automated optimal placement of log printing statements under specified overhead threshold," in SOSP, 2017, pp. 565--581.

[50]

D. Yuan, J. Zheng, S. Park, Y. Zhou, and S. Savage, "Improving software diagnosability via log enhancement," in ASPLOS, 2011, pp. 3--14.

Digital Library

[51]

P. He, Z. Chen, S. He, and M. R. Lyu, "Characterizing the natural language descriptions in software logging statements," in ASE, 2018, pp. 178--189.

Digital Library

[52]

R. Ding, H. Zhou, J. Lou, H. Zhang, Q. Lin, Q. Fu, D. Zhang, and T. Xie, "Log2: A cost-aware logging mechanism for performance diagnosis," in ATC, 2015.

[53]

P. He, J. Zhu, S. He, J. Li, and M. R. Lyu, "Towards automated log parsing for large-scale log data analysis," IEEE Trans. Dependable Sec. Comput. (TDSC), vol. 15, no. 6, pp. 931--944, 2018.

[54]

M. P. Stefan Thaler, Vlado Menkonvski, "Towards a neural language model for signature extraction from forensic logs," in ISDFS, 2017.

[55]

Y. Gao, S. Huang, and A. G. Parameswaran, "Navigating the data lake with DATAMARAN: automatically extracting structure from log datasets," in SIGMOD, 2018, pp. 943--958.

Digital Library

[56]

S. He, Q. Lin, J. Lou, H. Zhang, M. R. Lyu, and D. Zhang, "Identifying impactful service system problems via log analysis," in FSE, 2018, pp. 60--70.

Digital Library

[57]

W. Shang, Z. Jiang, H. Hemmati, B. Adams, A. Hassan, and P. Martin, "Assisting developers of big data analytics applications when deploying on hadoop clouds," in ICSE, 2013, pp. 402--411.

Digital Library

Cited By

Cheng HYing SDuan XYuan W(2024)DLLogInternational Journal of Intelligent Systems10.1155/2024/59619932024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/5961993
Shetty MChen YSomashekar GMa MSimmhan YZhang XMace JVandevoorde DLas-Casas PGupta SNath SBansal CRajmohan S(2024)Building AI Agents for Autonomous Clouds: Challenges and Design PrinciplesProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698525(99-110)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698525
Zhang SJi YLuan JNie XChen ZMa MSun YPei DFilkov VRay BZhou M(2024)End-to-End AutoML for Unsupervised Log Anomaly DetectionProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695535(1680-1692)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695535
Show More Cited By

Recommendations

A Survey on Automated Log Analysis for Reliability Engineering
Invited Tutorial

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems, because they are often the only data available ...
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in ...
Log-based anomaly detection without log parsing
ASE '21: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering

Software systems often record important runtime information in system logs for troubleshooting purposes. There have been many studies that use log data to construct machine learning models for detecting system anomalies. Through our empirical study, we ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIP '19: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice

May 2019

339 pages

Conference Chairs:
Helen Sharp
The Open University, UK
,
Mike Whalen
Amazon Inc.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
460
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)3

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng HYing SDuan XYuan W(2024)DLLogInternational Journal of Intelligent Systems10.1155/2024/59619932024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/5961993
Shetty MChen YSomashekar GMa MSimmhan YZhang XMace JVandevoorde DLas-Casas PGupta SNath SBansal CRajmohan S(2024)Building AI Agents for Autonomous Clouds: Challenges and Design PrinciplesProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698525(99-110)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698525
Zhang SJi YLuan JNie XChen ZMa MSun YPei DFilkov VRay BZhou M(2024)End-to-End AutoML for Unsupervised Log Anomaly DetectionProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695535(1680-1692)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695535
Xiao YLe VZhang HFilkov VRay BZhou M(2024)Demonstration-Free: Towards More Practical Log Parsing with Large Language ModelsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694994(153-165)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694994
Zhang LJia TWang KJia MYang YLi Y(2024)Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical StudyProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695403(538-548)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3695403
Astekin MHort MMoonen L(2024)A Comparative Study on Large Language Models for Log ParsingProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686684(234-244)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686684
Abraham ADani YZhang Kd'Amorim M(2024)Productionizing PILAR as a Logstash PluginCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3664460(689-691)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3664460
Gao ZXue ZHu XShang WXia Xd'Amorim M(2024)Easy over Hard: A Simple Baseline for Test Failures Causes PredictionCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663850(306-317)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663850
Sedki Id'Amorim M(2024)A Preliminary Study on the Privacy Concerns of Using IP Addresses in Log DataCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663791(527-531)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663791
Xie YZhang HBabar M(2024)LogSD: Detecting Anomalies from System Logs through Self-Supervised Learning and Frequency-Based MaskingProceedings of the ACM on Software Engineering10.1145/36608001:FSE(2098-2120)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660800
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents