Article

Finding failures by cluster analysis of execution profiles

Authors:

William Dickinson,

Andy PodgurskiAuthors Info & Claims

ICSE '01: Proceedings of the 23rd International Conference on Software Engineering

Pages 339 - 348

Published: 01 July 2001 Publication History

Publisher Site Get Access

Abstract

We experimentally evaluate the effectiveness of using cluster analysis of execution profiles to find failures among the executions induced by a set of potential test cases. We compare several filtering procedures for selecting executions to evaluate for conformance to requirements. Each filtering procedure involves a choice of a sampling strategy and a clustering metric. The results suggest that filtering procedures based on clustering are more effective than simple random sampling for identifying failures in populations of operational executions, with adaptive sampling from clusters being the most effective sampling strategy. The results also suggest that clustering metrics that give extra weight to unusual profile features are most effective. Scatter plots of execution populations, produced by multidimensional scaling, are used to provide intuition for these results.

References

[1]

Borg, I. and Groenen, P. Modern Multidimensional Scaling: Theory and Applications, Springer, 1997.

Digital Library

[2]

Davidson, T. jstyle.JSFormatter, http://www.bigfoot .com/~davidsont/j style, 1998.

[3]

Duran, J.W. and Ntafos, S.C. An evaluation of random testing. IEEE Transactions on Software Engineering, 10, 4 (July 1984), 438-444.

[4]

Elbaum, S., Malishevsky, A.G., and Rothermel, G. Prioritizing test cases for regression testing. Proceedings of the 2000 International Symposium on Software Testing and Analysis (Portland, OR, August 2000), 102-112.

Digital Library

[5]

Frmakl, P.G. mad Weyuker, E.J. A formal analysis of flae fault detecting ability of testing methods. IEEE Transactions on Software Engineering 19, 3 (March 1993), 202-213.

Digital Library

[6]

Frmakl, P.G. and Weyuker, E.J. Provable improvements on branch testing. IEEE Transactions on Software Engineering 19, 10 (October, 1993), 962-975.

Digital Library

[7]

GCC. The GCC Home Page, http://www.gnu.org /softwaxe/gcc/gcc.htxnl, Free Softwaxe Foundation, 2000.

[8]

GNU Java Softwaxe. http://www.guu.org/softwaxe/java/javasoftwaxe.htxnl, Free Software Foundation, 2000.

[9]

Hamlet, D. and Taylor, R. Partition testing does not inspire confidence. IEEE Transactions on Software Engineering 16, 12 (December 1990), 1402-1411.

Digital Library

[10]

Harrold, M.J., Roflaermel, G., Wu, R., and Yi, L. An empirical investigation of program spectra. ACM SIGPLAN- SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (Montreal, Canada, June 1998), 83- 90.

Digital Library

[11]

Jain, A.K. and Dubes, R.C. Algorithms for Clustering Data, Prentice Hall, 1988.

Digital Library

[12]

Leon, D., Podgurski, A., and White, L.J. Multivaxiate visualization in observation-based testing. Proceedings of the 22nd International Conference on Software Engineering (Limerick, Ireland, June 2000), ACM Press, 116-125.

Digital Library

[13]

Offutt, J., Voas, J., and Payne, J. Mutation Operators for Ada, Technical Report ISSE-TR-96-09, George Mason University, October, 1996.

[14]

Pavlopoulou, C. and Young, M. Residual test coverage monitoring. Proceedings of the 21st International Conference on Software Engineering (Los Angeles, CA, May 1999), ACM Press, 277-284.

Digital Library

[15]

Podgurski, A., Masri, W., McCleese, Y., Wolff; F.G., and Yaaag, C. Estimation of softwaxe reliability by stratified sampling. ACM Transactions on Software Engineering and Methodology 8, 9 (July, 1999), 263-283.

Digital Library

[16]

Podgurski, A. and Yaaag, C. Partition testing, stratified sampling, and cluster analysis. Proceedings of the First ACM Symposium on Foundations of Software Engineering (Los Angeles, CA, December 1993), ACM Press, 169-181.

Digital Library

[17]

Reps, T., Ball, T., Das, M., mad Lares, J. The use of program profiling for softwaxe maintenance with applications to flae Year 2000 Problem. Proceedings of the 6th European Software Engineering Conference and 5th ACM SIGSOFT Symposium on the Foundations of Software Engineering (Zurich, Switzerland, September 1997), ACM Press, 432- 449.

Digital Library

[18]

Robbins, J. Debugging Applications, Microsoft Press, 2000.

Digital Library

[19]

Roflaermel, G., Untch, R., Chu, C., and Haxrold, M.J. Testcase prioritization: an empirical study. Proceedings of the International Conference on Software Maintenance (August, 1999), 179-188.

Digital Library

[20]

Steven, J., Chandra, P., Fleck, B., and Podgurski, A. jRapture: a capture/replay tool for observation-based testing. Proceedings of the 2000 International Symposium on Software Testing and Analysis (Portland, Oregon, August 2000).

Digital Library

[21]

Weyuker, E.J. and Jeng, B. Analyzing pal-fition testing strategies. IEEE Transactions on Software Engineering 17, 7 (July 1991), 703-711.

Digital Library

Cited By

Song YZhang XXie XChen SLiu QGao R(2024)SURE: A Visualized Failure Indexing Approach Using Program Memory SpectrumACM Transactions on Software Engineering and Methodology10.1145/367695833:8(1-43)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3676958
Majumder SChakraborty JBai GStolee KMenzies T(2023)Fair Enough: Searching for Sufficient Measures of FairnessACM Transactions on Software Engineering and Methodology10.1145/358500632:6(1-22)Online publication date: 29-Sep-2023
https://dl.acm.org/doi/10.1145/3585006
Chen LWu DMa WZhou YXu BLeung H(2020)How C++ Templates Are Used for Generic ProgrammingACM Transactions on Software Engineering and Methodology10.1145/335657929:1(1-49)Online publication date: 30-Jan-2020
https://dl.acm.org/doi/10.1145/3356579
Show More Cited By

Index Terms

Finding failures by cluster analysis of execution profiles

Recommendations

Pursuing failure: the distribution of program failures in a profile space

Observation-based testing calls for analyzing profiles of executions induced by potential test cases, in order to select a subset of executions to be checked for conformance to requirements. A family of techniques for selecting such a subset is ...
Estimation of software reliability by stratified sampling

A new approach to software reliability estimation is presented that combines operational testing with stratified sampling in order to reduce the number of program executions that must be checked manually for conformance to requirements. Automatic cluster ...
Pursuing failure: the distribution of program failures in a profile space
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering

Observation-based testing calls for analyzing profiles of executions induced by potential test cases, in order to select a subset of executions to be checked for conformance to requirements. A family of techniques for selecting such a subset is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '01: Proceedings of the 23rd International Conference on Software Engineering

July 2001

844 pages

ISBN:0769510507

Chairman:
Hausi A. Müller
Univ. of Victoria, Canada

Copyright © Copyright (c) 2001 Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 July 2001

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE01

Sponsor:

SIGSOFT

ICSE01: 23rd International Conference on Software Engineering

May 12 - 19, 2001

Ontario, Toronto, Canada

Acceptance Rates

ICSE '01 Paper Acceptance Rate 47 of 268 submissions, 18%;

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

85
Total Citations
View Citations
1,190
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song YZhang XXie XChen SLiu QGao R(2024)SURE: A Visualized Failure Indexing Approach Using Program Memory SpectrumACM Transactions on Software Engineering and Methodology10.1145/367695833:8(1-43)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3676958
Majumder SChakraborty JBai GStolee KMenzies T(2023)Fair Enough: Searching for Sufficient Measures of FairnessACM Transactions on Software Engineering and Methodology10.1145/358500632:6(1-22)Online publication date: 29-Sep-2023
https://dl.acm.org/doi/10.1145/3585006
Chen LWu DMa WZhou YXu BLeung H(2020)How C++ Templates Are Used for Generic ProgrammingACM Transactions on Software Engineering and Methodology10.1145/335657929:1(1-49)Online publication date: 30-Jan-2020
https://dl.acm.org/doi/10.1145/3356579
Golagha MLehnhoff CPretschner AIlmberger HZhang DMøller A(2019)Failure clustering without coverageProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330561(134-145)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.1145/3293882.3330561
Kulah YDincer BYilmaz CSavas E(2019)SpyDetectorInternational Journal of Information Security10.1007/s10207-018-0411-718:4(393-422)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10207-018-0411-7
He SLin QLou JZhang HLyu MZhang DLeavens GGarcia APăsăreanu C(2018)Identifying impactful service system problems via log analysisProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3236024.3236083(60-70)Online publication date: 26-Oct-2018
https://dl.acm.org/doi/10.1145/3236024.3236083
Kim MCheung SKim SLeavens GGarcia APăsăreanu C(2018)Which generated test failures are fault revealing? prioritizing failures based on inferred precondition violations using PAFProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3236024.3236058(679-690)Online publication date: 26-Oct-2018
https://dl.acm.org/doi/10.1145/3236024.3236058
Feyzi FParsa S(2018)FPA-FLJournal of Systems and Software10.1016/j.jss.2017.11.002136:C(39-58)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.jss.2017.11.002
Feyzi FParsa S(2018)A program slicing-based method for effective detection of coincidentally correct test casesComputing10.1007/s00607-018-0591-z100:9(927-969)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s00607-018-0591-z
Chen YYing MLiu DAlim AChen FChen MBultan TSen K(2017)Effective online software anomaly detectionProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092730(136-146)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092730
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten