Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/2610384.2610418acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

An empirical study of injected versus actual interface errors

Published: 21 July 2014 Publication History

Abstract

The reuse of software components is a common practice in commercial applications and increasingly appearing in safety critical systems as driven also by cost considerations. This practice puts dependability at risk, as differing operating conditions in different reuse scenarios may expose residual software faults in the components. Consequently, software fault injection techniques are used to assess how residual faults of reused software components may affect the system, and to identify appropriate counter-measures. As fault injection in components’ code suffers from a number of practical disadvantages, it is often replaced by error injection at the component interface level. However, it is still an open issue, whether such injected errors are actually representative of the effects of residual faults. To this end, we propose a method for analyzing how software faults turn into interface errors, with the ultimate aim of supporting more representative interface error injection experiments. Our analysis in the context of widely used software libraries reveals that existing interface error models are not suitable for emulating software faults, and provides useful insights for improving the representativeness of interface error injection.

References

[1]
J. Andrews, L. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In ICSE, pages 402–411, 2005.
[2]
J. Arlat, J. Fabre, M. Rodríguez, and F. Salles. Dependability of COTS Microkernel-Based Systems. IEEE Trans. Comput., 51(2):138–163, 2002.
[3]
A. Avizienis, J. Laprie, B. Randell, and C. Landwehr. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Secure Comput., 1(1):11–33, 2004.
[4]
J. Barton, E. Czeck, Z. Segall, and D. Siewiorek. Fault Injection Experiments using FIAT. IEEE Trans. Comput., 39(4):575–582, 1990.
[5]
S. Chandra and P. M. Chen. How fail-stop are faulty programs? In FTCS, pages 240–249, 1998.
[6]
R. Chillarege, I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. Ray, and M. Wong. Orthogonal Defect Classification–A Concept for In-Process Measurements. IEEE Trans. Softw. Eng., 18(11):943–956, 1992.
[7]
J. Christmansson and R. Chillarege. Generation of an Error Set that Emulates Software Faults based on Field Data. In FTCS, pages 304–313, 1996.
[8]
J. Christmansson, M. Hiller, and M. Rimen. An Experimental Comparison of Fault and Error Injection. In ISSRE, pages 369–378, 1998.
[9]
D. Cotroneo, A. Lanzaro, R. Natella, and R. Barbosa. Experimental Analysis of Binary-Level Software Fault Injection in Complex Software. In EDCC, pages 162–172, 2012.
[10]
D. Cotroneo and R. Natella. Fault Injection for Software Certification. IEEE Security & Privacy, 11(4):38––45, 2013.
[11]
M. Daran and P. Thévenod-Fosse. Software Error Analysis: A Real Case Study Involving Real Faults and Mutations. ACM Soft. Eng. Notes, 21(3):158–171, 1996.
[12]
H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans. Softw. Eng., pages 733–752, 2006.
[13]
J. Durães and H. Madeira. Emulation of Software faults: A Field Data Study and a Practical Approach. IEEE Trans. Softw. Eng., 32(11):849–867, 2006.
[14]
S. Eldh, S. Punnekkat, H. Hansson, and P. Jönsson. Component Testing Is Not Enough–A Study of Software Faults in Telecom Middleware. In Testing of Softw. and Comm. Sys., pages 74–89. 2007.
[15]
A. K. Ghosh and M. Schmid. An approach to testing COTS software for robustness to operating system exceptions and errors. In ISSRE, pages 166–174, 1999.
[16]
M. Hiller, A. Jhumka, and N. Suri. EPIC: Profiling the propagation and effect of data errors in software. IEEE Trans. Comput., 53(5):512–530, 2004.
[17]
Hipp, Wyrick & Company, Inc. Well-Known Users of SQLite, 2013. http://www.sqlite.org/famous.html.
[18]
J. W. Hunt and T. G. Szymanski. A fast algorithm for computing longest common subsequences. Comm. ACM, 20(5):350–353, 1977.
[19]
T. Jarboui, J. Arlat, Y. Crouzet, K. Kanoun, and T. Marteau. Analysis of the Effects of Real and Injected Software Faults: Linux as a Case Study. In PRDC, pages 51–58, 2002.
[20]
Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng., 37(5):649–678, 2011.
[21]
G. Kanawati, N. Kanawati, and J. Abraham. FERRARI: A Flexible Software-Based Fault and Error Injection System. IEEE Trans. Comput., 44(2):248–260, 1995.
[22]
K. Kanoun and L. Spainhower. Dependability Benchmarking for Computer Systems. Wiley-IEEE Computer Society, 2008.
[23]
W.-I. Kao, R. Iyer, and D. Tang. FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults. IEEE Trans. Softw. Eng., 19(11):1105–1118, 1993.
[24]
P. Koopman and J. DeVale. The Exception Handling Effectiveness of POSIX Operating Systems. IEEE Trans. Softw. Eng., 26(9):837–848, 2000.
[25]
N. Laranjeiro, M. Vieira, and H. Madeira. A Technique for Deploying Robust Web Services. IEEE Trans. Services Comput., 7(1):68–81, 2014.
[26]
H. Madeira, D. Costa, and M. Vieira. On the Emulation of Software Faults by Software Fault Injection. In DSN, pages 417–426, 2000.
[27]
P. Marinescu and G. Candea. Efficient testing of recovery code using fault injection. ACM Trans. Computer Sys., 29(4):11:1–11:38, 2011.
[28]
R. Moraes, R. Barbosa, J. Durães, N. Mendes, E. Martins, and H. Madeira. Injection of Faults at Component Interfaces and Inside the Component Code: Are They Equivalent? In EDCC, pages 53–64, 2006.
[29]
R. Moraes, J. Durães, R. Barbosa, E. Martins, and H. Madeira. Experimental Risk Assessment and Comparison using Software Fault Injection. In DSN, pages 512–521, 2007.
[30]
R. Natella, D. Cotroneo, J. A. Durães, and H. S. Madeira. On Fault Representativeness of Software Fault Injection. IEEE Trans. Softw. Eng., 39(1):80–96, 2013.
[31]
N. Nethercote and J. Seward. Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation. ACM Sigplan Not., 42(6):89–100, 2007.
[32]
W. Ng and P. Chen. The Systematic Improvement of Fault Tolerance in the Rio File Cache. In FTCS, pages 76–83, 1999.
[33]
W. T. Ng and P. M. Chen. The design and verification of the rio file cache. IEEE Trans. Comput., 50(4):322–337, 2001.
[34]
ObjectWeb Consortium. CARDAMOM - An Enterprise Middleware for Building Mission and Safety Critical Applications, 2013. http://cardamom.ow2.org.
[35]
H. Patil, C. Pereira, M. Stallcup, G. Lueck, and J. Cownie. PinPlay: a framework for deterministic replay and reproducible analysis of parallel programs. In CGO, pages 2–11, 2010.
[36]
J. Seward. bzip2 Home Page, 2013. http://www.bzip.org.
[37]
D. J. Sheskin. Handbook of parametric and nonparametric statistical procedures. Chapman and Hall/CRC, 2003.
[38]
S. Tallam, C. Tian, R. Gupta, and X. Zhang. Enabling Tracing of Long-Running Multithreaded Programs Via Dynamic Execution Reduction. In ISSTA, pages 207–218, 2007.
[39]
D. Tang and H. Hecht. An Approach to Measuring and Assessing Dependability for Critical Software Systems. In ISSRE, pages 192–202, 1997.
[40]
M. Vieira and H. Madeira. A dependability benchmark for OLTP application environments. In VLDB, pages 742–753, 2003.
[41]
J. Voas. Certifying off-the-shelf software components. IEEE Computer, 31(6):53–59, 1998.
[42]
J. Voas, F. Charron, G. McGraw, K. Miller, and M. Friedman. Predicting How Badly “Good” Software Can Behave. IEEE Softw., 14(4):73–83, 1997.
[43]
E. J. Weyuker. Testing component-based software: A cautionary tale. IEEE Softw., 15(5):54–59, 1998.
[44]
S. Winter, C. Sârbu, N. Suri, and B. Murphy. The impact of fault models on software robustness evaluations. In ICSE, pages 51–60, 2011.

Cited By

View all
  • (2024)Automating Fault Test Cases Generation and Execution for Automotive Safety Validation via NLP and HIL SimulationSensors10.3390/s2410314524:10(3145)Online publication date: 15-May-2024
  • (2024)An Intelligent Secure Fault Classification and Identification Scheme for Mining Valuable Information in IIoTIEEE Systems Journal10.1109/JSYST.2024.343718518:3(1705-1716)Online publication date: Sep-2024
  • (2023)Fault Injection for TensorFlow ApplicationsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.317593020:4(2677-2695)Online publication date: 1-Jul-2023
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis
July 2014
460 pages
ISBN:9781450326452
DOI:10.1145/2610384
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Experimental Dependability Assessment
  2. FMECA
  3. Off-The-Shelf Software
  4. Software Components
  5. Software Fault/Error Injection
  6. Software Faults/Errors

Qualifiers

  • Research-article

Conference

ISSTA '14
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)22
  • Downloads (Last 6 weeks)3
Reflects downloads up to 22 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Automating Fault Test Cases Generation and Execution for Automotive Safety Validation via NLP and HIL SimulationSensors10.3390/s2410314524:10(3145)Online publication date: 15-May-2024
  • (2024)An Intelligent Secure Fault Classification and Identification Scheme for Mining Valuable Information in IIoTIEEE Systems Journal10.1109/JSYST.2024.343718518:3(1705-1716)Online publication date: Sep-2024
  • (2023)Fault Injection for TensorFlow ApplicationsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.317593020:4(2677-2695)Online publication date: 1-Jul-2023
  • (2023)Adaptive Tracing and Fault Injection based Fault Diagnosis for Open Source Server Software2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00076(729-740)Online publication date: 22-Oct-2023
  • (2023)Run-time failure detection via non-intrusive event analysis in a large-scale cloud computing platformJournal of Systems and Software10.1016/j.jss.2023.111611198(111611)Online publication date: Apr-2023
  • (2022)Fault Injection Analytics: A Novel Approach to Discover Failure Modes in Cloud-Computing SystemsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2020.302528919:3(1476-1491)Online publication date: 1-May-2022
  • (2022)Building Secured Software Defined Networks by Analyzing Anomaly Detection Algorithms on Various AttacksICT Analysis and Applications10.1007/978-981-19-5224-1_10(89-96)Online publication date: 6-Nov-2022
  • (2021)Software Fault Localization through Aggregation-Based Neural Ranking for Static and Dynamic Features SelectionSensors10.3390/s2121740121:21(7401)Online publication date: 7-Nov-2021
  • (2021)Measuring Software Maintainability with Naïve Bayes ClassifierEntropy10.3390/e2302013623:2(136)Online publication date: 22-Jan-2021
  • (2021)Dependability Assessment of the Android OS Through Fault InjectionIEEE Transactions on Reliability10.1109/TR.2019.295438470:1(346-361)Online publication date: Mar-2021
  • Show More Cited By

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media