Confirmation Bias and Time Pressure: A Family of Experiments in Software Testing
Pages 5203 - 5222
Abstract
<bold>Background</bold>: Software testers manifest confirmation bias (the cognitive tendency) when they design relatively more specification consistent test cases than specification inconsistent test cases. Time pressure may influence confirmation bias of testers per the research in the psychology discipline. <bold>Objective:</bold> We examine the manifestation of confirmation bias of software testers while designing functional test cases, and the effect of time pressure on confirmation bias in the same context. <bold>Method:</bold> We executed one internal and two external experimental replications concerning the original experimentation in Oulu. We analyse individual replications and meta-analyse our family of experiments (the original and replications) for joint results on the phenomena. <bold>Results:</bold> Our findings indicate a significant manifestation of confirmation bias by software testers during the designing of functional test cases. Time pressure significantly promoted confirmation bias among testers per the joint results of the family. The different experimental sites affected the results; however, we did not detect any effects of site-specific variables. <bold>Conclusion:</bold> Software testers should develop an outside-of-the-box thinking attitude to counter the manifestation of confirmation bias. Time pressure can be manoeuvred by centring manual suites on the designing and consequently the execution of inconsistent test cases, while automated testing focuses on consistent ones.
References
[1]
D. Arnott, “Cognitive biases and decision support systems development: A design science approach,” Inf. Syst. J., vol. 16, no. 1, pp. 55–78, 2006.
[2]
R. Mohanani, I. Salman, B. Turhan, P. Rodriguez, and P. Ralph, “Cognitive biases in software engineering: A systematic mapping study,” IEEE Trans. Softw. Eng., vol. 46, no. 12, pp. 1318–1339, Dec. 2018. [Online]. Available: https://ieeexplore.ieee.org/document/8506423/
[3]
R. Mohanani, P. Ralph, B. Turhan, and V. Mandic, “How templated requirements specifications inhibit creativity in software engineering,” IEEE Trans. Softw. Eng., vol. 48, no. 10, pp. 4074–4086, Oct. 2022.
[4]
G. Calikli and A. Bener, “Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance,” in Proc. 6th Int. Conf. Predictive Models Softw. Eng. (PROMISE), 2010, pp. 1–11. Accessed: Oct. 31, 2015. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1868328.1868344
[5]
G. Calikli, A. Bener, T. Aytac, and O. Bozcan, “Towards a metric suite proposal to quantify confirmation biases of developers,” in Proc. Int. Symp. Empirical Softw. Eng. Meas., 2013, pp. 363–372.
[6]
L. M. Leventhal, B. Teasley, D. S. Rohlman, and K. Instone, “Positive test bias in software testing among professionals: A review,” in Proc. Int. Conf. Human-Comput. Interact., 1993, pp. 210–218.
[7]
W. Stacy and J. MacMillan, “Cognitive bias in software engineering,” Commun. ACM, vol. 38, no. 6, pp. 57–63, 1995.
[8]
B. E. Teasley, L. M. Leventhal, C. R. Mynatt, and D. S. Rohlman, “Why software testing is sometimes ineffective: Two applied studies of positive test strategy,” J. Appl. Psychol., vol. 79, no. 1, pp. 142–155, 1994.
[9]
G. Calikli and A. Bener, “Empirical analysis of factors affecting confirmation bias levels of software engineers,” Softw. Qual. J., vol. 23, pp. 695–722, Dec. 2015. [Online]. Available: http://link.springer.com/10.1007/s11219-014-9250-6
[10]
G. Calikli, A. Bener, and B. Arslan, “An analysis of the effects of company culture, education and experience on confirmation bias levels of software developers and testers,” in Proc. ACM/IEEE 32nd Int. Conf. Softw. Eng., vol. 2, 2010, pp. 187–190.
[11]
M. V. Mäntylä, K. Petersen, T. O. A. Lehtinen, and C. Lassenius, “Time pressure: A controlled experiment of test case development and requirements review,” in Proc. 36th Int. Conf. Softw. Eng. (ICSE), 2014, pp. 83–94. Accessed: Nov. 18, 2015. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2568225.2568245
[12]
I. Salman, P. Rodriguez, B. Turhan, A. Tosun, and A. Gureller, “What leads to a confirmatory or disconfirmatory behaviour of software testers?” IEEE Trans. Softw. Eng., vol. 48, no. 4, pp. 1351–1368, Apr. 2022.
[13]
R. Ramač et al., “Prevalence, common causes and effects of technical debt: Results from a family of surveys with the IT industry,” J. Syst. Softw., vol. 184, Feb. 2022, Art. no.
[14]
H. Shah, M. J. Harrold, and S. Sinha, “Global software testing under deadline pressure: Vendor-side experiences,” Inf. Softw. Technol., vol. 56, no. 1, pp. 6–19, 2014.
[15]
M. Cataldo, “Sources of errors in distributed development projects: Implications for collaborative tools,” in Proc. ACM Conf. Comput. Supported Cooperative Work, 2010, pp. 281–290. Accessed: Mar. 5, 2018. [Online]. Available: http://portal.acm.org/citation.cfm?id=1718971
[16]
F. P. Seth, O. Taipale, and K. Smolander, “Organizational and customer related challenges of software testing: An empirical study in 11 software companies,” in Proc. IEEE 8th Int. Conf. Res. Challenges Inf. Sci. (RCIS), 2014, pp. 1–12.
[17]
N. Baddoo and T. Hall, “De-motivators for software process improvement: An analysis of practitioners’ views,” J. Syst. Softw., vol. 66, no. 1, pp. 23–33, 2003.
[18]
I. Hernandez and J. L. Preston, “Disfluency disrupts the confirmation bias,” J. Exp. Social Psychol., vol. 49, no. 1, pp. 178–182, 2013. [Online]. Available: http://linkinghub.elsevier.com/retrieve/pii/S002210311200176X
[19]
K. Ask and P. A. Granhag, “Motivational bias in criminal investigators’ judgments of witness reliability,” J. Appl. Social Psychol., vol. 37, no. 3, pp. 561–591, 2007.
[20]
I. Salman, B. Turhan, and S. Vegas, “A controlled experiment on time pressure and confirmation bias in functional software testing,” Empirical Softw. Eng., vol. 24, no. 4, pp. 1727–1761, Dec. 2018. [Online]. Available: http://link.springer.com/10.1007/s10664-018-9668-8
[21]
A. Santos et al., “A family of experiments on test-driven development,” Empirical Softw. Eng., vol. 26, no. 3, pp. 1–53, 2021. [Online]. Available: http://arxiv.org/abs/2011.11942
[22]
J. Carver, “Towards reporting guidelines for experimental replications: A proposal,” in Proc. 1st Int. Workshop Replication Empirical Softw. Eng., vol. 1, 2010, pp. 1–4. Accessed: Mar. 12, 2017. [Online]. Available: http://carver.cs.ua.edu/Papers/Conference/2010/2010_RESER.pdf
[23]
N. Juristo, S. Vegas, M. Solari, S. Abrah∼ao, and I. Ramos, “A process for managing interaction between experimenters to get useful similar replications,” Inf. Softw. Technol., vol. 55, no. 2, pp. 215–225, 2013.
[24]
A. Santos, O. Gomez, and N. Juristo, “Analyzing families of experiments in SE: A systematic mapping study,” IEEE Trans. Softw. Eng., vol. 46, no. 5, pp. 566–583, May 2020.
[25]
N. Juristo and O. S. Gómez, “Replication of software engineering experiments,” in Empirical Software Engineering and Verification: LASER 2008-2010. Lecture Notes in Computer Science, B. Meyer and M. Nordio, Eds., Berlin, Germany: Springer, vol. 7007, 2012, pp. 60–88. [Online]. Available: http://link.springer.com/10.1007/978-3-642-25231-0_2
[26]
N. Juristo and S. Vegas, “Using differences among replications of software engineering experiments to gain knowledge,” in Proc. 3rd Int. Symp. Empirical Softw. Eng. Meas. (ESEM), 2009, pp. 356–366.
[27]
M. G. Mendonça et al., “A framework for software engineering experimental replications,” in Proc. IEEE Int. Conf. Eng. Complex Comput. Syst., (ICECCS), 2008, pp. 203–212.
[28]
A. Santos, S. Vegas, M. Oivo, and N. Juristo, “A procedure and guidelines for analyzing groups of software engineering replications,” IEEE Trans. Softw. Eng., vol. 47, no. 9, pp. 1742–1763, Sep. 2019.
[29]
O. S. Gomez, N. Juristo, and S. Vegas, “Understanding replication of experiments in software engineering: A classification,” Inf. Softw. Technol., vol. 56, no. 8, pp. 1033–1048, 2014.
[30]
F. J. Shull, J. C. Carver, S. Vegas, and N. Juristo, “The role of replications in Empirical Software Engineering,” Empirical Softw. Eng., vol. 13, no. 2, pp. 211–218, Apr. 2008.
[31]
V. Mandić, J. Markkula, and M. Oivo, “Towards multi-method research approach in empirical software engineering,” in Proc. 10th Int. Conf. Product-Focused Softw. Process Improvement, 2009, pp. 96–110.
[32]
T. Gilovich, D. Griffin, and D. Kahneman, Heuristics and Biases, 8th ed. Cambridge, U.K.: Cambridge Univ. Press, 2002.
[33]
A. Tversky and D. Kahneman, “Judgement under uncertainty: Heuristics and biases,” Oregon Res. Inst. Res. Bull., Tech. Rep., 1973.
[34]
D. Arnott, “A taxonomy of decision biases,” School Inf. Manag. Syst., Monash Univ., Melbourne, Australia, Tech. Rep., 1998. Accessed: Nov. 18, 2015. [Online]. Available: http://www.sims.monash.edu.au/staff/darnott/biastax.pdf
[35]
D. Kahneman, D. Lovallo, and O. Sibony, “Before you make that big decision…,” Harvard Bus. Rev., pp. 51–60, Jun. 2011. Accessed: Jun. 8, 2019. [Online]. Available: http://website.aub.edu.lb/units/ehmu/Documents/before-you-make-that-big-decision.pdf
[36]
D. Arnott and S. Gao, “Behavioral economics for decision support systems researchers,” Decis. Support Syst., vol. 122, Feb. 2019, Art. no.
[37]
L. M. Leventhal, B. E. Teasley, and D. S. Rohlman, “Analyses of factors related to positive test bias in software testing,” Int. J. Human-Comput. Stud., vol. 41, pp. 717–749, 1994.
[38]
I. Salman, “The effects of confirmation bias and time pressure in software testing,” Ph.D. dissertation, Univ. Oulu, Oulu, Finland, 2019.
[39]
M. Kuutila, M. Mäntylä, U. Farooq, and M. Claes, “Time pressure in software engineering: A systematic review,” Inf. Softw. Technol., vol. 121, May 2020, Art. no. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S0950584920300045?via%3Dihub
[40]
M. Kuutila, M. Mantyla, U. Farooq, and M. Claes, “What do we know about time pressure in software development?” IEEE Softw., vol. 38, no. 5, pp. 32–38, Sep./Oct. 2021.
[41]
M. Kuutila, M. V. Mantyla, M. Claes, and M. Elovainio, “Reviewing literature on time pressure in software engineering and related professions: Computer assisted interdisciplinary literature review,” in Proc. IEEE/ACM 2nd Int. Workshop Emotion Awareness Softw. Eng. (SEmotion), 2017, pp. 54–59.
[42]
O. Hazzan, O. Hazzan, Y. Dubinsky, and Y. Dubinsky, “The software engineering timeline: A time management perspective,” in IEEE Int. Conf. Softw.-Sci., Technol. Eng. (SwSTE), 2007, pp. 95–103.
[43]
N. Nan and D. E. Harter, “Impact of budget and schedule pressure on software development cycle time and effort,” IEEE Trans. Softw. Eng., vol. 35, no. 5, pp. 624–637, Sep./Oct. 2009.
[44]
S. Linßen, D. Basten, and J. Richter, “Antecedents and consequences of time pressure in Scrum projects: Insights from a qualitative study,” in Proc. 51st Hawaii Int. Conf. Syst. Sci., 2018, pp. 4835–4844.
[45]
NASA, “NASA Task Load Index,” Human Mental Workload, vol. 1, no. 6, 2006, pp. 21–21. Accessed: Mar. 18, 2016. [Online]. Available: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=16243365
[46]
Human Performance Group at NASA's Ames Research Center, “NASA Task Load Index (TLX),” 1987. Accessed: May 4, 2022. [Online]. Available: https://humansystems.arc.nasa.gov/groups/TLX/publications.php
[47]
“7.2.6—Model assumptions and diagnostics assumptions—STAT 505.” Accessed: Jan. 6, 2023. [Online]. Available: https://online.stat.psu.edu/stat505/lesson/7/7.2/7.2.6
[48]
C. O. Fritz, P. E. Morris, and J. J. Richler, “Effect size estimates: Current use, calculations, and interpretation.” J. Exp. Psychol. General, vol. 141, no. 1, pp. 2–18, 2012. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/21823805
[49]
M. Borenstein, L. V. Hedges, J. P. Higgins, and H. R. Rothstein, Introduction to Meta-Analysis, 1st ed. Hoboken, NJ, USA: Wiley, Jan. 2009.
[50]
A. Field, J. Miles, and Z. Field, Discovering Statistics Using R. Newbury Park, CA, USA: Sage, 2012.
[51]
J. J. Randolph, “Free-marginal multirater kappa: An alternative to Fleiss fixed-marginal multirater kappa,” in Proc. Joensuu Univ. Learn. Instruction Symp., 2005, pp. 1–20.
[52]
J. J. Randolph, “Online kappa calculator,” 2008. Accessed: Feb. 8, 2018. [Online]. Available: http://justusrandolph.net/kappa/#dInfo
[53]
D. Falessi et al., “Empirical software engineering experts on the use of students and professionals in experiments,” Empirical Softw. Eng., vol. 23, no. 1, pp. 452–489, 2018.
[54]
I. Salman, A. T. Misirli, and N. Juristo, “Are students representatives of professionals in software engineering experiments?” in Proc. Int. Conf. Softw. Eng., vol. 1, 2015, pp. 666–676.
[55]
J. Hox, “Multilevel modeling: When and why,” in Classification, Data Analysis, and Data Highways. Berlin, Germany: Springer-Verlag, 1998, pp. 147–154.
[56]
A. Field, Discovering Statistics Using IBM SPSS Statistics, 5th ed. Newbury Park, CA, USA: Sage, 2018.
[57]
L. M. Leventhal, B. E. Teasley, and D. S. Rohlman, “Analyses of factors related to positive test bias in software testing,” Int. J. Human-Comput. Stud., vol. 41, no. 5, pp. 717–749, Nov. 1994. Accessed: Apr. 29, 2017. [Online]. Available: http://linkinghub.elsevier.com/retrieve/pii/S1071581984710792
[58]
A. Causevic, R. Shukla, S. Punnekkat, and D. Sundmark, “Effects of negative testing on TDD: An industrial experiment,” in Proc. Int. Conf. Agile Softw. Develop., 2013, pp. 91–105. Accessed: Oct. 31, 2015. [Online]. Available: http://link.springer.com/chapter/10.1007/978-3-642-38314-4_7
[59]
H. Topi, J. S. Valacich, and J. A. Hoffer, “The effects of task complexity and time availability limitations on human performance in database query tasks,” Int. J. Human Comput. Stud., vol. 62, no. 3, pp. 349–379, 2005.
[60]
M. V. Mäntylä and J. Itkonen, “More testers—The effect of crowd size and time restriction in software testing,” Inf. Softw. Technol., vol. 55, no. 6, pp. 986–1003, 2013.
[61]
I. Salman and B. Turhan, “Effect of time-pressure on perceived and actual performance in functional software testing,” in Proc. Int. Conf. Softw. Syst. Process (ICSSP), 2018, pp. 130–139.
[62]
R. Baskerville, L. Levine, J. Pries-Heje, and S. Slaughter, “How Internet software companies negotiate quality,” Computer, vol. 34, no. 5, pp. 51–57, May 2001.
[63]
J. Verner, J. Sampson, and N. Cerpa, “What factors lead to software project failure?” in Proc. 2nd Int. Conf. Res. Challenges Inf. Sci., Piscataway, NJ, USA: IEEE Press, Jun. 2008, pp. 71–80.
[64]
A. Deak, T. Stålhane, and G. Sindre, “Challenges and strategies for motivating software testing personnel,” Inf. Softw. Technol., vol. 73, pp. 1–15, May 2016.
[65]
M. Cataldo and J. D. Herbsleb, “Factors leading to integration failures in global feature-oriented development,” in Proc. 33rd Int. Conf. Softw. Eng., New York, NY, USA: ACM, May 2011, pp. 161–170.
[66]
D. N. Wilson and T. Hall, “Perceptions of software quality: A pilot study,” Softw. Qual. J., vol. 7, no. 1, pp. 67–75, 1998. [Online]. Available: http://link.springer.com/article/10.1023/B:SQJO.0000042060.88173.fe
[67]
F. Paetsch, A. Eberlein, and F. Maurer, “Requirements engineering and agile software development,” in Proc. 12th IEEE Int. Workshops Enabling Technol. Infrastructure Collaborative Enterprises (WETICE), 2003, pp. 308–313.
[68]
O. Albayrak, H. Kurtoglu, and M. Biçakçi, “Incomplete software requirements and assumptions made by software engineers,” in Proc. 16th Asia-Pacific Softw. Eng. Conf., 2009, pp. 333–339.
[69]
S. Vegas, I. Salman, P. Riofrío, and N. Juristo, “A method for aggregating families of experiments in software engineering a step by step guide,” Empirical Softw. Eng., early access, 2023.
[70]
P. Ralph, “Possible core theories for software engineering,” in Proc. 2nd SEMAT Workshop General Theory Softw. Eng. (GTSE), Piscataway, NJ, USA: IEEE Press, 2013, 2013, pp. 35–38.
[71]
P. Ralph, “Toward a theory of debiasing software development,” in Proc. Lecture Notes Bus. Inf. Process., vol. 93, 2011, pp. 92–105. [Online]. Available: http://link.springer.com/10.1007/978-3-642-25676-9
[72]
C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering. Berlin, Germany: Springer-Verlag, 2012.
[73]
National Bioethics Advisory Commission, Ethical and Policy Issues in International Research: Clinical Trials in Developing Countries. Washington, DC, USA: NBAC, 2012. Accessed: Oct. 22, 2018. [Online]. Available: https://bioethicsarchive.georgetown.edu/nbac/clinical/Vol1.pdf
Index Terms
- Confirmation Bias and Time Pressure: A Family of Experiments in Software Testing
Index terms have been assigned to the content through auto-classification.
Recommendations
Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance
PROMISE '10: Proceedings of the 6th International Conference on Predictive Models in Software EngineeringBackground: During all levels of software testing, the goal should be to fail the code. However, software developers and testers are more likely to choose positive tests rather than negative ones due to the phenomenon called confirmation bias. ...
Confirmation Bias in Complex Analyses
Most research works investigating the confirmation bias has used abstract experimental tasks where participants drew inferences from just a few items of evidence. The experiment reported in this paper investigated the confirmation bias in a complex ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© 2023 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
Publisher
IEEE Press
Publication History
Published: 08 November 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in