Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments

1. Basili, V.R., Selby, R.W., Hutchens, D.H.: Experimentation in Software Engineering. IEEE Transactions on Software Engineering SE-12(7), 733–743 (1986)

   Google Scholar 

2. Rombach, H.D., Basili, V.R., Selby, R.W.: Experimental Software Engineering Issues: Critical Assessment and Future Directions. In: Rombach, H.D., Selby, R.W., Basili, V.R. (eds.) Experimental Software Engineering Issues: LNCS, vol. 706, Springer, Heidelberg (1993)

   Google Scholar 

3. Basili, V.R.: The Role of Experimentation in Software Engineering: Past, Current, and Future. In: Proceedings of the 18th International Conference on Software Engineering, Berlin, Germany, March 25-29, pp. 442–449 (1996)

   Google Scholar 

4. Tichy, W.F.: Should Computer Scientists Experiment More? 16 Reasons to Avoid Experimentation. IEEE Computer 31(5), 32–40 (1998)

   MathSciNet  Google Scholar 

5. Zelkowitz, M.V., Wallace, D.R.: Experimental Models for Validating Technology. IEEE Computer 31(5), 23–31 (1998)

   Google Scholar 

6. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering – An Introduction. Kluwer Academic Publishers, Boston (1999)

   Google Scholar 

7. Juristo, N., Moreno, A.: Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Boston (2001)

   MATH  Google Scholar 

8. Shadish, W.R., Cook, T.D., Campbell, D.T.: Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston (2002)

   Google Scholar 

9. Montgomery, D.C.: Design and Analysis of Experiments, 4th edn. John Wiley & Sons, New York (1997)

   MATH  Google Scholar 

10. Harrison, W.: N=1 An Alternative for Software Engineering Research? Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research, Workshop, 5 June, 2000 at 22nd Int. Conf. on Softw. Eng. (ICSE), Limerick, Ireland, pp. 39–44 (2000)

    Google Scholar 

11. Potts, C.: Software-Engineering Research Revisited. IEEE Software 10(5), 19–28 (1993)

    Article  Google Scholar 

12. Glass, R.L.: The Software-Research Crisis. IEEE Software 11(6), 42–47 (1994)

    Article  Google Scholar 

13. Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Koren, E.F., Vokac, M.: Conducting Realistic Experiments in Software Engineering, ISESE 2002 (First International Symposium on Empirical Softwarfe Engineering), Nara, Japan, October 3–4, pp. 17–26. IEEE Computer Society, Los Alamitos (2002)

    Google Scholar 

14. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El-Emam, K., Rosenberg, J.: Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering 28(8), 721–734 (2002)

    Article  Google Scholar 

15. Rogers, E.M.: Diffusion of Innovations, 4th edn. The Free Press, New York (1995)

    Google Scholar 

16. Arisholm, E., Sjøberg, D.I.K.: A Controlled Experiment in Industry to Evaluate the Effect of Responsibility-Driven Design Principles on Software Changeability for Different Categories of Professional (2003) (in preparation)

    Google Scholar 

17. Tichy, W.F.: Hints for Reviewing Empirical Work in Software Engineering. Empirical Software Engineering 5(4), 309–312 (2000)

    Article  MathSciNet  Google Scholar 

18. Basili, V.R., Shull, F., Lanubile, F.: Building Knowledge through Families of Experiments. IEEE Transactions on Software Engineering 25(4), 456–473 (1999)

    Article  Google Scholar 

19. Arisholm, E., Sjøberg, D.I.K.: A Controlled Experiment in Industry to Evaluate the Effect of Responsibility-Driven Design Principles on Software Changeability for Different Categories of Professionals (2003) (in preparation)

    Google Scholar 

20. Anda, B., Sjøberg, D.I.K., Jørgensen, M.: Quality and Understandability in Use Case Models. In: Knudsen, J.L. (ed.) ECOOP 2001. LNCS, vol. 2072, pp. 402–428. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

21. Jørgensen, M.: A Review of Studies on Expert Estimation of Software Development Effort. To appear in Journal of Systems and Software (2004)

    Google Scholar 

22. Jørgensen, M., Sjøberg: Impact of Software Effort Estimation on Software Work. Journal of Information and Software Technology 43, 939–948 (2001)

    Article  Google Scholar 

23. Anda, B., Sjøberg, D.I.K.: Towards an Inspection Technique for Use Case Models. In: SEKE 2002 (Fourteenth International Conference on Software Engineering and Knowledge Engineering), Ischia, Italy, July 15-19, pp. 127–134 (2002)

    Google Scholar 

24. Arisholm, E., Sjøberg, D.I.K., Jørgensen, M.: Assessing the Changeability of two Object-Oriented Design Alternatives – a Controlled Experiment. Empirical Software Engineering 6(3), 231–277 (2001)

    Article  MATH  Google Scholar 

25. Karahasanovic, A., Sjøberg, D.: Visualizing Impacts of Database Schema Changes – A Controlled Experiment. In: 2001 IEEE Symposium on Visual/Multimedia Approaches to Programming and Software Engineering, Stresa, Italy, September 5-7, pp. 358–365. IEEE Computer Society, Los Alamitos (2001)

    Google Scholar 

26. Sieber, J.E.: Protecting Research Subjects, Employees and Researchers: Implications for Software Engineering. Empirical Software Engineering 6(4), 329–341 (2001)

    Article  MATH  Google Scholar 

27. Davis, M.: When is a Volunteer Not a Volunteer? Empirical Software Engineering 6(4), 349–352 (2001)

    Article  Google Scholar 

28. Fenton, N.: Conducting and Presenting Empirical Software Engineering. Empirical Software Engineering 6(3), 195–200 (2001)

    Article  Google Scholar 

29. Anda, B.: Comparing Effort Estimates Based on Use Case Points with Expert Estimates. In: Empirical Assessment in Software Engineering (EASE 2002), Keele, UK, April 8-10 (2002)

    Google Scholar 

30. Jørgensen, M., Sjøberg, D.I.K.: The Impact of Customer Expectation on Software Development Effort Estimates (2003) (submitted for publication)

    Google Scholar 

31. Vokác, M., Tichy, W., Sjøberg, D.I.K., Arisholm, E., Aldrin, M.: A Controlled Experiment Comparing the Maintainability of Programs Designed with and without Design Patterns – a Replication in a real Programming Environment (2003) (submitted for Publication)

    Google Scholar 

32. Deligiannis, I.S., Shepperd, M., Webster, S., Roumeliotis, M.: A Review of Experimental Investigations into Object-Oriented Technology. Empirical Software Engineering 7(3), 193–232 (2002)

    Article  MATH  Google Scholar 

33. Jørgensen, M., Bygdås, S.: An Empirical Study of the Correlation between Development Efficiency and Software Development Tools. Technical Journal of Norwegian Telecom 11, 54–62 (1999)

    Google Scholar 

34. Jørgensen, M.: An Empirical Study of Software Maintenance Tasks. Journal of Software Maintenance 7, 27–48 (1995)

    Article  Google Scholar 

35. Arisholm, E., Sjøberg, D.I.K.: Towards a Framework for Empirical Assessment of Changeability Decay. Journal of Systems and Software 53, 3–14 (2000)

    Article  Google Scholar 

36. Karahasanovic, A., Sjøberg, D.I.K.: Visualising Impacts of Change in Evolving Object-Oriented Systems: An Explorative Study. In: Proceedings of the International Workshop on Graph-Based Tools (GraBaTs 2002), Barcelona, Spain, pp. 22–31 (2002)

    Google Scholar 

37. Anda, B., Sjøberg, D.I.K.: Applying Use Cases to Design versus Validate Class Diagrams – A Controlled Experiment Using a Professional Modelling Tool. Simula Research Laboratory Technical Report, No. 2003–01 (2003)

    Google Scholar 

38. Prechelt, L., Unger, B., Tichy, W.F., Brössler, P., Votta, L.G.: A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions. IEEE Transactions on Software Engineering 27(12), 1134–1144 (2001)

    Article  Google Scholar 

39. Syversen, E., Anda, B., Sjøberg, D.I.K.: An Evaluation of Applying Use Cases to Construct Design versus Validate Design. In: Hawaii International Conference on System Sciences (HICSS-36), Big Island, Hawaii, January 6-9 (2003)

    Google Scholar 

40. Briand, L.C., Bunse, C., Daly, J.W.: A Controlled Experiment for Evaluating Quality Guidelines on the Maintainability of Object-Oriented Designs. IEEE Transactions on Software Engineering 27(6), 513–530 (2001)

    Article  Google Scholar 

41. Karahasanovic, A., Sjøberg, D.I.K., Jørgensen, M.: Data Collection in Software Engineering Experiments. In: Information Resources Management Association Conference, Soft. Eng. Track. 2001, pp. 1027–1028. Idea Group Publishing, Toronto (2001)

    Google Scholar 

42. Arisholm, E., Sjøberg, D.I.K., Carelius, G.J., Lindsjørn, Y.: A Web-based Support Environment for Software Engineering Experiments. Nordic Journal of Computing 9(4), 231–247 (2002)

    Google Scholar 

Download references