survey

Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming

Author:

Tim StorerAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 50, Issue 4

Article No.: 47, Pages 1 - 32

https://doi.org/10.1145/3084225

Published: 25 August 2017 Publication History

Abstract

The use of software is pervasive in all fields of science. Associated software development efforts may be very large, long lived, and complex, requiring the commitment of significant resources. However, several authors have argued that the “gap” or “chasm” between software engineering and scientific programming is a serious risk to the production of reliable scientific results, as demonstrated in a number of case studies. This article reviews the research that addresses the gap, exploring how both software engineering and research practice may need to evolve to accommodate the use of software in science.

References

[1]

Karen Ackroyd, Steve H. Kinder, Geoff R. Mant, Mike C. Miller, Christine A. Ramsdale, and Paul C. Stephenson. 2008. Scientific software development at a research facility. IEEE Softw. 25, 4 (July/August 2008), 44--51.

Digital Library

[2]

Stan Ahalt, Larry Band, Barbara Minsker, Margaret Palmer, Michael Tiemann, Ray Idaszak, Chris Lenhardt, and Mary Whitton. 2013. Water science software institute an open source engagement process, see Reference Carver [27], 40--47.

[3]

Benjamin A. Allan, Robert Armstrong, David E. Bernholdt, Felipe Bertrand, Kenneth Chiu, Tamara L. Dahlgren, Kostadin Damevski, Wael R. Elwasif, Thomas G. W. Epperly, Madhusudhan Govindaraju, Daniel S. Katz, James A. Kohl, Manoj Krishnan, Gary Kumfert, J. Walter Larson, Sophia Lefantzi, Michael J. Lewis, Allen D. Malony, Lois C. McInnes, Jarek Nieplocha, Boyana Norris, Steven G. Parker, Jaideep Ray, Sameer Shende, Theresa L. Windus, and Shujia Zhou. 2006. A component architecture for high performance scientific computing. Int. J. High Perform. Comput. Appl. 20, 2 (Summer 2006), 163--202.

Digital Library

[4]

Peter Bøgh Andersen, Florian Prange, and Søren Serritzlew. 2008. Software engineering as a part of scientific practice. Available at http://imv.au.dk/&sim;pba/Homepagematerial/publicationfolder/softwareengineering.pdf.

[5]

Richard G. Anderson, William H. Greene, B. D. McCullough, and H. D. Vinod. 2005. The Role of Data and Program Code Archives in the Future of Economic Research. Working Paper 2005-014C. Federal Reserve Bank of St. Louis, MO.

[6]

Sylwester Arabas, Michael R. Bareford, Lakshitha R. de Silva, Ian P. Gent, Benjamin M. Gorman, Masih Hajiarabderkani, Tristan Henderson, Luke Hutton, Alexander Konovalov, Lars Kotthoff, Ciaran McCreesh, Miguel A. Nacenta, Ruma R. Paul, Karen E. J. Petrie, Abdul Razaq, Daniël Reijsbergen, and Kenji Takeda. 2014. An Open and Reproducible Paper on Openness and Reproducibility of Papers in Computational Science (September 2014). Retrieved from https://github.com/larskotthoff/recomputation-ss-paper/.

[7]

Ritu Arora, Purushotham Bangalore, and Marjan Mernik. 2009. Developing scientific applications using generative programming, see Reference [24], 51--58.

[8]

Robert Axelrod. 1997. Advancing the art of simulation in the social sciences. In Simulating Social Phenomena, Rosaria Conte, Rainer Hegselmann, and Pietro Rainer (Eds.). Lecture Notes in Economics and Mathematical Systems, Vol. 456. Springer Verlag, 21--40.

[9]

Roscoe A. Bartlett. 2009. Integration strategies for computational science 8 engineering, see Reference Carver [24], 35--42.

[10]

Victor R. Basili, Jeffrey Carver, Daniela Cruzes, Lorin Hochstein, Jeffrey K. Hollingsworth, Forrest Shuill, and Marvin V. Zelkowitz. 2008. Understanding the high performance computing community: A software engineer’s perspective. IEEE Softw. 25, 4 (July/August 2008), 29--36.

Digital Library

[11]

Sean Bechhofer, Iain E. Buchan, David De Roure, Paolo Missier, John D. Ainsworth, Jiten Bhagat, Philip A. Couch, Don Cruickshank, Mark Delderfield, Ian Dunlop, Matthew Gamble, Danius T. Michaelides, Stuart Owen, David R. Newman, Shoaib Sufi, and Carole A. Goble. 2013. Why linked data is not enough for scientists. Future Gen. Comput. Syst. 29, 2 (2013), 599--611.

Digital Library

[12]

Kent Beck and Cynthia Andres. 2005. Extreme Programming Explained (2nd ed.). Addison Wesley/Pearson Education.

Digital Library

[13]

Mats Bergdahl, Manfred Ehling, Eva Elvers, Erika Földesi, Thomas Körner, Andrea Kron, Kornelia Mag, Peter Lohauß, Vera Morais, Anja Nimmergut, Hans Viggo Sæbø, Ulrike Timm, and Maria Jo ao Zilhão. 2007. Handbook on Data Quality Assessment Methods and Tools. European Commission, Wiesbaden.

[14]

David Bernholdt, Shishir Bharathi, David Brown, Kasidit Chanchio, Meili Chen, Ann Chervenak, Luca Cinquini, Bob Drach, Ian Foster, Peter Fox, Jose Garcia, Carl Kesselman, Rob Markel, Don Middleton, Veronika Nefedova, Line Pouchard, Arie Shoshani, Alex Sim, Gary Strand, and Dean Williams. 2005. The earth system grid: Supporting the next generation of climate modeling research. Proc. IEEE 93, 3 (March 2005), 485--494.

[15]

Robin M. Betz and Ross C. Walker. 2013. Implementing continuous integration software in an established computational chemistry software package, see Reference Carver [27], 68--74.

[16]

Charles Billie. 2002. Patterns in scientific software: An introduction. Comput. Sci. Eng. 4, 3 (May/June 2002), 48--53.

Digital Library

[17]

Stephen M. Blackburn, Amer Diwan, Matthias Hauswirth, Peter F. Sweeney, Jos Nelson Amaral, Vlastimil Babka, Walter Binder, Tim Brecht, Lubomr Bulej, Lieven Eeckhout, Sebastian Fischmeister, Daniel Frampton, Robin Garner, Laurie J. Hendren, Andy Georges, Michael Hind, Antony L. Hosking, Richard E. Jones, Tomas Kalibera, Philippe Moret, Victor Pankratius, Nathaniel Nystrom, and Petr Tuma. 2012. Can You Trust Your Experimental Results? Technical Report 1. Evaluate Collaboratory.

[18]

Martin Blom. 2012. Is scrum and XP suitable for CSE development? See Reference Sloot et al. [167], 1511--1517.

[19]

Ronald F. Boisvert and Ping Tak Peter Tang (Eds.). 2001. The Architecture of Scientific Software. Proceedings of the IFIP TC2/WG2.5 Working Conference on the Architecture of Scientific Software. IFIP Advances in Information and Communication Technology, Vol. 60. Springer.

[20]

Frederick P. Brooks, Jr. 1995. The Mythical Man-Month (9 ed.). Addison Wesley.

[21]

Alan C. Calder, Bruce Fryxell, T. Plewa, Robert Rosner, L. J. Dursi, V. G. Weirs, T. Dupont, H. F. Robey, J. O. Kane, B. A. Remington, R. P. Drake, G. Dimonte, M. Zingale, Francis X. Timmes, K. Olson, Paul Ricker, P. MacNeice, and H. M. Tufo. 2002. On validating an astrophysical simulation code. Astrophys. J. Suppl. Ser. 143 (November 2002), 201--229.

[22]

Jeffrey Carver, Dustin Heaton, Lorin Hochstein, and Roscoe Bartlett. 2013. Self perceptions about software engineering: A survey of scientists and engineers. Comput. Sci. Eng. 15, 1 (January/February 2013), 7--11.

Digital Library

[23]

Jeffrey C. Carver. 2009. Report: The second international workshop on software engineering for CSE. Comput. Sci. Eng. 11, 6 (November/December 2009), 14--19.

Digital Library

[24]

Jeffrey C. Carver (Ed.). 2009. Proceedings of the 2nd International Workshop on Software Engineering for Computational Science, in conjunction with the 2009 IEEE 31st International Conference on Software Engineering (ICSE’09). IEEE Computer Society.

[25]

Jeffrey C. Carver (Ed.). 2010. Proceedings of the 3rd International Workshop on Software Engineering for Computational Science and Engineering. Elsevier, Amsterdam, The Netherlands.

[26]

Jeffrey C. Carver (Ed.). 2011. Proceedings of the 4th International Workshop on Software Engineering for Computational Science, in Conjunction with the 2011 IEEE 33rd International Conference on Software Engineering (ICSE’11). ACM.

[27]

Jeffrey C. Carver (Ed.). 2013. Proceedings of the 5th International Workshop on Software Engineering for Computational Science, in Conjunction with the 2013 IEEE 35th International Conference on Software Engineering (ICSE’13). IEEE Computer Society.

[28]

Jeffrey C. Carver and Neil P. Chue Hong (Eds.). 2016. Proceedings of the International Workshop on Software Engineering for Science (SE4Science@ICSE’16). ACM.

[29]

Jeffrey C. Carver, Richard P. Kendall, Susan E. Squires, and Douglass E. Post. 2007. Software development environments for scientific and engineering software: A series of case studies. In Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Minneapolis, MN, 550--559.

Digital Library

[30]

Donatella Castelli, Paolo Manghi, and Costantino Thanos. 2013. A vision towards scientific communication infrastructures—on bridging the realms of research digital libraries and scientific data centers. Int. J. Dig. Libr. 13, 3-4 (2013), 155--169.

[31]

Parmit K. Chilana, Carole L. Palmer, and Andrew J. Ko. 2009. Comparing bioinformatics software development by computer scientists and biologists: An exploratory study, see Reference Carver [24], 72--79.

[32]

Trevor Cickovski, Thierry Matthey, and Jesús A. Izaguirre. 2008. Design patterns for generic object-oriented scientific software, see Reference SE-CSE [155].

[33]

Jon Claerbout and Martin Karrenbach. 1992. Electronic documents give reproducible research a new meaning. In Proceedings of the 1992 Meeting of the Society of Exploration Geophysics.

[34]

Thomas L. Clune and Richard B. Rood. 2011. Software testing and verification in climate model development. IEEE Softw. 28, 5 (September/October 2011), 49--55.

Digital Library

[35]

CMMI Product Team. 2010. CMMI for Development. Technical Report. Software Engineering Institute, Carnegie Mellon.

[36]

Christian Colberg, Todd Proebsting, Gina Moraila, Akash Shankaran, Zuoming Shi, and Alex M. Warren. 2013. Measuring Reproducibility in Computer Systems Research. Retrieved from http://reproducibility.cs.arizona.edu/tr.pdf.

[37]

Carlton A. Crabtree, A. Güneş Koru, and Carolyn Seaman. 2009. An empirical characterization of scientific software development projects according to the Boehm and Turner model: A progress report, see Reference Carver [24], 22--27.

[38]

Daniel J. Crichton, Chris A. Mattmann, Luca Cinquini, Amy Braverman, Duane Waliser, Michael Gunson, Andrew F. Hart, Cameron E. Goodale, Peter Lean, and Jinwon Kim. 2012. Shared satellite observations within the climate-modelling community. IEEE Softw. 29, 5 (September/October 2012), 73--81.

Digital Library

[39]

Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated testing of refactoring engines. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering, Ivica Crnkovic and Antonia Bertolino (Eds.). ACM Press, Cavtat near Dubrovnik, Croatia, 185--194.

Digital Library

[40]

Andrew Davison. 2012. Automated capture of experiment context for easier reproducibility in computational research. Comput. Sci. Eng. 14, 4 (2012), 48--56.

Digital Library

[41]

Diego Caminha B. de Oliveira, Alan Humphrey, Zvonimir Rakamari, Qingyu Meng, Martin Berzins, and Ganesh Gopalakrishnan. 2013. Practical formal correctness checking of million-core problem solving environments for HPC, see Reference Carver [27], 75--83.

[42]

Viktor K. Decyk and Henry J. Gardner. 2008. Object-oriented design patterns in fortran 90/95: Mazev1, mazev2 and mazev3. Comput. Phys. Commun. 178, 8 (2008), 611--620.

[43]

Viktor K. Decyk, Charles D. Norton, and Bolesaw K. Szymanski. 1997. How to express c++ concepts in fortan90. Sci. Program. 6, 4 (1997), 363--390.

Digital Library

[44]

Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on test data selection: Help for the practicing programmer. IEEE Comput. 11, 4 (1978), 34--41.

Digital Library

[45]

Shweta Deshpande. 2011. A Study of Software Engineering Practices for Micro Teams. Master’s thesis. Ohio State University.

[46]

Tom Doherty. 2007. Integration of the ATLAS VOMS system with the ATLAS Metadata Interface. (2007). Retrieved from ppewww.physics.gla.ac.uk/preprints/2007/01/.

[47]

Anshu Dubey, Katie Antypas, Alan Calder, Bruce Fryxell, Don Lamb, Paul Ricker, Lynn Reid, Katherine Riley, Robert Rosner, Andrew Siegel, Francis Timmes, Natalia Vladimirova, and Klaus Weide. 2013. The software development process of flash, a multiphysics simulation code, see Reference Carver [27], 1--8.

[48]

Paul F. Dubois. 2005. Maintaining correctness in scientific programs. Comput. Sci. Eng. 7, 3 (May/June 2005), 80--85.

Digital Library

[49]

Steve M. Easterbrook and Timothy C. Johns. 2009. Engineering the software for understanding climate change. Comput. Sci. Eng. 11, 6 (November/December 2009), 64--75.

Digital Library

[50]

Paul N. Edwards. 2010. A Vast Machine. MIT Press.

[51]

Khaled El Emam and A. Güneş Koru. 2008. A replicated survey of it software project failures. IEEE Softw. 25, 5 (September/October 2008), 84--90.

Digital Library

[52]

David Falessi and Forrest Shull. 2013. Towards flexible automated support to improve the quality of computational science and engineering software, see Reference Carver [27], 88--91.

[53]

Hans Fangohr, Maximilian Albert, and Matteo Franchin. 2016. Nmag micromagnetic simulation tool: Software engineering lessons learned, see Reference Carver and Hong [28], 1--7.

[54]

Stuart Faulk, Eugene Loh, Michael L. Van De Vanter, Susan Squires, and Lawrence G. Votta. 2009. Scientific computing’s productivity gridlock: How software engineering can help. Comput. Sci. Eng. 11, 6 (November/December 2009), 30--39.

Digital Library

[55]

Mohamed E. Fayad, Mauri Laitinen, and Robert P. Ward. 2000. Software engineering in the small. Commun. ACM 43, 4 (March 2000), 115--118.

Digital Library

[56]

Martin Fowler. 2000. Refactoring: Improving the Design of Existing Code. Addison Wesley, Pearson Education Inc, One Lake Street, Upper Saddle River, NJ 07458.

Digital Library

[57]

Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns: Elements of Reusable Object-oriented Software (1st ed.). Addison Wesley.

Digital Library

[58]

Henry Gardner and Gabriele Manduchi. 2007. Design Patterns for e-Science. Springer Verlag.

[59]

Matan Gavish and David Donoho. 2011. A universal identifier for computational results. Proc. Comput. Sci. 4 (2011), 637--647.

[60]

Ian P. Gent. 2013. The Recomputation Manifesto. Retrieved from http://recomputation.org/sites/default/files/Manifesto1_9479.pdf (April 2013).

[61]

Ian P. Gent, Stuart A. Grant, Ewan MacIntyre, Patrick Prosser, Paul Shaw, Barbara M. Smith, and Toby Walsh. 1997. How Not To Do It. Research Report 97.27. School of Computer Studies, University of Leeds.

[62]

Robert L. Glass. 1997. Software Runaways: Lessons Learned from Massive Software Project Failures (1st ed.). Prentice Hall.

[63]

Carole Goble and David De Roure. 2007. myExperiment: Social networking for workflow-using e-scientists. In Proceedings of the 2nd Workshop on Workflows in Support of Large-scale Science (WORKS’07), June 25, 2007, Monterey, CA, Ewa Deelman and Ian Taylor (Eds.). ACM Press, 1--2.

Digital Library

[64]

David Goldberg. 1991. What every computer scientist should know about floating point arithmetic. Comput. Surv. 23, 1 (1991), 5--48.

Digital Library

[65]

John A. Gunnels and Robert A. van de Geijn. 2001. Formal methods for high-performance linear algebra libraries, see Reference Boisvert and Tang [19], 193--210.

Digital Library

[66]

Jo Erskine Hannay, Carolyn MacLeod, Janice Singer, Hans Petter Langtangen, Dietmar Pfahl, and Greg Wilson. 2009. How do scientists develop and use scientific software? See Reference Carver [24], 1--8.

Digital Library

[67]

Les Hatton. 1997. The t experiments: Errors in scientific software. IEEE Comput. Sci. Eng. 4, 2 (1997), 27--38.

Digital Library

[68]

Les Hatton. 2007. The chimera of software quality. IEEE Comput. 40, 8 (2007), 102--103.

Digital Library

[69]

Dustin Heaton and Jeffrey C. Carver. 2015. Claims about the use of software engineering practices in science: A systematic literature review. Info. Softw. Technol. 67 (2015), 207--219.

Digital Library

[70]

Jette Henderson and Dewayne E. Perry. 2013. Exploring issues in software systems used and developed by domain experts, see Reference Carver [27], 96--99.

[71]

Thomas Herndon, Michael Ash, and Robert Pollin. 2013. Does High Public Debt Consistently Stiffle Economic Growth? A Critique of Reinhart and Rogoff. Working paper 322. Political Economy Research Institute, University of Massachusetts Amherst, Gordon Hall, 418 North Pleasant Street, Amherst, MA 01002.

[72]

Michael A. Heroux, Roscoe A. Bartlett, Vicki E. Howle, Robert J. Hoekstra, Jonathan J. Hu, Tamara G. Kolda, Richard B. Lehoucq, Kevin R. Long, Roger P. Pawlowski, Eric T. Phipps, Andrew G. Salinger, Heidi K. Thornquist, Ray S. Tuminaro, James M. Willenbring, and Alan Williams. 2005. An overview of the Trilinos project. ACM Trans. Math. Software 31, 3 (September 2005), 397--423.

Digital Library

[73]

Michael A. Heroux and James M. Willenbring. 2009. Barely sufficient software engineering: 10 practices to improve your cse software, see Reference Carver [24], 15--21.

[74]

Tony Hey, Stewart Tansley, and Kristin Tolle (Eds.). 2009. The Fourth Paradigm. Data Intensive Scientific Discovery. Microsoft Research, Redmond, Washington.

[75]

Christopher I. Higgins, Michael Koutroumpas, Richard O. Sinnott, John Watt, Thomas Doherty, Ally C. Hume, Andrew G. D. Turner, and David Rawnsley. 2009. Spatial data e-infrastructure. In Proceedings of the 5th International Conference on e-Social Science. Cologne, Germany.

[76]

Lorin Hochstein and Victor R. Basili. 2008. The asc-alliance projects: A case study of large-scale parallel scientific code development. IEEE Comput. 41, 3 (March 2008), 50--58.

Digital Library

[77]

Douglas Hoffman. 1998. Analysis of a taxonomy of test oracles. In Quality Week, 1998.

[78]

Douglas Hoffman. 1999. Heuristic test oracles. Softw. Test. Qual. Eng. 1 (March/April 1999), 29--32.

[79]

Daniel Hook and Diane Kelly. 2009. Testing for trustworthiness in scientific software, see Reference Carver [24], 59--54.

[80]

Daniel Alan Hook. 2009. Using Code Mutation to Study Code Faults in Scientific Software. Master’s thesis. Queen’s University, Kingston, Ontario, Canada.

[81]

House of Commons Science and Technology Committee 2010. The disclosure of climate data from the Climatic Research Unit at the University of East Anglia. Eighth Report of Session 2009-10. The Stationery Office Limited, London. Volume I Report, together with formal minutes.

[82]

House of Commons Science and Technology Committee 2010. The disclosure of climate data from the Climatic Research Unit at the University of East Anglia. Eighth Report of Session 2009-10. The Stationery Office Limited, London. Volume II Oral and written evidence.

[83]

William E. Howden. 1982. Validation of scientific programs. Comput. Surv. 14, 2 (June 1982), 193--227.

Digital Library

[84]

David W. Kane. 2003. Introducing agile development into bioinformatics: An experience report. In Proceedings of the Agile Development Conference. IEEE Computer Society, 132--139.

[85]

David W. Kane, Moses M. Hohman, Ethan G. Cerami, Michael W. McCormick, Karl F. Kuhlmman, and Jeff A. Byrd. 2006. Agile methods in biomedical software development: A multi-site experience report. BMC Bioinfo. 7 (2006), 723.

[86]

Upulee Kanewala and James M. Bieman. 2013. Techniques for testing scientific programs without an oracle, see Reference Carver [27].

[87]

Upulee Kanewala and James M. Bieman. 2014. Testing scientific software: A systematic literature review. Info. Softw. Technol. 56 (2014), 1219--1232.

Digital Library

[88]

Diane Kelly. 2015. Scientific software development viewed as knowledge acquisition: Towards understanding the development of risk-averse scientific software. J. Syst. Softw. 109 (2015), 50--61.

Digital Library

[89]

Diane Kelly, Robert Gray, and Yizhen Shao. 2011. Examining random and designed tests to detect code mistakes in scientific software. J. Comput. Sci. 2 (2011), 47--56.

[90]

Diane Kelly and Daniel Hook. 2011. Scientific software testing: Analysis with four dimensions. IEEE Softw. 28, 3 (May/June 2011), 84--90.

Digital Library

[91]

Diane Kelly, Daniel Hook, and Rebecca Sanders. 2009. Five recommended practices for computational scientists who write software. Comput. Sci. Eng. 11, 5 (September/October 2009), 48--53.

Digital Library

[92]

Diane Kelly and Rebecca Sanders. 2008. Assessing the quality of scientific software, see Reference SE-CSE [155].

[93]

Diane Kelly and Terry Shepard. 2004. Task-directed software inspection. J. Syst. Softw. 73, 2 (October 2004), 361--368.

Digital Library

[94]

Diane Kelly, Spencer Smith, and Nicholas Meng. 2011. Software engineering for scientists. Comput. Sci. Eng. 13, 5 (September/October 2011), 7--10.

Digital Library

[95]

Diane F. Kelly. 2007. A software chasm: Software engineering and scientific computing. IEEE Softw. 24, 5 (November/December 2007), 118--120.

Digital Library

[96]

Richard Kendall, Jeffrey C. Carver, Andrew Mark, Douglas Post, Clifford E. Rhoades, and Susan Squires. 2008. Developing a weather forecasting code: A case study. IEEE Softw. 25, 4 (July/August 2008), 59--65.

Digital Library

[97]

Richard P. Kendall, Douglass E. Post, Chris A. Atwood, Kevin P. Newmeyer, Lawrence G. Votta, Paula A. Gibson, Deborah L. Borovitcky, Loren K. Miller, Robert L. Meakin, Miles M. Hurwitz, Saikat Dey, John N. D’Angelo, Richard L. Vogelsong, Oscar A. Goldfarb, and Sunita B. Allwerdt. 2016. A risk-based, practice-centered approach to project management for HPCMP CREATE. Comput. Sci. Eng. 18, 1 (January/February 2016), 40--51.

Digital Library

[98]

Joshua Kerievsky. 2004. Refactoring to Patterns. Addison-Wesley.

[99]

Sarah Killcoyne and John Boyle. 2009. Managing chaos: Lessons learned developing software in the life sciences. Comput. Sci. Eng. 11, 6 (November/December 2009), 20--29.

Digital Library

[100]

Donald Knuth. 1984. Literate programming. Comput. J. 27, 2 (1984), 97--111.

Digital Library

[101]

Konstantin Kreyman, David Lorge Parnas, and Sanzheng Qiao. 1999. Inspection Procedures for Critical Programs that Model Physical Phenomena. CRL Report 368. Department of Computing and Software, Faculty of Engineering, McMaster University, Hamilton, Ontario, Canada.

[102]

John Krogstie, Arthur Jahr, and Dag I. K. Sjøberg. 2006. A longitudinal study of development and maintenance in Norway: Report from the 2003 investigation. Info. Softw. Technol. 48, 11 (2006), 993--1005.

[103]

Sotiria Lampoudi. 2012. The Path to Virtual Machine Images as First Class Provenance. Technical Report 2012-05. Department of Computer Science, University of California Santa Barbara.

[104]

Randall J. LeVeque, Ian M. Mitchell, and Victoria Stodden. 2014. Reproducible research for scientific computing: Tools and strategies for changing the culture. Comput. Sci. Eng. 14, 4 (2014), 13--17.

Digital Library

[105]

Yang Li. 2011. Reengineering a scientific software and lessons learned, see Reference Carver [26], 41--45.

[106]

Anders Lundgren and Upulee Kanewala. 2016. Experiences of testing bioinformatics programs for detecting subtle faults, see Reference Carver and Hong [28], 16--22.

[107]

Donald MacKenzie. 2001. Mechanizing Proof: Computing, Risk and Trust. The MIT Press, Massachusetts Institute of Technology, Cambridge, Massachusetts 02142.

[108]

Arjen Markus. 2006. Design patterns and fortran 90/95. ACM Fortran Forum 25, 1 (2006), 13--29.

Digital Library

[109]

Arjen Markus. 2008. Design patterns and fortran 2003. ACM Fortran Forum 27, 3 (2008), 2--15.

Digital Library

[110]

David Matthews, Greg Wilson, and Steve Easterbrook. 2008. Configuration management for large-scale scientific computing at the UK met office. Comput. Sci. Eng. 10 (2008), 56--64.

Digital Library

[111]

Chris A. Mattmann, Daniel J. Crichton, Nenad Medvidovic, and Steve Hughes. 2006. A software architecture based framework for highly distributed and data intensive scientific applications, Leon J. Osterweil, H. Dieter Rombach, and Mary Lou Soffa (Eds.). ACM Press, Shanghai, China, 721--730.

[112]

B. D. McCullough, Kerry Anne McGeary, and Teresa D. Harrison. 2006. Lessons from the JMCB archive. J. Money Credit Bank. 38, 4 (June 2006), 1093--1107.

[113]

Zeeya Merali. 2010. Error. Why scientific computing does not compute. Nature 467 (October 2010), 777--779.

[114]

Erika S. Mesh. 2015. Supporting scientific SE process improvement. In Proceedings of the 37th International Conference on Software Engineering—Volume 2. IEEE Press, Piscataway, NJ, 923--926.

[115]

Erika S. Mesh and J. Scott Hawker. 2013. Scientific software process improvement decisions: A proposed research strategy, see Reference Carver [27], 32--39.

[116]

Greg Miller. 2006. Scientific publishing: A scientist’s nightmare: Software problem leads to five retractions. Science 314, 5807 (2006), 1856--1857.

[117]

K. Jarrod Millman and Fernando Pérez. 2014. Developing open source scientific practice. In Implementing Reproducible Research, Victoria Stodden, Friedrich Leisch, and Roger D. Peng (Eds.). Chapman and Hall/CRC Press, 149--183.

[118]

Chris Morris. 2008. Some lessons learned reviewing scientific code, see Reference SE-CSE [155].

[119]

Hoda Naguib and Yang Li. 2012. (Position paper) applying software engineering methods and tools to CSE Research Projects, see Reference Sloot et al. [167], 1505--1509.

[120]

Peter Naur and Brian Randell (Eds.). 1968. Report on a Conference Sponsored by the NATO Science Committee. Garmisch, Germany.

[121]

Thomas Naylor and J. M. Finger. 1967. Verification of computer simulation models. Manage. Sci. 14, 2 (October 1967).

Digital Library

[122]

Nedialko S. Nedialkov. 2011. Implementing a rigorous ODE solver through literate programming. In Modeling, Design, and Simulation of Systems with Uncertainties, Andreas Rauh and Ekaterina Auer (Eds.). Springer, 3--19.

[123]

Vitor C. Neves, Vanessa Braganholo, and Leonardo Murta. 2013. Implicit provenance gathering through configuration management, see Reference Carver [27], 92--95.

[124]

Luke Nguyen-Hoan, Shayne Flint, and Ramesh Sankaranayana. 2010. A survey of scientific software development. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement (ESEM’10), Giancarlo Succi, Maurizio Morisio, and Nachiappan Nagappan (Eds.). ACM Press, Bolzano-Bozen, Italy.

Digital Library

[125]

Tom Nielsen, Tom Matheson, and Henrik Nilsson. 2011. Braincurry: A domain--specific language for integrative neuroscience. In Trends in Functional Programming Volume 10, Zoltán Horváth, Viktória Zsók, Peter Achten, and Pieter Koopman (Eds.). Intellect, The Mill, Parnall Road, Fishponds, Bristol, BS16 3JG, Chapter 11, 161--176.

[126]

Charles D. Norton, Viktor K. Decyk, Boleslaw K. Szymanski, and Henry Gardner. 2007. The transition and adoption of modern programming concepts for scientific computing in Fortran. Sci. Program. 15, 1 (Spring 2007), 1--27.

Digital Library

[127]

Piotr Nowakowskia, Eryk Ciepielaa, Daniel Harȩżlaka, Joanna Kocota, and Marek Kasztelnika, Tomasz Bartyńskia, Jan Meiznera, Grzegorz Dyka, and Maciej Malawski. 2011. The collage authoring environment. Proc. Comput. Sci. 4 (2011), 608--617.

[128]

William L. Oberkampf, Martin Pilch, and Timothy G. Trucano. 2007. Predictive Capability Maturity Model for Computational Modeling and Simulation. Technical Report SAND2007-5948. Sandia National Laboratories, Albuquerque, New Mexico 97185 and Livermore, California.

[129]

Emil Obreshkov. 2010. Software release build process and components in Atlas offline. In Proceedings of the Conference on Computing in High Energy and Nuclear Physics 2010.

[130]

Tom Oinn, Mark Greenwood, Matthew Addis, M. Nedim Alpdemir, Justin Ferris, Kevin Glover, Carole Goble, Antoon Goderis, Duncan Hull, Darren Marvin, Peter Li, Phillip Lord, Matthew R. Pocock, Martin Senger, Robert Stevens, and Anil Wipat and Chris Wroe. 2006. Taverna: Lessons in creating a workflow environment for the life sciences.Concurr. Comput.: Pract. Exp. 18, 10 (August 2006), 1067--1100.

[131]

Jeffrey L. Overbey, Stas Negara, and Ralph E. Johnson. 2009. Refactoring and the evolution of Fortran, see Reference Carver [24], 28--34.

[132]

Michael Pernice. 2013. Survey of Software Quality Assurance and Code Verification Practices in CASL. Technical Report CASL-U-2013-0079-000. Idaho National Laboratory, Oak Ridge Laboratory.

[133]

Jon Pipitone and Steve Easterbrook. 2012. Assessing climate model software quality: A defect density analysis of three models. Geosci. Model Dev. 5 (2012), 1009--1022.

[134]

Joe Pitt-Francis, Miguel O. Bernabeu, Jonathan Cooper, Alan Garny, Lee Momtahan, James Osborne, Pras Pathmanathan, Blanca Rodriguez, Jonathan P. Whiteley, and David J. Gavaghan. 2008. Chaste: Using agile programming techniques to develop computational biology software. Philos. Trans. Ser. A, Math. Phys. Eng. Sci. 366, 1878 (September 2008), 3111--3136.

[135]

Karl Popper. 2005. The Logic of Scientific Discovery. Routledge, 11 New Fetter Lane, London, EC4P 4EE.

[136]

Douglas E. Post. 2008. A new DoD initiative: The computational research and engineering acquisition tools and environment (CREATE) program. J. Phys.: Conf. Ser. 125 (2008), 012090.

[137]

D. E. Post and R. P. Kendall. 2004. Software project management and quality assurance practices for complex, coupled, multiphysics, massively parallel computational simulations: Lessons learned from ASCI. Int. J. High Perform. Comput. Appl. 18, 4 (2004), 399--416.

Digital Library

[138]

Douglass E. Post and Lawrence G. Votta. 2005. Computational science demands a new paradigm. Phys. Today 58, 1 (January 2005), 35--40.

[139]

Patrick Prosser and Chris Unsworth. 2010. Limited Discrepancy Search: Revisited (2010). Private communication.

[140]

H. M. Quiney and S. Wilson. 2005. Literate programming in quantum chemistry: A simple example. Int. J. Quant. Chem. 104, 4 (2005), 430--445.

[141]

James J. Quirk. 2005. Computational science “Same old silence, same old mistakes” “Something more is needed.” In Adaptive Mesh Refinement—Theory and Applications. Proceedings of the Chicago Workshop on Adaptive Mesh Refinement Methods, Sept. 3--5, 2003 (Lecture Notes in Computational Science and Engineering), Tomasz Plewa, Timur Linde, and V. Gregory Weirs (Eds.), Vol. 41. Springer Verlag, 3--28.

[142]

Krister Åhlander, Magne Haveraaen, and Hans Z. Munthe-Kaas. 2001. On the role of mathematical abstractions for scientific computing, see Reference Boisvert and Tang [19], 145--158.

[143]

Thomas Reichler and Junsu Kim. 2008. How well do coupled models simulate today’s climate. Bull. Am. Meteorol. Soc. 89, 3 (March 2008), 303--311.

[144]

Carmen M. Reinhart and Kenneth S. Rogoff. 2010. Growth in a time of debt. Am. Econ. Rev.: Papers Proc. 100 (May 2010), 573--578.

[145]

Hanna Remmel, Barbara Paech, Christian Engwer, and Peter Bastian. 2013. Design and rationale of a quality assurance process for a scientific framework, see Reference Carver [27], 58--67.

[146]

Christophe René, Thierry Priol, and Guillaume Alléon. 2001. Code coupling using parallel Corba objects, see Reference Boisvert and Tang [19], 105--118.

[147]

John Rice and Ronald F. Boisvert. 1996. From scientific software libraries to problem solving environments. IEEE Comput. Sci. Eng. 3 (1996), 44--53.

Digital Library

[148]

Patrick Roache. 2004. Building PDE codes to be verifiable and validatable. Comput. Sci. Eng. 6, 5 (September/October 2004), 30--38.

Digital Library

[149]

Damian W. I. Rouson, Helgi Adalsteinsson, and Jim Xia. 2010. Design patterns for multiphysics modeling in Fortran 2003 and C++. ACM Trans. Math. Software 37, 1 (January 2010), Article 3.

Digital Library

[150]

Damian W. I. Rouson, Jim Xia, and Xiaofeng Xu. 2012. Object construction and destruction design patterns in Fortran 2003, see Reference Sloot et al. [167], 1495--1504.

[151]

Christopher J. Roy and William L. Oberkampf. 2011. A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Comput. Methods Appl. Mech. Eng. 200, 25 (June 2011), 2131--2144.

[152]

Royal Swedish Academy of Sciences. 2013. Scientific Background on the Nobel Prize in Chemistry 2013. Development of Multiscale Models for Complex Chemical Systems. (October 2013). Retrieved from http://www.nobelprize.org/nobel_prizes/chemistry/laureates/2013/advanced-chemistryprize2013.pdf.

[153]

Rebecca Sanders and Diane Kelly. 2008. Dealing with risk in scientific software development. IEEE Comput. Softw. 25 (2008). Issue 4.

Digital Library

[154]

Ken Schwaber and Mike Beedle. 2001. Agile Software Development with SCRUM. Prentice Hall.

Digital Library

[155]

SE-CSE 2008. Proceedings of the 1st International Workshop on Software Engineering for Computational Science and Engineering. Leipzig Germany.

[156]

Judith Segal. 2005. When software engineers met research scientists: A case study. In Empirical Software Engineering. Springer Verlag, 517--536.

Digital Library

[157]

Judith Segal. 2008. Models of scientific software development, see Reference SE-CSE [155].

[158]

Judith Segal. 2009. Software development cultures and cooperation problems: A field study of the early stages of development of software for a scientific community. Comput. Support. Coop. Work (CSCW’09) 18, 5--6 (2009), 581--606.

[159]

Judith Segal. 2009. Some challenges facing software engineers developing software for scientists, see Reference Carver [24], 9--14.

[160]

Judith Segal and Chris Morris. 2008. Developing scientific software. IEEE Comput. Softw. 25 (July/August 2008), 18--20. Issue 4.

Digital Library

[161]

Simon Shackley, Peter Young, Stuart Parkinson, and Brian Wynne. 1998. Uncertainty, complexity and concepts of good science in climate change modelling: Are GCMS the best tool? Climate Change 38 (1998), 159--205.

[162]

A. Shaon, Jim Woodcock, and E. Conway. 2009. Tools and Guidelines for Preserving and Accessing Software as a Research Output Report II: Case Studies. Technical Report. The University of York.

[163]

Forrest Shull. 2011. Assuring the future? A look at validating climate model software. IEEE Softw. 28, 5 (September/October 2011), 4--8.

Digital Library

[164]

Jeremy Singer. 2011. A literate experimentation manifesto. In Proceedings of the ACM Symposium on New Ideas in Programming and Reflections on Software, Onward&excl; 2011, part of SPLASH’11, Robert Hirschfeld and Eelco Visser (Eds.). ACM Press, Portland, OR, 91--102.

Digital Library

[165]

Magnus Thorstein Sletholt, Jo Hannay, Dietmar Pfahl, Hans Christian Benestad, and Hans Petter Langtangen. 2011. A literature review of agile practices and their effects in scientific software development, see Reference Carver [26], 1--9.

[166]

Magnus Thorstein Sletholt, Jo Erskine Hannay, Dietmar Pfahl, and Hans Petter Langtangen. 2012. What do we know about scientific software development’s agile practices?Comput. Sci. Eng. 14, 2 (March/April 2012), 24--36.

Digital Library

[167]

Peter M. A. Sloot, G. Dick van Albada, and Jack Dongarra (Eds.). 2012. Proceedings of the International Conference on Computational Science (ICCS’10). Procedia Computer Science, Vol. 1. Elsevier, University of Amsterdam, The Netherlands.

[168]

Spencer Smith, Lei Lai, and Ridha Khedri. 2007. Requirements analysis for engineering computation: A systematic approach for improving reliability. Reliable Comput. 13, 1 (2007), 83--107.

[169]

W. Spencer Smith, Jacques Carette, and John McCutchan. 2008. Commonality analysis of families of physical models for use in scientific computing, see Reference SE-CSE [155].

[170]

Diomidis Spinellis and Henry Spencer. 2011. Lessons from space. IEEE Softw. 28, 5 (September/October 2011).

Digital Library

[171]

Nicholas Stern. 2007. The Economics of Climate Change. The Stern Review. Cambridge University Press.

[172]

Victoria Stodden, Peixuan Guo, and Zhaokun Ma. 2013. Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals. PLOS One 8, 6 (June 2013), e67111.

[173]

Victoria Stodden and Sheila Miguez. 2013. Best practices for computational science: Software infrastructure and environments for reproducible and extensible research. J. Open Res. Softw. 2, 1 (2013), e21.

[174]

Michael Stonebraker, Daniel Bruckner, Ihab F. Ilyas, George Beskales, Mitch Cherniack, and Stan Zdonik. 2013. Data curation at scale: The data tamer system. In Proceedings of the 6th Biennial Conference on Innovative Data Systems Research (CIDR’13). Online Proceedings. www.cidrdb.org, Asilomar, CA.

[175]

Dan Szymczak, Spencer Smith, and Jacques Carette. 2016. A knowledge-based approach to scientific software development: Position paper, see Reference Carver and Hong [28], 23--26.

[176]

Ken Thompson. 1984. Reflections on trusting trust. Commun. ACM 27, 8 (August 1984), 761--763.

Digital Library

[177]

Michael Thuné, Krister øAhlander, Malin Ljungberg, Markus Nordén, Kurt Otto, and Jarmo Rantakokko. 2001. Object-oriented modeling of parallel PDE solvers, see Reference Boisvert and Tang [19], 159--174.

[178]

T. G. Trucano, Douglass E. Post, M. Pilch, and W. L. Oberkampf. 2005. Software Engineering Intersections with Verification and Validation (V&V) of High Performance Computational Science Software: Some Observations. Technical Report SAND2005-3662P. Sandia National Laboratories, PO Box 5800, Albuquerque, New Mexico.

[179]

Medha Umarji, Carolyn Seaman, A. Gunes Koru, and Hongfang Liu. 2009. Software engineering education for bioinformatics. In Proceedings of the 2009 22nd Conference on Software Engineering Education and Training. IEEE Computer Society, 216--223.

Digital Library

[180]

Arie van Deursen, Paul Klint, and Joost Visser. 2000. Domain-specific languages: An annotated bibliography. ACM SIGPLAN Notices 35, 6 (June 2000), 26--36.

Digital Library

[181]

Pieter van Gorp and Steffen Mazanek. 2011. share: A web portal for creating and sharing executable research papers. Proc. Comput. Sci. 4 (2011), 589--597.

[182]

Mark Vigder, Darlene Stewart, and Janice Singer. 2008. Software automation in scientific research organizations, see Reference SE-CSE [155].

[183]

Alexey Voinov and Herman H. Shugart. 2013. “Integronsters,” integral and integrated modeling. Environ. Model. Softw. 39 (2013), 149--158.

Digital Library

[184]

Dali Wang, Tomislav Janjusic, Colleen Iversen, Peter E. Thornton, Misha Karssovski, Wei Wu, and Yang Xu. 2015. A scientific function test framework for modular environmental model development: Application to the community land model. In Proceedings of the 1st IEEE/ACM International Workshop on Software Engineering for High Performance Computing in Science (SE4HPCS’15), 2015, Jeffrey Carver, Paolo Ciancarini, and Neil Chue Hong (Eds.). IEEE Computer Society, 16--23.

Digital Library

[185]

Richard Y. Wang and Diane M. Strong. 1996. Beyond accuracy, what data quality means to data consumers. J. Manage. Info. Syst. 12, 4 (Spring 1996), 5--33.

Digital Library

[186]

Elaine J. Weyuker. 1982. On testing non-testable programs. Comput. J. 25, 4 (1982), 465--470.

[187]

Greg Wilson, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White, and Paul Wilson. 2014. Best practices in scientific computing. PLOS Biol. 12, 1 (January 2014), 1--7.

[188]

Greg Wilson and Andrew Lumsdaine. 2006. Software carpentry getting scientists to write better code by making them more productive. Comput. Sci. Eng. 8, 6 (November/December 2006), 66--69.

Digital Library

[189]

Greg Wilson and Andrew Lumsdaine. 2009. Software engineering and computational science. Comput. Sci. Eng. 11, 6 (November/December 2009), 12--13.

Digital Library

[190]

Marianne Winslett and Vanessa Braganholo. 2012. Erich Neuhold speaks out. SIGMOD Rec. 41, 2 (June 2012), 37--46.

Digital Library

[191]

William A. Wood and William L. Kleb. 2003. Exploring XP for scientific research. IEEE Softw. 20, 3 (May/June 2003), 30--36.

Digital Library

[192]

David Woollard, Chris Mattmann, and Nenad Medvidovic. 2009. Injecting software architectural constraints into legacy scientific applications, see Reference Carver [24], 65--71.

[193]

Yale Law School Roundtable on Data and Code Sharing. 2010. Reproducible research. Comput. Sci. Eng. 12, 5 (September/October 2010), 8--12.

Digital Library

[194]

Wen Yu and Spencer Smith. 2009. Reusability of FEA software: A program family approach, see Reference Carver [24], 43--50.

Cited By

Korkin SSayer AIbrahim ALyapustin A(2025)A practical guide to coding line-by-line trace gas absorption in Earth's atmosphereJournal of Quantitative Spectroscopy and Radiative Transfer10.1016/j.jqsrt.2025.109345337(109345)Online publication date: May-2025
https://doi.org/10.1016/j.jqsrt.2025.109345
Chen YWan ZZhuang YLiu NLo DYang X(2024)Understanding the OSS Communities of Deep Learning Frameworks: A Comparative Case Study of PyTorch and TensorFlowACM Transactions on Software Engineering and Methodology10.1145/370530334:3(1-30)Online publication date: 23-Nov-2024
https://dl.acm.org/doi/10.1145/3705303
Pertseva EChang MZaman UCoblenz MRoychoudhury APaiva AAbreu RStorey M(2024)A Theory of Scientific Programming EfficacyProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639139(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639139
Show More Cited By

Index Terms

Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming
1. Software and its engineering
  1. Software creation and management

Recommendations

A Software Chasm: Software Engineering and Scientific Computing

The domain-independent solutions proffered by the software engineering community have led to the isolation of the scientific-computing community.
Scientific Computing's Productivity Gridlock: How Software Engineering Can Help

Hardware improvements do little to improve real productivity in scientific programming. Indeed, the dominant barriers to productivity improvement are now in the software processes. To break the gridlock, we must establish a degree of cooperation and ...
Testing Scientific Programs

The Automated Testing System (ATS) is an open source, Python-based tool for automating the testing of applications, especially scientific simulations. It's especially designed to support the work of a team of subject-matter experts.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 50, Issue 4

July 2018

531 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3135069

Editor:
Sartaj Sahni
Department of Computer and Information Science and Engineering / University of Florida / Gainesville, FL

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2017

Accepted: 01 April 2017

Revised: 01 April 2017

Received: 01 August 2015

Published in CSUR Volume 50, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
1,321
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)7

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Korkin SSayer AIbrahim ALyapustin A(2025)A practical guide to coding line-by-line trace gas absorption in Earth's atmosphereJournal of Quantitative Spectroscopy and Radiative Transfer10.1016/j.jqsrt.2025.109345337(109345)Online publication date: May-2025
https://doi.org/10.1016/j.jqsrt.2025.109345
Chen YWan ZZhuang YLiu NLo DYang X(2024)Understanding the OSS Communities of Deep Learning Frameworks: A Comparative Case Study of PyTorch and TensorFlowACM Transactions on Software Engineering and Methodology10.1145/370530334:3(1-30)Online publication date: 23-Nov-2024
https://dl.acm.org/doi/10.1145/3705303
Pertseva EChang MZaman UCoblenz MRoychoudhury APaiva AAbreu RStorey M(2024)A Theory of Scientific Programming EfficacyProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639139(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639139
Alrashedy KBinjahlan A(2024)How do Software Engineering Researchers Use GitHub? An Empirical Study of Artifacts & Impact2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM63643.2024.00021(118-130)Online publication date: 7-Oct-2024
https://doi.org/10.1109/SCAM63643.2024.00021
Ahmed WHarbaoui A(2024)Is This Code the Best? Or Can It Be Further Improved? Developer Stats to the RescueIEEE Access10.1109/ACCESS.2024.347248112(144395-144411)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3472481
Blum VAsahi RAutschbach JBannwarth CBihlmayer GBlügel SBurns LCrawford TDawson Wde Jong WDraxl CFilippi CGenovese LGiannozzi PGovind NHammes-Schiffer SHammond JHourahine BJain AKanai YKent PLarsen ALehtola SLi XLindh RMaeda SMakri NMoussa JNakajima TNash JOliveira MPatel PPizzi GPourtois GPritchard BRabani EReiher MReining LRen XRossi MSchlegel HSeriani NSlipchenko LThom AValeev EVan Troeye BVisscher LVlček VWerner HWilliams-Young DWindus T(2024)Roadmap on methods and software for electronic structure based simulations in chemistry and materialsElectronic Structure10.1088/2516-1075/ad48ec6:4(042501)Online publication date: 15-Nov-2024
https://doi.org/10.1088/2516-1075/ad48ec
SAY B(2023)Bilimsel Yazılım Geliştirme Süreçleri için bir Yazılım Mühendisliği Yetkinlik EnvanteriA Software Engineering Competency Inventory for Scientific Software Development ProcessesTürkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi10.54525/tbbmd.115944316:2(170-176)Online publication date: 20-Nov-2023
https://doi.org/10.54525/tbbmd.1159443
González-Prieto ÁPerez JDiaz JLópez-Fernández D(2023)Reliability in software engineering qualitative research through Inter-Coder AgreementJournal of Systems and Software10.1016/j.jss.2023.111707202:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111707
Bernholdt DDoucet MGodoy WMalviya-Thakur AWatson G(2023)Experiential findings for sustainable software ecosystems to support experimental and observational scienceJournal of Computational Science10.1016/j.jocs.2023.10203371(102033)Online publication date: Jul-2023
https://doi.org/10.1016/j.jocs.2023.102033
Smith SMichalski PCarette JKeshavarz-Motamed Z(2023)State of the Practice for Lattice Boltzmann Method SoftwareArchives of Computational Methods in Engineering10.1007/s11831-023-09981-231:1(313-350)Online publication date: 1-Aug-2023
https://doi.org/10.1007/s11831-023-09981-2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents