A benchmark- and competition-based approach to software engineering research
Pages 43 - 46
Abstract
Many fields of study within computer science have benefited from the adoption of community-wide benchmarks and competitions. Software engineering has yet to fully embrace this approach. Case studies of existing uses of these techniques are presented, and a hypothetical application to software engineering based on research presented in last year's FSE are elaborated.
References
[1]
FOM mailing list. http://www.cs.nyu.edu/mailman/listinfo/fom, Last accessed August 2010.
[2]
HPC Challenge homepage. http://icl.cs.utk.edu/hpcc, Last accessed June 2010.
[3]
Top 500 homepage. http://top500.org,Last accessed June 2010.
[4]
Verify This: A collection of verification benchmarks. http://verifythis.cost-ic0701.org, Last accessed September 2010.
[5]
C. Barrett, L. de Moura, and A. Stump. SMT-COMP: Satisfiability Modulo Theories Competition. In K. Etessami and S. Rajamani, editors, 17th International Conference on Computer Aided Verification pages 20--23. Springer, 2005.
[6]
J. Burnim and K. Sen. Asserting and checking determinism for multithreaded programs. In ESEC/FSE '09: Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering pages 3--12, New York, NY, USA, 2009. ACM.
[7]
N. Chomsky. A review of B. F. Skinner's Verbal Behavior. Language 35(1):26--58, 1959.
[8]
K. Iagnemma and M. Buehler. Editorial for journal of field robotics special issue on the DARPA grand challenge: Editorial. J. Robot. Syst. 23(9):655--656, 2006.
[9]
V. Kahlon, N. Sinha, E. Kruus, and Y. Zhang. Static data race detection for concurrent programs with asynchronous calls. In ESEC/FSE '09: Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering pages 13--22, New York, NY, USA, 2009. ACM.
[10]
K. R. M. Leino and M. Moskal. VACID-0: Verification of ample correctness of invariants of data-structures, edition 0. In Verified Software: Theories, Tools and Experiments, VS-Tools and Experiments Workshop 2010.
[11]
C. T. S. Ranise. The SMT-LIB standard:Version 1. 2. Technical report, Department of Computer Science, University of Iowa,2006.
[12]
G. Sutclifle. The TPTP Problem Library and Associated Infrastructure: The FOF and CNF Parts, v3. 5. 0. Journal of Automated Reasoning 43(4):337--362, 2009.
[13]
G. Sutclifle and C. Suttner. The State of CASC. AI Communications 19(1):35--48,2006.
[14]
B. W. Weide, M. Sitaraman, H. K. Harton, B. M. Adcock, P. Bucci, D. Bronish, W. D. Heym, J. Kirschenbaum, and D. Frazier. Incremental benchmarks for software verification tools and techniques. In Verified Software: Theories, Tools, and Experiments (VSTTE)pages 84--98,2008.
Index Terms
- A benchmark- and competition-based approach to software engineering research
Recommendations
Process-Based Software Engineering: Building the Infrastructures
A recent trend in software engineering is the shift from a focus on laboratory-oriented software engineering to a more industry-oriented view of software engineering processes. This complements preceding ideas about software engineering in terms of ...
A Survey of Software Engineering Practice: Tools, Methods, and Results
The results of a survey of software development practice are reported and analyzed. The problems encountered in various phases of the software life cycle are measured and correlated with characteristics of the responding installations. The use and ...
Establishment of a National Benchmark of Software Engineering Practices
ISESS '99: Proceedings of the 4th IEEE International Symposium and Forum on Software Engineering StandardsSoftware engineering process benchmarking is one of the important methodologies in software process assessment and improvement. In collaboration with IBM, IVF has conducted a Swedish national benchmarking survey to derive a national benchmark of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2010
460 pages
ISBN:9781450304276
DOI:10.1145/1882362
- General Chair:
- Gruia-Catalin Roman,
- Program Chair:
- Kevin Sullivan
Copyright © 2010 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 November 2010
Check for updates
Author Tag
Qualifiers
- Research-article
Conference
SIGSOFT/FSE'10
Sponsor:
SIGSOFT/FSE'10: 18th ACM SIGSOFT Symposium on the Foundations of Software Engineering
November 7 - 8, 2010
New Mexico, Santa Fe, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 194Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in