Basili-Role of E in SE Irvine
Basili-Role of E in SE Irvine
Basili-Role of E in SE Irvine
Software Engineering
Victor R. Basili
University of Maryland
and
Fraunhofer Center - Maryland
Setting the Context
2
Motivation for Empirical Software Engineering
3
Motivation for Empirical Software Engineering
4
Motivation for Empirical Software Engineering
More specifically:
When are peer reviews more effective than functional testing?
When is an agile method appropriate?
When do I buy rather than make my software product elements?
5
Examples of Useful Empirical Results
Set
Analyze goals
results
Execute
process Choose
processes,
Provide process methods,
with feedback techniques,
Project and tools
learning
Analyze
results
8
The Experience Factory Organization
project
analysis, Disseminate
process
4. Execute Process modification
5. Analyze
data,
lessons
learned
9
The Experience Factory Organization
A Different Paradigm
10
SEL: An Example Experience Factory Structure
EF
PO
DEVELOPERS PROCESS ANALYSTS
(SOURCE OF EXPERIENCE) (PACKAGE EXPERIENCE FOR REUSE)
13
CeBASE Approach
Observation and
Empirical Data
Evaluation Studies
of Development
Technologies and
Techniques
15
CeBASE Basic Research Activities
16
Applied Research
NASA High Dependability Computing Program
Problem: How do you elicit the software dependability needs of
various stakeholders and what technologies should be applied to
achieve that level of dependability?
Research Problem 3
What set of technologies should be
applied to achieve the desired
Research Problem 1 quality? (Decision Support)
Can the quality needs be
understood and modeled?
System Developers
Research Problem 2
What does a technology do?
Can it be empirically demonstrated? 18
System User Issues
19
UMD - Unified Model of Dependability
The book search service (scope) should not have a response time
greater than 10 seconds (issue) more often than 1% of the cases
(measure); if the failure occurs, the system should warn the user and
recover full service in one hour. 20
UMD is a model builder
FAILURE HAZARD
concern - Type - Severity manifest
- Accuracy - People affected
- Response Time - Property only
- etc. - etc.
issue - Availability impact
- Stopping
- Non-Stopping
cause - Severity trigger
- High
- Low
- Impact mitigation
- warnings
- Type - alternative services
event - Adverse Condition
- Attack
reaction - mitigation services
- Recovery
- etc. - recovery time / actions
- Occurrence reduction
- guard services
21
UMD assimilates new experience
Characterizations (e.g., types, severity, etc.) of the basic UMD
modeling concepts of issue, scope, measure, and event depend on
the specific context (project and stakeholders).
22
UMD: a framework for engineering decisions
position
planned route
synthetized
conformance
Highlight non
Select flight
Display
MTBF Stopping Failures
aircraft
MTBF vs. Services
Display
Display
route
1.E+06
1.E+00
system
position
planned route
synthetized
conformance
Highlight non
Select flight
Display
aircraft
Display
route
Display
MTBF all failures
1.E+03
MTBF Stopping Failures
1.E+00
system
position
planned route
synthetized
conformance
Highlight non
Select flight
Display
aircraft
Display
route
Display
Requirements
Visualization scope
- Type
- Whole System
- Service
- Operational Profile measure
- Measurement Model
- MTBF
- Probability of Occurrence
- % cases
Computation of
- Distribution of transaction - MAX cases in interval X
- Workload volumes - Ordinal scale
aggregate values of
- etc. (rarely/sometimes/....)
concern
FAILURE
- Type
- Accuracy
- Response Time
- etc.
HAZARD
- Severity
- People affected
- Property only
- etc.
manifest
dependability
issue - Availability impact
- Stopping
- Non-Stopping
cause - Severity
- High
- Low
trigger
(availability, MTBF
event
- Type
- Adverse Condition
reaction
- Impact mitigation
- warnings
- alternative services
- mitigation services
per service, etc)
- Attack - Recovery
- etc. - recovery time / actions
- Occurrence reduction
- guard services
23
UMD
Technology Developer Issues
24
Example Technology Evolution
A process for inspections of Object-Oriented designs was
developed using multiple iterations through this method.
Early iterations concentrated on feasibility:
- effort required, results due to the process in the context of
offline, toy systems.
Is further effort justified?
Mid-process iterations concentrated on usability:
- usability problems, results due to individual steps in the
context of small systems in actual development.
What is the best ordering and streamlining of process
steps to match user expectations?
Most recent iterations concentrated on effectiveness:
- effectiveness compared to other inspection techniques
previously used by developers in the context of real systems
under development. Does the new techniques represent a
usable improvement to practice?
25
Using testbeds to transfer technology
Used to
Conduct empirical evaluations of emerging technology
Stress the technology and demonstrate its context of
effectiveness
Help the researcher identify the strengths, bounds, and limits
of the particular technology at different levels
Provide insight into the integration of technologies
Reduce costs by reusing software artifacts
Reduce risks by enabling technologies to mature
Assist technology transfer of mature results
26
Example Technology and Testbed Evolution
Testbed :
40 versions of TSAFE source code were created via fault
seeding
The faults were created to resemble possible errors that can
arise in using the concurrency controller pattern such as
making an error while writing a guarded command or
forgetting to call a concurrency controller method before
accessing a shared object
Results:
The experimental study resulted in a
Better fault classification
Identified strengths and weaknesses of the technology
Helped improve the design for verification approach
However, there was one type of fault that was difficult to catch
Three uncaught faults were created to test this
28
System Developer Issues
Problem: How do you improve the time and cost of developing high end
computing (HEC) codes?
Project Goal: Improve the buyers ability to select the high end computer
for the problems to be solved based upon productivity, where
productivity means
Time to Solution = Development Time + Execution Time
Partners: MIT Lincoln Labs, MIT, UCSD, UCSB, UMD, USC, FC-MD
30
HPCS Example Questions
32
HPCS Testbeds
33
Experimental artifacts Programming problems
Experimental Packages
34
Studies Conducted
Stanford U U Utah UIUC U Chicago
ASCI Alliance ASCI Alliance ASCI Alliance ASCI Alliance
MIT
3 studies
UCSB
3 studies
USC
4 studies
CalTech UMD
ASCI Alliance 6 studies
UCSD
1 study
Mississippi State
Iowa State 2 studies
1 study
35
Clearinghouse Project
36
Operational Concept
Best Practice Handling
Identification Quantification Analysis & Packaging &
Validation
& Qualification Synthesis Dissemination
BPCh Operations
Information Handlers System Upgrades & Maintenance
Initial Development
Backups & User Management
Support Team
BPCh IT Components
is!
BPCh Usage
e th
Tak Project characterization Access
Select Interface with
Information Appropriated
request data other resources
Role characterization Practice
Information Seekers
37
BPCh Process Components BPCh Roles BPCh IT Components BPCh Role Specific Interfaces
Behind the Scenes
BPCh recommendations based on
evidence from real programs.
Evidence From publications
Source: How trustable?
From interviews & feedback with users
From
Context: Used by avetted
safetyexpert
criticalguidebooks
program? In& astandards
DoD
environment? On a warfighter?
42