How to avoid drastic project change (using stochastic stability)

+

how to avoid drastic software process change
(using stochastic stability)
WVU: Tim Menzies, Steve Williams, Ous Elwaras
USC: Barry Boehm
JPL: Jairus Hihn

Apr 6, 2009

+
2
background


digression


internal vs drastic changes


what space do we explore?


what is interesting/different here?


results (on 4 projects)


this talk related work


conclusion & future work


questions? comments?


+
3
background


digression
















+ 4

stochastic
stability
 “For all is but a woven web of guesses.”
-- Xenophanes (570 – 480 BCE)


 Seek what holds true over the space of all guesses.
Surprisingly, happily, such stable conclusions exist.


 Bad idea for:
The safety-critical guidance system of a manned rocket.


 Good idea for:
Exploring the myriad space of possibilities associated with

software project management.

+ 5

stochastic stability
and option exploration
 Software project managers have more options than they think.

 It is possible (even useful) to push back against drastic change

 Q: can we found local options that out-perform drastic change?
A: yes, we can


Adjust current
project options

Fire people

Do it all again,
in ADA

Deliver late

+ 6

expectation
management
We explore models built from real project data


 We perform experiments with models about software project
Not experiments with actual projects.


 Such experiments are hard to do
Gone of the days of Victor Basili-style SEL experimentation

 where the researchers could tell the project what to do.
Software developers are more aggressive in selecting their own methods.


We hope to apply this to real “lab rats”, soon

Meanwhile, we sharpen our tools

 And publicize our results to date (see below).

+
7
background


digression
















+ 9

timm = nerd hippy (type 3)

+ 10


Type 1 Type 2 Type 3
Habitat: Habitat: Habitat:
Haight-Ashbury MIT (a) Berkeley (b) Mum’s living room, Helsinki
(c) Portland

Example: Richard Stallman Example: Linus Torvalds

Goals: What goals? Goals Goals: lecturing you on how to Goals: Finding out how we can better build
are a construct, man. Free do it better tools, together
your mind!

+ 11


Haight-Ashbury MIT (a) Berkeley (b) Mum’s living room, Helsinki
(c) Portland

Example: Richard Stallman Example: Linus Torvalds

Goals: What goals? Goals Goals: lecturing you on how to Goals: Finding out how we can better build
are a construct, man. Free do it better tools, together
your mind!

+ 12


Haight-Ashbury, MIT, Boston (a) Berkeley
San Francisco (b) Portland
(c) Mum’s living room, Helsinki

Examples: happily, very few Example: Richard Stallman Example: Linus Torvalds

Goals: Goals are a construct, Goals: lecturing you on how to Goals: Finding out how we can better build
man. Free your mind! do it better tools, together

+ 13

hippies share

(the “PROMISE”
project)

 Repeatable, improvable, (?refutable) software engineering experiments
“Put up or shut up”

 Submit the paper AND the data

 Activities
Annual conference: this year, co-located with ICSE

 Journal special issues: 2008,2009 Empirical Software Engineering
 On-line repository: http://promisedata.org/data: contributions welcome!

+ 14

plays well
with others
You have data? Ok then…..


With NASA, IV&V SE research chair

 2008, predicting software defects
 2001

With Dan Port

 ASE’08: softwareprocess models to
assess agile programming

with Andrian Marcus:

: SEVERIS: automatic audits for
 ICSM’08
text reports of software bugs.
 ICSM’09 (submitted): incorporating user
feedback for better concept location

with Barry Boehm

below
 See

with Jamie Andrews:

 TSE’09(submitted): genetic algorithms
to design test cases that maximize code
coverage

+ 15

plays well
with others

+ 16

Plays well
with others
(but not as good as some)

+
17
background


digression
















+ 18

internal vs
drastic changes
 Internal changes: within the space of current project options

 Drastic change: cry havoc and let slip the dogs of war

+ 19

internal vs
drastic changes

Internal
choices

+ 20

internal vs
drastic changes

Internal
choices

Drastic
changes

+ 21

internal vs
drastic changes

Internal
choices

Drastic
changes

Can internal choices out-perform drastic change?

+
22
background


digression
















Estimates = model(P, T)
+ 23

T = tunings

P = project

G = goals

Some project options

+

T = tunings

P = project

Note: controllability
assumption

G = goals


+ 25

Some tuning
options

Ranges seen in 161 projects, Learned
via regression, Boehm 2000

Decrease!
Increase!
effort!
effort!
acap, apex, ltex, pcap, pcon,
plex,sced, site,toool
Note: controllability cplx, data,
assumption docu pvol, rely, ruse, stor, time

G = goals


+ 26

Some tuning
options

Ranges seen in 161 projects, Learned
T = tunings
via regression, Boehm 2000

P = project
Decrease!
Increase!
effort!
effort!
acap, apex, ltex, pcap, pcon,
plex,sced, site,toool
Note: controllability cplx, data,
assumption docu pvol, rely, ruse, stor, time

An objective function

• Find least p from P that reduce effort (E) ,
defects (D), time to complete- in months (M)

G = goals

+
27
background


digression
















+ 28

It is possible to predict software

development effort [Boehm81,
Chulani99]

development models can be
 Software
used to debate trade-offs between
different management options:
[Boehm00], and many others besides.

decision making need not wait on
 Such
detailed local data domain collection
[Fenton08, Menzies08]

AI is useful for software engineering


tools can explore and rank more
 AI
options that humans [Menzies00], and
many more besides

tools might be better than standard
 AI
methods (many papers).

options found by the AI tools are
 The
better than (at least some) management
repair actions (this paper)

+ 29


Chulani99]

 Software

Such decision making need not wait on




 AI
many more besides

AI tools might be better than standard


The options found by the AI tools are


+ 30


Relative impact above lowest value
Chulani99]
team
flex
 Software
resl
used to debate trade-offs between pmat
different management options: prec
[Boehm00], and many others besides. e = b + 0.01 * sum( scalers )
cplx
acap
Such decision making need not wait on pcap Effort = a * KLOC ^e

time
detailed local data domain collection * prod( multipliers )
[Fenton08, Menzies08] pcon
rely
Defaults <a,b> = <2.94, 0.91>
site

docu
Local calibration [Boehm81]:
apex
• Tune <a,b> using local data
tool
 AI
pvol
stor
many more besides Bayesian calibration [Chulani99]:
sced
• Combine expert intuition with
ltex
historical data
 data
methods (many papers). plex
ruse
 0.00 2.00 4.00 6.00
Estimates are within 30% of actual, 69% of the time

+ 31


Chulani99]

 Software
[Boehm00], and many others besides




 AI
many more besides





+ 32


Boehm’s value-based SE challenge:
• Most SE techniques are “value-neutral”
Chulani99]
(euphuism for “useless”).
• tune recommendations, process decisions
 Software
to the particulars of company
e.g. [Huang06]: mapped a business into Boehms’ models
developed a “risk exposure measure” combining
(a) racing delivery to market
(b) delivered software defects
Ran two scenarios:.


 AI
many more besides





+ 33


Chulani99]

 Software




 AI
many more besides





+ 34


Chulani99]

 Software
Yet another victim of
the data drought


 AI
many more besides Fenton07: “....much of the current software
metrics research is inherently irrelevant to
the industrial mix ... any software metrics

program that depends on some extensive
metrics collection is doomed to failure.”

e.g. After 26 years, Boehm collected less
than 200 sample projects for the COCOMO
effort database

+ 35

For project options P and internal model tunings T:

Chulani99]
Tuning uses local data to constraint T
development models can be • e.g. Boehm’s local calibration constrains <a,b>
 Software
different management options: If T’s variance dominates P’s variance, then you must tune?

T)
Such decision making need not wait on Estimates = model(P,
But what about if P’s variance dominates?

 AI
options that humans [Menzies00], and Then control estimates by controlling P:
many more besides • Keep “t” random (no local data for tuning)
• Find the smallest “p” from P (random project)
 that most changes estimates.
[Menzies08]:
The options found by the AI tools are • var(P) dominates in Boehm’s COCOMO models

better than (at least some) management • just changing P yields similar estimates to standard methods
repair actions (this paper) • local data collection is useful, not mandatory.

UAI finds “p” in Boehm’ s effort/time/defect predictors.

+ 36


Harman: search-based SE
 Mark
 Many management decisions are over-
Chulani99]
constrained
 No solution satisfies all users, all criteria.
 Software
 E.g. better, faster, cheaper, pick any two
tools.
 Many
 Data mining to learn defect predictors (see Jan

IEEE TSE ’07)
detailed local data domain collection  Genetic algorithms for test case generation
(recall work with Andrews).
 Simulated annealing for software process

planning (my ASE’07 paper).
 AI search for project planning (see below)
tools can explore and rank more  Abduction (a.k.a. partial evaluation + constraints
 AI
many more besides





+ 37


Chulani99]

 Software


The “one slide” rule


 AI
many more besides





+ 38


development effort [Boehm81, At each “x”, AI
Chulani99]
search (*) finds
and takes next best
 Software
decision.
After each decision,
run Boehm’s models
100 times (for the as-
yet undecided, select
their values at
random).

Prune spurious final
decisions with a
 AI
back-select.
many more besides





+ 39


development effort [Boehm81, At each “x”, AI
Chulani99]
search (*) finds
and takes next best
 Software
decision.
After each decision,
run Boehm’s models
100 times (for the as-
yet undecided, select
their values at
random).

Prune spurious final
decisions with a
 AI
back-select.
many more besides




Search methods = simulated annealing, beam, issamp,
keys, a-star, maxwalksat, dfid, LDS…

+ 40


Chulani99]

 Software





 AI
many more besides



repair actions (this paper) IMPORTANT: this representation allows managers to perform
trade-off relations on our recommendations

+ 41

Conventional

development effort [Boehm81, optimizations:
Chulani99] 1. One solution
2. Constraining all
development models can be choices
 Software
AI search :
1) N solutions

2) Provide
neighborhood
information


 AI
many more besides

AI tools might be better than standard Mark Harman: “Solution robustness may be as important as

methods (many papers). solution functionality.

 “For example, it may be better to locate an area of the
better than (at least some) management search space that is rich in t solutions, rather than
identifying an even better solution that is surrounded by a
set of far less t solutions. ”

+ 42


• Other uncertainty-in-SE-estimates
Chulani99]
• Typically Bayes nets.
• Usually, single goal
 Software
• defects [Fenton08]
different management options: • effort [Pendharkar05]

• Little (?no) trade off analysis to understand

• neighborhood of solution
[Fenton08, Menzies08] • minimal solution

AI is useful for software engineering • We explore multiple, possibly competing, goals


• Better AND faster AND cheaper
 AI
many more besides


• We offer neighborhood solutions
• We offer trade offs between solution size and effectiveness
The options found by the AI tools are • We can work in very dimension space

.

+ 43


Even advanced visualization methods
fail after 5-10 dimensions
Chulani99]

 Software





 AI
many more besides





+ 44

[Coarfo00] and [Gu97]: AI methods faster*100
 than (e.g.) int. programming for this kind of task.
Chulani99]
Our tools are like optimizers
 Software that make so assumption of
used to debate trade-offs between linearity, continuity,
different management options: single maxima or
even smoothness.

Traditional gradient descent
optimizers assume smooth
surfaces.


But what of local-maxima?
 AI
many more besides And what if our
shapes are not smooth?

[Baker07]: learn
methods (many papers). <a,b> for Boehm’s b
model in NASA
The options found by the AI tools are data 1000 times,

better than (at least some) management pick 90% of
repair actions (this paper) data at random.

a

+ 45


Chulani99]
P(ground)=
 Software





 AI
many more besides





+ 46


Chulani99]
P(ground)=
 Software


detailed local data domain collection Generate consequences of drastic change
[Fenton08, Menzies08] • Monte Carlo simulation of the above.



Contrast with good selection of internal choices
 AI
Control estimates by controlling P:
many more besides
• Keep “t” random (no local data for tuning)

• Find the smallest “p” from P (random project)
that most improves scores from estimates.


+
47
background


digression
















+
54
background


digression
















+ 55

Related Work
Other search-based SE:
Other parametric cost models: 

Focuses on few tools: SA, GA, tabu.
PRICE-S, SLIM, or SEER- SEM 

(we started with SA, but moved on)
Not open source (have secrets) 

Exciting new generation of tools
?possibly over-elaborated. 

 constraint satisfaction algorithms
May generate range of estimates

 stochastic SAT solves to explore
 but no search to find better options.

Other instance-based effort tools:

Other COCOMO work

ANGEL: Shepperd’s nearest neighbor

 Very focused on regression methods & tuning
No parametric form.

 Problems with data drought, tuning
No way to describe shape of the theory,

variance, performance variance
no way to explore perturbations of that
theory

Discrete methods: insightful? fun!


+
56
background


digression
















+ 57

Conclusion

 Software project managers have more options than they think.

 It is possible (even useful) to push back against drastic change

 Q: can we found local options that out-perform drastic change?
A: yes, we can


Adjust current Use models whose estimates are
project options dominated by project variance;

Fire people

Control estimates via P:
Do it all again,
• Keep “t” random (no need for local
in ADA
tuning data)

• Using AI, find the fewest parts of P
Deliver late
that most change estimates

+ 58

Future work

 Constraint logic programming
Haven’t shown the dark side of our models


Procedural kludges regarding conditional co-dependencies


 Much recent interest in constrained regression
Users offer hints on what kinds of theories they’d accept


These hints bias the search algorithms


 Is
CLP a general/useful framework for adding “hinting” to
our current tools?

+
59
background


digression
















+ 60

my research question:
why are humans so successful?
 The world is a very complex place: how do dumb humans get by?

 How did dummies like me (?and you) build things as complex as:
The internet?


The international domestic airplane network?


 The Apollo moon rocket? (400K parts, 2K contractors, worked flawlessly)

+ 61

why do dummies get by?
 Answer #1: we don’t get by
Computers crash

 Economic systems fail
 Sure, we get some failures
Zipf’s law: reuse frequency
 But why don’t we fail all the time?
of library functions: LINUX,
Sun OS, Mac OS
 Answer #2: some of us aren’t so dumb
Kepler, Descartes, Newton, Planck,

 So few of them, too many “Menzies”

 Answer #3: the world is not as complex as it
appears
Key variables: a few things set the rest

 Most possible differences, aren’t
 More regularities than you might expect

Distribution of, change-
proned classes:
Koffice, mozilla

+ 62

applications of “keys”

 Rewrite all your algorithms, assuming keys
 Reduce complex problems to simper ones

 simpler, faster code

 Reduce complex answers to simpler ones

 shorter, clearer, theories

+ 63

It is possible to predict software In effort estimation, useful for bridging expert vs

development effort [Boehm81, model-based estimation methods
Chulani99]
Support Jorgensen's Expert Judgment Best Practices:
development models can be 1. evaluate estimation accuracy,
 Software
2. avoid conflicting estimation goals
different management options: 3. ask the estimators to justify and
[Boehm00], and many others besides. criticize their estimates
4. avoid irrelevant and unreliable
estimation information

5. use documented data from previous
development tasks
6. find estimation experts with relevant
domain background

7. estimate top-down +bottom-up
8. use estimation checklists
9. combine estimates of many
 AI
experts +estimation strategies
10. assess the uncertainty of the estimate
many more besides
11. provide feedback on estimation accuracy
12. provide estimation training opportunities

(M. Jorgensen. A review of studies on expert estimation of software
development effort. Journal of Systems and Software, 2004).


+ 64


Chulani99]
 Software
Cross-val

development tasks
domain background

 AI
many more besides



+ 65


Chulani99]
 Software
[Boehm00], and many others besides Feature selection,
criticize their estimates
Instance-based
estimation information learning

development tasks
domain background

 AI
many more besides



+ 66


Chulani99]
 Software
[Boehm00], and many others besides criticize their estimates
development tasks
[Fenton08, Menzies08] Data mining
domain background

 AI
many more besides



+ 67


Chulani99]
 Software
[Boehm00], and many others besides criticize their estimates
development tasks
domain background

7. estimate top-down +bottom-up Ensemble
8. use estimation checklists learning
 AI
many more besides



+ 68


Chulani99]
 Software

development tasks
domain background

 AI
options that humans [Menzies00], and Report variance
many more besides in cross-
validation



How to avoid drastic project change (using stochastic stability)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie How to avoid drastic project change (using stochastic stability)

Ähnlich wie How to avoid drastic project change (using stochastic stability) (20)

Mehr von CS, NcState

Mehr von CS, NcState (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How to avoid drastic project change (using stochastic stability)