A Method Assessment Framework

1. © T.McBride and B.Henderson-Sellers, 2011
A Method Assessment
Framework
Tom McBride
Brian Henderson-Sellers
School of Software
University of Technology, Sydney

2. © T.McBride and B.Henderson-Sellers, 2011
There is a need for a method
assessment framework
• There has been little discussion of how the
quality of a method might be assessed, or when
• A growing number of communities are
developing methods without the aid of method
engineers
• We believe that such initiatives can be supported
by an assessment framework that separates the
concerns of different phases in the life cycle of a
method (design, enactment, performance)
ME'11 2

3. © T.McBride and B.Henderson-Sellers, 2011
A method can be assessed at design,
enactment and performance
• Method design
– When a method is created in order to achieve a specific purpose
but in a range of circumstances
e.g. a method suited to developing life critical software using
medium to large teams under tight schedule and budget
constraints
• Method enactment
– When a method is tailored to the specific circumstances of a
project
– Project planning once the project requirements, schedule, team,
budget, resources, environment have been decided.
• Method performance
– When the method is being used
ME'11 3

4. © T.McBride and B.Henderson-Sellers, 2011
Process design and enactment
Process Model or Method
Metamodel
Process
Tailored
process
Performed
process
As enacted by
real people on a
specific project
As documented
As standardized
Method
Design
Method
Enactment
Process Model or Method
Metamodel
Process
Tailored
process
Performed
process
As enacted by
real people on a
specific project
As documented
As standardized
Process Model or Method
Metamodel
ProcessProcess
Tailored
process
Performed
process
As enacted by
real people on a
specific project
As documented
As standardized
Method
Design
Method
Enactment

5. © T.McBride and B.Henderson-Sellers, 2011
What is method quality
• Is the method suitable for its intended
purpose in the circumstances for which is
intended?
• Is the method complete?
– Work products missing, created but not used,
used but not created
• Is the method actually needed for the
particular application
ME'11 5

6. © T.McBride and B.Henderson-Sellers, 2011
Method design
• Software development methods are created to
address a range of circumstances
– Waterfall development
– Spiral development
– SCRUM
• Not intended to be used without modification.
• Although necessarily general, their usefulness and
ability to achieve their primary purpose of developing
software in the claimed circumstances should be
assessed.
• For example: If the method is claimed to be suited to
developing life critical software applications, is it?
ME'11 6

7. © T.McBride and B.Henderson-Sellers, 2011
Different approaches suit different
contingencies
• Large projects with dispersed teams without tight
collaboration generally favours plan-based
methods (Waterfall)
• Exploratory systems with high risk generally
favour a Spiral, iterative or incremental
approach.
• When business architect and software developer
must jointly explore problem and develop the
system, SCRUM or other agile method is
favoured
ME'11 7

8. © T.McBride and B.Henderson-Sellers, 2011
The primary purpose of a method is to
achieve some objective
• The method must contain the processes that
can conceivably achieve the intended purpose
e.g.
– Develop, deploy, maintain software
– Deploy, operate and manage IT services
• Such primary processes have been the focus
of many proposed situational method
engineering methods
ME'11 8

9. © T.McBride and B.Henderson-Sellers, 2011
Consider issues for performance
management
• A method should include the means to set and
manage performance goals
• Performance goals must be set
– Usually the agreed requirements, schedule, budget
and other project specific objectives
– Over-arching organizational goals
– Statutory requirements
• Performance must be monitored and managed
• Can be achieved through including specific
process fragments
ME'11 9

10. © T.McBride and B.Henderson-Sellers, 2011
Resources must be deployed, work
coordinated
• Performance goals do not determine how they will
be met
• Resources must be deployed to their best effect
• Work must be coordinated through the processes
– Static coordination through plans, work product flows
– Dynamic through meetings, reviews and other exchanges
• Usually achieved through project management
processes
• Resource deployment and work coordination
processes must be adequate for their intended range
of circumstances
• Are they?
ME'11 10

11. © T.McBride and B.Henderson-Sellers, 2011
Method enactment
• Each project has considerable knowledge of the
specific project contingencies, constraints and
performance objectives
• Each project must decide precisely how the
resources will be deployed to perform the
specific activities necessary to achieve the
specific objectives in the specific circumstances
– Shift activities among processes to compensate for
personnel weaknesses or to take advantage of
strengths
– Any resource excess or shortage also influences how
the method is enacted
ME'11 11

12. © T.McBride and B.Henderson-Sellers, 2011
Existing methods have little to say about
distributed software development
• How are processes enacted to deal with the
problems of distributed development
• Existing software development methods have
little to say about distributed development yet
they are used on such projects
e.g. verification could be performed by the
developer on exit from a process or by the
acquirer on entry to the next process in the
workflow
ME'11 12

13. © T.McBride and B.Henderson-Sellers, 2011
Method assessment at enaction
Helps avoid any tendency to use a familiar
method that is not fully appropriate to the
particular situational constraints
Helps to facilitate a more objective judgement
about the appropriateness of the method

14. © T.McBride and B.Henderson-Sellers, 2011
Method performance
• An enacted method should achieve the best possible
outcome in practice
• Does it?
• Need to gather evidence to prove or disprove that the
method is the best possible
• Two distinct issues
– Right method, poorly performed
– Acceptable performance, wrong method
• Dominant assumption is that the method is right, but
poorly performed
• How would anyone detect a wrong method?
ME'11 14

15. © T.McBride and B.Henderson-Sellers, 2011
Existing process assessment methods
range from the formal to informal
• Formal process assessment methods
– SPICE, CMMI, Six Sigma, ISO 9001
– Not specific to the circumstances
– Rely on finding common problems
• Informal assessments are done all the time
– SCRUM has a retrospective
– People learn and adjust what they do
– BUT this doesn’t provide guidance on what to look
for
ME'11 15

16. © T.McBride and B.Henderson-Sellers, 2011
Measures of method performance
• Some interest in measuring process
effectiveness, efficiency, alignment,
coordination, governance BUT no agreed
measures
• May be difficult to determine whether these
attributes are being achieved
• May be easier and more possible to
determine the effect of their absence –
defects, schedule overruns, other failures
• Difficult to establish links between detected
faults and their causes
ME'11 16

17. © T.McBride and B.Henderson-Sellers, 2011
Advantages of this framework
• Separates the concerns of each phase (design,
enactment, performance)
• Identifies what can reasonably be achieved
and assessed at each phase
• Provides guidance for development of
assessment techniques and tools
ME'11 17

18. © T.McBride and B.Henderson-Sellers, 2011
Future Research
• Linkages between actual software
development practices and their tailoring to
SME body of knowledge
• Development of metrics appropriate to each
of the three lifecycle phases (design,
enactment, performance)

19. © T.McBride and B.Henderson-Sellers, 2011
Conclusion
• Proposed a framework to enable assessment of
methods at different phases of their life cycle
• The framework allows method engineers to assess the
congruence of their developed, enacted or performed
method according to the claimed range of known
constraints and contingencies
• The framework highlights some fruitful areas for
research
• The framework underlines the need to better link
actual practices and their tailoring to method
engineering
ME'11 19