An Empirical Study on the Relationship between the Use of Agile Practices and the Success of Scrum Projects A. César C. França, Fabio Q. B. da Silva, Leila M. R. de Sousa Mariz Abstract-In this article, factors considered critical for the success of projects managed using Scrum are correlated to the results of software projects in industry. Using a set of 25 factors compiled in by other researchers, a cross section survey was conducted to evaluate the presence or application of these factors in 11 software projects that used Scrum in 9 different software companies located in Recife-PE, Brazil. The questionnaire was applied to 65 developers and Scrum Masters, representing 75% (65/86) of the professionals that have participated in the projects. The result was correlated with the level of success achieved by the projects, measured by the subjective perception of the project participant, using Spearman's rank correlation coefficient. The main finding is that only 32% (8/25) of the factors correlated positively with project success, raising the question of whether the factors hypothesized in the literature as being critical to the success of agile software projects indeed have an effect on project success.Given the limitations regarding the generalization of this result, other forms of empirical results, in particular case-studies, are needed to test this question. Paper presented at Empirical Software Engineering and Measurement Conference, Bolzano, 2010. http://www.haseresearch.com

1. S
O
P
E
I
Influences
Is influenced
An Empirical Study on the Relationship between the Use of
Agile Practices and the Success of Scrum Projects
A. César C. França; Fabio Q. B. da Silva; Leila Mariz
<cesarfranca@gmail.com; fabio@cin.ufpe.br;
leilamariz@yahoo.com.br>

2. S
O
P
E
I
Influences
Is influenced
Research Project
2
(empirical) research project on the
influence of human factors on software
engineering
Long Term Objective: to understand
the influence of personality, behavior,
and cognition on individuals, teams,
projects, processes, and organizations
in software engineering.

3. S
O
P
E
I
Influences
Is influenced
Research Project
3
(empirical) research project on the
influence of human factors on software
engineering
Centro de Informática – UFPE
Since 2003 ...
1 professor/researcher
4 PhD students
18 master students

4. S
O
P
E
I
Influences
Is influenced
Context
4
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority
T. Chow, e D. Cao, "A Survey
Study of Critical Success Factors
in Agile Software Projects.", The
Journal of Systems and Software,
n. 81, 2007, pp. 961–971

5. S
O
P
E
I
Influences
Is influenced
Context and Goals
5
Critical Success
Factors
Agile Attibutes
Delivery strategy A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and
expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management
style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement
management process
A14 Following agile-oriented project management
process
A15 Following agile-oriented configuration
management process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-
face meetings
A18 Honoring regular working Schedule
Team environment A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority
Is the adoption of the
25 agile attributes
related to the
success of software
development projects
managed using
Scrum?

6. S
O
P
E
I
Influences
Is influenced
Study Design
6
Method
Cross sectional survey

7. S
O
P
E
I
Influences
Is influenced
Study Design
7
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Method
Cross sectional survey

8. S
O
P
E
I
Influences
Is influenced
Study Design
8
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner

9. S
O
P
E
I
Influences
Is influenced
Study Design
9
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception of use of agile attribute
Perception of project success

10. S
O
P
E
I
Influences
Is influenced
Study Design
10
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25: level of use of Agile Attribute
S: project success
Scale
Five point Likert type

11. S
O
P
E
I
Influences
Is influenced
Development
11
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25 and S
Study Design
Time
July-August, 2009

12. S
O
P
E
I
Influences
Is influenced
Development
12
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25 and S
Study Design
Time
July-August, 2009Context
9 firms
11 projects
Recife Scrum Group

13. S
O
P
E
I
Influences
Is influenced
Development
13
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25 and S
Study Design
Time
July-August, 2009
Context
9 firms
11 projects
Recife Scrum Group
Participants (62)
48 Scrum Team
10 Scrum Master
4 Product Owner

14. S
O
P
E
I
Influences
Is influenced
Development
14
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25 and S
Study Design
Time
July-August, 2009
Context
9 firms
11 projects
Recife Scrum Group
Participants (62)
48 Scrum Team
10 Scrum Master
4 Product Owner
Questionnaire
65 questionnaire sent
62 answered
95% response rate

15. S
O
P
E
I
Influences
Is influenced
Statistical Analysis
15
Method
Cross sectional survey
Context
Small software firms
Located in Recife.br
Finished Scrum Project
Perspective/Population
Scrum Team, Scrum
Master, Product Owner
Questionnaire
Closed questions
Perception use agile attribute
Perception of project success
Variables
A01 ... A25 and S
Study Design
Time
July-August, 2009
Context
9 firms
11 projects
Recife Scrum Group
Participants (62)
48 Scrum Team
10 Scrum Master
4 Product Owner
Questionnaire
65 questionnaire sent
62 answered
95% response rate
Correlation
Spearman's rank
correlation coefficient
Factor Analysis
PCA
Development

16. S
O
P
E
I
Influences
Is influenced
Results: Correlation
16
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority

17. S
O
P
E
I
Influences
Is influenced
Results: PCA
17
C1
Management style
A17 Strong communication focus with daily face-to-face meetings
A10 Managers knowledgeable in agile
A18 Honoring regular working Schedule
A11 Managers who have adaptative management style
A20 Coherent, self-organizing team-work
C2
Software process
A23Good customer relationship
A13Following agile-oriented requirement management process
A15 Following agile-oriented configuration management process
A14Following agile-oriented project management process
A02Delivering most important features first
C3
Team Structure
A22Projects with no multiple independent teams
A21Projects with small team
C4
Technology
A03 Well-defined coding Standards up front
A07 Correct integration testing
A12 Appropriate technical training to team
A04 Pursuing simple design
C5
Team capability
A09Team members with great motivation
A08Team members with high competence and expertise
C6
Customer commitment
A24Strong customer commitment and presence
A06Right amount of documentation
C7
Delivery strategy
A01Regular delivery of software
C8
Team Location
A19Collocation of the whole team
A05 Rigorous refactoring activities
C9
Customer awareness
A25Customer having full authority
A16Good progress tracking mechanism

18. S
O
P
E
I
Influences
Is influenced
Discussion
18

19. S
O
P
E
I
Influences
Is influenced
“Human Factors”
19
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority

20. S
O
P
E
I
Influences
Is influenced
Correlated “Human Factors”
20
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority

21. S
O
P
E
I
Influences
Is influenced
Correlated “Human Factors”
21
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority
Not a surprise. Happens with non Agile
methodologies.

22. S
O
P
E
I
Influences
Is influenced
Correlated “Human Factors”
22
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority
Agrees with research on team work!
Not ease to achieve in SE!

23. S
O
P
E
I
Influences
Is influenced
“Surprises”
23
Critical Success
Factors
Agile Attibutes
Delivery
strategy
A01 Regular delivery of software
A02 Delivering most important features first
Agile software
engineering
techniques
A03 Well-defined coding Standards up front
A04 Pursuing simple design
A05 Rigorous refactoring activities
A06 Right amount of documentation
A07 Correct integration testing
Team capability A08 Team members with high competence and expertise
A09 Team members with great motivation
A10 Managers knowledgeable in agile
A11 Managers who have adaptative management style
A12 Appropriate technical training to team
Project
management
process
A13 Following agile-oriented requirement management process
A14 Following agile-oriented project management process
A15 Following agile-oriented configuration management
process
A16 Good progress tracking mechanism
A17 Strong communication focus with daily face-to-face
meetings
A18 Honoring regular working Schedule
Team
environment
A19 Collocation of the whole team
A20 Coherent, self-organizing team-work
A21 Projects with small team
A22 Projects with no multiple independent teams
Customer
involvement
A23 Good customer relationship
A24 Strong customer commitment and presence
A25 Customer having full authority

24. S
O
P
E
I
Influences
Is influenced
Four Areas of Further Research
24
Self-organizing Teams
What are they in SE?
How to build one?
How to keep them cohesive?

25. S
O
P
E
I
Influences
Is influenced
Four Areas of Further Research
25
Motivation
How is motivation related to
performance?
How is motivation related to
success?
Self-organizing
Teams
What are they in SE?
How to build one?
How to keep them
cohesive?

26. S
O
P
E
I
Influences
Is influenced
Four Areas of Further Research
26
Motivation
How is motivation
related to
performance?
How is motivation
related to success?
Leadership Style
How the style of
leadership and
management affects
success?
Self-organizing
Teams
What are they in SE?
How to build one?
How to keep them
cohesive?

27. S
O
P
E
I
Influences
Is influenced
Four Areas of Further Research
27
Leadership Style
How the style of
leadership and
management affects
success?
Personality and
Behavior
Does personality matter?
How different team work
behavior affects team
work effectiveness?
Motivation
How is motivation
related to
performance?
How is motivation
related to success?
Self-organizing
Teams
What are they in SE?
How to build one?
How to keep them
cohesive?

28. S
O
P
E
I
Influences
Is influenced
Conclusions
• Limitations and Threats to Validity:
– Restricted context
– Small number of participants, projects, and firms
• Contributions:
– New research questions
– More detailed factor structure
28

29. S
O
P
E
I
Influences
Is influenced
An Empirical Study on the Relationship between the Use of
Agile Practices and the Success of Scrum Projects
A. César C. França; Fabio Q. B. da Silva; Leila Mariz
<cesarfranca@gmail.com; fabio@cin.ufpe.br;
leilamariz@yahoo.com.br>
Thank you!

30. S
O
P
E
I
Influences
Is influenced
An Empirical Study on the Relationship between the Use of Agile Practices and the Success
of Scrum Projects by A. César C. França, Fabio Q. B. da Silva, Leila M. R. de Sousa Mariz is
licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported
License.
Based on a work at Empirical Software Engineering and Measurement Conference, Bolzano,
2010.
Permissions beyond the scope of this license may be available
at http://www.haseresearch.com