test

1. Globally Distributed Agile Release Trains
Adjusting the Scaled Agile Framework® (SAFe®) for distributed
environments
Peter van Buul
Master thesis

3. Globally Distributed Agile Release Trains
Adjusting the Scaled Agile Framework® (SAFe®) for distributed
environments
Thesis
submitted in partial fulfilment of the
requirements for the degree of
Master of Science
in
Computer Science
Information Architecture Track
by
Peter van Buul
born in Leiden
at the Delft University of Technology,
to be defended publicly on Thursday December 15, 2016 at 16:00.
Delft University of Technology
Faculty Electrical Engineering, Mathematics and Computer Science
Thesis committee: Prof.dr.ir. Rini van Solingen TU Delft, supervisor university
Dr. Georgios Gousios TU Delft
Ir. Hendrik-Jan van der Waal Prowareness, supervisor company
An electronic version of this thesis is available at http://repository.tudelft.nl/

5. Globally Distributed Agile Release Trains
Adjusting the Scaled Agile Framework® (SAFe®) for distributed
environments
Author: Peter van Buul
Student ID: 1512269
Email: petervanbuul@gmail.com
Abstract
Thesis committee:
Prof.dr.ir. Rini van Solingen TU Delft, supervisor university
Dr. Georgios Gousios TU Delft
Ir. Hendrik-Jan van der Waal Prowareness, supervisor company
SAFe is a framework that applies both agile and lean practices
for developing software. The current trend is that increasingly more organizations
develop their software in a globally distributed setting. Although SAFe is being
deployed in such a setting, SAFe was not originally developed for such a setting but
for a co-located setting. Therefore, this research investigates the application of
SAFe in globally distributed settings. Five problems are discovered that can be
expected to fail when SAFe is applied in distributed settings: incorrect execution of
SAFe, language barriers, time zone differences, increased communication effort, and
inefficient communication tools. Given these problems, four SAFe elements are
identified that can be expected to fail when SAFe is applied in distributed settings:
the PI planning, the inspect & adapt meeting, the DevOps team, and the system team.
Finally, a customization of SAFe for distributed settings is proposed. This
customization is focused on solving the discovered problems for the elements
identified to fail.

7. i
Preface
This thesis presents the research that is conducted as part of the Information Architecture Master
track of the Computer Science programme of the Delft University of Technology. A workplace during
this research was provided by Prowareness. The research is focused on the Scaled Agile Framework®
(SAFe®) in globally distributed environments. SAFe and Scaled Agile Framework are registered
trademarks of Scaled Agile Inc. referred in [1]1
.
SAFe is a framework that applies both agile and lean practices for developing software. This
framework is publicly available and widely used in mainly the IT industry. The current trend is that
increasingly more organizations develop their software in a globally distributed setting. Although SAFe
is being deployed in such a setting, SAFe was not originally developed for such a setting, but for a co-
located setting. Therefore, it is interesting to research the application of SAFe in globally distributed
settings.
This research has discovered that the following five problems can be expected when SAFe is applied
in a distributed setting: incorrect execution of SAFe, language barriers, time zone differences,
increased communication effort, and inefficient communication tools.
Given these five problems, this research identified that the following four elements of SAFe can be
expected to fail when SAFe is applied in a distributed setting: the PI planning, the inspect & adapt
meeting, the DevOps team, and the system team.
This research is concluded by proposing a customization of SAFe for distributed settings. This
customization is focused on solving the discovered problems for the elements identified to fail. The PI
planning and inspect & adapt can be customized by having a Release Train Engineer at each location,
using a video conferencing system, using a digital program board & digital PI objectives, and extending
the PI planning over multiple days if needed. The DevOps team can be customized by enabling the
team to travel regularly to the other locations. Finally, the system team can be distributed over all
locations.
First, and foremost, I would like to thank my supervisor, Rini van Solingen, for guiding me in giving
direction to this research, and providing critical feedback during this research. Without his guidance
and feedback, the systematic approach and critical view in this report would not have been possible.
Second, I would like to thank my supervisor at Prowareness, Hendrik-Jan van der Waal, for giving his
insights on SAFe, and his critical feedback on the results of this research. Third, because most of my
time I have spent working at Prowareness, I would like to thank all my colleagues for always being
there to help me, and for being a sparring partner to discuss ideas. Fourth, I would like to thank the
participants of the focus groups for their contributions to this research.
Last, but not least, I would like to thank my girlfriend, my family, and my friends for always supporting
me during this project. Without these loving and caring people, taking an interest in my progress,
reviewing my work, and helping me to finish the project, this result would not have been possible.
Finally, I present you my thesis, I hope you will enjoy reading it.
1
It should be noted that all information regarding SAFe in this thesis is as interpreted by the author, and verified
with SAFe experts. Any information is therefore not officially supported by Scaled Agile, Inc., unless quoted
directly from Scaled Agile, Inc., in which case this is specified.
Peter van Buul
Leiden, the Netherlands
December 1, 2016

8. 2
Table of Contents
Chapter 1 Introduction ........................................................................................................................4
1.1. Context....................................................................................................................................4
1.2. Research questions.................................................................................................................6
1.3. Reading guide..........................................................................................................................7
Chapter 2 Research background..........................................................................................................8
2.1. Globally Distributed Software Engineering.............................................................................8
2.2. SAFe.........................................................................................................................................9
2.3. Agile scaling frameworks ......................................................................................................14
2.4. Research scope .....................................................................................................................18
Chapter 3 Research methodology .....................................................................................................21
3.1. Research approach................................................................................................................21
3.2. Systematic Literature Review ...............................................................................................22
3.3. Multiple informant methodology .........................................................................................24
3.4. Focus group...........................................................................................................................26
Chapter 4 Distributed SAFe problems ...............................................................................................30
4.1. Problems of Distributed Agile Development: Systematic Literature Review .......................30
4.2. Distributed SAFe problems: Multiple informant methodology............................................37
Chapter 5 Identification of failing SAFe elements .............................................................................42
5.1. Identification based on theory: Literature............................................................................42
5.2. Identification based on theory: Expert focus group .............................................................43
5.3. Identification based on practice: Practitioner focus group ..................................................49
5.4. Result of triangulation ..........................................................................................................56
Chapter 6 Customizations of SAFe.....................................................................................................59
6.1. Customizations of SAFe based on theory: Literature ...........................................................59
6.2. Customizations of SAFe based on practical experience: Practitioner focus group ..............61
6.3. Combining theory and practical experience.........................................................................65
Chapter 7 Discussion..........................................................................................................................68
7.1. Answers to the research questions.......................................................................................68
7.2. Limitations.............................................................................................................................69
7.3. Reflection..............................................................................................................................73
7.4. Recommendations for future research.................................................................................76
Chapter 8 Conclusion.........................................................................................................................78
8.1. Summary...............................................................................................................................78
8.2. Conclusion.............................................................................................................................78

9. 3
Bibliography ..........................................................................................................................................81
List of tables..........................................................................................................................................93
List of figures.........................................................................................................................................98
Appendix A Systematic Literature Review protocol.......................................................................100
Appendix B Multiple informant protocol.......................................................................................102
Appendix C Expert focus group: protocol ......................................................................................103
Appendix D Practitioner focus group: protocol..............................................................................109
Appendix E Multiple informant execution.....................................................................................115
Appendix F List of SAFe elements..................................................................................................117
Appendix G Description of the Agile Release Train elements........................................................120
Appendix H Rejected studies..........................................................................................................126
Appendix I Problems and challenges of accepted studies............................................................128
Appendix J Result of SLR reviews ..................................................................................................133
Appendix K Problem groups...........................................................................................................135
Appendix L Ungrouped problems..................................................................................................142
Appendix M Expert focus group: invitation letter...........................................................................143
Appendix N Expert focus group: attachment .................................................................................145
Appendix O Expert focus group: execution ....................................................................................156
Appendix P Expert focus group: individual votes round 2.............................................................177
Appendix Q Expert focus group: consequences per element round 3...........................................179
Appendix R Expert focus group: individual votes focus group: round 4........................................185
Appendix S Expert focus group: visualization dot voting consequences: round 4 ........................190
Appendix T Survey..........................................................................................................................198
Appendix U Results of the survey...................................................................................................205
Appendix V Practitioner focus group: invitation letter ..................................................................219
Appendix W Practitioner focus group: forms..................................................................................220
Appendix X Practitioner focus group: execution ...........................................................................225
Appendix Y Practitioner focus group: categorization participants ................................................242
Appendix Z Practitioner focus group: individual votes round 2 ....................................................243
Appendix AA Practitioner focus group: individual solutions round 3...........................................248
Appendix BB Practitioner focus group: group solutions round 3.................................................255
Appendix CC Practitioner focus group: individual votes round 4 ................................................257

10. 4
Chapter 1 Introduction
In this chapter a short introduction to the research is given. First, the context for this research is
presented. Second, the research questions are described. Finally, a reading guide is given.
1.1. Context
This thesis presents the research that is conducted as part of the Information Architecture Master
track of the Computer Science programme of the Delft University of Technology. A workplace during
this research was provided by Prowareness. This research is done in the research chair on Global
Software Engineering, in the Software Engineering group of the Software Technology department in
the Faculty Electrical Engineering, Mathematics and Computer Science of Delft University of
Technology.
The research presented in this thesis is on the Scaled Agile Framework® (SAFe®) in globally distributed
environments. SAFe and Scaled Agile Framework are registered trademarks of Scaled Agile Inc.
referred in [1]2
. To provide context, first, distributed is briefly described. Second, a brief overview of
SAFe is presented. Third, the original way of software development is briefly sketched. Finally, the
agile scaling frameworks that are replacing this original way are briefly mentioned. A more elaborate
description of the agile scaling frameworks and SAFe can be found in Chapter 2.
1.1.1.Distributed
Because of the globalization of business in the 21st
century, increasingly more companies develop
software in a globally distributed setting [2], [3], [4], [5], [6], [7], and [8]. When working in a globally
distributed environment, different problems and challenges can occur regarding, among other things,
communication, coordination, and time zone differences, according to [3] and [9]. Despite these
problems, the use of fully distributed teams can be successful, as presented in [10], [11], and [12].
However, the research of [10], [11], and [12], is focused on Scrum, with a small team. SAFe, on which
the research presented in this thesis focusses, is executed with multiple teams, possibly having
different problems.
1.1.2.SAFe
In this section, SAFe is briefly described. A full overview of SAFe can be found at the SAFe website
provided by Scaled Agile, Inc. www.scaledagileframework.com [1]. SAFe is a framework that applies
both agile and lean practices for developing software. This framework is publicly available and widely
used in mainly the IT industry. In the 10th
Annual State of Agile report by VersionOne from 2015 [13],
27% of the respondents name SAFe as the method to scale agile. This ranks SAFe as the second most
used scaling method and the most used scaling framework.
As stated previously, the current trend is that increasingly more organizations develop their software
in a globally distributed setting. Although SAFe is being deployed in such a setting, as seen in multiple
case studies [14], [15], [16], and [17], SAFe was not originally developed for such a setting, but for a
co-located setting. Therefore, it is interesting to investigate the application of SAFe in globally
distributed settings.
2
It should be noted that all information regarding SAFe in this thesis is as interpreted by the author, and verified
with SAFe experts. Any information is therefore not officially supported by Scaled Agile, Inc., unless quoted
directly from Scaled Agile, Inc., in which case this is specified.

11. 5
The main workflow in SAFe for delivering value to a customer is using Value Streams. Deployment of
these Value Streams is done using Agile Release Trains which are continuously delivering new versions
of a solution to the customer. The Agile Release Train is a team of teams. In the Agile Release Train,
the agile teams are aligned via a single vision, roadmap, and program backlog. The Agile Release Train
iterates in a so-called program increment, PI, which lasts 8 to 12 weeks during which there are 4 to 6
two-week team iterations. During the team iterations the teams continuously add value to the
solution by finishing fully tested stories. At the end of each team iteration the integrated solution is
demoed.
The SAFe website states: “Along with the various Agile methods, the Manifesto provides the Agile
foundation for effective, empowered, self-organizing-teams. SAFe extends this foundation to the level
of teams of teams” [18]. The Agile Manifesto [19] is key to SAFe, and reads as follows:
1.1.3.Traditional project management frameworks
To put SAFe in context, a traditional software development process, waterfall, is presented. As well as
traditional project management frameworks such as, PRINCE2 and Rational Unified Process (RUP) are
described. These traditional frameworks use an upfront planning for projects. This upfront planning
however, does not work if the environment changes. Though both RUP and PRINCE2 can be used in
agile projects, these frameworks were not designed as such, [20], [21].
In his article in 1970, [22], Royce describes the waterfall model as a model that is at that time widely
used in the manufacturing industry. The waterfall model works in 7 steps, which are executed one
after another, starting with the creation of system requirements and finishing with deploying to
operations. Strikingly, in his article Royce expresses his concerns about the waterfall model: “I believe
in this concept, but the implementation described above is risky and invites failure.”. To counter this,
he proposes feedback and interaction between the steps. Though this is a good idea, no case has been
found were this is done in practice. In practice, there are rigid agreements without feedback and
interaction between the steps.
The upfront planning from traditional frameworks is symbolized in the PRINCE2 acronym which stands
for PRojects IN Controlled Environments [23]. This controlled environment only changes when the
controlling entity changes the environment according to plan, while agile frameworks such as SAFe
are designed to respond to unexpected change [19].
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Figure 3: Research improvement cycleWe are uncovering better ways of developing software by
doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Figure 1: Agile Manifesto from [19]
Figure 2: Agile Manifesto from [19]

12. 6
RUP is a traditional approach, of which the goal is: “to produce, within a predictable schedule and
budget, high-quality software that meets the needs of its end users.” [24]. RUP prescribes processes
and follows a predefined plan which is not in line with the Agile Manifesto. Moreover, the sixth best
practice in RUP is: “Control changes to software” [25], controlling the response to change rather than
embracing change.
Though none of the traditional frameworks explicitly considers a distributed setting, it seems they can
all be used in distributed development. In PRINCE2, delivery can be done by a supplier which is not
necessarily co-located. RUP and waterfall contain steps, hand overs or toll gates in which the project
is given to the next team based on a set of predefined requirements. Because of these hand over
points the next team could be located in a different location.
1.1.4.Agile scaling frameworks
Agile scaling frameworks are currently replacing these traditional frameworks, because the upfront
planning of the traditional frameworks can not handle unplanned changes as much as needed. Besides
SAFe, there are multiple agile scaling frameworks available which are designed to handle change. Four
frequently used frameworks are Large-Scale Scrum (LeSS) [26], Disciplined Agile Delivery (DAD) [27],
Nexus [28], and Spotify [29]. An elaborate description of these frameworks is provided in 0.
Each of these four frameworks scale the development department, however this scaling ends there.
SAFe scales reasoning with Value Streams which, if needed, include the business and other
stakeholders. This enables SAFe to scale on an organizational level rather than only on the IT level.
1.2. Research questions
The goal of the research presented in this thesis is to investigate if and how the Scaled Agile
Framework (SAFe) should be adjusted for distributed settings. To this end the following research
questions have been formulated.
1. What problems can be expected when SAFe is applied in distributed settings?
2. What SAFe elements can be expected to fail when applied in distributed settings?
3. What would SAFe look like, when customized for distributed settings?
These questions are answered in sequence, as the previous answer provides the required input for
the next answer. The answer to the first question results in distributed SAFe problems. Based on these
problems the answer to the second research question identifies which SAFe elements can be expected
to fail. To answer the third research question, these failing elements serve as input on how to
customize SAFe. When problems are identified for this customized version it is possible to repeat this
process with this input, starting again with research question one. Thus, a circle is created that can be
used to continuously improve distributed SAFe, as visualized in Figure 4. However, in this research
each step is executed only once, and the customized version remains a theoretical proposal for now.
RQ 2: Failing
SAFe
elements
RQ 3:
Customized
distributed
SAFe
RQ 1:
Distributed
SAFe
problems
Figure 4: Research improvement cycle

13. 7
1.3. Reading guide
This thesis presents the result and approach taken for the research. The research background is given
in Chapter 2. The approach is described in detail in Chapter 3. Substantiation of the results and the
data is presented in Chapter 4 to Chapter 6. Discussion on both the results and approach can be found
in Chapter 7. The conclusions of the research can be read in Chapter 8.
Chapter 2 presents the research background for the research. First, different definitions of distributed
are presented. Second, SAFe is explored and a high-level description of the framework is given, as
interpreted by this author, and verified with SAFe experts. Third, to give some perspective the
different agile scaling frameworks are described. After this exploration, the scope of the research is
set to fit within the timeframe of the master thesis project.
Chapter 3 describes the approach taken and the methodologies that are used to answer the research
questions. For each methodology, the conditions, strengths, limitations, and protocols are presented.
Chapter 4 presents an answer to the first research question: “What problems can be expected when
SAFe is applied in distributed settings?”. First, the Distributed Agile Development problems are
presented. Second, the distributed SAFe problems are presented.
Chapter 5 gives an answer to the second research question: “What SAFe elements can be expected to
fail when applied in distributed settings?”. First, failing elements are identified based on the
distributed SAFe problems that were discovered in the previous chapter. Second, failing elements are
identified in an expert focus group. Third, failing elements are identified in a practitioner focus group.
Finally, these 3 identifications are combined.
Chapter 6 describes an answer to the third research question: “What would SAFe look like, when
customized for distributed settings?”. First, solutions are identified based on the failing elements
discovered in the previous chapter. Second, the solutions identified in the practitioner focus group are
presented. Finally, a proposal on how to customize SAFe is presented.
Chapter 7 discusses the results. First, the extent to which the research questions are answered is
presented. Second, the limitations of the research methodologies are discussed. Third, the answers to
the research questions are reflected upon. Finally, recommendations for future research are
presented.
Chapter 8 gives a summary of the research questions and presents the conclusions of the research.

14. 8
Chapter 2 Research background
In this chapter the research background is presented. First, the research area of Globally Distributed
Software Development is described. After which, two different definitions of distributed are given and
the definition that is applied in this research is clarified. Second, SAFe is explored and a high-level
description of the framework is presented. Third, the different agile scaling frameworks are given.
Finally, it is described how the scope of the research is adjusted to fit within the timeframe of this
master thesis project.
2.1. Globally Distributed Software Engineering
Because of the globalization of business in the 21st
century, increasingly more companies develop
software in a globally distributed setting [2], [3], [4], [5], [6], [7], and [8]. The area of research on
software development in a globally distributed setting is referred to in various ways. In [3] these are
listed as: Global software development (GSD), global software engineering (GSE), globally distributed
software engineering (GDSE), and globally distributedsoftware development (GDSD). Common among
all these is the globally distributed setting in which SAFe is also being applied, according to case studies
[14], [15], [16], and [17].
At the same time as this trend of globalization, agile software development methodologies have
become the most used approach for software development [13]. The application of these popular
agile methods in globally distributed settings is called Distributed Agile Development, [7], [30] and
[31]. SAFe is such an agile methodology. The Systematic Literature Review, done as part of this
research will thus focus on Distributed Agile Development.
When working in a globally distributed environment, different problems and challenges can occur
according to [3] and [9]. Examples of these problems are: difficulties with coordinating in multiple time
zones [32], [33], and [34], insufficient communication tools [7], [31], [35], and [10], and delay in
communication [6], [36], [4], and [37]. In the Systematic Literature Review, the problems of
Distributed Agile Development are reviewed.
Despite these problems, the use of fully distributed teams can be successful, as presented in [10], [11],
and [12]. The conclusion of these papers is “it is possible to create a distributed/outsourced Scrum
with the same velocity and quality as a collocated team”.
To solve the problems of distributed, different solutions are applied. For example from [9], regular
traveling to meet face to face [7], [31], [38], [39], [40], [32], [35], frequent communication [7], [41],
[42], [33], [35], and [10], and using different communication channels [7], [32], [39], [33], [10], [12],
and, [43].
2.1.1.Definition of distributed
In [44], Thomas Allen presents the Allen Curve which describes the probability of communication
related to distance. From this research originates the critical distance of 50 meters for weekly technical
communication. Because of the advances in telecommunication, Thomas Allen revisited this research
later in [45]. His research shows that his previous conclusion holds. Therefore, the definition of
distributed based on the Allen Curve remains: when the distance between working places is more than
50 meters it is called distributed.

15. 9
Globally distributed is different than distributed as specified by Thomas Allen. With globally
distributed the working places are located at locations which are possibly in different countries or time
zones. Though the definition of distributed as presented by Thomas Allen does not exclude this,
problems such as time zone differences and cultural differences do rarely occur over a distance of 50
meters.
In this research, a globally distributed setting is considered. Therefore, from this point on, when talking
about distributed, globally distributed is meant.
2.2. SAFe
The first version of SAFe was launched in 2011. The version considered in this research is the latest
available version of SAFe at the start of this research (SAFe 4.0), which was launched at the 3th of
January 2016. A full overview of SAFe 4.0 can be found at the SAFe website provided by Scaled Agile,
Inc. www.scaledagileframework.com [1]. The SAFe 4.0 big picture shows almost all elements that are
in SAFe and is shown in Figure 5. Some elements are not made visible in this picture, for example the
PO sync and Scrum of Scrums.
Figure 5: Four level SAFe picture from [1]
2.2.1.Core values, SAFe principles, and lean-agile mindset
SAFe is built upon the core values and SAFe principles, which are the foundations of SAFe. Therefore,
first the core values and SAFe principles are discussed.

16. 10
For organizations that want to adopt SAFe, the core values describe the culture that these
organizations need to develop. These values have to become the heart of the organization. The four
core values are:
 Alignment
 Built-in Quality
 Transparency
 Program Execution
The core value Alignment is to make sure that everyone has the same goal and vision so that they can
align on that. This explains why everyone does what they do. Built-in Quality is to make sure that
quality standards are high and maintained. This is required because “you can’t scale crappy code” [46].
Transparency is to give insight into what is being done. These insights provide data on which decisions
can be based to give direction for a project, rather than steering based on gut feeling. Lastly, Program
Execution is to focus on the program by putting the program before the team. This enables the
program to continuously deliver value.
The SAFe principles are the guidelines for decision making when working with SAFe. The nine SAFe
principles are:
1. Take an economic view
2. Apply systems thinking
3. Assume variability; preserve options
4. Build incrementally with fast, integrated learning cycles
5. Base milestones on objective evaluation of working systems
6. Visualize and limit Work In Progress, reduce batch sizes and manage queue lengths
7. Apply cadence, synchronize with cross-domain planning
8. Unlock the intrinsic motivation of knowledge workers
9. Decentralize decision-making
The lean-agile mindset is needed to support lean and agile development at scale in an entire
organization. The mindset consists of two parts, thinking lean and embracing agility. The core values,
SAFe principles and lean-agile mindset are explained in detail in [1].
2.2.2.The four levels of SAFe
With the introduction of SAFe 4.0 there are two different ways of implementing the SAFe framework,
three level SAFe and four level SAFe. The difference between these versions is an extra level: the value
stream level. The four levels in SAFe are, in order of scale: the team level, the program level, the value
stream level, and the portfolio level.
2.2.2.1. Team level
At the team level are the agile teams that build software. These agile teams use either Scrum or
Kanban with an iteration length of 2 weeks, as long as they keep to the iteration length. Each iteration
contains the following three meetings: iteration planning, iteration retrospective and team demo. On
a daily basis each team has a daily Scrum to synchronize teamwork. These meetings correspond to the
meetings known in Scrum.
2.2.2.2. Program level
At the program level a single Agile Release Train, or ART, is managed. An Agile Release Train contains
all necessary components, including the different agile teams, to deliver end to end value to the
customer. The only way to deliver value in SAFe is using a Value Stream. So, if a single Agile Release

17. 11
Train can deliver the full solution to a customer this train spans the entire Value Stream. In this case
three level SAFe is in place.
The Agile Release Train iterates in a so-called program increment, or PI, of 8 to 12 weeks consisting of
4 to 6 two week iterations from the team level. During the team iterations the Scrum Masters and
Product owners meet twice during a Scrum of Scrums and a PO sync. At the end of each team iteration
the integrated solution is demoed in the system demo done by the system team.
During each program increment all teams and persons which are part of the Agile Release Train come
together during three consecutive days for three events. First, the program increment planning (PI
planning) event during which the next program increment is planned. Second, the Inspect and Adapt
meeting during which previous program increment is reviewed. Third, the Solution Demo, in which
the fully integrated solution is demoed. Again, these meetings correspond to the meetings known in
Scrum.
2.2.2.3. Value stream level
The value stream level is used when a single Agile Release Train cannot deliver the full solution. Then
multiple Agile Release Trains work together to deliver the full solution of a Value Stream. This is four
level SAFe. The value stream level iterates in the same cadence as the program level following the
program increment. A pre- and post-program increment planning event is added to synchronize the
different Agile Release Trains. The Solution Demo is used to demo the fully integrated solution of all
Agile Release Trains. Though the added meetings do not correspond with the meetings in Scrum, the
meetings do follow the pattern of the Scrum meetings: planning (PI planning), retrospective (Inspect
and Adapt), and demo (solution demo).
2.2.2.4. Portfolio level
At the portfolio level, the highest level of SAFe, all Value Streams supported by the organization are
handled. The portfolio level ensures that budgeting on the Value Streams is handled and that the
Value Streams realize the strategic themes of the enterprise.

18. 12
2.2.3.Events
The following events are defined in the SAFe framework. These events are visualized on a timeline in
Figure 6.
 Daily event
 Daily Scrum
 Iteration events
 Iteration Planning
 Scrum of Scrums
 PO Sync
 Team Demo
 System Demo
 Iteration Retrospective
 Program increment events
 Pre-PI Planning
 PI Planning
 Post PI Planning
 Solution Demo
 Inspect and Adapt
Figure 6: SAFe meeting timeline, created based on [1]

19. 13
2.2.4.Flow of a trigger/idea in SAFe
To deliver value SAFe uses Value Streams. A Value Stream starts with a trigger from customers and
results in customers having a solution which adds value to the organization.
Triggers from customers start at the portfolio level and are formulated as epics. If an epic is accepted,
it is put on the portfolio backlog. Epics that are on the portfolio backlog are going to be realized by the
Value Streams. All epics are in line with the strategic themes of the organization, and therefore the
portfolio level is connected to the organization.
In the value stream level, epics that are defined on the portfolio level are split into capabilities. The
size of these capabilities is so that they can be picked up in a single program increment by the Agile
Release Trains that are part of the Value Stream.
In the program level, the epics or capabilities coming from the portfolio level or value stream level are
split into features. A feature is planned and reviewed at the program increment boundaries. These
features are split into stories which can be picked up by a team during an iteration. Teams can pick up
multiple stories during one iteration. A graphic representation of this flow is shown in Figure 7.
Figure 7: Flow of a trigger in SAFe based on [1]

20. 14
2.3. Agile scaling frameworks
There are multiple other frameworks that scale agile. The company Agile Scaling maintains and
updates a matrix [47] in which many of these frameworks are described and compared. This matrix
can be found online via http://www.agilescaling.org/ask-matrix.html. Four frequently used
frameworks are Large-Scale Scrum (LeSS) [26], Disciplined Agile Delivery (DAD)3
[27], Nexus [28], and
Spotify [29]. This section describes these frameworks to give insight into what frameworks, other than
SAFe, are used in practice, and because these frameworks could offer a solution that SAFe does not.
2.3.1.Large-Scale Scrum
The goal of LeSS is “to apply Scrum to very large, multisite, and offshore product development” [48].
There are two versions of the framework, LeSS, which supports up to 10 teams, and LeSS Huge for
more teams. LeSS Huge is applied in settings up to a few thousand people spread over multiple sites.
LeSS attempts to stay similar to Scrum. Thus, LeSS uses one product backlog, one Definition of Done,
one Potentially Releasable Increment, one Product Owner (possibly multiple Area Product Owners),
and one sprint. Teams are not specialized, so teams have to communicate new insights that are gained
to other teams. The structure of LeSS is shown in Figure 8.
Figure 8: LeSS scaling model from [26]
2.3.2.Disciplined Agile Delivery
The goal of DAD is “to fill in the process gaps that Scrum purposely ignores” [27].
DaD uses a “hybrid approach which extends Scrum with proven strategies from Agile Modeling (AM),
Extreme Programming (XP), Unified Process (UP), Kanban, Lean software Development, Outside In
Development (OID) and several other methods.” [27] to scale agile. DAD uses four lifecycles which
organizations can apply according to their needs. As stated in [49] the four lifecycles are:
 An agile/basic version that extends the Scrum Construction lifecycle with proven ideas from
RUP
 An advanced/lean lifecycle
3
Note that, in this research Distributed Agile Development is also discussed, this should not be confused with
Disciplined Agile Delivery which is as abbreviated DAD.

21. 15
 A lean continuous delivery lifecycle
 An exploratory “Lean Startup” lifecycle
The structure of DAD is shown in Figure 9.
Figure 9: DAD scaling model from [50]
2.3.3.Nexus
“Nexus is an exoskeleton that rests on top of multiple Scrum Teams” [28]. This way Nexus aims to solve
the integration of complex software that results in working software. Nexus combines a maximum of
10 teams in a single unit of development, called a Nexus. If multiple of these units of development are
used it is called Scaled Professional Scrum. Nexus extends Scrum with a Nexus Integration Team, the
Nexus Sprint Backlog and additional events, for example the Nexus Daily Scrum. The Nexus framework
is visualized in Figure 10.

22. 16
Figure 10: Nexus scaling model from [51]
2.3.4.Spotify
Spotify scales agile using squads, tribes, chapters, and guilds. How these are related is visualized in
Figure 11. The squad is “the basic unit of development” [29], similar to a Scrum team. Multiple squads
working in related areas are called tribes. The people with similar skills within the tribe are a chapter,
for example testers. Lastly, guilds are all who are interested in a certain topic across tribes, for example
the automated testing guild contains testers and developers from multiple tribes.
Figure 11: Spotify scaling model from [29]

23. 17
2.3.5.Comparing the different scaling frameworks to SAFe
When compared to SAFe, the other agile scaling frameworks, Nexus, LeSS and Spotify provide only
few artefacts, roles, and events in addition to those of regular Scrum. The four lifecycles in DAD contain
more elements than the previous three, however still less than SAFe. Same as SAFe, Spotify does not
prescribe Scrum, but leaves the way of working of a team to the teams themselves. LeSS states that it
can be used in a distributed setting, whereas the other frameworks do not mention this. And although
Spotify is used successfully in a distributed setting, it does not contain specific elements that support
working distributed.
Each of these four frameworks scale the development department, however the scaling ends there.
SAFe scales reasoning with Value Streams which, if needed, include the business and other
stakeholders which are needed to deliver value. Consequently, SAFe has more roles, events, artefacts
and practices than the other frameworks. This enables SAFe to scale on an organizational level rather
than only on the IT level.
In the 10th
Annual State of Agile report by VersionOne from 2015 [13], SAFe is named as the most used
scaling method with 27%. Both LeSS and DAD are also mentioned, but are used significantly less, 4%
and 6% respectively, as shown in Figure 12. Nexus and Spotify are not mentioned in this report.
Figure 12: 10th State of Agile report - Scaling agile from [13]
Note that in this figure, Scrum of Scrums is mentioned as most used scaling method. However, this
method is a single meeting, not a framework. Thus, SAFe is the most used framework.

24. 18
2.4. Research scope
This research is done as part of a master thesis project, for which the duration is fixed. SAFe is
described in 2.2. Based on this description and the illustration provided in Figure 5, 80 roles, events,
artefacts, and best practices can be identified. This list of 80 elements is presented in Appendix F.
Researching all 80 elements within in the timeframe of such a project is not feasible. Thus, the
research must be scoped, this is done by looking at essential SAFe.
Essential SAFe is the bare minimum required to apply SAFe which is still SAFe. This was first presented
on the 9th
of February 2016 in [52], on which this research was initially based. This picture is shown in
Figure 13. An update was presented on the 23rd
of June 2016 in [53]. Although there are some
differences, these differences have no impact on this research.
Figure 13: Essential SAFe from [52]
Essential SAFe consists of the program level and the team level. The value stream level and portfolio
level are not part of essential SAFe. The focus of agile delivery is to deliver value via working software.
To deliver value, SAFe uses Value Streams, which are supported by one or more Agile Release Trains.
As such, we consider the Agile Release Train as the core delivery construct of SAFe. Moreover, when
not using the Agile Release Train, an implementation cannot be considered to be a SAFe
implementation [53]. This research will therefore focus on the Agile Release Train, and in particular
the extent to which distributed impacts the Agile Release Train.
Another part of essential SAFe is the team level. Distribution at the team level entails that within the
team the members are distributed. However, in this case the field of research is Distributed Scrum.
This is a different topic, and cannot be considered as being distributed SAFe. Therefore, the team level
is omitted in this research. Distribution within a team has been researched extensively, for example
in [10], [11], and [12].
Thus, the scope of this research will be set to the Agile Release Train. The elements which are part of
the Agile Release Train have been numbered and are visualized in Figure 14.

25. 19
Figure 14: Agile Release Train elements numbered, modified and reproduced with permission from © 2011-2016 Scaled Agile,
Inc. All rights reserved. Original Big Picture graphic found at scaledagileframework.com.
Below a list of the 34 elements of the Agile Release Train4
Agile Release Train best practices:
 Level transcending best practices
1. Core values
2. Lean-agile mindset
3. SAFe principles
4. Implementing 1-2-3
5. Lean-agile leaders
6. Communities of Practice
7. Value Stream coordination
8. Weighted Shortest Job First
9. Release any time
• Program level best practices
10. Agile Release Train / Value Stream
11. Architectural runway
12. Program Kanban
Agile Release Train events:
• Program level events
13. System Demo
14. Solution Demo
4
A description of all 34 elements can be found in 0.

26. 20
15. Inspect and Adapt
16. PI Planning
Agile Release Train roles:
• Program level roles
17. Release Train Engineer
18. System Architect
19. Product Management
20. Business Owners
 Level transcending roles
21. Customer
 Spanning palette
22. DevOps
23. System team
24. Release Management
25. Shared Services
26. User Experience
Agile Release Train artefacts:
• Spanning palette artefacts
27. Vision
28. Roadmap
29. Metrics
30. Milestones & Releases
 Program level artefacts
31. PI Objectives
32. Feature
33. Enabler
 Level transcending artefacts
34. Epics
A description of all 34 elements can be found in 0.

27. 21
Chapter 3 Research methodology
In this chapter the approach and different research methodologies used during this research are
described. First, the approach taken to answer each research question is presented, as well as the
applicability of that approach to answer the research question. Second, the conditions, strengths,
limitations, and protocols of the methodologies presented in the approach are given.
3.1. Research approach
This section describes the approach taken to answer each research question. As well as the
applicability of this approach to provide an answer for the research question.
3.1.1.Approach for research question 1
There is no literature found on the problems of distributed SAFe, based on a literature search done by
the author5
. Therefore, to answer the first research question “What problems can be expected when
SAFe is applied in distributed settings?”, a Systematic Literature Review was done on the topics of
Distributed Agile Development and Distributed Scrum. To relate these problems to SAFe and discover
distributed SAFe problems, a multiple informant methodology with consensual approach was used to
filter the problems.
The Systematic Literature Review is applicable to answer this question as it is a method to evaluate
and interpret all available research relevant to the topic of Distributed Agile Development. This
corresponds with the method as stated in [54]: “A systematic literature review (often referred to as a
systematic review) is a means of identifying, evaluating and interpreting all available research relevant
to a particular research question, or topic area, or phenomenon of interest.”. Besides this, the result
of this Systematic Literature Review provides a background for the next steps of this research. This
corresponds to one of the reasons to do a Systematic Literature Review in [54].
Not all Distributed Agile Development problems are equally relevant in distributed SAFe. To discover
the relevant distributed SAFe problems, the problems of the Systematic Literature Review have been
filtered using a multiple informant methodology. The use of multiple informants enables the
researcher to objectively reach a decision, using relatively few resources, according to [55] and [56].
Thus, the use of multiple informants is applicable to discover the distributed SAFe problems, providing
an answer to the first research question: “What problems can be expected when SAFe is applied in
distributed settings?”.
3.1.2.Approach for research question 2
To answer the second research question: “What SAFe elements can be expected to fail when applied
in distributed settings?” triangulation was used. First, the author identified failing elements based on
the distributed SAFe problems found in previous research question. Such identification, based only on
the insights of one person, cannot be considered sufficiently substantiated. Therefore, this has been
substantiated using triangulation. Second, a focus group with both SAFe, and distributed experts was
done to find an answer based on theory: the expert focus group. And third, a focus group with
5
At the 15th
of February 2016 Google Scholar was searched using the following queries: ““Scaled Agile
Framework” AND distributed AND problems”, and ““Scaled Agile Framework” AND SAFe”. The first
query yielded 96 hits, the second query 122.

28. 22
practitioners was done to find an answer based on practice: the practitioner focus group. The overlap
of these identification methods provides an answer to the research question.
To provide an answer to this question based on theory, expertise on both SAFe and distributed is
required. A focus group “involves engaging a small number of people in an informal group discussion
(or discussions), ‘focused’ around a particular topic or set of issues.” [57]. Within these discussions, the
interactions among the participants can yield important data [58], as in these interactions the
experience of the different participants is combined. By combining the experience of both distributed
and SAFe experts in a focus group the use of a focus group is applicable for answering this research
question.
The second focus group aims to answer the question based on practical experience. Practical
experience with SAFe differs for each person, as each organization is unique. To answer the question
properly, these differences must be taken into account. As mentioned previously, a focus group
combines the experience of the participants. With experts from different companies participating in
the focus group the use of a focus group is applicable for answering this research question based on
practice.
Additionally, a survey was done in an attempt to provide an answer based on practice to the question:
“What SAFe elements can be expected to fail when applied in distributed settings?”. However, due to
low response the results of this survey have not been used in this research. The survey and the results
of the survey can be found in Appendix T and Appendix U, so that these can be used for future
research. To emphasize, the survey is not used in this research.
3.1.3.Approach for research question 3
To answer the third research question: “What would SAFe look like, when customized for distributed
settings?” insights from the author are combined with focus group results. The discussions within the
focus group can provide new insights on possible solutions. This does not provide a sufficiently
substantiated answer, which is due to the time constraints of this research not possible. However, the
answer does provide an indication where to focus future research. To this end, providing an indication
for future research, the use of a focus group is applicable.
3.2. Systematic Literature Review
A Systematic Literature Review is done to provide a background on which to base the following steps
of the research. In previous research [9], the author has explored the problems of Distributed Scrum
using a Systematic Literature Review. To gain an insight as broad as possible, this research explores
the topics of Distributed Agile Development and Distributed Scrum by reviewing Systematic Literature
Reviews. Doing yet another Systematic Literature Review is not expected to give new insights, as many
reviews have already been done on Distributed Agile Development and Distributed Scrum. For this
reason, a review of existing Systematic Literature Reviews is done, using the approach of a Systematic
Literature Review.
3.2.1.Conditions for a Systematic Literature Review
The main condition for using a Systematic Literature Review is that there should be sufficient literature
on the topic of the review to avoid coincidental findings. Another condition for using a Systematic
Literature Review is that it is undertaken in accordance to a predefined search strategy [54].
3.2.2.Strengths of a Systematic Literature Review
According to [54] and [59], the main strength of a Systematic Literature Review is that the predefined
search strategy makes it less likely that the results are biased. Additionally, using a Systematic

29. 23
Literature Review enables the researcher to analyze a wide range of variables, according to [54]. This
wide range of variables provides the researcher with a broad view of the topic as well as insight into
the current state of the field of research.
3.2.3.Limitations of a Systematic Literature Review
The major disadvantage mentioned in [54] and [59] is that Systematic Literature Reviews require a lot
more effort than traditional literature reviews. This is a limitation if time is limited. Besides this, a
limitation is that the results of the review might be biased. Another limitation is that the frame of
reference is different for each person. Result of this is that the execution of the protocol might be
slightly different when applied by another researcher. These small differences can lead to different
papers being accepted and different data being extracted. Finally, the use of a Systematic Literature
Review is limited to previously published research.
Besides these limitations, if any of the conditions is not fulfilled, this becomes a limitation as well and
should be handled as such.
3.2.4.Systematic Literature Review protocol
A review protocol is created based on the procedures and guidelines, presented by Kitchenham and
Charters in [59] and [54]. A short summary of the steps in the protocol is presented below and these
steps are visualized in Figure 15. The review protocol can be found in Appendix A.
In the first step, the search results from the search in Google Scholar6
, on Systematic Literature
Reviews discussing the topics Distributed Agile Development and Distributed Scrum, have been
filtered. The search results have been accepted or rejected based on the criteria described in the
protocol. This filtering resulted in a list of accepted papers.
In the second step, data has been extracted from the accepted papers. The extracted data consists of
the standard information, as described in [54], as well as additional information. The standard
information extracted is: study title, study author(s), study year, and publication details. This is
extended with additional information, namely, the problems or challenges presented in the study.
In the third step, the extracted data of the different studies was combined. Similar problems or
challenges have been grouped together, and presented as in Table 1.
Table 1: Problem groups example
Distributed Agile Development problems Times mentioned
Problem A
Description of problem A
12
Problem B
Description of problem B
11
6
Google Scholar has been used for this search because it has indexed many different databases, including those
of different universities, providing a broad view of the available literature

30. 24
Figure 15: Visualization Systematic Literature Review protocol
3.3. Multiple informant methodology
To discover the relevant distributed SAFe problems, the problems of the Systematic Literature Review
were filtered using a multiple informant methodology with a consensual approach.
3.3.1.Conditions for using multiple informants
The first condition for using multiple informants is regarding the selection of the informants. According
to [55] there are 4 selection criteria. First, the informants should be likely to be able to answer all
questions under investigation. Second, members of the organization should nominate the informants
as the most knowledgeable in within the organization. Third, the selected informants themselves
should think they are competent to answer the questions. Fourth, the duration that the informants
have worked with the topic should be long enough that the answers of the informants are plausible.
The second condition is that the right number of informants should be consulted. According to [56],
using 3 informants with different backgrounds is sufficient to eliminate less relevant problems. This is
supported in [55], which gives multiple studies that indicate that 2 or 3 informants are sufficient.
Though in [55], it is stated that this only holds if the selection criteria are applicable to every informant.
3.3.2.Strengths of using multiple informants
The strength of using multiple informants enables a researcher to make a decision using relatively few
resources. Using multiple informants instead of a key informant reduces the impact of bias and
random errors, according to [55]. Also, this is less costly than using a focus group or other group
decision schemes, according to [56]. Another advantage of using a consensual approach is that the
researcher does not have to perform the aggregation himself.

31. 25
3.3.3.Limitations of using multiple informants
Using multiple informants, a limitation is that one could wonder whether the informants are able to
individually judge the topic sufficiently. Another limitation could be that the informants as a group do
not provide a broad enough view to make a sufficiently substantiated decision.
As a consensual approach has been taken, one could ask whether the group consensus provides the
best answer. According to [55], some studies indicate that the group consensus performs better than
the average of the individual informants, but is outperformed by the best informant [60], [61]. Also, it
is concluded that the unweighted mean of the individual informants tends to outperform group
consensus [60].
Besides these limitations, if any of the conditions is not fulfilled, this becomes a limitation as well and
should be handled as such.
3.3.4. Multiple informant protocol
The consulted informants made two mappings in which Distributed Agile Development problems were
mapped on the core values of SAFe. On both mappings consensus was reached. For this a protocol
was used and is presented in Appendix B. A short summary of the steps in the protocol is presented
below and is visualized in Figure 16.
In the first step, the informants mapped the Distributed Agile Development problems on the core
values of SAFe based on consideration. Based on this mapping, problems that are considered by SAFe,
were filtered out. This results in problems of Distributed Agile Development that are not considered
in SAFe, and are thus SAFe threats.
In the second step, the informants mapped the distributed SAFe threats on the core values based on
impact. Based on this mapping, the threats which have a low impact were filtered out. This results in
problems which are not considered by SAFe and have a high impact on SAFe, thus distributed SAFe
problems.
Figure 16: Visualization of multiple informant protocol

32. 26
3.4. Focus group
To answer the question: “What SAFe elements can be expected to fail when applied in distributed
settings?”, two focus groups were used. Using a focus group provides insights of experts with different
backgrounds. The first focus group consisted of experts of different fields. This group answered the
question based on theory. The second focus group consisted of practitioners. This second group
answered the question based on practice.
3.4.1.Conditions for a focus group
The main condition for a focus group is that it should be composed of 6 to 12 people with different
backgrounds and preferably different genders as stated in [62]. Although this book focuses on focus
group interviews, the rationale is the same.
The second condition is that if the participants are asked to individually judge a topic, all participants
should be able to do this based on their expertise.
The last condition is that participants should not have been previously involved in the research. If the
participants have been previously involved in the research, the results of the focus group could be
compromised. Participants could then have information on the topic that could influence the outcome
of the focus group. An example of this could be that a participant might strive towards a certain result
to understate the statements from their previous participation.
3.4.2.Strengths of a focus group
As stated in 3.1.2, the strength of a focus group is that the interactions in a focus group combine the
experience of the participants. This is reinforced in [63]: “Focus group interactions reveal not only
shared ways of talking, but also shared experiences, and shared ways of making sense of these
experiences”.
3.4.3.Limitations of a focus group
The main limitation of using a focus group is that the data analysis is complex and can lead to
unwarranted conclusions. According to [64]: “just as using counts by themselves can be problematic,
mainly reporting and describing the themes that emerge from an analysis of focus groups also can be
misleading because, to the extent that any themes that might stem from dissenters are ignored, it can
lead to unwarranted analytical generalizations”. It must be explained how the data analysis was done
to avoid unwarranted conclusions.
Besides these limitations, if any of the conditions is not fulfilled, this becomes a limitation as well and
should be handled as such.
3.4.4. Expert focus group protocol
For each of the focus groups a protocol has been made to structure the execution of the focus group.
A short summary of the steps in the expert focus group protocol is presented below, these steps are
visualized in Figure 17 and Figure 18. The expert focus group protocol can be found in Appendix C.
In the first part of the expert focus group protocol the participants identify which Release Train
Elements are challenged by distributed, which is done in two rounds. First the group identifies the
specifically challenged Agile Release Train elements. For this the group filters the Agile Release Train
elements, first in groups, then plenary. For the group filtering, the participants are divided in two
groups, based on expertise. Each group consists of at least a distributed expert, a SAFe expert and a
practitioner. Second, the group ranks the elements using dot voting. From this, the high risk Agile
Release Train elements are identified. This first part is visualized in Figure 17.

33. 27
In the second part of the expert focus group protocol the participants identify the consequences of
the high risk Agile Release Train elements. First, consequences are discovered during a plenary
discussion. Next, these consequences are ranked using dot voting. This second part is visualized in
Figure 18.
Figure 17: Expert focus group - part 1: identifying failing elements
Figure 18: Expert focus group - part 2: identifying consequences

34. 28
3.4.5.Practitioner focus group protocol
A short summary of the steps in the practitioner focus group protocol is presented below, these steps
are visualized in Figure 19 and Figure 20. The practitioner focus group protocol can be found in
Appendix D.
In the first part of the practitioner focus group protocol the participants identify which Release Train
Elements are challenged by distributed. Same as in the first focus group, this is done in two rounds.
First the group identifies the specifically challenged Agile Release Train elements. For this the group
filters the Agile Release Train elements, first in groups, then plenary. For the group filtering, the
participants were divided in two groups, based on earlier participation in the research. Those who
participated in the survey were in one group, those who did not previously participate on the other
group. Second, the individuals ranked the elements using dot voting. From this, the high risk Agile
Release Train elements were identified. This first part is visualized in Figure 19.
In the second part of the practitioner focus group protocol the participants identify solutions to
prevent the elements from failing. First, solutions are discovered during a discussion in two groups.
The participants are divided over the groups randomly. Next, these solutions are ranked individually
using dot voting. This second part is visualized in Figure 20.
Figure 19: Practitioner focus group - part 1: identifying failing elements

35. 29
Figure 20: Practitioner focus group - part 2: identifying solutions

36. 30
Chapter 4 Distributed SAFe problems
In this chapter an answer is presented to the first research question: “What problems can be expected
when SAFe is applied in distributed settings?”. The research was split in two steps. First, a Systematic
Literature Review on the problems of Distributed Agile Development and Distributed Scrum was done.
Second, a multiple informant methodology was used to filter these problems and uncover the
distributed SAFe problems.
4.1. Problems of Distributed Agile Development: Systematic Literature Review
To provide a background on which to base the following steps of the research the literature is searched
for problems of Distributed Agile Development and Distributed Scrum. For this Systematic Literature
Review a protocol is used. The protocol is described in 3.2.4, page 23, and is visualized in Figure 21.
Figure 21: Visualization Systematic Literature Review protocol
4.1.1.Conditions for the Systematic Literature Review
As stated in 3.2.1 page 22, the first condition for conducting a Systematic Literature is that there
should be sufficient literature on the topic of the review. To check if this condition was fulfilled for the
topic of distributed SAFe, Google Scholar7
was searched on the 15th
of February 2016 using the queries:
7
Google Scholar has been used for this search because it has indexed many different databases, including
those of different universities, providing a broad view of the available literature

37. 31
““Scaled Agile Framework” AND distributed AND problems”, and ““Scaled Agile Framework” AND
SAFe”. The first query yielded 96 hits, the second query 122.
From this search two studies discussing problems were found, [65] and [66]. The studies discuss the
challenges of transitioning from a traditional organization to an organization where agile is scaled.
However, the studies do not cover the distributed aspect required for this research. Therefore, a
different approach is taken to find problems on distributed SAFe. For this approach, problems of
Distributed Agile Development and Distributed Scrum are researched in this Systematic Literature
Research. On these topics there is enough literature to fulfill the condition.
The second condition is that a predefined search strategy is used for the review. The protocol, as
summarized in 3.2.4, page 23, fulfills the condition that a predefined search strategy is used.
4.1.2.SAFe transformation problems
In the literature search on the 15th
of February 2016 two papers were identified that discuss the
challenges of transitioning from a traditional organization to an organization where agile is scaled, [65]
and [66]. In [65] distribution is not mentioned as a common challenge for scaling agile. Although for
example, inter-team communication, team coordination, and dependency management are
mentioned. These problems overlap with problems regarding coordination and communication that
can occur when being distributed. In [66] distribution is mentioned as a challenge. Four levels are used
towards agility at scale, agile delivery, team agility, product agility and enterprise agility. For these
levels there are different challenges mentioned, distribution being one of them, which is mentioned
both at team and at enterprise level.
4.1.3.Problems of Distributed Scrum
SAFe applies the Plan-Do-Check-Adjust cycle at scale, which is the basis of Scrum and Agile, as shown
in Figure 22. Both Scrum and SAFe have their meetings structured based on this cycle, resulting in
similar meetings and the use of backlogs. Due to the similarities between Scrum and SAFe, problems
that occur in Distributed Scrum are expected to also be present in distributed SAFe. Additionally, SAFe
scales agile. Therefore, problems of Distributed Agile Development are researched as well.
Figure 22: Plan-Do-Check-Adjust cycle in SAFe from [1]

38. 32
A Systematic Literature Review [9], done prior to this research, discovered problem groups in
Distributed Scrum. These problems are listed ordered by their class, as classified in [9], in Table 2.
Table 2: Problem groups of Distributed Scrum ordered by class from [9]
Problem group Class
No syncing between sites Coordination
Planning a meeting with everyone present is difficult Coordination
Integration difficulties Coordination
Lack of focus Coordination
Coordinating in multiple time zones is difficult Coordination
Multiple Product Owners not in sync Coordination
Misunderstanding Culture
Difference in reporting impediments Culture
Different holidays Culture
Silence / passivism Culture
Different work practices Culture
Incorrect execution of Scrum Scrum
Scrum of Scrums not effectively used Scrum
Features not being deployment ready at end of sprint Scrum
Managing customers new to agile Scrum
Not communicating all information to team Communication
Product Owner not present Communication
Informal contact is lost Communication
Meetings at the office outside office hours Time zone
No transparency between sites Time zone
Time differences Time zone
Hardware and tools not sufficient Technical
It should be noted that these problems can also happen when working co-located, for example
“incorrect execution of Scrum” can also happen with co-located teams. Additionally, there is overlap
between these problems, for example “meetings at the office outside office hours” are due to “time
differences” which is a separate problem. Also, the classification itself, as presented by [9] could be
argued, for example the problem “coordinating in multiple time zones is difficult” is classified as
coordination, but could also be classified as time zone.
Although these observations are correct, choices were made in [9] regarding when problems are
included, the way the problems are grouped, and the way they are classified. This research does not
reevaluate these choices because these choices are substantiated in [9]. The problems as presented
are used in this research.
4.1.4.Problems of Distributed Agile Development
SAFe is designed to scale agile development, therefore the problems of Distributed Agile Development
can be expected to occur in SAFe as well. In their thesis, Dullemond and van Gameren listed the
challenges of Distributed Agile Development [3]. These challenges are depicted and based on their
most applicable class, as classified in [3], in Table 3.

39. 33
Table 3: Challenges of Distributed Agile Development grouped by most applicable class from [3]
Challenge Class
Lack of informal communication Geographic dispersion
Increased effort to initiate contact Geographic dispersion
Reduced hours of collaboration Control and coordination breakdown
Lack of shared understanding Control and coordination breakdown
Increased dependency on technology Control and coordination breakdown
Increased complexity of the technical infrastructure Control and coordination breakdown
Communication delay Loss of communication richness
Loss of cohesion Loss of teamness
Reduced trust Loss of teamness
Perceived threat from low-cost alternatives Loss of teamness
Increased team size Loss of teamness
Differences in language Cultural differences
Differences in ethical values Cultural differences
Differences in organizational vision Cultural differences
Differences in managing individualism and collectivism Cultural differences
Differences in terms of agreement Cultural differences
Differences in time perception Cultural differences
Differences in quality assessment Cultural differences
Differences in design Cultural differences
Same as for the previous study, the choices made in [3] regarding the challenges and their classes are
not reevaluated in this research. The challenges, as presented in [3] are used.
4.1.5.Extending the search
Besides these two Systematic Literature Reviews, an additional Systematic Literature Review was
done. The protocol that was used is described in 3.2.4, page 23. The results of the search are
presented in Table 4.
Table 4: Google Scholar search results
Query # hits # accepted Accepted papers
“Distributed Agile Development” AND (problems
OR challenges) AND “systematic literature review”
124 9 [67], [68], [69], [70],
[71], [72], [73], [74],
[75]
"Distributed Scrum" AND (problems OR challenges)
AND "systematic literature review"
140 10 [68], [69], [70], [67],
[72], [73], [71], [76],
[74], [77]
The rejected papers including the reason of rejection can be found in Appendix H. A list of all 184
problems and challenges identified by the accepted papers can be found in Appendix I.
In addition to the studies found, three studies were discovered that have investigated Systematic
Literature Reviews as well, [78], [79] and [80]. The studies that have been found in these studies have
also been tested on the acceptance criteria. No new papers have been found in this search. The results
of these tests can be found in Appendix J.

40. 34
4.1.6.Combining the results
The results as discussed in the previous sections were combined by grouping the similar problems or
challenges. Problems that occurred more than once have been grouped together. In Table 5 the 29
grouped problems are listed and summarized. How these problems are grouped can be found in
Appendix K. There are 25 problems that could not be grouped, these can be found in Appendix L.
It should be noted that some of these problems can also occur when working co-located. For example,
language barriers can also occur when working co-located with team members from different
nationalities. Thus, the problems that are presented are not exclusively occurring when working
distributed. However, the studies that have been reviewed mention these problems as problems that
can occur when working distributed. Moreover, that a problem can occur when working co-located,
does not mean that a problem cannot occur when working distributed. For this reason, these problems
can be expected to occur when SAFe is applied in distributed settings.
Table 5: Problem groups
# Distributed Agile Development problems Times mentioned
1 Problems due to inefficient communication tools
The most mentioned problem group is problems due to inefficient
communication tools. Both the hardware and tools used for communication
are part of the communication infrastructure. These problems are quite severe
as there is a high dependency on this infrastructure for communication.
12
2 Problems due to unavailability of people
Within this group the main person that is mentioned that is not available is the
Product Owner. This results in difficulties with communication between the
Product Owner and the developers. It is also mentioned that the Scrum Master
or developers are not available, resulting in communication difficulties as well.
11
3 Problems due to lack of synchronous communication
Lack of synchronous communication makes collaboration difficult because
there are less hours in which to collaborate. This lack of synchronous
communication comes from the asynchronous nature of distributed working
which results in few overlapping work hours.
10
4 Problems due to different execution of work practices
The different work practices can come from personal preferences or cultural
differences. These different work practices result in differences in design,
quality assessment, and terms of agreement. This makes it difficult to
coordinate and collaborate.
9
5 Problems due to language barriers
Many studies report problems due to language barriers. If the language used
for communication is not someone’s native language it can be hard for this
person to follow a conversation or express themselves. Also, speakers from
different countries might have different dialects that can be hard to follow for
others.
8
6 Problems due to lacking technical infrastructure
Lacking technical infrastructure is caused by increased complexity of the
infrastructure in big projects. This tends to lead to a weak infrastructure that
makes it difficult for the developers to work with.
8
7 Problems due to loss of cohesion
The loss of cohesion can make that teams no longer feel that they are one
team. This can lead to poor team dynamics and less productivity.
8

41. 35
# Distributed Agile Development problems Times mentioned
8 Problems due to misinterpretation
Misinterpretation can come from misunderstanding during communication
because of small communication bandwidth. Or because information is not
accessible or even hidden. This can result in reduced cooperation and loss of
information.
8
9 Problems due to lack of agile training
If customers or developers do not have a similar understanding of agile this
creates a gap between the skill levels of the involved parties. This gap makes it
difficult for these parties to work together.
7
10 Problems due to reduced trust
Many studies mention that there is reduced trust between team members
when working in a distributed setting. This reduced trust can lead to lack of
productivity and loss of teamness.
7
11 Problems due to time zone differences
Time zone differences can lead to having meetings outside office hours and
reduced availability for synchronous communication.
7
12 Problems due to people differences
People differences can be many things, difference in time perception, notion
of authority, individualism, or ethical values. These differences can make it
difficult to work together.
6
13 Problems due to lack of traditional management
Managing an agile project can be difficult because of the lack of traditional
management processes to steer the project. This lack of processes can cause
problems if teams do not function autonomously.
6
14 Problems due to difficulties with coordination
Working together with multiple sites in multiple time zones is difficult, this
increases coordination costs and can lead to unnecessary delays and conflicting
work.
6
15 Problems due to shared ownership and responsibility
In agile, teams get shared ownerships over their own projects, also giving them
responsibility for these projects. When the teams work distributed, this can
cause problems as the teams do not feel this responsibility which could result
in avoidance of accountability.
6
16 Problems due to incorrect execution of Scrum*
Not executing Scrum properly results in many problems. Features that are not
ready at the end of the sprint and teams that get no feedback on their work
because they do not hold retrospectives or reviews.
6
17 Problems due to cultural differences - organizational and national
Differences in culture can be differences of national culture between sites but
also differences in organizational culture between sites. These differences can
make it harder for sites to cooperate.
5
18 Problems due to the loss of informal contact
When working distributed the chitchat at the coffee corner is lost. This loss of
informal contact can create a lack of awareness of what is going on with the
team members, leading to less communication and collaboration.
5
19 Problems due to lack of collective vision
If there is no collective vision, teams miss the big picture. This results in less
focus and commitment of the teams.
4

42. 36
# Distributed Agile Development problems Times mentioned
20 Problems due to lack of requirement documents
Scrum does not provide formal documentation in the form of requirement
documents, this can cause problems if the customer wants to work with fixed
requirements. Or teams can have issues with communication if important
decisions are not documented leading to unclear requirements.
4
21 Problems due to lack of visibility
It is difficult to evaluate the current state of a project. This lack of visibility
makes it difficult create trust between sites.
4
22 Problems due to difficulties in knowledge sharing
Knowledge sharing when working distributed is difficult as knowledge is spread
over the different sites. If sharing of this knowledge is not done between sites
this can lead to a lack of domain knowledge in some sites.
4
23 Problems due to increased communication effort
Initiating contact in a distributed environment takes an increased effort as this
cannot be initiated face to face, some tool must be used. This creates
communication overhead and increases communication costs.
4
24 Problems due to increased team size
When the team size is increased, it becomes more difficult to work together as
team.
3
25 Problems due to different holidays
Different countries, cultures and religions have different holidays. When these
holidays are not overlapping, it is difficult to synchronize work between the
distributed sites.
3
26 Problems due to difficulties with agile decision making
Agile decision making is different from traditional decision making because
teams get more decision-making power. This results in management having to
let go and trust the teams to make the right decision.
3
27 Problems due to increased number of teams
When the number of teams increases, this creates difficulties for agile
practices. Agile practices must be scaled for this increased number of teams.
3
28 Problems due to silence of participants
During meetings, some participants can remain silent and passive due to
linguistic or cultural differences.
2
29 Problems due to increased number of sites
Working distributed means working with multiple sites. This can cause all kinds
of problems with communication and coordination. Many of these problems
are listed in this table.
2
4.1.6.1. *Note on: problems due to incorrect execution of Scrum
From the literature search the problem “problems due to incorrect execution of Scrum” is mentioned
multiple times. This problem, however, is Scrum specific and mentioned in the literature as such.
Looking at this problem from a framework perspective the problem is incorrect application of the
methods presented by the framework (Scrum) in a distributed setting. The problem described is thus
“problems due to incorrect execution of the methods presented by the framework”, in short
“problems due to incorrect execution of the framework”
This incorrect execution of methods in the framework can also be applied to distributed SAFe. For
readability in this thesis, the problem is described as “problems due to incorrect execution of SAFe”.

43. 37
Although this step is logical, it should be clearly noted that this generalization is not substantiated in
any way.
4.2. Distributed SAFe problems: Multiple informant methodology
In this section, the 27 problems of Distributed Agile Development mentioned more than twice (see
Table 5) are filtered in two stages to discover the distributed SAFe problems. This filtering is done
using a multiple informant methodology with a consensual approach. The protocol used for this is
summarized in 0, and visualized in Figure 23, below.
Figure 23: Visualization of multiple informant protocol
It should be noted that not all 29 Distributed Agile Development problems from the Systematic
Literature Review are used. Problems mentioned only twice are not considered, problems mentioned
thrice are. On one hand, dismissing problems that are not mentioned often early in the process could
result in an important problem being filtered out. On the other hand, taking all problems further in
the research could result in solving a problem which does not occur often. If only two out of 13
Systematic Literature Reviews mention a problem, it can be considered a coincidence as the others
should have found the problem as well. Three could also be considered a coincidence. However, this
chance is smaller. Moreover, dismissing those problems would result in four problems being dismissed
early in the process. In this stage of the research this is not preferred.
4.2.1.Conditions for using multiple informants
To check whether the first condition regarding the selection of the informants is fulfilled, the four
selection criteria presented in 3.3.1 should be met. All four criteria have been met. The first criterion
is met as all informants are Safe Program Consultants (SPC’s) and therefore have sufficient knowledge
of SAFe to create the mappings. The second criterion is met as the SPC’s are regarded as the most
knowledgeable on SAFe. The third criterion is met. Three informants did think they were competent.
During one of the sessions, one informant did not think he was competent to make the mapping. In

44. 38
consultation with the informant it was decided to exclude this informant from the results. The fourth
criterion was met as the informants have all implemented SAFe in different companies.
The second condition is that the right number of informants should be consulted. For the multiple
informants methodology 3 informants that have implemented SAFe in different companies were
consulted. As stated in 3.3.1, 3 informants with different backgrounds are sufficient to eliminate less
relevant problems.
4.2.2.Problem - Core Value mapping: consideration
The problems were first mapped on the core values of SAFe based on consideration. A high value (++)
means that the problem is strongly considered in that core value. In this mapping the core values are
seen as a means to an end. All elements in the SAFe framework support these core values. The
mapping thus provides insight into which problems might not be covered in SAFe. In Table 7 this
mapping is presented.
Table 6: Legend for Table 7: degree of consideration
Explanation
-- Problem is not considered at all
- Problem is not really considered
-/+ Problem partly considered
+ Problem is considered
++ Problem is strongly considered
Table 7: Problems mapped to core values on consideration
Core value
Problem
Alignment Built-in
Quality
Transparency Program
Execution
Result*
Technical
Inefficient communication
tools
- - - - -
Lacking technical
infrastructure
-/+ ++ -/+ + ++
Coordination
Unavailability of people + -- - -/+ +
Different execution of work
practices
-/+ ++ + ++ ++
Time zone differences -- -- -- -- --
Difficulties with
coordination
+ -/+ -/+ + +
Increased number of teams ++ + ++ ++ ++
Communication
Lack of synchronous
communication
- -- - - -
Loss of informal contact - - - - -
Misinterpretation ++ -- ++ -/+ ++
Increased communication
effort
-- -- -- -- --

45. 39
Core value
Problem
Alignment Built-in
Quality
Transparency Program
Execution
Result*
Cultural
Language barriers -- -- -- -- --
Different holidays -- -- -- -- --
Cultural differences -
organizational and national
+ -- + -/+ +
Agile expertise
Lack of agile training - -- - -/+ -/+
Lack of traditional
management
-/+ -/+ -/+ + +
Difficulties with agile
decision making
+ -- + -/+ +
Incorrect execution of SAFe - - - - -
Lack of requirement
documents
+ + -/+ + +
Lack of visibility + - ++ - ++
Teamness
Reduced trust ++ + ++ + ++
Loss of cohesion + + + + +
People differences - - - - -
Shared ownership and
responsibility
++ + + + ++
Increased team size + -- -- + +
Difficulties in knowledge
sharing
+ -- + + +
Lack of collective vision ++ -- + -/+ ++
*Note on result: The value in the result column is the maximum value of the four other columns.
This table has been discussed and verified with three SAFe program consultants from Prowareness
using the protocol previously discussed in 0, page 25. It should be noted that in the execution of the
protocol some of the informants gave different values for the mapping. However, none of these
differences effected the result column. And, when reaching a consensus on the values, this also did
not affect the result.
From this table the following problems can be extracted which are not really considered or not
considered in SAFe and therefore, are threats to SAFe.
1. Incorrect execution of SAFe
2. Language barriers
3. Different holidays
4. People differences
5. Time zone differences
6. Increased communication effort
7. Loss of informal contact
8. Lack of synchronous communication
9. Inefficient communication tools

46. 40
4.2.3.Problem - Core Value mapping: impact
Another mapping between the problems and core values is done in which the core values are
viewed as the things that SAFe tries to achieve, the goals of SAFe. In this mapping the impact of a
problem on the core value as a goal is mapped. This mapping is presented in Table 9.
Table 8: Legend for Table 9: degree of impact
Explanation
-- Problem has no impact
- Problem has little impact
-/+ Problem has moderate impact
+ Problem has severe impact
++ Problem has very severe impact
Table 9: Problems mapped to core values on impact
Core value
Problem
Alignment Built-in
Quality
Transparency Program
Execution
Result*
Technical
Inefficient communication
tools
++ + ++ ++ ++
Coordination
Time zone differences ++ -- + + ++
Communication
Lack of synchronous
communication
+ + - - +
Loss of informal contact + - - - +
Increased communication
effort
+ + + ++ ++
Cultural
Language barriers ++ + + ++ ++
Different holidays -- -- - - -
Agile expertise
Incorrect execution of SAFe ++ + + ++ ++
Teamness
People differences - - -/+ -/+ -/+
*Note on result: The value in the result column is the maximum value of the four other columns.
This table has been discussed and verified with three SAFe program consultants from Prowareness
using the protocol previously discussed in 0, page 25. It should be noted that, same as for the previous
mapping, in the execution of the protocol, in some cases the informants gave different values for the
mapping. The informants deviated more from each other with filling in this mapping, and this resulted
in some differences, also for the result column. However, these small changes did not have effect on
the next steps of the process.

47. 41
Selecting from the 9 problems that are not considered in the core values, the problems that have a
severe impact on the core values are the following.
1. Incorrect execution of SAFe
2. Language barriers
3. Time zone differences
4. Increased communication effort
5. Inefficient communication tools
Note that, as stated previously, these problems do not exclusively occur when working distributed.
However, that a problem can occur when working co-located, does not mean that a problem cannot
occur when working distributed. Thus, these problems are expected to occur when applying SAFe in
distributed settings.

48. 42
Chapter 5 Identification of failing SAFe elements
In this chapter an answer is presented to the second research question: “What SAFe elements can be
expected to fail when applied in distributed settings?”. To do this, failing elements were identified
based on the distributed SAFe problems. Because such identification, based only on the insights of
one person, cannot be considered sufficiently substantiated this will be substantiated using
triangulation. First, the previously mentioned identification was done based on insights gained from
answering the first research question. Second, an identification was made based on theory using a
focus group in which distributed and SAFe experts participated. Third, an identification was made
based on practice using a focus group in which practitioners participated. Finally, the insights of the
three identification methods are combined to reach a conclusion on what SAFe elements can be
expected to fail in distributed settings.
5.1. Identification based on theory: Literature
The elements in SAFe that are part of the Agile Release Train (presented in 0) can be categorized in
two groups: elements which are expected to fail when applied in distributed settings and elements
which are not expected to fail. These elements are selected based on the problems found in previous
research. In Figure 24 coloring has been applied based on this categorization. The elements that are
expected to fail are colored red, the elements that are not expected to fail are colored green. This
does not mean that the elements that are green will never fail. However, their failure is not deemed
likely, given the problems identified in Chapter 4.
Figure 24: Agile Release Train elements failing - result of identification based on literature, created based on [1]

49. 43
5.1.1.Elements expected to fail
The argumentations of why the elements are classified as expected to fail are described below. Only
the elements that are expected to fail are presented.
5.1.1.1. Agile Release Train
If the Agile Release Train is spread over multiple time zones it can be difficult to adhere to all
prescribed practices, which could result in an incorrect execution of SAFe. Incorrect execution of SAFe
is one of the problems according to the literature. An example of this, mentioned in the literature for
distributed Scrum, is that a retrospective is not done because it is difficult when working in multiple
time zones. Applying this to SAFe would mean, not doing Inspect and Adapt because it is difficult. This
will probably have a high impact on the Agile Release Train, possibly even running it of its tracks.
Consequence of the increased communication effort is that there is less communication between the
teams of the Agile Release Train during the program increment. This lack of communication can cause
the teams to go in different directions. These are examples of why the Agile Release Train might fail.
5.1.1.2. System Demos
System demos are given every iteration. To give the system demo, the system should be integrated, if
this is not possible partial integration can be used, but this is not preferable. The system team does
this integration with assistance from the other teams if needed. However, if the teams are spread over
multiple time zones, or locations, teams are not always available to assist the system team with this
integration. Therefore, the integration could fail. Additionally, key stakeholders should be present at
the system demo. If these are spread over multiple locations, it can be more difficult to get all
stakeholders to be present and actively participating. Because of these problems the system demo
might fail.
5.1.1.3. PI planning, Inspect & Adapt, and Solution Demo
The program increment planning is essential for aligning the Agile Release Train. The inspect & adapt
is essential to continuously improve the program, and the solution demo is essential to gain feedback
on the created solution. Not doing any of these three events are likely to derail the Agile Release Train
in no time due to lack of alignment, improvement, and feedback. When doing the events distributed,
inefficient communication tools and language barriers can cause the communication during the events
to be problematic. If the communication becomes too big of a problem, it could also derail the train.
Even when doing the events co-located, language barriers can still make communication problematic.
Therefore, the program increment planning, inspect and adapt, and solution demo might fail.
5.1.1.4. Spanning palette teams
The teams that are part of the spanning palette: DevOps, system team, release management, shared
services, and user experience are available for all teams of the Agile Release Train. When the teams
of the Agile Release Train are spread over multiple time zones, the spanning palette needs to be
available during multiple time zones. However, if these time zones span a time of, for example, 14
hours the teams have to be available for 14 hours. This is not necessarily a problem, but this extra load
on these teams can cause them to fail. Therefore, the spanning palette teams might fail.
5.2. Identification based on theory: Expert focus group
To identify which SAFe elements fail, a focus group has been used. This focus group has been executed
according to the protocol presented in Chapter 3, page 26. This section describes how the conditions
for using a focus group are fulfilled. The results of focus group, and an analysis of these results are also
presented. A detailed description of the execution of the focus group can be found in Appendix O.

50. 44
5.2.1.Conditions for the expert focus group
The main condition for using a focus group, as stated in 3.4.1 page 26, is that it should be composed
of 6 to 12 people from different backgrounds. With the summer holiday season nearby the choice was
made to try and organize the meeting before the holidays, as otherwise two months would pass before
another option would arise. Therefore, there was little time to recruit participants, thus the session
was held with 6 participants of different backgrounds and genders.
The second condition is that if the participants are asked to individually judge a topic, all participants
should be able to do this based on their expertise. The participants have been asked to vote
individually in the second round, thus they should all be able to individually judge the topic. The
participants for this focus group have three different backgrounds: SAFe program consultants, Release
Train Engineers with distributed experience (practitioners), and distributed experts from the academic
world. The level of expertise of the participants on the topics, distributed and SAFe, is different. This
difference could be a limitation and will be discussed in the limitations section.
The last condition is regarding previous involvement in the research. For the focus group, none of the
participants have been previously involved with the research. Therefore, no influence from this can
be expected.
5.2.2.Expert focus group results
In this section the results of round 1 & 2 of the focus group are presented. The results of the other
rounds are not presented as those results are not used in the research but can be found in Appendix
Q to Appendix S. A full description of the execution of the focus group, including the results of the
other rounds, can be found in Appendix O. The protocol used in the focus group is visualized in Figure
25.
Figure 25: Expert focus group protocol visualization