Embarking on salesforce.com's large scale transformation to our Adaptive Development Methodology presented many unique challenges due to the scale of the rollout as well as the need to align a large number of teams (30+) to a common release date, 3-4 times per year. To specifically address the latter challenge, we created a lightweight release framework in order to optimize on-time, high quality delivery to our customers and partners. The fundamentals of this framework and approach emphasize supporting and enabling the development teams rather than controlling them, through a release framework--simple, lightweight, unambiguous, visible everywhere. By implementing this lightweight release framework and making it highly visible, we've promoted rhythm and flow within and across all scrum teams. In doing so, we have been able to develop a stable rhythm for delivering on-time, high-quality releases in a complex environment, while not compromising our Agile principles.

1. Fast & Predictable – A Lightweight Release Framework Promotes
Agility through Rhythm and Flow
Amy Farrow; Steve Greene
salesforce.com
afarrow@salesforce.com; sgreene@salesforce.com
Our answer to this problem was a lightweight
Abstract
Release Framework and the introduction of a Release
Manager for the overall product release.
Embarking on salesforce.com's large scale
transformation to our Adaptive Development
The result is a highly effective Technology
Methodology presented many unique challenges due
organization, operating on a consistent iteration and
to the scale of the rollout as well as the need to align a
release rhythm, enabling flow of information within
large number of teams (30+) to a common release
the organization, reducing cognitive dissonance,
date, 3-4 times per year. To specifically address the
realizing organic alignment with the rhythm, and
latter challenge, we created a lightweight release
optimization of the whole of the product release.
framework in order to optimize on-time, high quality
delivery to our customers and partners.
2. Release Framework: Guiding Goals &
The fundamentals of this framework and approach Principles
emphasize supporting and enabling the development
teams rather than controlling them, through a release A very clear set of goals and principles provided us
framework--simple, lightweight, unambiguous, visible guidance us in defining the Release Framework.
everywhere. By implementing this lightweight release
framework and making it highly visible, we've 2.1. Consistent, On-Time Delivery, No
promoted rhythm and flow within and across all scrum Exceptions
teams. In doing so, we have been able to develop a
stable rhythm for delivering on-time, high-quality One of the founding principles of our ADM
releases in a complex environment, while not implementation was that we would deliver on-time,
compromising our Agile principles. high quality releases consistently to deliver value to
our customers and partners. To achieve this end
1. Introduction required discipline within the management team to
reject any requests for exceptions in the first two
Salesforce.com initiated a transition to the Adaptive releases we delivered with ADM. This change
Development Methodology, based upon scrum, in dramatically simplified the process and focused
October 2006. Shortly after the initial rollout to 30+ Product Owners on the task of prioritizing effectively
product teams it was clear that we had unleashed the to deliver as much value as possible in the timeframe
power of our individual teams through their new found available to them. In addition to this, in order to deliver
self-organization and focus on short sprints. However, frequently and to converge to a common release date,
since we aggregate the work of many sprints into a the scope of the release and the scope of work for each
single release it was also becoming clear that our team working in the release had to be negotiable.
scrum teams were operating independently without
respect for the context of the larger release. Herding In the process of doing this, it became readily
30+ teams working in the same codeline to a apparent that we needed to adopt a concept of “Scrum
predictable time-boxed release date is very challenging Team Equality”, meaning that in the scope of the
in any traditional development model let alone an agile release and schedule, the work that one team was doing
development model where we value self-organization was no more important than another team’s. In the
over centralized control. same respect, one feature, no matter how important,

2. would not cause us to delay a release of 30+ teams’ environment we were very sensitive to imposing heavy
work that was ready to go. milestones on our teams. However, we still believed
that there were key milestones that would align our
teams to our on-time, high quality release objectives
2.2. Scale for Today, Tomorrow, 10x
while enhancing the agility of our teams. Those
milestones are as follows:
Salesforce.com is a consistently and rapidly growing
organization so we needed to create a framework that
1. Release Kickoff – Alignment from the Start: At the
would enable us to scale our ability to successfully
beginning of each release, which coincided with the
manage a release that today includes 30+ scrum teams
event of opening of the codeline, the Release
but tomorrow will easily surpass 60 scrum teams and
Operations team organizes an R&D wide release
beyond. We want to have a model that could give 10X
kickoff meeting. The purpose of the meeting is to
scalability. At the time we embarked on the ADM
provide visibility to the release plan for each team,
rollout, there was not a lot of experience in the industry
highlight dependencies, generate excitement about
related to enterprise agility at scale. There was little
what’s coming next and align the teams on the next
professional advice available on how to scale an agile
release rhythm.
organization like Salesforce.com with so many teams
working in the same codeline.
At this meeting our most senior executives participate
by talking about key business objectives of the release
2.3. Optimize the Whole of the Release
including exciting new innovations and how those
innovations will drive customer satisfaction. Their
We needed to ensure that the process or framework we
commitment to the success of the release is clear.
put into place would promote visibility and
optimization of the whole release, or of the suite of
In addition, each of our Chief Product Owners present
product functionality that we deliver to our customers
the release plans for their scrum teams (features,
and our partners.
architecture and infrastructure work, refactoring, etc).
In a short amount of time they are challenged to
With such a large organization working on the same
convey their plans concisely and effectively. The level
codeline, it was imperative that our global priorities
of information team members derive from the kickoff
were consistent and clear to all scrum teams in order to
enables them to be effective in planning for their own
maintain quality of the release throughout the
teams, dependencies they had on others or others had
development cycle. The priorities were as follows:
on them, and to foresee any possible impacts to plan
their time more effectively.
1. Maintain existing functionality and quality of the
overall codeline on a daily basis. All automation
We value the importance of the organization getting a
regression tests must pass at a 99% passing
look into what is planned to enable them to effectively
threshold on every checkin.
communicate and work with each other, but we also
2. Building new functionality is secondary to
valued the time that we had to contribute to this, so we
codeline stability.
time-boxed the kickoff to 1.5 hours.
While each scrum team had to invest time to
After the Release Kickoff, all release plans are
maintain the quality of the codeline on a daily basis,
available for all teams working on the release and are
they also experienced benefits from it as well.
publicly shared with the entire Technology
organization.
9. Release Framework – A Blueprint
2. Feature Freeze
The Release Framework we created became a
blueprint for all major releases which enabled Product
Feature Freeze marked the end of the product
Owners to prioritize adequately and enabled teams to
development sprints and the start of what we call the
define realistic release plans.
Release Sprint. All new product features are expected
to meet our “definition of done” at the end of each
9.1. Light on Milestones sprint but also be potentially releasable at this
milestone. This milestone also coincides with the
Part of the release framework included the creation of a beginning of the Release Sprint. Between Kickoff and
few key milestones. In our self-organizing, agile Feature Freeze, teams are clear on the goals they need

3. to meet but are able to self-organize to achieve their that release even though not necessarily involved in the
goals in that timeframe. release in its earlier stages. This “handoff” of
ownership to release managers late in the game
3. Release Freeze exposes weaknesses in the waterfall approach
promoting an environment of blame and frustration
This milestone marks the end of the Release Sprint and when problems occur and releases are delayed.
means the overall release has moved from a potentially
releasable state to a releasable state. To differentiate When we introduced the Release Manager role at
the activities from the Product Development Sprints salesforce.com, we believed that it was important that
and Release Sprints we clearly documented the the role be engaged from the inception of the release
activities that did not contribute value when completed through to production deployment partnering with the
each month and actually made more sense to only rest of the organization to ensure success. The Release
execute once per release. Manager provides a framework within which self-
organizing teams could be independent and agile while
at the same get a clear understanding of the context of
4. Release
the release with a simple escalation path if something
was preventing them from achieving their goals, as
The final important milestone is the release!
related to the release. To achieve this Release
Managers transformed away from a traditional role of
This was the extent of the milestones we used to create
“driver of the release” to a servant leader role, clearing
a release framework to promote self organization in
the path of systemic release issues that slowed down or
achieving the milestones to deliver a high quality, on-
halted team success.
time release. The rest was left to the teams.
It was important to us when we created the release
9.2. Rhythmic Cycles
framework that we introduced the concept of this role
but needed to reset expectations around it, since we
In order to support teams in establishing and
were creating a new definition for this role, with the
understanding their velocity and what they would be
following key characteristics and responsibilities:
able to achieve in a given release, it was important to
create a consistent, reusable template for our releases
• Engage throughout the lifecycle of the release,
such that they exhibited similar characteristics and
from inception through to deployment.
mimicked the iterative monthly cycles in iterative 4
• Promote regular release visibility and keep the
month cycles.
health, quality and release schedule highly visible
to the organization.
The following diagram depicts this rhythmic cycle:
• Resolve systemic release issues that that slowed
down or halted team success.
Release Release
• Promote team/product equality with respect to the
Sprint Sprint Sprint Sprint Sprint Sprint
overall release schedule.
Jan Feb March April May June
Release & Sprint Rhythm
We initially equated this to role to “Scrum Master for
the Release” and this sums up what we were aiming to
The rhythmic cycle provided a mechanism for organic
achieve – a Servant Leader who would uphold the
self-organization across, within, and amongst teams
values and principles of ADM and our Release
around how to collaborate better on their work. All
Framework to enable the organization to deliver.
teams are on a monthly cycle with sprints and all
releases are on the same 4 month cycle. This
consistent rhythm at a daily, sprint and release level is 10. Release Framework – In Action
very powerful for our organization reducing cognitive
dissonance and eliminating waste across teams. Defining the Release Framework was an important
step which led to determination of some of the
9.3. Release Manager as Servant Release mechanics in action that would enable and promote
Leader this framework.
At many non-agile software development companies,
release managers own the “end game” or delivery of

4. automation results and incoming defect rates
10.1. Release Schedule Visibility
normalized by scrum teams and number of developers
and quality engineers. The analysis of aggregated data
A simple method that we use to expand visibility of
at a release level (with comparisons to our previous
our release schedule is posting large visually striking
releases) provides a higher level of visibility into
posters of the overall schedule throughout our
release trends and quality hot spots in the overall
workspace. The posters clearly communicate to and
codeline. This analysis has been useful for scrum
are a quick reference to the entire organization on a
teams and the management team to remove systemic
daily basis as they walk through our workspace. These
obstacles that are impacting all teams relative to the
posters help teams understand how all the release
overall release as well as providing a guide for
pieces fit together, and to give better insight into the
coaching. The Release Manager takes the lead on
overall mechanism for delivery in the SaaS model.
analyzing and providing this information along with
This simple technique has significantly increased
recommended action if any. All of this is necessary in
awareness of the key dates, milestones and rhythm of
a large scale agile environment, however, done with a
the release.
light touch to ensure that we don’t violate our
principles of self-organization for teams.
10.2. Release Health & Quality Visibility
10.4. Communication
The Release Manager promotes daily visibility of the
overall release quality and status through “Daily
Communication is a key element of our Adaptive
Drumbeats” that are made available to the entire R&D
Development Methodology. At the beginning of our
organization. Thresholds are defined across any given
transition, Scrum Team and Scrum of Scrums
day in the release cycle and thresholds are also defined
communication was effective in understanding both
at monthly intervals when the release is expected to be
progress and issues or blockers facing the teams.
potentially releasable.
However, since we had three Scrum of Scrums we
found that there were challenges with communication
Here’s a snapshot:
across Scrum of Scrums as it related to the overall
release. As a result, the model we adopted to increase
effectiveness and communication across all teams
working on the release was not for the teams to come
to the Release Manager but rather for the Release
Manager to go to the teams, through the weekly Scrum
of Scrums. This became a forum for bi-directional
The difference here is that the biggest challenge with information sharing and issue resolution.
on-time delivery in large projects or environments
where 30+ teams are working on a release is that the 10.5. Push Decision Making to the Team -
complexity is high. Promoting regular visibility into Scrum Teams Decide when “Release Ready”
aggregate release status while scrum masters promote
visibility of individual team status ensured that we kept As with any release, there comes a time when we need
both the local (team) and global (release) status in our to decide whether or not the release is ready to be
purview at all times. This framework approach also deployed to customers. In a traditional approach, this
enabled team agility and ownership in understanding decision is often made behind closed-doors at the
their dependencies and impacts, rather than another executive level. We believed that this decision should
team resolving them on their behalf. be pushed down to the individual Scrum Teams to
promote even more agility. It was important for each
10.3. Release-by-Release Comparison Visibility of the Scrum Teams to contribute to the release
readiness decision because they held the greatest level
Weekly reporting of release-by-release comparisons of detail about readiness for each features. As such,
add another dimension of understanding of the current we expanded our Sprint “Definition of Done” to create
state of the release – both how the release itself is a Release “Definition of Done”. At the Release Freeze
progressing day by day as well as how this release on a milestone, we expected each scrum team to
weekly basis compared to previous releases, communicate to the Release Manager that they had met
normalized to make the comparisons meaningful and the Release “definition of done” criteria and that their
insightful. We regularly analyze aggregated bug debt,

5. product area was ready to be released. We do continue Development Methodology, we experienced tension
to maintain release signoff and decisions at the with a few of the Product Owners who continued to
executive level; however, the framework we have expect the release date to be negotiable if their features
created promotes a shared level of responsibility were not completed to their expectations. Guided by
between the individual scrum teams and the executive our desire to deliver value to our customers and
team. partners on a regular basis (through time-boxed
releases and variable scope), we upheld the
organizational will and resisted these demands.
11. Challenges
Executive level support and commitment to both our
Adaptive Development Methodology and to our
There were some challenges with creating this
Release Framework were the keys to overcoming this
lightweight Release Framework that are worth
challenge. Over time, this issue dissipated as teams
mentioning.
understood that release dates were not negotiable.
We’ve seen significant behavior change and this issue
11.1. Release Manager / Scrum Master Role
and is no longer part of the discussion
Confusion.
12. Conclusion
When we first introduced the Release Manager role,
there was some role confusion between the Release
A lightweight Release Framework can be very useful
Manager and the Scrum Masters for individual teams.
in scaling at the enterprise level. By providing this
Since both roles were new to the organization at the
framework we have increased the level of agility and
same time, it took some time for us to evolve the roles
self-organization throughout our organization. On the
and the relationships between them. Once we did and
vast continuum from rigid structure and control to self-
clarified the Scrum Master responsibility to the Scrum
organization, the key to success is to constantly strive
Team and the Release Manager responsibility to the
for openness and flexibility (that comes from self-
overall Release, we were able to achieve an effective
organization) and to implement a simple, lightweight
operating model within the organization.
release framework that supports and enhances those
principles of self-organization. Finding the right
11.2. Scrum Team Equality
balance is an art form and depends upon cultural issues
within the organization and requires a high level of
The concept of Scrum Team Equality was a hard one
trust. Finding that complementary balance is
to grasp initially given the sheer number of teams and
challenging yet worthwhile as it can unleash your
Product Owner pride in the importance of their
organization’s potential.
particular product area and feature set. In the first two
agile on-time releases we delivered with our Adaptive