Continuous Deliveryand Continuous Agile by Andy Singleton - Agile Maine Day 2016

1. Continuous Delivery
and Continuous Agile
From Andy Singleton,
http://continuousagile.com
www.assembla.com

2. The secret weapon that Silicon Valley is using to
disrupt and destroy competitors
• Retailer X deploys changes to their monolithic online
ordering app once every six weeks. Ops holds for three
weeks to make sure the complete system is stable.
• Amazon has thousands of services and more than 1000
service teams. They release something about once every
11.6 seconds. In the time that Retailer X takes to try
one new release, Amazon has made 100,000 changes.
• Amazon hosting competitor: “It’s an emergency”.
• 6 Apex Predators in the top 10 public companies worth
$2.4 trillion: Google, Amazon, Facebook, Microsoft, GE,
Apple

3. Survey on Continuous Delivery
46% think their
competitors
have adopted
Continuous
Delivery

4. Problems solved by Continuous
• Stressful releases
• Speed to market
• Competitiveness
• Quality
• Scaling – building large systems

5. Not Scrum
• Continuous improvement versus Cadence
Unblock your teams
• Web services versus monoliths
Smaller, faster testing, faster releases
• Technical practices, not team practices
HUGE productivity gains from automation
• Tech leads versus self-managing teams
One leader, ready to build services
• MAXOS versus SAFe
Scale different

6. 改善
1. リリースを頻繁にする
2. 改善する
/ 月割りのリリース

7. Kaizen
1. Release more frequently
2. Improve

8. Continuous Agile Agenda
1) DevOps - Continuous delivery of software –
Build/Test/Deploy
2) Continuous, metric-driven product
management
3) MAXOS enterprise management
4) Making the Move

9. Continuous Delivery
or “DevOps”

10. Scrum Sprint
Validate and
Sort Deliverables
Select a
Sprint Plan
Completed
= observed
velocity
Expand stories
and estimate
tasks
Collect
Ideas
Close Sprint
Into next
sprint
Work on
tasks

11. Scrumban Iteration
Validate and
Sort Deliverables
Pull Deliverables
When Ready
Collect
Ideas
Stabilize
Release
Into next
release
Feature
Freeze
Triage

12. Idealized ScrumBan Release
Copyright 2012 Perforce Software, Inc. - Assembla, Inc.

13. Kanban / Continuous
Validate and
Sort Deliverables
Pull Deliverables
When Ready
Collect
Ideas
Continuous
Release
Stay within
WIP limit
Release Features
Continuously

14. Program Launch
Releases
Plan Program Test Doc Deploy
Skip
Automate
& Blend Lag Automate
Pull
CI & CD
End up with
Waterfall to Continuous
Measure

15. Code Contribution Patterns
Manage code if possible. People are hard to manage and
can’t be automated. They want to contribute.
• Centralized continuous delivery
– No branches, finds and fixes problems as early as possible
• Distributed continuous delivery
– Release every change with its own branch and test
• Temporary branches
– Combines benefits of centralized and distributed
• MAXOS
– Use centralized continuous integration to manage a
massively scalable IT system

16. Centralized CI/CD
Contributor Commits – “as early as possible” to find problems
Continuous
Integration tests
Fail - alarm
Release Candidate
Test System
Release
QA Testing
Pass

17. Continuous delivery at Edmunds.com
with PERFORCE

18. The big question: How to test?
• We release software in batches so that we can test it.
• In continuous delivery, we might get as little at 10
minutes to test a release candidate

19. Test Layering
Monitor your released software: Errors, Usage
volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to
ask for test scripts.
Continuous integration: Run automated tests
before using human review time
Unit tests in the development environment
Switch new features and architecture
More frequent releases can increase quality

20. 9 (sparse) Layers at Edmunds

22. Feature Switch and Unveil
Hidden
Programmer sees a change locally.
Change is tested in the main
version but not seen.
Test
Story Owner and testers see the
change on test systems.
Beta
Insiders see it and use it. Story
Owner can show it to selected
users for feedback or A/B testing.
UNVEIL!
The big event. Communicate with
all users. Measure reaction.
Onecodeversion
Nospecialtestbuilds
Nolong-runningbranches

23. Role: Developer
• Developers have more power and responsibility.
• Developers have more responsibility for testing.
• Developers (not QA or PM) decide when to release.
This is a strong finding.
• Incentives are correct. Developer might have to
come back from Friday night beers to fix a problem.
This provides a motivation to make good decisions
and automate testing.
• Features can be released but hidden. Product
Managers and Marketers will unveil when they are
ready. Unblock!
Programmers approve releases, not QA

24. Role: IT Vendor
Old Agreement
• Fixed Price
• Fixed deliverables
• Large probability of late delivery
New Agreement
• Fixed price
• Fixed Time
• Variable deliverables
– Small guaranteed deliverable
– Weekly meetings to agree on
improvements
Advantages
• No late delivery
• Deliverable is better
than original
specification
• Start sooner and finish
sooner

25. Role: Product Manager/Owner
• Batch -> Continuous
• Requirements -> User Experience
• Strategy -> Measurement
– Usage measurements are so important, so
underutilized
– Double your productivity

26. Batch Agile Continuous
Explains the resurgence of vertical integration
(eg Apple, Tesla). Requires CI

27. Making one good change is not a team sport
Product Owner -> Story Owner

28. What you realize when you begin to be very
data-driven is a very large percentage of our
ideas are bad.
Microsoft Product CTO Adam Pisoni

29. Measure everything
• Measurements are power. They are impossible to ignore.
They win arguments.
• Double your development capacity for zero extra cost!
Users ignore 50% of all new features. If you can find and
ignore these features, you double your capacity.
• Measure everything possible about usage. Easy with an
online service.
• Collect measurements from installed products and
optionally transmit them.
• Use your marketing department. They measure.

30. Make Product, not Quality
Continuous
Release
Inventory
Release
Improve Quality
Improve Product Usage

31. Matrix of Services
(MAXOS)
Breaking the scale barrier

32. The Services Megatrend
Desktop Web App Cloud Services
App
DB
Service
Service
Service

33. Scale it like Google
• 15,000 developers, 5,000 projects, one current version of
the code (2013). They can go from an idea to a release in 48
hours
• Vast Internet system divided into thousands of "services"
• Most programming done by teams of 3-4
• Centralized process with single version of the test system –
run 100 million test cases daily
• Before developers release a change, they test it with the
most recent version of all the other services. If a test script
finds conflicts, it tells developers who to contact to resolve
them

34. Matrix of Services - MAXOS
Prioritized
Backlog
Current
Work
Each team
releases
when ready
Hundreds
of releases
per day
Service team
Production
service
Service team
Production
service
Service team
Production
service
Feedback on speed, errors, usage, and requests
Test as one system
Integration
test env
Integration
test env
Integration
test env

35. Coordinate without big meetings
Continuous Integration between
latest dev version of each service
• Continuous integration
helps teams coordinate.
• See dependencies
between “producers”
and “consumers”
• Errors and conflicts show
related team contact info
• Meetings and changes
negotiated between two
teams, not many
Prioritized
Backlog
Current
Work
Service team
Service team
Service team
Integration
test env
Integration
test env
Integration
test env
Machines can replace layers of management

36. Teams are largely self-managing
Prioritized
Backlog
Current
Work
Service team
Integration
test env
Up to 50% of work
from backlog
At least 50% of work is self-planned
Problems get fixed quickly
Production
service
Production
service
Production
service
Feedback: quality, reliability,
speed, user support
Production
service
Production
Server
Sense, respond, self manage
minimize planning

37. Hubspot – Great at Mid-scale
• Transformed a monolithic app to 200 services over one
year
• 3-person programming teams. Each of 20 teams is
responsible for about 10 services
• Dev teams responsible for design, programming,
testing, release, monitoring, and responding to
production problems. No full-time QA. Shared PM
and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy,
manage, and monitor their services
• Feedback from customer support also grouped by team

38. Team Building
Prioritized
Backlog
Current
Work
Each team
releases
when ready
Service team
Production
service
Service team
Production
service
Add capacity fast by building around
Single-function programmer/tech leads
Integration
test env
Integration
test env
Teams are not permanently multifunctional

39. Ways to Scale
Scrum + SAFe
• Add more hierarchy
• Complex
multifunction teams
• Hold big meetings and
teleconferences
• Block everyone into
one cadence
• Coordinate big
releases
Top Tech Companies
 Automate management,
as well as testing and
deployment.
 Dev-lead teams
 Communicate peer to
peer
 Unblock teams to move as
fast a possible
 Release more frequently

40. SAFe versus MAXOS

41. How to move to CD and Services
• Measure
• Release more frequently
• Assign blame and other organizational and
authority hacks
• Automate
• Organize the fast layer
• Reverse Conway’s Law
• Manage products, not programs

42. Measure
People respond directly to measurements. They
are very difficult to ignore. They are also MORE
IMPORTANT than internal process discussions.
• Usage and customer value
• Errors
• Fix times
• Release frequency

43. Release more frequently
1. Release more frequently
2. Improve

44. Authority Hacks
Training, persuasion, browbeating, and “culture
change” will not work. They don’t change the game.
You first change the game. Then, people figure out
how to play the game and win.
• Need automated tests? Mandatory code review
• Need higher quality and shorter test cycles?
Force programmers to push the “release button”,
not QA
• Stop your boss from launching big waterfall
projects? Give him monthly budgeting, not
annual.

45. Code Review gets you tests
Automate
If the machine works, people will use it

46. Core IT
annual budget
Reliability & security
mission
API Layer
Marketing
Fast (Awesome Mode) IT
monthly budget
Mission to respond to opportunities

47. Service team
Production
service
Service team
Production
service
Production
service
Integration
test env
Integration
test env
Integration
test env
Fast IT
Core IT (stable service)

48. Reverse Conway’s Law
Boss
Dept A Dept B
Part
A
Part
B
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1

49. Manage Products, not Programs
• Strong product managers deliver value to
users as directly as possible
• The Matrix of Service teams respond to
product managers. The front end teams
provide the product. They act as “consumers”
to pull what they need from back-end teams
• Service teams support multiple products,
dissolving the program hierarchy

50. Product Matrix
Boss
Dept A Dept B
Part
A
Part
B
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1

51. People first, or Technology first?
• Your comments