ACCELERATE
AGILE DEVOPS / BUSINESS CASE FOR CI/CD, TDD,
Microservices, K8s, AUTOMATION, ETC.

The promise of CI/CD and Agile/DevOps culture
What we are after!

evOps, Continuous Deployment and Continuous Integration (CI/C
Why we care?

ABOUT RICK• Author of best-selling agile development book, early adopter of TDD, DevOps,
Agile, etc.
• Highest leadership scores of any senior director in a 2,000 person org (happiest,
most productive team). Highest performing team in a 1,000+ person group: Jenkins
pipelines, Event Hub/EEL, High-code coverage, PR process, Trunk based git like
GitHub flow, full automation, etc.
• Two awards from CIO at fortune 500, amazing results (G.O.A.T and Engineering
Excellence)
• Amazing results finishing projects deemed impossible under tight deadlines. Grew
team from 12 to 50+, Talent Magnet due to culture of excellence
• Written open source software used by millions
• Early adopter and proponent of MicroServices and streaming high-speed streaming,
12 factor deployment, container orchestration, in-memory compute and uService
architecture
• Speaker at conferences on microservice development, a Java Champion (chosen
from 10,000,000 Java Developers), parsers, distributed data grids, books, articles,
etc.
• Mentoring, consulting, papers, blogs, specifications, JSRs for distributed compute,
streaming

SLIDE DECK BASED ON
BOOK ACCELERATE AND
MORE• PAST
EXPERIENCE
• LATEST
TRENDS

OUTLINE
• How we got here
• History of MicroServices
• DevOps / Agile
• CI/CD
• Kubernetes
• Best practices
• Continuous delivery
• Continuous integration
• Lean management and
monitoring / KPIs
• SCM / Version Control
/ GitOps / Immutable
infrastructure
• Trunk-based
development

BRIEF HISTORY OF MICROSERVICES
AND AGILE,CI/CD
Brief history of time

HOW WE GOT HERE
• Web pages that were brochures
• eCommerce
• Legacy integration
• Rush to Webify businesses
• SOA: Wrap legacy systems as services to use from web
• Virtualization, Virtualization2.0, Cloud, Containers, and now
Container orchestration

“Now you can run a JVM in a Docker image which is
just a process pretending to be an OS running in an
OS that is running in the cloud which is running inside
of a virtual machine which is running in Linux server
that you don’t own that you share with people who you
don’t know.”
Microservices Architecture

“The Java EE container is no longer needed because
servers are not giant refrigerator boxes that you order
from Sun and wait three months for (circa 2000). Don’t fight
classpath, classloader hell of Java EE. … your whole … OS is
now an ephemeral container (Docker). Deliver an image with all
the libs you need, don’t deploy to a Java EE server which has
to be versioned and configured. You are only running one
service in it anyway.… Let it go. …One issue with enterprise
components is they assume the use of hardware servers
which are large monoliths and you want to run a lot of
things on the same server. Well, turns out in (today), that
makes no sense. Operating systems and servers are
ephemeral, virtualized resources and can be shipped like a
component. We have EC2 images AMIs, OpenStack, Vagrant,
Kubernetes and Docker. The world changed. Move on….
Microservices just recognize this trend so you are not
developing like you did when the hardware, cloud
orchestration, multi-cores, and virtualization was not there.
You did not develop code in the 90s with punch cards did
you? So don’t use an EAR file or a WAR file (today).”
Microservices Architecture

MICROSERVICES
• Focus is building small, reusable, scalable services
• Adopt the Unix single purpose utility approach to service
development
• Small so they can be released more often and are written to be
malleable
• Easier to write
• Easier to change
• Go hand in hand with continuous integration and continuous
delivery
• Heavily REST based and messagingWhat is microservice architecture?Microservices Architecture

KEY INGREDIENT OF
MICROSERVICES
• independently deployable, small, domain-driven services
• Own their data (no shared databases)
• communication through a well-defined wire protocol
usually JSON over HTTP (curl-able interfaces)
• well defined interfaces and minimal functionality
• avoiding cascading failures and synchronous calls -
reactive designing for failure
What is microservice architecture?

SOA VS. MICROSERVICES
• SOA and Microservices have common goals and
purposes
• Refinement to meet goals of polyglot devices and
3rd generation virtualization (cloud, container,
container orchestration)
• Parts of SOA that worked well
• MS: Web technologies to provide scalability,
modular, domain-drive, small, and continuously
deployable cloud-based services
What is microservice architecture?
It’s not the daily
increase but daily
decrease. Hack away
at the unessential. --
Bruce Lee

SOA VS. MICROSERVICES
“Microservices Architecture is taking what perhaps started out as SOA and
applying lessons learned as well as pressure to support polyglot devices,
deploy more rapidly and the architecture liquidity that cloud computing and
virtualization/containerization provide. You mix all that together and you
can see where Microservices Architecture started. Microservices
Architecture is in general less vendor driven than SOA and more needs
driven by demands of application development and current cloud
infrastructure.” —Rick Hightower 2013 :)

UNIX PHILOSOPHY
• Microservices compares to Unix philosophy,
• Ken Thompson, Unix creator, said Unix has a
philosophy of:
one tool, one job
• “Unix philosophy emphasizes building short, simple,
clear, modular, and extendable code that can be
easily maintained and repurposed by developers
other than its creators” What is microservice architecture?

COPING WITH FAILURE
• Avoid synchronous calls to avoid cascading
failures
• Microservices tend to embrace streams,
queues, actor systems, event loops and
other async calls
• Spend more time with distributed logging /
log aggregation w/ MDC and now distributed
tracing

MICROSERVICES
MONITORING
• User Experience KPIs
• Debugging (requests per second,
#threads, #connections, failed auth,
expired tokens, etc.)
• Circuit Breaker (monitor health, restarts,
act/react)
• Cloud Orchestration (monitor load, spin
up instances)
• Health checks and observable KPIs
"doveryai no proveryai"
(trust, but verify)
Microservice Monitoring

“Just remember Microservices are not a new
thing, and they are not cool or hip.
Microservices are obvious evolutionary
architecture to address the revolutionary things
that already happened: web, cloud, mobile,
server virtualization, OS containerization,
container orchestration, multi-core servers,
cheaper and cheaper RAM, 64 bit computing,
10GBE, 100GBE, etc.”
Microservices Architecture

MICROSERVICES ARE
CONTINUOUSLY DEPLOYABLE
SERVICES
• Microservices focus on business capability,
and a refocus on object oriented
programming roots and organizing code
around business domains with data and
business rules co-located in the same
process or set of processes

CONTINUOUSLY
DEPLOYABLE
MICROSERVICES• Focus of Microservices is on breaking up applications into
small (micro) reusable services which might be useful to
other services or other applications.
• Services can be deployed independently
• Each of these services to be tweaked, and then
redeployed independently
• This is where the "micro" part of microservices comes
into play
What is microservice architecture?

DEVOPS, AGILE, TDD,
CI/CDBrief history of time

XP, AGILE, SCRUM, TDD,
CI/CD
• TDD, CI and CI/CD
• Test Driven Development and Agile
• XP, Agile, Scrum
• CI/CD (Jenkins and the tools that came
before)
• CI/CD needs automated testing

DEVOPS
• DevOps aka DevSecOps
• Heroku and the birth of 12 factor deployment, DevOps, KPIs, SRE
• YBYOI vs. SRE vs. DevOps and where do you fit?
• SRE: Observability, Log aggregation, KPIs/Metrics, Distributed/Trace
logging
• Container Orchestration: Yarn, Mesos, Marathon, Nomad, Borg and
Kubernetes
• What is GitOps? What is immutable infrastructure?
• What is cloud native?

KUBERNETES / K8S
• Services, Stateful sets, Namespaces, Tags, Ingress, Egress
• Helm/Kustomize for packaging an app or set of related uServices and deploy to
K8s
• Multi-cloud support and just cloud support
• Monitoring built-in (or at least easily pluggable)
• Easy to ramp up (or easier)
• Supports flexible deployment models
• Integrates with Cloud providers services or runs standalone (on prem or cloud)
• What came before and now: Heroku/PaaS/IAAS/EC2, Docker, Docker Swarm,
Mesos/Marathon, Nomad, ECS, EKS, etc. K8s is the current mindshare champ.

WINNING THE RACE
Why we want to do it? Why it is important!

ACCELERATION IN
PRACTICE
• Make more money
• Deliver more
• Less Burnout
• Grow the value of the
company
• Make customers happy

ORGANIZATIONAL
PERFORMANCE
• High performers 2x the rate will exceed organizational
performance goals as low performers:
• 2x profitability
• 2x productivity
• 2x market share
• 2x number of customers

ORGANIZATIONAL
PERFORMANCE PART II
• High performers twice as likely to exceed non-commercial
performance goals as low performers
• 2x better quantity of products and services
• 2x operating efficiency
• 2x customer satisfaction
• 2x quality of products/services
• 2x achieving organizational/mission goals

ORGANIZATIONAL
PERFORMANCE PART III
50% increase in market
capitalization compared to low
performers!

SOFTWARE DELIVERY
PERFORMANCE
• Deploy frequency, Lead time, mean time to restore (MTTR),
and change fail percentage do well to predict overall software
delivery performance
• Improving software delivery performance improves tempo and
stability
• Software delivery performance improves organizational
performance and quality/customer satisfaction
• Deploy frequency is highly correlated with continuous delivery
and use of version control best practices

SOFTWARE DELIVERY
PERFORMANCE II
• Lead time is highly correlated with good version control
and automated testing
• MTTR is highly correlated with version control and
monitoring
• Software delivery performance is negatively correlated
with deployment pain
• Software delivery performance is correlated with
organizational investment in DevOps

QUALITY
Source, Forsgren PhD, Nicole. Jez Humble, Gene Kim, Accelerate . IT Revolution Press. Kindle Edition.
st amount of manual work across all practices - configuration m

CULTURE
• 5 factors most associated with burnout are
negatively impacted by bad software delivery
performance
• Deployment pain and poor software delivery
practices cause organizational burnout

IMPROVE CULTURE BY
IMPROVING PRACTICES• Technical practices predict continuous delivery
• Improve organizational culture, identity, job
satisfaction, software delivery performance, less
burnout, less deployment pain, and less time
spent on rework!
• High performers spend 50% less time
remediating security issues than low
performers

TRUNK BASED DEVELOPMENT
(LIKE GITHUB FLOW)
• ​High performers have shortest integration times
and branch lifetimes
• Branch life and integration typically lasting hours
or a day
• Low performers have longest integration times
and branch lifetimes
• Branch life and integration typically lasting days or
weeks

ARCHITECTURE
• Loosely coupled, well-encapsulated
architecture drives IT performance.
• 2017 dataset biggest contributor to
continuous delivery was loosely coupled,
well-encapsulated architecture

LEAN PRODUCT
MANAGEMENT
CAPABILITIES• Experimental approach to product
development highly correlates with
continuous delivery
• Lean product development capabilities
predict improvements in organizational
culture like reduced burnout higher software
delivery performance and overall
organizational performance

HOW WE GET THERE
Rules of the road

ACCELERATE DEVOPS
• Continuous delivery
• Architecture
• Product and process
• Lean Management and monitoring
• Cultural

CONTINUOUS DELIVERY
CAPABILITIES
1. Implement continuous delivery / continuous deployment
2. Version control all production artifacts
3. Automate your deployment pipeline
4. Implement continuous integration
5. Use trunk-based development methods (like Github flow
instead of git flow)
6. Implement test automation
7. Shift left on security

PRODUCT AND PROCESS
CAPABILITIES
• Gather and implement customer feedback
• Make the flow of work through the system
visible
• Work in small batches
• Foster and enable team experimentation

CULTURE CAPABILITIES
• Support a generative culture
• Encourage and support learning
• Encourage collaboration
• Make work as meaningful as possible
• Support and encourage transformational
leadership

ACCELERATE DEVOPS
Source, Forsgren PhD, Nicole. Jez Humble, Gene Kim, Accelerate . IT Revolution Press. Kindle Edition.

ARCHITECTURAL
CAPABILITIES TO
ACCELERATE
• Use loosely coupled architecture
• Release new services on demand
without outages
• Empower the team to select tools; trust team
to pick the best tools

LEAN MANAGEMENT AND
MONITORING
CAPABILITIES• Have a light weight change approval process
• Monitor application and system KPIs to inform business
decisions
• Proactively check system health
• Preemptively detect and mitigate problems
• Improve process and work within WIP limits
• Set up visible dashboards to monitor/communicate WIP,
quality, applications and systems

EXAMPLE MONITORING
• Hygieia
• Productivity dashboards
• Jenkins dashboards
• Runtime KPIs
• Customer KPIs

LEAN MANAGEMENT
• Process: Small Batches
• Decompose into features that allow for rapid development
• MVP - prototype with just enough features to proved business value or enable validated
learning
• Quickly gather customer requirements (A/B testing, customer satisfaction surveys, etc.)
• Team experimentation
• Lean Management: Change approval
• Lean Management: Proactive notification
• Lean Management: Monitoring and KPIs
• Lean Management: WIP limits, visualizing work

Companies with regulatory requirements or strict CCB
Can focus on Continuous Delivery
Continuous Deployment can be part of a workflow and
Based on the Continuous Delivery

VERSION CONTROL - SCM
• GitOps - keeping application code, system configuration,
application configuration, and scripts for automating
build and configuration in version control.
• Factors together predict IT performance
• Key component of continuous delivery
• GitOps - keeping system and application config in git
(versioned) correlates high with delivery performance

DEPLOYMENT
AUTOMATION
• Deployment automation
• Containers, config, immutable infrastructure
• Comprehensive configuration management
(automation scripts), continuous integration and
continuous testing
• Key metric of GitOps is how much diffs exists in
system config from git to deploy

CONTINUOUS
INTEGRATION
• Continuous integration
• Relies on SCM and Deployment automation
• Relies on automated tests
• Unit
• Integration
• Acceptance

TRUNK-BASED
DEVELOPMENT
• Trunk-based development
• Like GitHub Flow but shorter lived branches
• Fewer active branches that never outlive a sprint
• Branch-off master per feature, bug fix, etc.
• More PRs more often
• No code freezes; integration periods less than a day
• Polar opposite of git flow

SHIFT LEFT ON SECURITY
• Integrating security into design and testing
phases
• Security reviews of applications, including
the infosec team in design and demo
process
• Using pre-approved security libraries and
packages, and testing security features as a
part of automated testing suite

CONTINUOUS DELIVERY
• The ability to deliver
• Build quality in
• Work in small batches
• Automate repetitive tasks including testing &
deployments
• Pursue continuous improvement
• Ownership
• Comprehensive configuration management
• Continuous integration
• Continuous testing
You can’t skip steps.
There is investment
up front.
Today’s speed up can
be tomorrows painted
yourself
In a corner.

KNOWING THE ROAD
Best practices

PREFER MICROSERVICES
BUT…
• Know when to employ Monolith and when not to
• Can speed up MVP and prototypes but at a cost
• Why this makes CI/CD slower?
• Automated tests needed for CI/CD
• Smaller Monoliths and SOA is a move in the right
direction

PREFER MICROSERVICES
• Refactoring to Microservices is a journey
• Know when to employ Microservices
• Fits the CI/CD well
• Fits small batch well
• How Micro can mean different things as you get better at
Microservices
• Why today's micro could be tomorrows Monolith?
• Adoption is a Journey

EMBRACE OBSERVABILITY
FROM THE START
• Log aggregation
• Time series data base
• Log all KPIs for clusters
• Log all KPIs for applications and services
• Alerting
• Know when and how to employ distributed tracing
• Distributed/Trace logging

CI/CD
• CI/CD
• Deploy often (daily or more)
• Test often (after every checkin run all
automated tests)
• Block PRs from merging until they pass tests

TESTS TO CREATE: TDD
• Unit tests
• Perf testing JMH
• Functional tests
• At the HTTP
layer
• At the Spring
Boot layer
• Acceptance tests
• Smoke integration tests
• Full integration tests
• Synthetic testing
• Code Coverage (sonarqube)
• Security/Vulnerability dependency license
checks
• Aqua, Fortify
• Full integration perf testing

WHAT IS A PR? AND HOW
TO ENSURE QUALITY WITH
IT• A PR is a pull-request
• PR gives other developers a chance to review code before it
committed to master
• PR via small batch (why small? JIRA story or even a tasks or two)
• With GitHub and WebHooks you can block PRs from merging
• Code coverage met, build works, unit tests run, other checks via
Jenkins
• Review and approved by at least two people

TESTING IS A MUST!
• You can't do CI/CD without automated
testing
• Testing allows you to move quickly with
confidence

EMBRACE SMALL BATCH
WORK
• Goal of three PRs per week
• Goal of one to two tasks from JIRA per PR
• Use JIRA # in commits and PR comments

AUTOMATED
DEPLOYMENT
• Merging into master triggers a deploy to integration and sends a Team message
• Approval from Product Manager pushes code to Prod or Demo
• Checking into main branch from PR
• Artifacts and scripts for deployment checked into git and should not be modified
• Puts code staging area to be checked by Product Manager
• into containers and deploys to cluster (some ephemeral some not)
• Once checked by Infosec and Product Manager: Canary Deploys to 3 to 5% of
traffic and is monitored (end goal)
• Then more and more as it is monitored and is ok

DEMO
Jenkins
Minikube
Spring Boot

THE RACE IS WON
Conclusion

TRANSFORMATION IS A
BUSINESS IMPERATIVE• You can’t afford not to transform
• Transformation requires a deep understanding of
practices
• Having a team called DevOps is not doing DevOps
per se
• Culture of DevOps, Agility, Lean, MVP, etc. is a
clear win
• There are guides, books, practices, and information

Read Accelerate by Forsgren PhD, Nicole. Jez Humble, Gene Kim.
Also read The Loop Approach: How to Transform your organization from
The inside out! By Sebastian Klein and Ben Hughes
Also read Cloud Native DevOps with Kubernetes by John Arundel and Justin Domingus