Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

TB
Full‐day Tutorial
6/4/2013 8:30 AM

"Continuous Delivery:
Rapid and Reliable Releases with
DevOps Practices"

Presented by:
Bob Aiello
CM Best Practices Consulting

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Bob Aiello
CM Best Practices Consulting
Editor-in-chief of CM Crossroads and author of Configuration Management Best Practices:
Practical Methods that Work in the Real World, Bob Aiello is a consultant and software
engineer specializing in software process improvement, including software configuration and
release management. He has more than twenty-five years of experience as a technical
manager at top New York City financial services firms, where he held company-wide
responsibility for configuration management. Bob served as vice chair of the IEEE 828
Standards Working Group on CM Planning and a member of the IEEE Software and Systems
Engineering Standards Committee (S2ESC) Management Board. Contact Bob
at bob.aiello@ieee.org, via LinkedIn, or visit cmbestpractices.com.

Continuous Delivery (DevOps Best Practices)
Bob Aiello, Principal Consultant and Author of

Configuration Management Best Practices :

Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiello
http://cmbestpractices.com

1

CM Best Practices Consulting © 2013

Who am I?
• CM/DevOps Lead & Consultant for over
25 years
• Editor-in-Chief at CM Crossroads
• Author of CM Best Practices
• IEEE Management Board
• Tools and process agnostic
• The guy called in the middle of the night
when the release doesn’t work!

http://cmbestpractices.com © 2013

2

April 9, 2013

Goals of this Course
• Understand Continuous Delivery
• Configuration Management roots
• Control Dependencies & Configuration
• Continuous Integration
• Build and Deployment Automation
• Deployment Pipeline is an Art!
Agile Release Train

3

April 9, 2013

DevOps Focus
• Understand DevOps Best Practices
• A Little History of DevOps
• Scope of DevOps and how to get started
• The People side of DevOps
Establish your own plan for DevOps!

4

April 9, 2013

And Don't Forget
• Delivery Ecosystem
• Components & Dependencies
• Test vs Verification & Validation (V&V)
• Don't forget the Data
• Establish IT governance and compliance
So what is Continuous Delivery?

5

April 9, 2013

Continuous Delivery
• Methodology for getting software from
development to release
• Focus on the Deployment Pipeline
• Rapid incremental deployment
• Minimize Risk
• Many small deployments better than
big bang

6

April 9, 2013

Continuous Integration
• What is Continuous Integration
• Why does CI work?
• Martin Fowler reminds us to test
• Let's consider the ergonomics


7

April 9, 2013

Lessons from Aviation
• Cockpit of a plane
• Controls are easy to read
• Traceability
• Designed to avoid mistakes
How does this relate builds?

8

April 9, 2013

Ergonomics of the Build
• “Bob-proof” your build
• Implicit verification and validation
• Avoid the possibility of mistakes
• Each step should be easy to
understand
• One step should not break the stream
• Use dashboards and reports to
communicate status

9

April 9, 2013

Knight Capital Group
• August 1st 2012 trading disaster
• Related to NYSE systems upgrade
• Resulted in a $440 million loss
• Loss grew as customers left the firm
• Knight Capital Group merged with
GETCO holding company
DevOps doesn't cost $440 million

10

April 9, 2013

Introducing the Trusted Base
• Ensure that you know exactly what
you built
• Verify that the release gets deploy
• Ensure that there are no unauthorized
changes
Understanding Continuous Integration

11

April 9, 2013

Features of CI
• Source Code Management
• Building the Code
• Database integration
• Testing
• Source code inspection
• Deployment
Controlling isolation

12

April 9, 2013

Controlled Isolation
• Developers work in sandboxes
• Deliver (actually published) your
changes
• Rebase before you deliver
• Private builds
• Manage variants
Consider the ergonomics

13

April 9, 2013

Ergonomics of CI/CD
• Small incremental changes
• Early Warning
• Reduce risk
• Easier to triage
• Easier to backout
What about life support systems?

14

April 9, 2013

I was once asked...
• What if you were upgrading a life
support system and your loved one
was impacted
• How do we ensure that there are no
mistakes
• Join me in Detroit in July
How do we keep CI lean?

15

April 9, 2013

Keeping CI Lean
• Too many builds
• Too much noise
• Tag interesting builds
What about pilots and landings?


16

April 9, 2013

Pilots Abort Landings
• Recently I was on a plane when
another plane was on the same runway
• You want to fail early when necessary
• Abort bad builds and identify the
cause
What are the best practices?

17

April 9, 2013

Seven Practices, Duvall p. 39
• Commit code frequently
• Don't commit broken code
• Fix broken builds immediately
• Write automated developer tests
• All tests & inspections must pass
• Run private builds
• Avoid getting broken code

18

April 9, 2013

Chicago Board Options Exchange
• Planned systems upgrade
• Problem with staging software
• Employees reported there was a
problem
• CBOE did not fail over
Could DevOps have helped the CBOE?

19

April 9, 2013

What is DevOps?
• New Term for...
• Portmanteau
• Agile Systems Administration
• Agile Operations
• Group of Principles
Now that we cleared that up!

20

April 9, 2013

New Term
• Group of concepts
• Been around for a while
• Use case is compelling
• Stimulating discussion
• Necessary to meet demand


21

April 9, 2013

Portmanteau
• Combination of two words
• Development
• Operations
Development and Operations have
very different goals

22

April 9, 2013

Conflict Between Dev & Ops
• Development focused on delivering new
functionality
• Operations is focused on providing
continuous (reliable) services
• Manage risk!
One time I was asked to break the rules

23

April 9, 2013

Trying to make the deadline
• Trading system was tested and
passed
• Few bugs discovered
• I was asked to deliver a different
version than was tested
How does DevOps help balance?

24

April 9, 2013

DevOps is also
• Emerging Best Practices
• Collaboration between Dev & Ops
• Application and Systems Deployment
• Software and Systems Development
But is DevOps Agile?

25

April 9, 2013

What about Agile?
• Agile Systems Administration
• Agile Operations
• Waterfall needs DevOps too!
Release Antipatterns...

26

April 9, 2013

Release Antipatterns
Deploying software manually

Deploying to a production-like
environment only after development is
complete

Manual configuration of production
environment.


So what really is DevOps?

27

April 9, 2013

DevOps is Really...
• Developer and Operations
collaboration
• Crossfunctional team
• Knowledge Management
• Better communication
Time to get rid of silos

28

April 9, 2013

What is Ops?
• Blanket term
• Systems engineers
• Systems administrators
• Operations staff
What's honesty got to do with all this?

29

April 9, 2013

Agile on What We Know
• Don't try to define requirements we do
not yet understand
• Last responsible moment
• Requirements documents that are
unusable
Deming says, “drive out fear”

30

April 9, 2013

What is a Deployment Pipeline?
• Build once
• Deploy the same way to every
environment
• Smoke is essential
• Deploy to a copy of Production
• Manage the pipeline
Environment management

31

April 9, 2013

Deployment Pipeline
A deployment pipeline is … an
automated implementation of your
application’s build, deploy, test and
release process
Jez Humble and David Farley’s
Continuous Delivery, p 3.

32

April 9, 2013

Aim of the Pipeline
• Makes building, deploying, testing and
releasing software visible to everyone involved
• Improves feedback so that problems are
identified, and so resolved, as early in the
process as possible
• Enables teams to deploy and release any
version of their software to any environment at
will through a fully automated process (p. 4)

33

April 9, 2013

Antipatterns
• Deploying Software Manually
• Deploying to Production-like
environment only after Development is
complete
• Manual Configuration of Production
Environments
Continuous Deployment, p. 7 – 10

34

April 9, 2013

Agile Release Train (ART)
Making each product a successful and
routine event – an event that is indeed
planned and eagerly anticipated yet one
one that happens almost on autopilot
Dean Leffingwell’s Agile Software
Requirements, p. 299

35

April 9, 2013

How Do We Implement?
Are deployment pipelines practical?
How do we figure out the details?
Is it worth the time and effort?
What are the benefits?
What are the risks?

It's really all about knowledge...

36

April 9, 2013

Knowledge Management
• There are always a few experts
• They are not working when system
glitch
• Building a Knowledge Management
System
• But What if some people do not want
to share? (caution silos ahead)

37

April 9, 2013

Beware of Silos
• The SAs see file systems
• The DBAs have a different view
• WebSphere Admins
• InfoSec helped us secure (so much
nothing worked)
DevOps is about sharing knowledge!

38

April 9, 2013

Where did DevOps start?
• O'Reilly Velocity Conference 2008
• Web Performance and Operations
• “Infrastructure as Code”


39

April 9, 2013

Need for Rapid Change
• 2009 Presentations on developer /
operations collaboration at large shops
along with safe rapid change of Web
environments


40

April 9, 2013

DevOps Days
• Patrick Debois – DevOpsDays in
2009
• Tools (actually toolchains) have
brought together “the three layers of
what you need for agile movement
(principles, process and practices)”

41

April 9, 2013

Let's Get Into Some Details
• How do we implement?
• How do we make pragmatic choices?
• How do we do this in the real world?
My experience taking down NYSE

42

April 9, 2013

I Was Once Accused
• Promoting the wrong shell scripts
• Taking down the NYSE
• Stopping the World Economy
Principles of Software Delivery

43

April 9, 2013

Continuous Delivery
• Configuration Management focus
• Version control
• Dependency and configuration control
Principles of Software Delivery

44

April 9, 2013

Software Delivery Principles
• Create repeatable, reliable process
• Automate as much as possible
• If it hurts, do it more often!
• Build quality in from the beginning
(Deming)
any more?

45

April 9, 2013

More Principles
• Done means released
• Everyone is responsible for the
delivery process
• Continuous improvement
• Version control is key
So what is CM?

46

April 9, 2013

Configuration Management
• Configuration Identification
• Status Accounting
• Change Control
• Configuration Audit
Tracking and Controlling Changes to
Configuration Items

47

April 9, 2013

Configuration Identification
• Provides a specific and unique
identity to each configuration item (e.g.
binary, config file, documentation)
• Selecting the configuration items for a
system and recording their functional
and physical characteristics (Sevocab)

48

April 9, 2013

Status Accounting
• Tracking the status of a configuration
item throughout its lifecycle.
• Recording and reporting of
information needed to manage a
configuration effectively (Sevocab)

49

April 9, 2013

Change Control
• Establishing checkpoints including
gatekeeping (e.g. Production, QA,
UAT) and configuration control.
• Identifying, documenting, approving or
rejecting, and controlling changes to
the project baselines (Sevocab)

50

April 9, 2013

Configuration Audit
• Inspect and identify the exact version
of any configuration item (physical &
functional)
• Independent examination of the
configuration status to compare with
the physical configuration (Sevocab)

51

April 9, 2013

Functional description of CM
• Easier to understand in the context of
a lifecycle
• Consisting of six core CM functions
• Closely matches the job descriptions
of the people doing the work
• Can be tailored to your needs
So what are the six functions?

52

April 9, 2013

CM Functions
• Source Code Management
• Build Engineering
• Environment Configuration
• Change Control
• Release Engineering
• Deployment
Let's start with a brief overview

53

April 9, 2013

My buddy from Harvard
• Builds are too complex to automate
• Some folks do not want to see
automation as being possible
• You may have to shadow or ask to
drive
• Document the procedures and then
Script your build...

54

April 9, 2013

Build Principles
• Create a script for each stage of the
build process
• Use the right technology to deploy
(find out what others are doing)
• Use your operating systems native
tools


55

April 9, 2013

More Build Principles
• Idempotent – reliable and no side
effects
• Evolve your deployment system
incrementally
• Start with “attended automation”


56

April 9, 2013

Some other tips
• Relative paths (watch your paths)
• Eliminate manual steps
• Traceability from binary to source
• Test targets should not fail the build
What do I do with binaries?


57

April 9, 2013

Managing Binaries
• Binaries can be rebuilt
• Based upon baselines
• Verifiable (I hope)
• Don't belong in the VCS with source
• Definitive Media Libraries
• Release Repos
Managing Variants in the Code

58

April 9, 2013

Version Control Features
• Provides history and security
• Model the architecture
• Reduce complexity
• Model the process
More on streams

59

April 9, 2013

Source Code Management

• Control of every configuration item
(e.g. source code, config, binaries,
compile and runtime dependencies).
• Much more than just checkin and
checkout (version control)
• Provides sanity to the development
process (reduces cognitive complexity)

60

April 9, 2013

Terminology
• Configuration items (CIs) include
binaries, source code, config files and
even documents
• ISO 1007 notes end user function
• Bob says, “anything where getting the
wrong version would be bad”

61

April 9, 2013

What is Control?
• In CM, control is managing the
evolution of a CI throughout its lifecycle
• Change Control
• Configuration Control
Is control really the right word?

62

April 9, 2013

Principles
• Code is locked down and can never
be lost
• Code is baselined marking specific
milestones
• Managing variants using branches
• Code changed on a branch can be
merged

63

April 9, 2013

More Principles
• Processes are repeatable Agile and
Lean
• Traceability and tracking of all
changes
• Improves productivity and quality


64

April 9, 2013

Best Practices
• How do we establish source code
management that adheres to these
principles?
• Better question is how does CM add
value and help facilitate the
development effort?


65

April 9, 2013

Streams
• Provides a clear usage paradigm
• Model components and architecture
• Control flow of changesets
• Snapshots create baseline of code


66

April 9, 2013

Streams
Ability to load a particular snapshot

Strong security authorization and
entitlements

Complete history and traceability


How about task based development?


67

April 9, 2013

Defect & Task Tracking
• Track changesets to workitem
• Traceability to who made the change
• Makes release notes a breeze to
create
• Ties back to requirements and test
cases
• Allows for ALM and workflow
automation

68

April 9, 2013

InfoSec Scans Code
• Source Code Inspection
• Are coding practices creating risk?
• Are passwords being hardcoded
• Scan for complexity
• Code quality
Managing globally distributed teams

69

April 9, 2013

Globally Distributed team
• Managing work for a globally
distributed team
• Effective communication
• Better coordination
• Traceability
• Visibility


70

April 9, 2013

Build on Commit
• Nightly builds often enough
• Build on demand
• Pre-flight (private) builds
• Build framework
Tame the complexity and communicate
via dashboards

71

April 9, 2013

Deploy to Environment
• Run automated tests
• Monitor the environment
• Build the Ops Knowledgebase
• Building our deployment framework
Infrastructure as code

72

April 9, 2013

Infrastructure as Code
• Provisioning Servers
• Fundamental in the Cloud
• What about private clouds?
• Managing the OS
Puppet and chef

73

April 9, 2013

Puppet/Chef
• Automate provisioning, patching and
configuration of operating system and
application components
• Systems integration framework
• Scalable and extensible
• Used in other deployment frameworks
www.puppetlabs.com www.opscode.com

74

April 9, 2013

CIS Benchmark
• Center for Internet Security (CIS)
• Consists of hundreds of
recommended configurations
• Code is included to verify the
configuration
This is all about taming complexity

75

April 9, 2013

Taming Complexity
• Understand the technology
• Automate everything
• Do it more often
• Move upstream
• Build a framework
By the time we get to Production...

76

April 9, 2013

Build Once
• Build once – deploy everywhere
• Ensure bits are identical
• Build based upon baseline
• Embed immutable version IDs
• Configuration audit
Automated deployments

77

April 9, 2013

Deployment Frameworks
• Starts with scripting
• Many dependencies
• Taming complexities
• Test each step
Traceable, Repeatable Process

78

April 9, 2013

Deploy the Same Every
Environment
• Write a deployment framework
• Practice the deploy
• Well oiled machine
• Repeatable and traceable
DevOps Focus

79

April 9, 2013

DevOps
• Moving automation upstream
• Communicating with stakeholders
• Building knowledge
• Infrastructure as code
Smoke testing is required

80

April 9, 2013

SmokeTest
• Test the deploy itself
• Put in the first trade
• Verify what changed
• Work with QA & Testing
Environments need to be similar to
Production

81

April 9, 2013

Deploy into Copy of Production
• You need a dress rehearsal
• Need to verify automation works
• Need to know the deploy will work
• Manage risks and unknowns
Deploys need to be verifiable

82

April 9, 2013

Changes Through the Pipeline
• Every commit triggers
• Build and deploy automation
• Testing the release
What are the recommended practices?

83

April 9, 2013

Pipeline Practices
• Only build binaries once
• Deploy the same way to every
environment
• Smoke test
Changes should propagate instantly
continuously
The process itself must be testable

84

April 9, 2013

Verification and Validation
• Does it meets requirements?
• Are the requirements correct?
• Deming – build in quality
• Each step is testable
When problems occur...

85

April 9, 2013

Stop the Line
• Need to detect defects early
• Stop the process immediately
• Easier to diagnose
• Easier to fix
Kanban for the Deployment Pipeline

86

April 9, 2013

Kanban
• Push
• Pull
• Implement through workflow
automation
Delivery Environment as an Ecosystem

87

April 9, 2013

Delivery Ecosystem
• Understanding Operations
• Managing Infrastructure
• Server Provisioning & Configuration
• Managing Middleware
• Virtualization & Cloud
DevOps should focus on Ops

88

April 9, 2013

Operations
• Key stakeholder
• Often outgunned and kept in the dark
• Building a knowledgebase
• Automating detection and response
How can Ops get ahead of the curve?

89

April 9, 2013

Operations in DevOps
• Infrastructure as Code
• Provisioning servers
• Monitoring the environment
• Monitoring events
InfoSec is also key

90

April 9, 2013

InfoSec
• Key stakeholder
• Often misinformed
• Policies don't secure systems
• Many incidents show this is a problem
area
Securing the trusted base

91

April 9, 2013

Securing the Trusted Base
• Builds are baselined
• Version IDs are embedded
• Configuration audit
• Non-repudiation
Security is quality

92

April 9, 2013

Build Security In
• Security should be considered from the
beginning
• Security and quality are tightly coupled
• Provision servers using standards
• Control & Detect unauthorized changes
Manage components & dependencies

93

April 9, 2013

Managing Components
• Code should be designed into
components
• Reduces complexity
• Interfaces are essential
• Part of environment management
Managing Big Builds

94

April 9, 2013

Managing Big Builds
• Big builds may require multiple
pipelines
• Treat the team as internal products
• Handle this as COTs
Configuration can be complex

95

April 9, 2013

Managing Configuration
• Many ways to handle this
• Configuration files (httpd.conf)
• Properties files (.properties)
• XML as configuration (server.xml)
• Default as production (so you don't
forget!)
Managing Dependencies

96

April 9, 2013

Managing Dependencies
• Maven and Ivy help identify
dependencies
• Need to be able to identify versions
• Monitor and detect issues
• Often controlled through data
But you have to test

97

April 9, 2013

Testing Topology
• Unit Testing
• Functional
• Regression
• Integration
• User Acceptance
What about non functional testing?

98

April 9, 2013

Non-functional Testing
• Capacity
• Performance
• Scalability

Shoemakers children...

99

April 9, 2013

Testing the Pipeline
• You need to test the automation
including build, package and
deployment
• Fail early!
• Trust but verify
Don't forget the data

100

April 9, 2013

Internal Audit Requirements
• Managing baselines
• Traceability
• Change control
• Seperation of controls
Regulatory Requirements

101

April 9, 2013

Conducting an Assessment
• What is going well
• What can be improved?
Assess to industry standards and
frameworks


102

April 9, 2013

Regulatory
• Section 404 of the Sarbanes-Oxley
Act of 2002
• SSAE-16 (formerly SAS-70)
• Finra
• Office of the Currency
Standards and Frameworks

103

April 9, 2013

Industry Standards
• IEEE 828
• EIA 649-B
• ISO 12207 or 15288
• ISO 9001
Frameworks also provide guidance

104

April 9, 2013

Frameworks
• Cobit for Sox Compliance
• ITIL for IT Service Management
• CMMI (less common in financial
services)


105

April 9, 2013

Globally Distributed team
• Managing work for a globally
distributed team
• Effective communication
• Better coordination
• Traceability
• Visibility


106

April 9, 2013

The CD/CI/CM Process
• Should be Lean
• Processes need to be reviewed
• Tailor down or tailor up
• More collaboration and consensus
building
• Use standards and frameworks


107

April 9, 2013

Assessment
• First step is to assess current
practices - “As-Is”
• Compare to industry standards and
frameworks
• Determine “To-Be”
• Create a plan for improving your CM
processes

108

April 9, 2013

IT Governance & Compliance
• IT Governance needs to be in
alignment with corporate governance
• Financial reports needs to be
accurate
• Separation of controls
• Security measures to prevent
unauthorized access
• Audit in place for intrusion detection

109

April 9, 2013

Sox Compliance
• Section 404 of the Sarbanes Oxley
Act of 2002
• Using ISACA Cobit 4.1
• 34 high level IT controls
• PCI compliance
• SAS-70


110

April 9, 2013

ISO 9001
• Establishes the quality management
system
• ISO 90003 is the software standard in
the 9000 family of standards
• Uses ISO 12207 (or 15288) to specify
lifecycle processes
• ISO 10007 for CM
• IEEE 828, EIA 649-A, Mil Std coming!

111

April 9, 2013

Which Standards?
• IEEE 828 – CM Planning
• EIA 649-A – Non compliance
• ISO 90003 to support QMS
• Full lifecycle ISO 12207
Tailor !


112

April 9, 2013

Moving Upstream
• Dev to CM to QA to Ops
• Cross functional focus
• Speed up development
• Build a great deployment architecture
• Give it to Devs as a service!


113

April 9, 2013

Frameworks
• ITIL v3 including CMDBs, federated
CMDBs, CMS, DML…
• Cobit for SOX
• CMMI ->>>> Agile


114

April 9, 2013

How Do We Improve
• CSI is well - continuous
• Inclusive
• Transparent
• Learning from mistakes
Retrospectives are essential

115

April 9, 2013

Retrospective
• After action review
• Need open and honest evaluation
• Opportunity to improve the process
• Drives the entire release process


116

April 9, 2013

Plan for Improvement
• Improve training and use case for
source code management
• Improvement build automation
• Setup or improve continuous
integration
• Automate package and deployment
• Create procedures for configuration
audit

117

April 9, 2013

CM/Devops
• Flexible technical background
• Good knowledge of development
• Knowledge of QA/Ops
• Strong automation skills
• Some systems administration
• Ability to work across silos


118

April 9, 2013

Toolsmith/Devops
• Strong technical background
• Strong scripting skills
• Diving deep into the tools including
troubleshooting
• Understands toolchains and finds
flexible solutions
• Process orientation – focus on
traceability

119

April 9, 2013

Goals of this Course
• Understand Continuous Delivery
• Configuration Management roots
• Control Dependencies & Configuration
• Continuous Integration
• Build and Deployment Automation
• Deployment Pipeline is an Art!
Agile Release Train

120

April 9, 2013

DevOps Focus
• Understand DevOps Best Practices
• A Little History of DevOps
• Scope of DevOps and how to get started
• The People side of DevOps
Establish your own plan for DevOps!

121

April 9, 2013

And Don't Forget
• Delivery Ecosystem
• Components & Dependencies
• Test vs Verification & Validation (V&V)
• Don't forget the Data
• Establish IT governance and compliance
So what is Continuous Delivery?

122

April 9, 2013

Continuous Delivery
• Methodology for getting software from
development to release
• Focus on the Deployment Pipeline
• Rapid incremental deployment
• Minimize Risk
• Many small deployments better than
big bang

123

April 9, 2013

Continuous Delivery (DevOps Best Practices)
Bob Aiello, Principal Consultant and Author of

Configuration Management Best Practices :

Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiello
http://cmbestpractices.com

124

CM Best Practices Consulting © 2013

Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Ähnlich wie Continuous Delivery: Rapid and Reliable Releases with DevOps Practices (20)

Mehr von TechWell

Mehr von TechWell (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Continuous Delivery: Rapid and Reliable Releases with DevOps Practices