This presentation will show the combination of two ideas that can create 2 to 3 order of magnitude efficiencies in service delivery. We will discuss an example used in an insurance company that has experienced these efficiencies. Josh Corman will present the concept of using Open Source and Toyota Supply Chain principles as a weapon for eliminating operational costs of service delivery. By applying first order principles like fewer suppliers (e.g, less logging frameworks) and image manifests (i.e., bill of materials) he will show how an organization can cut down on bugs and issue resolution times. John Willis will then cover how these principles fit like peanut butter and chocolate when used in an immutable delivery model based on Docker. This presentation was the third highest rated session at the 2015 Devops Enterprise Summit.

2. Immutable Awesomeness?
Where Containers Collide
with SW Supply Chains
John Willis - @botchagalupe
Joshua Corman - @joshcorman
#dockercon

3. Devops and Immutable
Infrastructure
John Wills
Director of Ecosystem Development
IMMUTABLE

4. @botchagalupe
• a.k.a. John Willis
• 35 Years in IT Operations
• Exxon, Canonical, Chef, Enstratius,
Socketplane, Docker
• Director of Ecosystem Development at
Docker
• Devopsdays Core Organizer
• Devopscafe on iTunes
• Devops Enterprise Summit Core Organizer

5. Devops
Devops is a movement
motivated to turn human
capital into high
performance
organizational capital.

6. @joshcorman
• 20 Years in SW & Security
• IBM ISS, The 451 Group,
Akamai, Sonatype
• Founder, Rugged Software
• Founder, I Am the Cavalry
• Adjunct Professor, Carnegie
Mellon University Heinz College

8. h/t$@petecheslock$DevOpsDays$Aus4n$2015$

13. Beyond Heartbleed: OpenSSL in 2014
(31 in NIST’s NVD thru December)
CVE-2014-3470 6/5/2014 CVSS Severity: 4.3 MEDIUM ! SIEMENS *
CVE-2014-0224 6/5/2014 CVSS Severity: 6.8 MEDIUM ! SIEMENS *
CVE-2014-0221 6/5/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0195 6/5/2014 CVSS Severity: 6.8 MEDIUM
CVE-2014-0198 5/6/2014 CVSS Severity: 4.3 MEDIUM ! SIEMENS *
CVE-2013-7373 4/29/2014 CVSS Severity: 7.5 HIGH
CVE-2014-2734 4/24/2014 CVSS Severity: 5.8 MEDIUM ** DISPUTED **
CVE-2014-0139 4/15/2014 CVSS Severity: 5.8 MEDIUM
CVE-2010-5298 4/14/2014 CVSS Severity: 4.0 MEDIUM
CVE-2014-0160 4/7/2014 CVSS Severity: 5.0 MEDIUM ! HeartBleed
CVE-2014-0076 3/25/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0016 3/24/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0017 3/14/2014 CVSS Severity: 1.9 LOW
CVE-2014-2234 3/5/2014 CVSS Severity: 6.4 MEDIUM
CVE-2013-7295 1/17/2014 CVSS Severity: 4.0 MEDIUM
CVE-2013-4353 1/8/2014 CVSS Severity: 4.3 MEDIUM
CVE-2013-6450 1/1/2014 CVSS Severity: 5.8 MEDIUM
…
As#of#today,#internet#scans#
by#MassScan##reveal#300,000#
of#original#600,000#remain#
unpatched#or#unpatchable#

16. Product Vulnerability Disclosures Following the HeartBleed Announcement (Circle Size Indicates CVSS Severity Score)
F5
New OpenSSL Disclosures (Both CVSS Level 10)
Here
IBM
Cisco
IBM
McAfee
Initial 'HeartBleed' OpenSSL Disclosure (CVSS Level 5 (underscored))
NumberofProductsIncludedinAnnouncement
0
10
20
30
40
50
60
70
80
90
100
110
120
Days Since HeartBeed Announcement
0 10 20 30 40 50 60 70 80 90 100 110 120
!!
X!Axis:!!Time!(Days)!following!ini6al!HeartBleed!disclosure!and!patch!availability!
Y!Axis:!!Number!of!products!included!in!the!vendor!vulnerability!disclosure!
Z!Axis!(circle!size):!!Exposure!as!measured!by!the!CVE!CVSS!score!
!
COMMERCIAL!RESPONSES!TO!OPENSSL!

18. Actual Exploitation 2015 VZ DBIR

19. Quality?)
Security?)
Maintainability?)
Repeatability?)
Raw)innova6on))
Innova&on'at''
any'cost'
Net)innova6on))
Net'value'to'the'
organiza&on'

21. Supply&chain&advantage&
Source:(Toyota(Supply(Chain(
Management:(A(Strategic(
Approach(to(Toyota’s(
Renowned(System,(by(Ananth(
Iyer(and(Sridhar(Seshadri(
Toyota&
Advantage&
Toyota&
Prius&
Chevy&
Volt&
Unit%Retail%Price% 61%& $24,200% $39,900%
Units%Sold/Month% 13x& 23,294% 1,788%
In?House%ProducBon% 50%& 27%% 54%%
Plant%Suppliers% 16%&& 125% 800%
Firm@Wide(Suppliers( 4%# 224( 5,500(

22. Use their highest
quality parts
Use fewer, better
suppliers
Track which parts
you use & where

23. Demo?
#DOES15

27. @bglpe

28. @bglpe
Immutable Infrastructure

29. @bglpe
Immutable
Infrastructure
Myth

30. @bglpe
“The least-cost way to ensure that the behavior of any
two hosts will remain completely identical is always to
implement the same changes in the same order on both
hosts.”
Order Matters

31. @bglpe
Management Methods
• Divergence
• Convergence
• Congruence

32. @bglpe
Why (When) Does Order
Matter?
• Circular Dependancies
• Right Command Wrong Order
• Right Package Wrong Order

33. @bglpe
Package Example

34. @bglpe
Immutable Infrastructure
“Model”
• No CRUD allowed for…
• Packages
• Configuration Files
• Application Software
• Data (RUD)

35. @bglpe
How To Do Immutable?
• Provision a new server.
• Test the new server.
• Change the reference to the new server.
• Keep the old server around for rollback.

36. @bglpe
The Immutable “Trombone”
• Golden Images
• Virtual Desktop Infrastructure (VDI)
• Virtual Images
• Phoenix Servers vs Snowflake Servers
• Infrastructure as Code
• Bake vs Fry
• Containers

37. Immutable Delivery

38. Immutable Delivery

40. V4L Principles
• Variety
• Determine your variety of offerings
based on operational efficiency and
market demand
• Velocity
• Maintain a steady flow through all
processes of the supply chain
• Variability
• Manage inconsistencies carefully to
reduce cost and improve quality
• Visibility
• Ensure the transparency of all
processes to enable continuous
learning and improvement

41. Left to Right Flow
•Variety
• Lean Startup
• Minimal Viable Product
• Pivot
• Build Measure Learn
• Customer Development
Methodology

42. Left to Right Flow
• Velocity
• Developer Flow
• Integration Flow
• Deployment Flow
https://upload.wikimedia.org/wikipedia/commons/7/74/Continuous_Delivery_process_diagram.png

43. Left to Right Flow
• Variation
• Converged Isolation
• Immutable Infrastructure
• Immutable Delivery
https://en.wikipedia.org/wiki/Standard_deviation

44. Left to Right Flow
• Visibility
• Containerization
• Microservices
• Data Gravity

45. @bglpe
Visibility
• Where and when was it built and why
• What was its ancestor images
• How do I start, validate, monitor and update it
• What git repo is being built, what hash of that git repo was built
• What are all the tags this specific container is known as at time
of build
• What’s the project name this belongs to
• Have the ability to have arbitrary user supplied rich metadata

46. Immutable Infrastructure

48. @joshcorman
6 - Personel best
10 - When he arrived
4 - Basic supply chain hygiene
1 - Fewer suppliers
0.1 - Docker and Immutable Delivery

52. References
DOCKER AND THE THREE WAYS OF DEVOPS PART 1: THE FIRST WAY – SYSTEMS THINKING
https://blog.docker.com/2015/05/docker-three-ways-devops/
DevOpsDays Chicago Sept 2015 - State of the DevOps by John Willis
https://www.youtube.com/watch?t=16&v=319wIaAiaHM
Guns Germs and Microservices
https://vimeo.com/129822162
Become More Agile and Get Ready for DevOps by Using Docker in Your Continuous Integration
Environments
https://www.gartner.com/doc/3016317/agile-ready-devops-using-docker
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
Immutable Infrastructure with Docker and EC2 by Michael Bryzek (Gilt)
https://www.youtube.com/watch?v=GaHzdqFithc
Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results
http://www.amazon.com/Toyota-Kata-Managing-Improvement-Adaptiveness/dp/0071635238