Immutable infrastructure and Toyota Supply Chain.

1. Immutable Awesomeness?
Where Containers Collide
with SW Supply Chains
John Willis - @botchagalupe
Joshua Corman - @joshcorman
#dockercon

2. Devops and Immutable
Infrastructure
John Wills
Director of Ecosystem Development
IMMUTABLE

3. @botchagalupe
• a.k.a. John Willis
• 35 Years in IT Operations
• Exxon, Canonical, Chef, Enstratius,
Socketplane, Docker
• Director of Ecosystem Development at
Docker
• Devopsdays Core Organizer
• Devopscafe on iTunes
• Devops Enterprise Summit Core Organizer

4. Devops
Devops is a movement
motivated to turn human
capital into high
performance
organizational capital.

5. @joshcorman
• 20 Years in SW & Security
• IBM ISS, The 451 Group,
Akamai, Sonatype
• Founder, Rugged Software
• Founder, I Am the Cavalry
• Adjunct Professor, Carnegie
Mellon University Heinz College

7. h/t$@petecheslock$DevOpsDays$Aus4n$2015$

12. Beyond Heartbleed: OpenSSL in 2014
(31 in NIST’s NVD thru December)
CVE-2014-3470 6/5/2014 CVSS Severity: 4.3 MEDIUM ! SIEMENS *
CVE-2014-0224 6/5/2014 CVSS Severity: 6.8 MEDIUM ! SIEMENS *
CVE-2014-0221 6/5/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0195 6/5/2014 CVSS Severity: 6.8 MEDIUM
CVE-2014-0198 5/6/2014 CVSS Severity: 4.3 MEDIUM ! SIEMENS *
CVE-2013-7373 4/29/2014 CVSS Severity: 7.5 HIGH
CVE-2014-2734 4/24/2014 CVSS Severity: 5.8 MEDIUM ** DISPUTED **
CVE-2014-0139 4/15/2014 CVSS Severity: 5.8 MEDIUM
CVE-2010-5298 4/14/2014 CVSS Severity: 4.0 MEDIUM
CVE-2014-0160 4/7/2014 CVSS Severity: 5.0 MEDIUM ! HeartBleed
CVE-2014-0076 3/25/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0016 3/24/2014 CVSS Severity: 4.3 MEDIUM
CVE-2014-0017 3/14/2014 CVSS Severity: 1.9 LOW
CVE-2014-2234 3/5/2014 CVSS Severity: 6.4 MEDIUM
CVE-2013-7295 1/17/2014 CVSS Severity: 4.0 MEDIUM
CVE-2013-4353 1/8/2014 CVSS Severity: 4.3 MEDIUM
CVE-2013-6450 1/1/2014 CVSS Severity: 5.8 MEDIUM
…
As#of#today,#internet#scans#
by#MassScan##reveal#300,000#
of#original#600,000#remain#
unpatched#or#unpatchable#

15. Product Vulnerability Disclosures Following the HeartBleed Announcement (Circle Size Indicates CVSS Severity Score)
F5
New OpenSSL Disclosures (Both CVSS Level 10)
Here
IBM
Cisco
IBM
McAfee
Initial 'HeartBleed' OpenSSL Disclosure (CVSS Level 5 (underscored))
NumberofProductsIncludedinAnnouncement
0
10
20
30
40
50
60
70
80
90
100
110
120
Days Since HeartBeed Announcement
0 10 20 30 40 50 60 70 80 90 100 110 120
!!
X!Axis:!!Time!(Days)!following!ini6al!HeartBleed!disclosure!and!patch!availability!
Y!Axis:!!Number!of!products!included!in!the!vendor!vulnerability!disclosure!
Z!Axis!(circle!size):!!Exposure!as!measured!by!the!CVE!CVSS!score!
!
COMMERCIAL!RESPONSES!TO!OPENSSL!

17. Actual Exploitation 2015 VZ DBIR

18. Quality?)
Security?)
Maintainability?)
Repeatability?)
Raw)innova6on))
Innova&on'at''
any'cost'
Net)innova6on))
Net'value'to'the'
organiza&on'

20. Supply&chain&advantage&
Source:(Toyota(Supply(Chain(
Management:(A(Strategic(
Approach(to(Toyota’s(
Renowned(System,(by(Ananth(
Iyer(and(Sridhar(Seshadri(
Toyota&
Advantage&
Toyota&
Prius&
Chevy&
Volt&
Unit%Retail%Price% 61%& $24,200% $39,900%
Units%Sold/Month% 13x& 23,294% 1,788%
In?House%ProducBon% 50%& 27%% 54%%
Plant%Suppliers% 16%&& 125% 800%
Firm@Wide(Suppliers( 4%# 224( 5,500(

21. Use their highest
quality parts
Use fewer, better
suppliers
Track which parts
you use & where

22. Demo?
#DOES15

26. @bglpe

27. @bglpe
Immutable Infrastructure

28. @bglpe
Immutable
Infrastructure
Myth

29. @bglpe
“The least-cost way to ensure that the behavior of any
two hosts will remain completely identical is always to
implement the same changes in the same order on both
hosts.”
Order Matters

30. @bglpe
Management Methods
• Divergence
• Convergence
• Congruence

31. @bglpe
Why (When) Does Order
Matter?
• Circular Dependancies
• Right Command Wrong Order
• Right Package Wrong Order

32. @bglpe
Package Example

33. @bglpe
Immutable Infrastructure
“Model”
• No CRUD allowed for…
• Packages
• Configuration Files
• Application Software
• Data (RUD)

34. @bglpe
How To Do Immutable?
• Provision a new server.
• Test the new server.
• Change the reference to the new server.
• Keep the old server around for rollback.

35. @bglpe
The Immutable “Trombone”
• Golden Images
• Virtual Desktop Infrastructure (VDI)
• Virtual Images
• Phoenix Servers vs Snowflake Servers
• Infrastructure as Code
• Bake vs Fry
• Containers

36. Immutable Delivery

37. Immutable Delivery

39. V4L Principles
• Variety
• Determine your variety of offerings
based on operational efficiency and
market demand
• Velocity
• Maintain a steady flow through all
processes of the supply chain
• Variability
• Manage inconsistencies carefully to
reduce cost and improve quality
• Visibility
• Ensure the transparency of all
processes to enable continuous
learning and improvement

40. Left to Right Flow
•Variety
• Lean Startup
• Minimal Viable Product
• Pivot
• Build Measure Learn
• Customer Development
Methodology

41. Left to Right Flow
• Velocity
• Developer Flow
• Integration Flow
• Deployment Flow
https://upload.wikimedia.org/wikipedia/commons/7/74/Continuous_Delivery_process_diagram.png

42. Left to Right Flow
• Variation
• Converged Isolation
• Immutable Infrastructure
• Immutable Delivery
https://en.wikipedia.org/wiki/Standard_deviation

43. Left to Right Flow
• Visibility
• Containerization
• Microservices
• Data Gravity

44. @bglpe
Visibility
• Where and when was it built and why
• What was its ancestor images
• How do I start, validate, monitor and update it
• What git repo is being built, what hash of that git repo was built
• What are all the tags this specific container is known as at time
of build
• What’s the project name this belongs to
• Have the ability to have arbitrary user supplied rich metadata

45. Immutable Infrastructure

47. @joshcorman
6 - Personel best
10 - When he arrived
4 - Basic supply chain hygiene
1 - Fewer suppliers
0.1 - Docker and Immutable Delivery

51. References
DOCKER AND THE THREE WAYS OF DEVOPS PART 1: THE FIRST WAY – SYSTEMS THINKING
https://blog.docker.com/2015/05/docker-three-ways-devops/
DevOpsDays Chicago Sept 2015 - State of the DevOps by John Willis
https://www.youtube.com/watch?t=16&v=319wIaAiaHM
Guns Germs and Microservices
https://vimeo.com/129822162
Become More Agile and Get Ready for DevOps by Using Docker in Your Continuous Integration
Environments
https://www.gartner.com/doc/3016317/agile-ready-devops-using-docker
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
Immutable Infrastructure with Docker and EC2 by Michael Bryzek (Gilt)
https://www.youtube.com/watch?v=GaHzdqFithc
Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results
http://www.amazon.com/Toyota-Kata-Managing-Improvement-Adaptiveness/dp/0071635238