2. Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
CONFIDENTIAL 2
5. Hardware
OS Kernel
OS File system
Userspace
Container
Appprocess
Appprocess
Appprocess
Appprocess
Appprocess
Container
Appprocess
Appprocess
Linux Containers
55
OS-level Isolation
• Isolation at individual kernel subsystem
level (e.g. filesystem, process table, etc)
• User-level process (LXC, libcontainer)
orchestrates these subsystems to create
a container
Existed for Many Years
• Solaris Zones, FreeBSD Jails, OpenVZ
Why?
• Process isolation
• Reproducible environment
• Enables management at scale
6. The Problem in 2014
Sta$c
website
Web
frontend
User
DB
Queue
Analy$cs
DB
Background
workers
API
endpoint
nginx
1.5
+
modsecurity
+
openssl
+
bootstrap
2
postgresql
+
pgv8
+
v8
hadoop
+
hive
+
thriJ
+
OpenJDK
Ruby
+
Rails
+
sass
+
Unicorn
Redis
+
redis-‐sen$nel
Python
3.0
+
celery
+
pyredis
+
libcurl
+
ffmpeg
+
libopencv
+
nodejs
+
phantomjs
Python
2.7
+
Flask
+
pyredis
+
celery
+
psycopg
+
postgresql-‐client
Development
VM
QA
Server
Public
Cloud
Disaster
Recovery
Contributor’s
Laptop
Produc$on
Servers
Multiplicity
of Stacks
Multiplicity
of hardware
environments
Produc$on
VM
Cluster
Customer
Data
Center
Do services
and apps
interact
appropriately
?
Can I
migrate
smoothly
and quickly?
66
7. Let’s create a shipping container system for
applications
Multiplicity
of Stacks
Multiplicity
of hardware
environments
Do services
and apps
interact
appropriately
?
Can I
migrate
smoothly
and quickly?
Sta$c
website
Web
frontend
User
DB
Queue
Analy$cs
DB
Development
VM
QA
Server
Public
Cloud
Contributor’s
Laptop
Produc$on
VM
Cluster
Customer
Data
Center
An engine that enables any
payload to be encapsulated
as a lightweight, portable,
self-sufficient container…
…that can be manipulated
using standard operations and
run consistently on virtually
any hardware platform
77
8. Container Fits Well with DevOps Lifecycle
8
Development
Package &
Repository
Test Automation
Integrated Dev.
Env.
Continuous
Integration
UAT
Continuous Delivery Platform
ProductionSys. Int. Test
Code Dev &
Check-in
Build, Integration
and Testing
Repository
Mgmt
Deployment &
Testing
Promotion &
Governance
Production
Deployment
Build &
Integration
9. is a “Shipping Container” for Code
9
Ops ♥ Consistent operations on code
Uniform start, stop, logging, monitoring
Devs ♥ Consistent environment
OS, libs, layering on other containers
9
10. ü On-premise
ü Client-server, stateful, scaleup
ü Tier 1/Converged HW
ü Classic NAS & SAN
ü Relies on infrastructure availability
ü Human-driven
The Rise of Third Platform Applications
10
ü On/Off premise
ü Elastic, stateless, scale-out
ü Commodity/disaggregated HW
ü DAS, HDFS, Object, Flash, NVM
ü Built-in application resiliency
ü API-Driven/DevOps infrastructure
11. One School of Thought: Containers or VMs?
11
VMs Containers
16. VM and Container Isolation are Better Together
16
VMs Containers
• Hardware level isolation
• Focused on security and
multi-tenancy
• 15 years in production,
battle tested
• OS level isolation
• Focused on environmental
consistency
• Emerging, still maturing
Great for security Great for reproducibility
VMs
rs
Best of both worlds
17. VMs are Lightweight and Efficient
17
Forking
Fast
Sub-second VM
provisioning time
Ready to Go
Clone a running container
in warmed up state
Efficient
Lower resource usage
through sharing
Binaries
& Libraries
App A
OS
VM Debunk the Myth
• VM overhead < 5%
• VM is lightweight
• OS tends to be heavier
Looking ahead
• Thinner OS emerging
• Project Fargo
18. Containers & VMware NSX
• Unified operational model for
VMs & containers
• Programmable, datacenter-wide
connectivity
• Enterprise-grade security with
micro-segmentation.
• Native Open vSwitch support
for containers
18
Any Application
(without modification)
Virtual Networks
VMware NSX Network Virtualization Platform
Logical L2
Any Network Hardware
Any Cloud Management Platform
Logical
Firewall
Logical
Load Balancer
Logical L3
Logical
VPN
Any Hypervisor
19. Distributed and Reliable Storage for Container
HOST
Stateless
Container
HOST HOST HOST
Stateless
Container
HOST
VSAN Distribute, Reliable Storage
Snapshots, Clones, QoS, Remote Replication
Boot
Image
Boot
Image
Boot
Image
Container PROVISIONING AND MANAGEMENT • Simple data persistence
• Easy deployment of
containers on cluster
• Reliable, high
performance storage
• Tolerant of host/disk
failures
• Fast container create
leveraging snapshots and
clones in VSAN
• Quality of Service
Controls
Stateless
Container
Boot
Image
Stateless
Container
Boot
Image
20. DBsTraditional Apps
Sharing Infrastructure Efficiently
20
Container cluster
• Unified platform to run all your apps
• Dynamically allocate resource based on demands and SLA
• Strong security and performance isolation
Database cluster Traditional Apps
Silo’ed cluster leads to server/cluster sprawling, increases cost
Scenario 1: Multiple workloads Scenario 2: Multiple tenants
Containerized apps Tenant/LOB 1
Tenant 1 Tenant 2 Tenant 3
Data Center Virtualization
SDDC Platform
21. Hybrid Platform
21
vCloud Air
Data Centers
On-premises
Data Centers
Data
vCloud
Plug-in
Security
Apps, Tools, Services
Multi-
tenant
Secure Connectivity
Management
Dedicated
Software-Defined Data Center
23. Container Fits Well with Devops Lifecycle
23
Development
Package &
Repository
Test Automation
Integrated Dev.
Env.
Continuous
Integration
UAT
Continuous Delivery Platform
ProductionSys. Int. Test
Code Dev &
Check-in
Build, Integration
and Testing
Repository
Mgmt
Deployment &
Testing
Promotion &
Governance
Production
Deployment
Build &
Integration
24. Manage VMs and Containers at Scale is Key
24
ü On-premise
ü Client-server, stateful, scaleup
ü Tier 1/Converged HW
ü Classic NAS & SAN
ü Relies on infrastructure availability
ü Human-driven
ü On/Off premise
ü Elastic, stateless, scale-out
ü Commodity/disaggregated HW
ü DAS, HDFS, Object, Flash, NVM
ü Built-in application resiliency
ü API-Driven/DevOps infrastructure
Web tier
App tier
DB tier
Load
Balancer
Authentication
Session
Store
Licensing
MonitoringProvisioning
DNSContent
Database
x3
Web
Server
x3
…
25. Separation of Infrastructure and Apps Concerns
25
Developers
“Write code, not tickets”
Infrastructure Team
“IT as a service provider”
Focus
Deliver IT resources to rest of company
Challenge
Agility for devs, while maintaining control
Role
Enable rapid delivery of dev sandboxes, pre-provision
3rd Platform Services (Kubernetes, Pivotal CF, etc)
Focus
Frictionless development, rapid innovation
Challenge
Write code, without worrying about infrastructure details
Role
Self-service access to new resources (i.e. new cluster),
comply with company policies and regulations
26. Separation of Infrastructure and Apps Concerns
Developers
“Write code, not tickets”
Infrastructure Team
“IT as a service provider”
Focus
Deliver IT resources to rest of company
Challenge
Agility for devs, while maintaining control
Role
Enable rapid delivery of dev sandboxes, pre-provision
3rd Platform Services (Kubernetes, Pivotal CF, etc)
Focus
Frictionless development, rapid innovation
Challenge
Write code, without worrying about infrastructure details
Role
Self-service access to new resources (i.e. new cluster),
comply with company policies and regulations
Architecturally, makes
sense to separate
infrastructure and
app management
Infrastructure
Management
Infrequent/no access
by developers; devs
shouldn’t care
Application
Management
Lightweight, fast; call
infrastructure manager
when needed
26
27. Lifecycle: Self-service, Governance, Automation
27
Data Center Virtualization
SDDC Platform
Benefit: Common portal, catalog, permissions for developers and LOBSelf-Service
Benefit: Compliance consistently enforced across entire datacenterGovernance
Benefit: Same tools for automating traditional and new app lifecyclesAutomation
DBsTraditional Apps Containerized apps Tenant/LOB 1
28. Operations: Service Availability and Traceability
28
Hardware
vSphere, NSX, vSAN/vVOL
OS
App
Virtual HW
OS
Virtual HW
App
Lib
App
Lib
App
Lib
App
Lib
Performance
monitoring
Capacity
management
Log
management…
Instrument all
layers of stack
Inputs: Metrics
and log data
Delivering better
service levels,
availability, root
cause analysis, …
Linux OS
App
Lib
App
Lib
App
Lib
30. The Rise of Third Platform Applications
30
ü On-premise
ü Client-server, stateful, scaleup
ü Tier 1/Converged HW
ü Classic NAS & SAN
ü Relies on infrastructure availability
ü Human-driven
ü On/Off premise
ü Elastic, stateless, scale-out
ü Commodity/disaggregated HW
ü DAS, HDFS, Object, Flash, NVM
ü Built-in application resiliency
ü API-Driven/DevOps infrastructure
31. Management
vCloud Automation Center,
vCenter Operations, Log Insight
Container
Repositories
3rd Platform Apps Stack & DevOps Process
31
Container-optimized Linux
Container Packaging
Container Cluster Scheduler
App Definition, Policies, and Provisioning
Infrastructure
ESXi, NSX, Virtual SAN
vCloud Hybrid Service
Developer Production
Dev’s Laptop
Optional: Type 2 Hypervisor
Linux
Developer Tools
Container Packaging
32. Open Ecosystem: 3rd Platform Developer Stack
32
Management
vCloud Automation Center,
vCenter Operations, Log Insight
Container-optimized Linux
Container Packaging
Container Cluster Scheduler
App Definition, Policies, and Provisioning
Infrastructure
ESXi, NSX, Virtual SAN
vCloud Hybrid Service
Developer Production
Container
Repositories
Dev’s Laptop
Optional: Type 2 Hypervisor
Linux
Developer Tools
Container Packaging
E.g. Hashicorp Vagrant, Jenkins, github, etc
E.g. Docker/Docker Hub
E.g. RedHat, Ubuntu, Boot2Docker
E.g. Fusion, Workstation, Player, VirtualBox
33. Container
Repositories
Developer Production
Dev’s Laptop
Optional: Type 2 Hypervisor
Linux
Developer Tools
Container Packaging
Open Ecosystem: 3rd Platform Production Stack
E.g. Pivotal CF, Fig
Terraform, Shipyard
E.g. Kubernetes, libswarm, Mesos, Fleet
E.g. CoreOS, Atomic, Ubuntu
Management
vCloud Automation Center,
vCenter Operations, Log Insight
Container-optimized Linux
Container Packaging
Container Cluster Scheduler
App Definition, Policies, and Provisioning
Infrastructure
ESXi, NSX, Virtual SAN
vCloud Hybrid Service
E.g. Docker
33
35. Containers at Google
• Everything at Google runs in Linux application containers
• A decade of production container experience
• We start more than 2 billion a week
• Containers have changed the game
• Separation of infra and applications ops
• Increased efficiency
35
36. A few lessons learned...
1: Declarative trumps imperative
Imperative: run this container on this server
Declarative: run between 2 and 100 copies; keep latency < 2ms
Pros
• Repeatable and eventually consistent deployment and update
• Fire-and-forget app management (self scaling, self healing)
• Dynamic scheduling yields better efficiency
Cons
• Tracing action/reaction can be hard (“is it done?”)
• Diagnostics can be tough (“what happened?”)
So
• We need a cluster manager
• Strong integration with container metrics, logging, etc helps
36
37. A few things we have learned...
2: Prepare for more production services
The system known as Borg made it easier to run production services
at scale...so our engineers wrote a lot more
Pros
• Strong shift to dev and away from ops
• Radically simpler infrastructure operations
But…
• Governance gets harder as service number increases
• Managing, finding, versioning
So…
• We need a cluster manager
• It needs mechanism to deal with large numbers of services
37
38. So we created Kubernetes...
• OSS project created by Google, but owned by the
community
• Google style cluster management
• Move from static containers to dynamic management
lightweight
modular/
extensible
portable
:
38
39. And where do VMs fit in?
• Needed to run untrusted and unconstrained workloads
• Linux syscall layer is large and difficult to defend
• VMs surface can be aggressively defended
• VMware has been doing this for 15 years
• Critical for multi-tenant cloud use with untrusted tenants
• E.g. VMware vCloud Air
• In Google Cloud Platform
• VMs create ‘idealized’ infrastructure
• Containers package and run applications
• Kubernetes stitched together VMs to create a mini-Google
cluster
39
40. What is next?
• Make it work everywhere
• Operationalize
• Extend services for distributed systems development
40
49. Case study: ITBM leveraging containers on SDDC and vCloud Air
49
In our front-end, over dozen micro-services
run in Docker containers on CoreOS VMs:
• Web Server x3
• Database x3
• Content
• Load Balancer
• Session Store
• Authentication
• Licensing
• DNS
• Provisioning
• Monitoring
• …
Registry
ClusterCluster
DNS
• Our backend is processing customer data, acquiring more data
from online sources, and generating content for the front-end
• It is also validating the content and serves as staging environment
• Some services use fleet and some are managed by Mesos
fleet & etcd
ClusterCluster
Mesos
HDFS
Cluster
Registry Jenkins
Binaries and content are packaged in Docker
containers during build and moved to staging
and pro-duction using Jenkins Pipeline
vCloud Air
Data Center Virtualization
SDDC Platform
50. Data Center Virtualization
SDDC Platform
Case Study:
ITBM Leveraging Containers on SDDC & vCloud Air
50
fleet & etcd
ClusterCluster
Mesos
HDFS
Registry
ClusterCluster
DNS
Registry
IT Benchmarking Service (ITBM)
SaaS application to measure IT process against peers or common recognized patterns
• Build and content generation on private cloud (SDDC), Customer-facing modules on vCloud Air
• All services running in Dockers on CoreOS VMs
vCloud Air
51. Software-Defined Data Center
• Single platform for running and
managing traditional + modern apps
• Enterprise grade: security,
performance, operational efficiency
• Ability to extend applications to the
hybrid cloud
• Support for community-led projects
(Big Data, OpenStack, containers)
51
The Open Platform for Modern Applications
VMware Bridges These Two Worlds
Resilience
Security
QoS
Openness
Portability
Agility
Traditional Apps Modern Apps
OS
App
Virt. HW
OS
App
Virt. HW
OS
App
Virt. HW
OS
App
Virt. HW
App
Container OS
App
OpenStack API
Open
Container API
Software-Defined Data Center
ESXi, NSX, VSAN
On-premise Off-premise
IT Faces Conflicting Demands
52. In Summary
• VMware is focused on helping companies run and manage their applications,
whether they are packaged in VMs or containers
• A software-defined datacenter is the best place to run and manage all
application types
• Docker, Google, Pivotal, VMware are working together to help companies
efficiently run and operationalize containerized applications
52