1. State of the Container Ecosystem
prepared for Huawei
Vinay Rao
Former Chief Architect
Formation Data Systems Inc

2. Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl +
bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg +
libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg +
postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
Docker Basics: The Challenge
Multiplicityof
Stacks
Multiplicityof
hardware
environments
Production Cluster
Customer Data Center
Doservicesand
appsinteract
appropriately?
CanImigrate
smoothlyand
quickly?

3. Docker Basics: The Matrix From Hell
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developm
ent VM
QA Server
Single
Prod
Server
Onsite
Cluster
Public
Cloud
Contribut
or’s
laptop
Customer
Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?

4. Static website Web frontendUser DB Queue Analytics DB
Developm
ent VM
QA server Public Cloud Contributor’s
laptop
Docker is a shipping container system for code
MultiplicityofStacks
Multiplicityof
hardware
environments
Production
Cluster
Customer Data
Center
Doservicesandapps
interact
appropriately?
CanImigrate
smoothlyandquickly
…that can be manipulated using
standard operations and run
consistently on virtually any
hardware platform
An engine that enables
any payload to be
encapsulated as a
lightweight, portable, self-
sufficient container…

5. Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developm
ent VM
QA Server
Single
Prod
Server
Onsite
Cluster
Public
Cloud
Contribut
or’s
laptop
Customer
Servers
Docker eliminates the matrix from Hell

6. Why Developers Care
• Build once…(finally) run anywhere
• A clean, safe, hygienic and portable runtime environment for your app.
• No worries about missing dependencies, packages and other pain points
during subsequent deployments.
• Run each app in its own isolated container, so you can run various
versions of libraries and other dependencies for each app without worrying
• Automate testing, integration, packaging…anything you can script
• Reduce/eliminate concerns about compatibility on different platforms,
either your own or your customers.
• Cheap, zero-penalty containers to deploy services? A VM without the
overhead of a VM? Instant replay and reset of image snapshots? That’s
the power of Docker

7. Why it works—separation of concerns
• Dan the Developer
• Worries about what’s “inside” the
container
• His code
• His Libraries
• His Package Manager
• His Apps
• His Data
• All Linux servers look the same
• Oscar the Ops Guy
• Worries about what’s “outside” the
container
• Logging
• Remote access
• Monitoring
• Network config
• All containers start, stop, copy,
attach, migrate, etc. the same way

8. App
A
Containers vs. VMs
Hypervisor (Type 2)
Host OS
Server
Gues
t
OS
Bins/
Libs
App
A’
Gues
t
OS
Bins/
Libs
App
B
Gues
t
OS
Bins/
Libs
AppA’
Docker
Host OS
Server
Bins/Libs
AppA
Bins/Libs
AppB
AppB’
AppB’
AppB’
VM
Container
Containers are isolated,
but share OS and, where
appropriate, bins/libraries
Gues
t
OS
Gues
t
OS
…result is significantly faster
deployment, much less overhead,
easier migration, faster restart

9. What makes containers lightweight?
App
A
Original App
(No OS to take
up space, resources,
or require restart)
AppΔ
Bins
/
App
A
App
A’
Modified App
Copy on write
capabilities allow
us to only save the
diffs
Between container A
and container
A’
VMs
Every app, every copy of an
app, and every slight modification
of the app requires a new virtual server
Copy of
App
No OS. Can
Share bins/libs
App
A’
Guest
OS
Guest
OS
VMs Containers
Bins/
Libs
Bins/
Libs
Bins/
Libs
App
A
Guest
OS
Bins/
Libs

10. Basics of the Docker system
Source Code
Repository
Dockerfile
For
A
Docker Engine
Docker
Containe
r
Image
Registry
Build
Docker
Host 2 OS (Linux)
ContainerA
ContainerB
ContainerC
ContainerA
Push
Search Pull
Run
Host 1 OS (Linux)

11. Docker Basics: The Internals
Docker Daemon
(dockerd)
libcontainer
cgroups namespace
AUFS
BTRFS
VFS
User
Space
Kernel

12. Docker Basics: The Internals
• Namespaces provides the isolated workspace of a container.
• The pid namespace: Process isolation (PID: Process ID).
• The net namespace: Managing network interfaces (NET: Networking).
• The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication).
• The mnt namespace: Managing filesystem mount points (MNT: Mount).
• The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System)
• Docker uses control groups or cgroups to limit an application to a specific set of
resources. For example, you can limit the memory available to a specific
container.
• Docker Engine uses UnionFS to provide the building blocks for containers.
Docker Engine can use AUFS, btrfs, vfs, and DeviceMapper.

13. Docker Basics: Filesystems
Union File System
Snapshot
FileSystem
Copy-on-write Block
Dev
Provisioning
Superfast
supercheap
fast/cheap fast/cheap
Small Files
Superfast
supercheap
fast/cheap fast/costly
Large Files
slow(first-time)
Inefficient
fast/cheap fast/cheap
Diffing superfast superfast slow
Memory usage Efficient Efficient Inefficient
Implementation AUFS BTRFS ZFS

14. Container ecosystem components
• Container Orchestration
• Kubernetes
• Docker Swarm
• Apache Mesos
• Networking
• Flannel
• Weave
• Storage
• Native filesystems.
• Volume Plugins for NAS/SAN, DAS
• Storage virtualization platforms - portworx, quobyte
• Container security
• Application Lifecycle Management
• Rancher Labs
• Robin Systems

15. Kubernetes Container Orchestration

16. Key Concepts of Kubernetes
● Pod - A group of Containers
● Labels - Labels for identifying pods
● Kubelet - Container Agent
● Proxy - A load balancer for Pods
● etcd - A metadata service for service discovery
● cAdvisor - Container Advisor provides resource
usage/performance statistics
● Replication Controller - Manages replication
of pods
● Scheduler - Schedules pods in worker nodes
● API Server - Kubernetes API server

17. Docker Swarm Architecture

18. Native Docker Networking - Not cloud ready!!
• Docker Containers on different machines cannot reach each
other
• Docker on different machines can assign conflicting network
ranges and IP addresses.
• Solution is to allocate unique ports on the machine’s own IP
address, which are then forwarded or proxied to the containers.
• Flannel and Weave are the real cloud-ready solutions using
overlay networking architecture.

19. Container Networking with Flannel
• Flannel is a basic overlay network that works by assigning a
range of subnet addresses (usually IPv4 with a /24 or /16
subnet mask).
• Flannel gives each host an IP subnet (/24 by default) from
which the Docker daemon is able to allocate IPs to the
individual containers.
• Allows each container to have its own IP host address allowing
true distributed container clustering on different hosts.

20. Container Networking with Flannel
•

21. Container Networking using Weave

22. Container Networking - Weave
• Weave Net use standard port numbers, (for example MySQL’s default is port 3306),
• Weave Net uses Encapsulation based overlay. It has a Weave router and bridge modules
in each host.
• Weave Net implements service discovery by providing a fast “micro DNS” server at each
node. You simply name containers and everything ‘just works’, including load balancing
across multiple containers with the same name.
• Weave Net is bundled with a Docker Network plugin that doesn’t require an external cluster
store. You can get started right away and you can start and stop containers even when
there are network connectivity problems.
• Operates in Partially Connected Networks - works well in a mix of legacy systems and
containerized apps environment

23. Storage for Containers
• A data volume is a specially-designated directory within one or
more containers that bypasses the Union File System.
• Volumes are initialized when a container is created. If the
container’s base image contains data at the specified mount point,
that existing data is copied into the new volume upon volume
initialization.
• Data volumes can be shared and reused among containers.
• Changes to a data volume are made directly.
• Changes to a data volume will not be included when you update an
image.
• Data volumes persist even if the container itself is deleted.

24. Persistence
IOPs
Connectors
SSD
HDD SAN
EBS
S3
Admins Realize that Apps Don’t Want Volumes
1. Don’t make data protection a science project
2. Stop trying to connect legacy SAN/NAS/SDS to containers – They are not
container or application aware
3. Your applications want on-demand services delivered programmatically via
the scheduler
4. Storage services need to be “app-aware”, for example Cassandra-aware –
Traditional storage is not.

25. Storage for Containers - Missing pieces
• What is really required is:
• A storage virtualization layer that encapsulates SAN, NAS, DAS or public-cloud
storage elements such as S3/EBS
• Specify application level QoS as IOPS/throughput and enforcement down the
stack.
• Control snapshot/cloning schedules per data volume based on application
characteristics such as data churn, RPO/RTO, business criticality.
• Control tiering between various storage media in the backend to match the
application performance requirements.
• Storage container motion capability for migration of storage containers to a
different pod or across data centers.
• Seamless integration with application management layers.
• Integration with container orchestration layers

26. PortWorx (PX): Data services management
WordPres
s
WordPres
s
PX
MySQL
MySQL
persistence
encryption
CoS
namespace
Auth/ACLs
HA
Operational Experience
Delivered as a lightweight
container with support for
volumes and graph. Runs on
any server, any cloud.
Converged Performance
Data collocated with
container compute
Container Granular Controls
Encryption, CoS, replication,
snapshots are done at
container granularity
Hybrid Cloud Ops
Cloud agnostic deployment
and data accessibility across
multiple clouds
S3 Object

27. Portworx vs ScaleIO or CEPH : Unique Architectural Tenet
x86 Servers x86 Servers
rack
…
x86 Servers x86 Servers
rack
Portworx Places an Application’ss Data
Local to a Few Topology Constrained
Servers. Cassandra runs converged.
x86 Servers x86 Servers
rack
…
x86 Servers x86 Servers
rack
x86 Servers
ScaleIO and CEPH shard an application’s
data across many servers to get capacity.
This breaks convergence. You also don’t
have application granular settings on the
volumes.

28. : Complete Container Management Platforms
RUN CONTAINERS
• with the most complete set of container and infrastructure
management capabilities
MANAGE APPLICATIONS
• by simplifying day to day application lifecycle management
INNOVATE WITH CONTAINERS
• without compromising flexibility by empowering developers with
fast access to the latest tools
Production ready
✔ 20 million+
downloads
✔ Open platform for
innovating
✔ Easy to use interface
✔ Multi-tenant
✔ Role based access
✔ 24X7 support

29. RancherOS: Complete Container Management Platform
Application Catalog
Container Orch. & SchedulingUser Mgmt
RBAC
AD/LDAP
SAML
Ops Mgmt
CI/CD
Registries
Monitoring
Networking
Multi-tenant Environments
Environment 1 Environment N
Infrastructure Services
Storage
……. ..Environment 2
Security DNS/LB

30. RancherOS: Volume Plugin Manager
• Rancher is your Docker volume plugin manager and
provides the following:
• Distribution – automatically deploys volume plugins to desired hosts
managed by Rancher so they are made available immediately
• Version Management and Upgrades – provides version
management and allows users to automatically update plugins on
all hosts with a single click (or API call)
• Lifecycle Management – Provides health checks and automatic
redeployment of plugins if things go wrong
• Plugin-aware Container Scheduling – provides container scheduling
based on deployment of plugins so you never schedule containers
on hosts that have no access to storage

31. Diamanti - Appliances for converged container clouds
• Network and Storage QoS -Enforced performance policies
across container cloud users
• Multi-tenant isolation with role-based access policies.
• Built-in Kubernetes and Docker with zero effort integration
• Add container cloud capacity in seconds, one appliance at a time
• Utilization and capacity monitoring for all containers and
resources
• Selectable Docker network configuration per container
• On-demand low-latency persistent flash storage per Docker
container

32. Conclusions
• Integrate storage and networking stacks to support
production (persistent) workloads
• Keep a check on Operational complexity
• Application lifecycle management integration is key.
• Container orchestration enables cross-cloud
migration in a hybrid cloud model.