Weitere ähnliche Inhalte Ähnlich wie EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications (20) Kürzlich hochgeladen (20) EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications1. 1© Copyright 2016 EMC Corporation. All rights reserved. 1© Copyright 2016 EMC Corporation. All rights reserved.
STATE OF THE CONTAINER ECOSYSTEM
WITH PERSISTENT APPLICATIONS
STEVE WONG
DEVELOPER ADVOCATE –EMC{code}
2. 2© Copyright 2016 EMC Corporation. All rights reserved.
Old: monolithic, big,
slow to change,
long lived
APPLICATIONS ARE CHANGING
New: many instances,
rapidly implemented,
updated, deployed
3. 3© Copyright 2016 EMC Corporation. All rights reserved.
SOLUTION: ENTER THE CONTAINER
4. 4© Copyright 2016 EMC Corporation. All rights reserved.
TODAY’S AGENDA
Two rapidly
developing areas
fields within the
container space:
1. Persistent
Storage
2. Schedulers
5. 5© Copyright 2016 EMC Corporation. All rights reserved.
WHERE DOES STORAGE FIT INTO THIS?
… THE SHELL GAME
Containers and this new
“cattle” thing magically made
my storage management
headaches disappear, right?
6. 6© Copyright 2016 EMC Corporation. All rights reserved.
IF YOUR CONTAINER ARE CATTLE,
YOU HAVE NOTHING TO LOSE …
Go ahead, use local storage.
• The test: I don’t need
persistence beyond the life
of the container instance or
host
7. 7© Copyright 2016 EMC Corporation. All rights reserved.
I HAVE SERVICES THAT AREN’T CATTLE
WHAT IF I RUN THESE THE SAME WAY?
A portion of the container host’s
filesystem can be exposed to a
container.
• Availability Risk
– Migrate container to another host – your
storage is gone
– Host goes down – your service goes down
• Scale Limitation
– Need more storage than the host has?
Sorry…
8. 8© Copyright 2016 EMC Corporation. All rights reserved.
SOLUTION: MOUNT OF EXTERNAL STORAGE
9. 9© Copyright 2016 EMC Corporation. All rights reserved.
HOW DO I MOUNT EXTERNAL STORAGE?
THE REX-RAY PROJECT
• Open Source - extensible
• Supports many container
platforms, including cloud hosted
• Supports many storage platforms
• Managed and configurable by
container host,
not inside-the-container
configuration
10. 10© Copyright 2016 EMC Corporation. All rights reserved.
TODAY’S AGENDA
Two rapidly developing
areas fields within the
container space:
1. Persistent Storage
2. Schedulers
..we’ll come back to
storage in a minute
11. 11© Copyright 2016 EMC Corporation. All rights reserved.
GOT CONTAINERS AT SCALE?
YOU NEED A SCHEDULER…
12. 12© Copyright 2016 EMC Corporation. All rights reserved.
ASSUME 1000’S OF CONTAINERS
• How do you place an
assortment of
containers of
different sizes and
shapes onto your
servers efficiently?
13. 13© Copyright 2016 EMC Corporation. All rights reserved.
ASSUME 1000’S OF SERVICES
• How do you
find them?
• How do they
find each
other?
16. 16© Copyright 2016 EMC Corporation. All rights reserved.
WHAT’S A SCHEDULER?
• Manages compute nodes
– Takes reports from each backend server, as to
whether they are still available, and what their
capabilities and capacities are.
• Manages incoming work request queue
– Takes user’s, batch, and API job requests
– Based on requirements, decides which
backend server(s) can fulfill each request
– If multiple backend servers are suitable,
engages optimization algorithms in favor of
random placement
on demand
scheduled
batch
17. 17© Copyright 2016 EMC Corporation. All rights reserved.
MANY SCHEDULERS ARE AVAILABLE
• Differences
– Management “philosophy”
• “Tell me How” vs “Tell me What”
– aka Imperative vs Declarative, Explicit vs Implicit, declare procedure vs declare outcome
– Container technology(s) used
– Maturity
– Supported scale
– Support for external storage volume mounts
– Degree of “convergence” with a specific application stack
• Aka degree to which scheduler is “opinionated”
18. 18© Copyright 2016 EMC Corporation. All rights reserved.
SCHEDULERS COMPARED
Swarm Kubernetes Mesos Cloud
Foundry
Management
style
procedure outcome outcome outcome
Maturity new new years years
Scale 1,000? 1000 10,000+ 5,000
Extensibility limited limited Well defined limited
Container
Engine
Docker Docker
Roadmap:
CoreOS
Linux+Docker
Roadmap:CoreO
S
Garden
External
storage
support
DVD RexRay in host RexRay in host RexRay
Application
specific
framework
availability
Roadmap:Auror
a for generic
proposed 20+
applications,
Marathon and
Aurora for
generic
Multiple
language
platforms
19. 19© Copyright 2016 EMC Corporation. All rights reserved.
KUBERNETES
• Started by Google
• Primary container integration is with Docker, but
enhancing support for CoreOS in upcoming release
• Implicit aka Declarative configuration philosophy
– Declare desired outcome
– User defined Key/value labels on objects can be used to
support this
20. 20© Copyright 2016 EMC Corporation. All rights reserved.
KUBERNETES
• pods group containers together’
– pods are the smallest management unit
– provides “plumbing” to allow containers to interoperate
(micro-service per container) inside the pods
• replication controllers handle container lifecycle
• labels provide a way to find and query containers
and other objects
• services assemble a set of containers performing a
function
21. 21© Copyright 2016 EMC Corporation. All rights reserved.
KUBERNETES
• monitors health of services, with auto-healing
• Load balancing built in
• External volume mounts supported in POC / release
candidate with additional external storage features in
active planning stage
22. 22© Copyright 2016 EMC Corporation. All rights reserved.
SWARM
• Started by Docker Inc.
• Stays with Docker CLI/API originally developed for
single host, and allows using it across a multiple
node cluster
• Explicit aka Imperative configuration philosophy
– Declare desired how to achieve outcome
• Compose uses a yaml file to group containers
23. 23© Copyright 2016 EMC Corporation. All rights reserved.
SWARM
• Opportunities for enhancement:
– Container communication across host boundaries
– External persistent volume management
24. 24© Copyright 2016 EMC Corporation. All rights reserved.
MESOS
• Mesos is the scheduler that provides maximal
flexibility
• It allows you to run sub-schedulers (aka
Frameworks), many of which are available off-the
shelf – or you can implement your own
• It can support Docker containers, but also supports
non Docker workloads.
25. 25© Copyright 2016 EMC Corporation. All rights reserved.
MESOS FRAMEWORKS
• Marathon
– Supports generic workloads that run in a linux container
– Supports Docker containers
• Hadoop
• Spark
• Kafka
• Many more
26. 26© Copyright 2016 EMC Corporation. All rights reserved.
MESOS STORAGE SUPPORT
• External volume mounts supported now
• Full global external storage resource management,
using an offer/accept model is in active planning
stage now
27. 27© Copyright 2016 EMC Corporation. All rights reserved.
CLOUD FOUNDRY
• Cloud Foundry delivers management at the
application level
• Containers are used “under the covers” but you
aren’t exposed to the details and don’t have to
manage them
28. 28© Copyright 2016 EMC Corporation. All rights reserved.
CONTAINER SCHEDULERS AS PAAS STACKS
• A variety of PAAS stacks exist that attempt to give
you “everything you need” to create an app on
Amazon, or your own hardware, provided you accept
the constraints imposed by the platform.
• But marketing people hate to call these constraints.
Instead, the PaaS stack is described as opinionated
– Some container schedulers are more opinionated than
others. Basically the platform makes technology choices for
you, and you don’t get to change these.
29. 29© Copyright 2016 EMC Corporation. All rights reserved.
CLOUD FOUNDRY
• Cloud Foundry is far more opinionated than other
container schedulers – and depending on your
objective, and staffing capacity,
this could be a good thing
30. 30© Copyright 2016 EMC Corporation. All rights reserved.
TODAY’S AGENDA
Two rapidly developing
areas fields within the
container space:
1. Persistent Storage
2. Schedulers
..we’ll come back to
storage in a minute
31. 31© Copyright 2016 EMC Corporation. All rights reserved.
A SCHEDULER SHOULD INTERFACE WITH
STORAGE AT MULTIPLE LEVELS
Create
Utilize
Retire
Lifecycle management with
governance
Mount managementAssignment to applications
(tenancy)
33. 33© Copyright 2016 EMC Corporation. All rights reserved.
WHAT POLLY STORAGE SCHEDULER DOES…
• Manages interfaces to storage providers
– Multiple concurrent providers anticipated
• Mixed vendors
– Including public cloud
– Takes reports from each provider, and what their
capabilities and capacities are
– Uses admin defined key/value labels to enable
• Outcome based placement
• Concurrent use of multiple schedulers, with quotas
• Pool advertisement, with suballocation
34. 34© Copyright 2016 EMC Corporation. All rights reserved.
STORAGE SCHEDULER = DELIVER SCALE OUT
Scale out is not just about adding capacity.
A Storage Scheduler will let you:
• Add, remove, replace storage provider platforms
– in or out of public cloud
• Add, remove, replace schedulers
• This is about giving you flexibility and
shifting the burden of keeping up to vendors
35. 35© Copyright 2016 EMC Corporation. All rights reserved.
DEMO
• Demo attach of a persistent volume under
Kubernetes
• Shutdown container
• Re-attach same volume to a Mesos hosted container,
under Linux containerizer instead of Docker
36. 36© Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container World
Wednesday 3PM
Joshua Bernstein
VP of Technology for ETD
Tobi Knaup
CEO & Co-Founder of Mesosphere
Guru Session
37. 37© Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode
@cantbewong
emccode.com
community.emccode.com
cantbewong.com
Come visit us at
Booth #1044
or in the vLab
Questions?
38. 38© Copyright 2016 EMC Corporation. All rights reserved. 38© Copyright 2016 EMC Corporation. All rights reserved.
Hinweis der Redaktion Takeaways: Containers are a huge opportunity to improve efficiency
Taking advantage of this opportunity at scale presents some challenges
Container schedulers are a popular and proven way to handle the challenge of deploying containers at massive scale. Mesos, Kubernetes, Swarm and CloudFoundry are examples your organization might find it worthwhile to use more than one of these.
In the past year, containerization technology has advanced to allow persistent applications to run in a container.
The EMC{code} Polly project delivers supplements container schedulers to deliver true scale out for persistent applications running on containers.
Scale out is not just about adding capacity.
A Storage Scheduler will let you:
run multiple container platforms
run multiple schedulers
Utilize multiple storage providers
in or out of public cloud
This is about giving you flexibility and shifting the burden of keeping up to vendors
These two topics interact and as we’ll see, when addressed properly, link together These two topics interact and as we’ll see, when addressed properly, link together Objective: On each cluster node, pack jobs in tight to avoid waste in the form of unutilized resource
Playing Tetris with manual load placement might seem fun for a minute or two, but this gets old quick
Container placement ABOLUTELY must be automated,
Your old system of submitting tickets to reserve an IP and create a DNS entry isn’t going to cut it The old 3 tier model was relatively simple,
You had your presentation or UI layer, your application or business logic layer, and your persistence or database layer.
Every thing was deterministically placed, and you knew where things were for troubleshooting, and the components knew where there counterparts were
Microservices isn’t quite N squared Metcalf’s law, but it heads that direction – a lot more interconnects.
And to improve efficiency, you have dynamic non-deterministic placement on to cluster nodes.
This is why I’m thinking of setting up a side business on Etsy to sell these cool microservices liquor flasks.It’s also why you need something called a scheduler These two topics interact and as we’ll see, when addressed properly, link together Schedulers can also enable end use self service,
With controls, audit, and health monitoring