Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
1© Copyright 2016 EMC Corporation. All rights reserved. 1© Copyright 2016 EMC Corporation. All rights reserved.
DEEP DIVE ...
2© Copyright 2016 EMC Corporation. All rights reserved.
• David vonThenen, Developer Advocate - EMC {code}
– Twitter: @dvo...
3© Copyright 2016 EMC Corporation. All rights reserved.
• Quick Review of Mesos
• Frameworks and Two Layer Scheduling
• Pr...
4© Copyright 2016 EMC Corporation. All rights reserved. 4© Copyright 2016 EMC Corporation. All rights reserved.
QUICK REVI...
5© Copyright 2016 EMC Corporation. All rights reserved.
• From a resource perspective, it’s a cluster manager:
– Pools res...
6© Copyright 2016 EMC Corporation. All rights reserved.
SCHEDULERS ARE DIFFICULT TO BUILD
7© Copyright 2016 EMC Corporation. All rights reserved.
• Master
– Manages Mesos Agents
– Enables fine-grained sharing of ...
8© Copyright 2016 EMC Corporation. All rights reserved.
MESOS ARCHITECTURE
9© Copyright 2016 EMC Corporation. All rights reserved.
• Generic
– Anything that can be run on a standard Linux shell
– P...
10© Copyright 2016 EMC Corporation. All rights reserved.
GENERIC APPLICATIONS
11© Copyright 2016 EMC Corporation. All rights reserved. 11© Copyright 2016 EMC Corporation. All rights reserved.
FRAMEWOR...
12© Copyright 2016 EMC Corporation. All rights reserved.
• On the surface…
• Allows specialization of
workload by providin...
13© Copyright 2016 EMC Corporation. All rights reserved.
• Scheduler
– a scheduler that registers with the master to be of...
14© Copyright 2016 EMC Corporation. All rights reserved.
FRAMEWORK ARCHITECTURE
15© Copyright 2016 EMC Corporation. All rights reserved.
FRAMEWORK ARCHITECTURE
16© Copyright 2016 EMC Corporation. All rights reserved.
• Odds are you have been using at least 1 framework
without reali...
17© Copyright 2016 EMC Corporation. All rights reserved.
• Zooming into the “Marathon” box…
• Marathon is a Scheduler
• La...
18© Copyright 2016 EMC Corporation. All rights reserved.
MESOS FRAMEWORKS
19© Copyright 2016 EMC Corporation. All rights reserved.
• 2 layer scheduling or sub-
scheduling of resources
• Manages th...
20© Copyright 2016 EMC Corporation. All rights reserved.
BUILDING HADOOP CLUSTERS
21© Copyright 2016 EMC Corporation. All rights reserved.
MANAGING HADOOP CLUSTERS
22© Copyright 2016 EMC Corporation. All rights reserved.
APPLICATIONS USING HADOOP
23© Copyright 2016 EMC Corporation. All rights reserved.
THIS SEEMS SILLY
24© Copyright 2016 EMC Corporation. All rights reserved.
• Allows you to subsume the default scheduler such
that:
• You ca...
25© Copyright 2016 EMC Corporation. All rights reserved.
• The details of your applications are hidden from the
end user!
...
26© Copyright 2016 EMC Corporation. All rights reserved.
• CouchBase Framework (doesn’t exist, let’s imagine it
does)
– Co...
27© Copyright 2016 EMC Corporation. All rights reserved.
SKYNET
28© Copyright 2016 EMC Corporation. All rights reserved. 28© Copyright 2016 EMC Corporation. All rights reserved.
PRODUCTI...
29© Copyright 2016 EMC Corporation. All rights reserved.
• Containers are ephemeral therefore stateless
• Quickly found th...
30© Copyright 2016 EMC Corporation. All rights reserved.
• Persistent
– Containers can come and go,
but continue where the...
31© Copyright 2016 EMC Corporation. All rights reserved.
• REX-Ray
– Provides a vendor agnostic storage orchestration engi...
32© Copyright 2016 EMC Corporation. All rights reserved.
• We can run “real world” applications
• Persistence – DBs
• High...
33© Copyright 2016 EMC Corporation. All rights reserved.
LET’S FACE IT…
34© Copyright 2016 EMC Corporation. All rights reserved.
• Trouble is… this still represents the “old way” of
thinking
• P...
35© Copyright 2016 EMC Corporation. All rights reserved.
• Mesos frameworks can proactively monitor for thing
going sidewa...
36© Copyright 2016 EMC Corporation. All rights reserved. 36© Copyright 2016 EMC Corporation. All rights reserved.
DEMO
37© Copyright 2016 EMC Corporation. All rights reserved.
• Running on Amazon EC2
– Mesos
– Marathon
– Mesos DNS
• Producti...
38© Copyright 2016 EMC Corporation. All rights reserved.
• Elastic Search Framework
– Create and mount external volume
– P...
39© Copyright 2016 EMC Corporation. All rights reserved.
• Deploy Elastic Search Framework using REST API
– Observe Schedu...
40© Copyright 2016 EMC Corporation. All rights reserved.
DEMO: AUTO DEPLOY ELASTIC SEARCH
41© Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container World
Wednesday 3PM
Joshua ...
42© Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode
@dvonthenen
emccode.com
community.emccode.com
Come visit...
43© Copyright 2016 EMC Corporation. All rights reserved. 43© Copyright 2016 EMC Corporation. All rights reserved.
EMC World 2016 - code.14 Deep Dive with Mesos and Persistent Storage for Applications
Nächste SlideShare
Wird geladen in …5
×

EMC World 2016 - code.14 Deep Dive with Mesos and Persistent Storage for Applications

Persistent applications that can be complex to operate and scale tend to be perfect for Apache Mesos. Internal direct attached storage and external storage are both options to run your applications. This talk will outline patterns for using these to allow deployment of managed frameworks and tasks, while maintaining fault tolerance and scalability.

  • Als Erste(r) kommentieren

EMC World 2016 - code.14 Deep Dive with Mesos and Persistent Storage for Applications

  1. 1. 1© Copyright 2016 EMC Corporation. All rights reserved. 1© Copyright 2016 EMC Corporation. All rights reserved. DEEP DIVE WITH MESOS & PERSISTENT STORAGE FOR APPLICATIONS DAVID VONTHENEN, EMC {code}
  2. 2. 2© Copyright 2016 EMC Corporation. All rights reserved. • David vonThenen, Developer Advocate - EMC {code} – Twitter: @dvonthenen – Blog: http://dvonthenen.com – Github: https://github.com/dvonthenen PRESENTER
  3. 3. 3© Copyright 2016 EMC Corporation. All rights reserved. • Quick Review of Mesos • Frameworks and Two Layer Scheduling • Production Applications • Demo • Q&A AGENDA
  4. 4. 4© Copyright 2016 EMC Corporation. All rights reserved. 4© Copyright 2016 EMC Corporation. All rights reserved. QUICK REVIEW OF MESOS
  5. 5. 5© Copyright 2016 EMC Corporation. All rights reserved. • From a resource perspective, it’s a cluster manager: – Pools resources to be centrally managed as a single unit • From an application perspective, it’s a scheduler: – Dispatches workloads to consume pooled resources • Often described as a Data Center Operating System (DCOS) – Static versus Dynamic Partition of Resource – Efficient utilization of CPU, Memory, Disk • Proven resource scheduler for production workloads QUICK REVIEW OF MESOS
  6. 6. 6© Copyright 2016 EMC Corporation. All rights reserved. SCHEDULERS ARE DIFFICULT TO BUILD
  7. 7. 7© Copyright 2016 EMC Corporation. All rights reserved. • Master – Manages Mesos Agents – Enables fine-grained sharing of resources (CPU, RAM, …) across frameworks in the form of resource offers. • Zookeeper – Enables highly reliable distributed coordination (cluster state) • Agent (aka Slave) – The entity that will be running the task MESOS COMPONENTS
  8. 8. 8© Copyright 2016 EMC Corporation. All rights reserved. MESOS ARCHITECTURE
  9. 9. 9© Copyright 2016 EMC Corporation. All rights reserved. • Generic – Anything that can be run on a standard Linux shell – Provides a mechanism for deploying applications packages – Supports Docker workloads • Framework – A plug-in to Mesos that allows you to implement a sub- scheduler – specialize a workload by providing custom placement and resource allocation APPLICATION SUPPORT
  10. 10. 10© Copyright 2016 EMC Corporation. All rights reserved. GENERIC APPLICATIONS
  11. 11. 11© Copyright 2016 EMC Corporation. All rights reserved. 11© Copyright 2016 EMC Corporation. All rights reserved. FRAMEWORKS AND TWO LAYER SCHEDULING
  12. 12. 12© Copyright 2016 EMC Corporation. All rights reserved. • On the surface… • Allows specialization of workload by providing custom placement and resource allocation • Provides monitoring of these workloads to protect and react when things fail MESOS FRAMEWORK
  13. 13. 13© Copyright 2016 EMC Corporation. All rights reserved. • Scheduler – a scheduler that registers with the master to be offered resources • Executor – an executor process that is launched on slave nodes to run the framework’s tasks FRAMEWORK COMPONENTS
  14. 14. 14© Copyright 2016 EMC Corporation. All rights reserved. FRAMEWORK ARCHITECTURE
  15. 15. 15© Copyright 2016 EMC Corporation. All rights reserved. FRAMEWORK ARCHITECTURE
  16. 16. 16© Copyright 2016 EMC Corporation. All rights reserved. • Odds are you have been using at least 1 framework without realizing it… AS IT TURNS OUT… • Marathon is a framework – A framework for performing container orchestration for long running tasks • Chronos is a framework – A framework for performing batch or scheduled tasks
  17. 17. 17© Copyright 2016 EMC Corporation. All rights reserved. • Zooming into the “Marathon” box… • Marathon is a Scheduler • Launching a Marathon task, the “generic” applications are created inside a “generic” (or default) executor • Dogfooding! MARATHON DISSECTED
  18. 18. 18© Copyright 2016 EMC Corporation. All rights reserved. MESOS FRAMEWORKS
  19. 19. 19© Copyright 2016 EMC Corporation. All rights reserved. • 2 layer scheduling or sub- scheduling of resources • Manages the lifecycle of your workload through application specific monitoring • What does this mean specifically? DIGGING DEEPER
  20. 20. 20© Copyright 2016 EMC Corporation. All rights reserved. BUILDING HADOOP CLUSTERS
  21. 21. 21© Copyright 2016 EMC Corporation. All rights reserved. MANAGING HADOOP CLUSTERS
  22. 22. 22© Copyright 2016 EMC Corporation. All rights reserved. APPLICATIONS USING HADOOP
  23. 23. 23© Copyright 2016 EMC Corporation. All rights reserved. THIS SEEMS SILLY
  24. 24. 24© Copyright 2016 EMC Corporation. All rights reserved. • Allows you to subsume the default scheduler such that: • You can install and configure your application • Resources can be allocated in a way that optimizes your applications • Allows operation real-time automated improvements for your applications – Scaling up for load balancing peak times – Scaling back to free up resources 2 LAYER SCHEDULING
  25. 25. 25© Copyright 2016 EMC Corporation. All rights reserved. • The details of your applications are hidden from the end user! – Deployment – Operations – Growth – Disaster recovery – Performance • Just a consumer of the application as a service! THE NON-TECHNICAL VERSION
  26. 26. 26© Copyright 2016 EMC Corporation. All rights reserved. • CouchBase Framework (doesn’t exist, let’s imagine it does) – CouchBase is a NoSQL DB • Monitor the CouchBase instances, when nearing full – Dynamically add an node using “server-add” – Then call “rebalance” • Reactive and Proactive but fully automated • I think we had all seen this movie… FOR EXAMPLE
  27. 27. 27© Copyright 2016 EMC Corporation. All rights reserved. SKYNET
  28. 28. 28© Copyright 2016 EMC Corporation. All rights reserved. 28© Copyright 2016 EMC Corporation. All rights reserved. PRODUCTION APPLICATIONS
  29. 29. 29© Copyright 2016 EMC Corporation. All rights reserved. • Containers are ephemeral therefore stateless • Quickly found the need for state – Configuration – Data – Logging / Auditing • Containerizers like Mesos and Docker introduced exporting to local storage • Failure of a node? hardware? A better way? EVOLUTION OF STATEFUL
  30. 30. 30© Copyright 2016 EMC Corporation. All rights reserved. • Persistent – Containers can come and go, but continue where they left off – Application Data, Logs, Audit Trail, etc • External – Storage lives outside the Mesos Agent node – High Availability – Tolerates node failure PERSISTENT EXTERNAL STORAGE
  31. 31. 31© Copyright 2016 EMC Corporation. All rights reserved. • REX-Ray – Provides a vendor agnostic storage orchestration engine – AWS, GCE, ScaleIO, XtremIO, many many more – Github: https://github.com/emccode/rexray • mesos-module-dvdi – Provides a hook to Mesos agent nodes to provision, attach, and detach external storage – Github: https://github.com/emccode/mesos-module-dvdi HOW WE ACHIEVE THIS?
  32. 32. 32© Copyright 2016 EMC Corporation. All rights reserved. • We can run “real world” applications • Persistence – DBs • Highly Available, Fault Tolerant • Audit trail – Transaction Logs • Not worry about the “2am call” STATELESS TO STATEFUL
  33. 33. 33© Copyright 2016 EMC Corporation. All rights reserved. LET’S FACE IT…
  34. 34. 34© Copyright 2016 EMC Corporation. All rights reserved. • Trouble is… this still represents the “old way” of thinking • PREVENT THE CRASH OR FAILURE! • Problem is you can’t • What is the alternative? STATEFUL 2.0?
  35. 35. 35© Copyright 2016 EMC Corporation. All rights reserved. • Mesos frameworks can proactively monitor for thing going sideways • Leveraging external persistent storage and frameworks fail fast and recover quicker! • Skynet! FRAMEWORKS + STATEFUL 2.0
  36. 36. 36© Copyright 2016 EMC Corporation. All rights reserved. 36© Copyright 2016 EMC Corporation. All rights reserved. DEMO
  37. 37. 37© Copyright 2016 EMC Corporation. All rights reserved. • Running on Amazon EC2 – Mesos – Marathon – Mesos DNS • Production HA Cluster – 3 x Master + Zookeeper Nodes (t2.micros) – 5 x Slave Nodes with Docker Support (t2.medium) DEMO CONFIGURATION
  38. 38. 38© Copyright 2016 EMC Corporation. All rights reserved. • Elastic Search Framework – Create and mount external volume – Pour some data into Elastic Search – Github: https://github.com/mesos/elasticsearch • Elastic Search Instances backed by Persistent External Storage – Using REX-Ray – Using mesos-module-dvdi DEMO: FRAMEWORK + EXTERNAL STORAGE
  39. 39. 39© Copyright 2016 EMC Corporation. All rights reserved. • Deploy Elastic Search Framework using REST API – Observe Scheduler creation – Using browser, log into the Elastic Search Framework UI – Observe Executor nodes get created and attaching to EBS volumes • Simulate Failure – Kill EC2 Instance – Watch executor recreation and reattach to EBS volumes DEMO: FRAMEWORK + EXTERNAL STORAGE
  40. 40. 40© Copyright 2016 EMC Corporation. All rights reserved. DEMO: AUTO DEPLOY ELASTIC SEARCH
  41. 41. 41© Copyright 2016 EMC Corporation. All rights reserved. Data Persistence in the New Container World Wednesday 3PM Joshua Bernstein VP of Technology for ETD Tobi Knaup CEO & Co-Founder of Mesosphere Guru Session
  42. 42. 42© Copyright 2016 EMC Corporation. All rights reserved. @EMCcode @dvonthenen emccode.com community.emccode.com Come visit us at Booth #1044 or in the vLab Questions?
  43. 43. 43© Copyright 2016 EMC Corporation. All rights reserved. 43© Copyright 2016 EMC Corporation. All rights reserved.

×