Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (19)
Ähnlich wie London HUG 14/4 - Deploying and Discovering at Scale with Consul and Nomad
Ähnlich wie London HUG 14/4 - Deploying and Discovering at Scale with Consul and Nomad (20)
Mehr von London HashiCorp User Group
Mehr von London HashiCorp User Group (7)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
London HUG 14/4 - Deploying and Discovering at Scale with Consul and Nomad
- 1. HASHICORP Deploying and Discovering at Scale with Consul & Nomad
- 3. HASHICORP Agenda • Nomad Overview • Nomad Architecture
- 4. HASHICORP Agenda • Consul Overview • Consul Architecture • Prepared Queries
- 6. HASHICORP Advantages of a Scheduler Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 7. HASHICORP Advantages of a Scheduler Bin Packing Over-Subscription Job Queueing Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 8. HASHICORP Advantages of a Scheduler Abstraction API Contracts Standardization Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 9. HASHICORP Advantages of a Scheduler Priorities Resource Isolation Pre-emption Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 10. HASHICORP
- 11. Nomad HASHICORP Cluster Scheduler Easily Deploy Applications Job Specification
- 12. HASHICORP example.nomad # Define our simple redis job job "redis" { # Run only in us-east-1 datacenters = ["us-east-1"] # Define the single redis task using Docker task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }
- 13. HASHICORP Job Specification Declares what to run
- 14. HASHICORP Job Specification Nomad determines where and manages how to run
- 15. HASHICORP Job Specification Abstract work from resources
- 16. Nomad HASHICORP Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Large Scale Operationally Simple
- 17. HASHICORP Thousands of regions Tens of thousands of clients per region Thousands of jobs per region
- 18. HASHICORP Built on Experience gossip consensus
- 19. HASHICORP • Cluster Management • Gossip Based (P2P) • Membership • Failure Detection • Event System
- 20. HASHICORP • Gossip Protocol • Large Scale • Production Hardened • Operationally Simple
- 21. HASHICORP • Service configuration and discovery • Monitoring at scale • High-availability
- 22. HASHICORP • Service configuration and discovery • Monitoring at scale • High-availability
- 23. HASHICORP • Service configuration and discovery • Monitoring at scale • High-availability
- 24. HASHICORP • Service configuration and discovery • Monitoring at scale • High-availability
- 25. HASHICORP
- 26. HASHICORP • Multi-Datacenter • Raft Consensus • Large Scale • Production Hardened • Coordination (Locking) • Central Servers + Distributed Clients • Network Tomography • Prepared Queries
- 27. HASHICORP Prepared Queries • Multiple instances of a given service exist in multiple datacenters • Clients can talk to any of them, and always prefer the instances with lowest latency • Policies can change, desire to not have the clients know the details of how to locate a healthy service
- 28. HASHICORP Prepared Queries • New query namespace, similar to services • Register queries to answer for parts of this namespace • Clients use APIs, or “.query.consul” DNS lookups to run queries
- 29. HASHICORP gossip consensus Mature Libraries Design Patterns No Scheduling Logic
- 30. HASHICORP Built on Research gossip consensus
- 31. Nomad HASHICORP Inspired by Google Omega Optimistic Concurrency Internal State and Coordination Service and Batch workloads Pluggable Architecture
- 32. HASHICORP Single Region Architecture SERVER SERVER SERVER CLIENT CLIENT CLIENTDC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC
- 33. HASHICORP Multi Region Architecture SERVER SERVER SERVER FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION REGION B GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING REGION A SERVER FOLLOWER SERVER SERVER LEADER FOLLOWER
- 34. Nomad HASHICORP Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary
- 36. HASHICORP Evaluations ~= State Change Event
- 37. HASHICORP Create / Update / Delete Job Node Up / Node Down Allocation Failed
- 38. HASHICORP External Event Evalua?on Crea?on Evalua?on Queuing Evalua?on Processing Op?mis?c Coordina?on State Updates
- 39. HASHICORP Server Architecture Omega Class Scheduler Pluggable Logic Internal Coordination and State Multi-Region / Multi-Datacenter
- 40. HASHICORP Client Architecture Broad OS Support Host Fingerprinting Pluggable Drivers
- 41. HASHICORP Fingerprinting Operating System Hardware Applications Environment Type Examples Kernel, OS, Versions CPU, Memory, Disk Java, Docker, Consul AWS, GCE
- 42. HASHICORP Fingerprinting Constrain Placement and Bin Pack
- 43. HASHICORP Fingerprinting “Task Requires Linux, Docker, and PCI-Compliant Hardware” expressed as Constraints
- 44. HASHICORP Fingerprinting “Task needs 512MB RAM and 1 Core” expressed as Resource Ask
- 45. HASHICORP Drivers Execute Tasks Provide Resource Isolation
- 46. HASHICORP Containerized Virtualized Standalone Docker Qemu / KVM Java Jar Static Binaries Rocket
- 47. HASHICORP Containerized Virtualized Standalone Docker Rocket Windows Server Containers Qemu / KVM Hyper-V Xen Java Jar Static Binaries C#
- 49. Nomad HASHICORP Operational Simplicity: Single Binary No Dependencies Highly Available
- 50. Nomad HASHICORP Cluster Scheduler Easily Deploy Applications Job Specification
- 51. Nomad HASHICORP Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 52. Nomad HASHICORP • Million Container Challenge • hashicorp.com/c1m.html • github.com/hashicorp/c1m
- 53. HASHICORP Thanks! We’ll do a quick demo then answer questions… Jon Benson @jm_benson