Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

How Self-Healing Nodes and Infrastructure Management Impact Reliability

37 Aufrufe

Veröffentlicht am

Self-healing does not equal self-healing. There are multiple layers to it, whether a self-healing infrastructure, cluster, pods, or Kubernetes. Kubernetes itself ensures self-healing pods. But how do you ensure your applications, whose reliability depends on every single layer, are truly reliable?

This presentation covers the different self-healing layers, what Kubernetes does and doesn't do (at least not by default), and what you should look out for to ensure true reliable applications. Hint: infrastructure provisioning plays a key role.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

How Self-Healing Nodes and Infrastructure Management Impact Reliability

  1. 1. How Self-Healing Nodes and Infrastructure Management Impact Reliability Oleg Chunikhin |CTO
  2. 2. Introductions Oleg Chunikhin CTO, Kublr ✔ 20+ years in software architecture & development ✔ Working w/ Kubernetes since its release in 2015 ✔ CTO at Kublr—an enterprise ready container management platform ✔ Twitter @olgch; @kublr Like what you hear? Tweet at us!
  3. 3. Automation Ingress Custom Clusters Infrastructure Logging Monitoring Observability API Usage Reporting RBAC IAM Air Gap TLS Certificate Rotation Audit Storage Networking Container Registry CI / CD App Mgmt Infrastructure Container Runtime Kubernetes OPERATIONS SECURITY & GOVERNANCE What’s Kublr? @olgch, @kublr
  4. 4. Building a Reliable System with Kubernetes • Day 1: trivial, K8S will restart my pod! • Day 30: this is not so simple... • What Kubernetes does, can, and doesn’t do • Full stack reliability: tools and approaches @olgch, @kublr
  5. 5. Kubernetes Cluster K8S Architecture Refresher: Components The Master, agent, etcd, API, overlay network, and DNS Master API Server etcd data controller manager scheduler etcd kubectl Worker kubelet container runtime overlay network cluster DNS kube-proxy @olgch, @kublr
  6. 6. Cluster K8S Architecture Refresher: API Objects Nodes, pods, services, and persistent volumes Node 1 Node 2 Pod A-1 Cnt1 Cnt2 Pod A-2 Cnt1 Cnt2 Pod B-1 Cnt3 SrvA SrvB Persistent Volume @olgch, @kublr
  7. 7. Reliability Tools: Pod Probes and Controllers Probes Liveness and readiness check • TCP, HTTP(S), exec Controllers ReplicaSet • Maintain specific number of identical replicas Deployment • ReplicaSet + update strategy, rolling update StatefulSet • Deployment + replica identity, persistent volume stability DaemonSet • Maintain identical replicas on each node (of a specified set) Operators @olgch, @kublr
  8. 8. • Resource framework • Standard: CPU, memory, disk • Custom: GPU, FPGA, etc. • Requests and limits • Kube and system reservations • no swap • Pod eviction and disruption budget (resource starving) • Pod priority and preemption (critical pods) • Affinity, anti-affinity, node selectors & matchers Reliability Tools: Resource & Location Mgmt @olgch, @kublr
  9. 9. Reliability Tools: Autoscaling Horizontal pod autoscaler (HPA) Vertical pod autoscaler (VPA) • In-place updates - WIP (issue #5774) Cluster Autoscaler • Depends on infrastructure provider - uses node groups AWS ASG, Azure ScaleSets, ... • Supports AWS, Azure, GCE, GKE, Openstack, Alibaba Cloud @olgch, @kublr
  10. 10. Applications/pods/containers “Middleware” • Operations: monitoring, log collection, alerting, etc. • Lifecycle: CI/CD, SCM, binary repo, etc. • Container management: registry, scanning, governance, etc. Container Persistence: cloud native storage, DB, messaging Container Orchestrator: Kubernetes • “Essentials”: overlay network, DNS, autoscaler, etc. • Core: K8S etcd, master, worker components • Container engine: Docker, CRI-O, etc. OS: kernel, network, devices, services, etc. Infrastructure: “raw” compute, network, storage Full Stack Reliability @olgch, @kublr
  11. 11. Architecture 101 • Layers are separate and independent • Disposable/restartable components • Reattachable dependencies (including data) • Persistent state is separate from disposable processes Pets vs cattle - only data is allowed to be pets (ideally) @olgch, @kublr
  12. 12. If a node has a problem... • Try to fix it (pet) • Replace or reset it (cattle) Infrastructure and OS Tools • In-cluster: npd, weaveworks kured, ... • hardware, kernel, servicer, container runtime issues • reboot • Infrastructure provider automation • AWS ASG, Azure Scale Set, ... • External auto node recovery logic • Custom + infrastructure provider API • Cluster management solution • (Future) cluster API @olgch, @kublr
  13. 13. Components • etcd • Master: API server, controller manager, scheduler • Worker: kubelet, kube-proxy • Container runtime: Docker, CRI-O Already 12 factor Kubernetes Components: Auto-Recovery Monitor liveliness, automate restart • Run as services • Run as static pods Dependencies to care about • etcd data • K8S keys and certificates • Configuration @olgch, @kublr
  14. 14. API Server controller manager scheduler API Server controller manager scheduler etcd data etcd etcd data etcd K8S multi-master • Pros: HA, scaling • Cons: need LB (server or client) etcd cluster • Pros: HA, data replication • Cons: latency, ops complexity etcd data • Local ephemeral • Local persistent (survives node failure) • Remote persistent (survives node replacement) Kubernetes Components: Multi-Master API Server etcd data controller manager scheduler etcd kubectl kubelet @olgch, @kublr
  15. 15. • Persistent volumes • Volume provisioning • Storage categories • Native block storage: AWS EBS, Azure Disk, vSphere volume, attached block device, etc. • HostPath • Managed network storage: NFS, iSCSI, NAS/SAN, AWS EFS, etc. • Some of the idiosyncrasies • Topology sensitivity (e.g. AZ-local, host-local) • Cloud provider limitations (e.g. number of attached disks) • Kubernetes integration (e.g. provisioning and snapshots) Container Persistence @olgch, @kublr
  16. 16. • Integrates with Kubernetes • CSI, FlexVolume, or native • Volume provisioners • Snapshots support • Runs in cluster or externally • Approach • Flexible storage on top of backing storage • Augmenting and extending backing storage • Backing storage: local, managed, Kubernetes PV • Examples: Rook/Ceph, Portworx, Nuvoloso, GlusterFS, Linstor, OpenEBS, etc. Cloud Native Storage @olgch, @kublr
  17. 17. Data pool mon config data config data monmon config data Cloud Native Storage: Rook/Ceph raw data osd raw data osd raw data mdsosd Data pool Image Image Ceph Filesystem Components Abstractions Ceph rgw S3/Swift Object Store mgr Rook Operator CSI plugins osdosdganesha NFS CephCluster Block Pool Object Store Filesystem NFS Object Store User Provisioners rbd-mirror @olgch, @kublr
  18. 18. Operations: monitoring, log collection, alerting, etc. Lifecycle: CI/CD, SCM, binary repo, etc. Container management: registry, scanning, governance, etc. Deployment options: • Managed service • In Kubernetes • Deploy separately Middleware @olgch, @kublr
  19. 19. • Region to region; cloud to cloud; cloud to on-prem (hybrid) • One cluster (⚠) vs cluster per location (✔) Tasks • Physical network connectivity: VPN, direct • Overlay network connectivity: Calico BGP peering, Native routing • Cross-cluster DNS: CoreDNS • Cross-cluster deployment: K8S federation • Cross-cluster ingress, load balancing: K8S federation, DNS, CDN • Cross-cluster data replication • native: e.g. AWS EBS, Snapshots inter-region transfer • CNS level: e.g. Ceph geo-replication • database level: e.g. Yugabyte geo-replication, sharding, ... • application level Something Missing? Multi-Site @olgch, @kublr
  20. 20. To Recap... • Kubernetes provides robust tools for application reliability • Underlying infrastructure and Kubernetes components recovery is responsibility of the cluster operator • Kubernetes is just one of the layers • Remember Architecture 101 and assess all layers accordingly • Middleware, and even CNS, can run in Kubernetes and be treated as regular applications to benefit from K8S capabilities • Multi-site HA, balancing, failover is much easier with K8S and the cloud native ecosystem. Still requires careful planning! @olgch, @kublr
  21. 21. Q&A @olgch, @kublr
  22. 22. Oleg Chunikhin CTO | Kublr oleg@kublr.com Thank you! Take Kublr for a test drive! kublr.com/deploy Free non-production license @olgch, @kublr