Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Centralizing Kubernetes and Container Operations

2.119 Aufrufe

Veröffentlicht am

While developers see and realize the benefits of Kubernetes, how it improves efficiencies, saves time, and enables focus on the unique business requirements of each project; InfoSec, infrastructure, and software operations teams still face challenges when managing a new set of tools and technologies, and integrating them into an existing enterprise infrastructure.

These meetup slides go over what’s needed for a general architecture of a centralized Kubernetes operations layer based on open source components such as Prometheus, Grafana, ELK Stack, Keycloak, etc., and how to set up reliable clusters and multi-master configuration without a load balancer. It also outlines how these components should be combined into an operations-friendly enterprise Kubernetes management platform with centralized monitoring and log collection, identity and access management, backup and disaster recovery, and infrastructure management capabilities.  This presentation will show real-world open source projects use cases to implement an ops-friendly environment.

Check out this and more webinars in our BrightTalk channel: https://goo.gl/QPE5rZ

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Centralizing Kubernetes and Container Operations

  1. 1. Centralizing Kubernetes and Container Operations Oleg Chunikhin | CTO, Kublr
  2. 2. Introductions Oleg Chunikhin CTO, Kublr • Nearly 20 years in the field of software architecture and development. • Joined Kublr as the CTO in 2016. • Kublr is an enterprise Kubernetes management and operations platform that helps accelerate Kubernetes adoption and containerized applications management for enterprises.
  3. 3. History • Custom software development company • Dozens of projects per year • Varying target environments: clouds, on-prem, hybrid • Unified application delivery and ops platform wanted: monitoring, logs, security, multiple env, ...
  4. 4. Docker and Kubernetes to the Rescue • Docker is great, but local • Kubernetes is great... when it is up and running • Who sets up and operates K8S clusters? • Who takes care of operational aspects at scale? • How do you provide governance and ensure compliance?
  5. 5. Enterprise Kubernetes Needs Developers SRE/Ops/DevOps/SecOps • Self-service • Compatible • Conformant • Configurable • Open & Flexible • Org multi-tenancy • Single pane of glass • Operations • Monitoring • Log collection • Image management • Identity management • Security • Reliability • Performance • Portability
  6. 6. Kubernetes Management Platform Wanted • Portability – clouds, on-prem, air-gapped, different OS’ • Centralized multi-cluster operations saves resources – many environments (dev, prod, QA, ...), teams, applications • Self-service and governance for Kubernetes operations • Reliability – cluster self-healing, self-reliance • Limited management profile – cloud and K8S API • Architecture – flexible, open, pluggable, compatible • Sturdy – secure, scalable, modular, HA, DR etc.
  7. 7. Central Control Plane: Operations K8S Clusters Cloud(s) Data center API UI Log collection Operations Monitoring Authn and authz, SSO, federation Audit Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev
  8. 8. Central Control Plane: Operations
  9. 9. Infrastructure Automation Cluster: Self-Sufficiency Central control plane MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store Secrets discovery Simple orchestration and configuration agent
  10. 10. Cluster: Portability • (Almost) everything runs in containers • Simple (single-binary) management agent • Minimal store requirements • Shared, eventually consistent • Secure: RW files for masters, RO for nodes • Thus the store can be anything: S3, SA, NFS, rsynced dir, provided files, ... • Minimal infra automation requirements • Configure and run configuration agent • Enable access to the store • Can be AWS CF, Azure ARM, BOSH, Ansible, ... • Load balancer is not required for multi-master; each agent can independently fail over to a healthy master Infrastructure Automation MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store Secrets discovery
  11. 11. Cluster: Reliability • Rely on underlying platform as much as possible • ASG on AWS • IAM on AWS for store access • SA on Azure, S3 on AWS • ARM on Azure, CF on AWS • Minimal infrastructure SLA tolerate temporary failures • Multi-muster API failover on nodes • Resource management, memory requests and limits for OS and k8s components Infrastructure Automation MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store
  12. 12. Central Control Plane: Logs and Metrics K8S Clusters Cloud(s) Data center API UI Operations Authn and authz, SSO, federation Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev Log collection Monitoring Audit
  13. 13. Centralized Monitoring and Log Collection. Why Bother? • Prometheus and ELK are heavy and not easy to operate; need attention and at least 4-8 Gb RAM... each, per cluster • Cloud/SaaS monitoring is not always permitted or available • Existing monitoring is often not container-aware • No aggregated view and analysis • No alerting governance
  14. 14. K8S Monitoring with Prometheus • Discover nodes, services, pods via K8S API • Query metrics from discovered endpoints • Endpoint are accessed directly via internal cluster addresses Kubernetes Cluster Prometheus Nodes K8S API Grafana Pods Discovery Srv Metrics
  15. 15. Centralized Monitoring Cluster registry PROMETHEUSGrafana K8S Proxy API nodes, pods, service endpoints Ship externally Ship externally Prometheus config Prometheus data Configurator Control plane KUBERNETES CLUSTER Prometheus (collector) Prometheus (collector)
  16. 16. Centralized Monitoring: Considerations • Prometheus resource usage tuning • Long-term storage (m3) • Configuration file growth with many clusters • Metrics labeling • Additional load on API server
  17. 17. Centralized Monitoring
  18. 18. K8S Logging with Elasticsearch • Fluentd runs on nodes • OS, K8s, and container logs collected and shipped to Elasticsearch • Kibana for visualization Kubernetes Cluster Elasticsearch Kibana Pods Logs
  19. 19. Prometheus (collector) RabbitMQ Centralized Log Collection Cluster registry K8S Proxy API Port forwarding MQTT Ship externally Messaging config Configurator Control plane RabbitMQ Shovel ElasticsearchLogstash Fluentd KUBERNETES CLUSTER filter filter analyze Ship externally MQTT Forwarder filter
  20. 20. Centralized Log Collection: Considerations • Tune Elasticsearch resource usage • Take into account additional load on API server • Log index structure normalization { "data": { "elasticsearch": { "version": "6.x" } } } { "flatData": [ { "key": "elasticsearch.version", "type": "string", "key_type": "elasticsearch.version.string", "value_string": "6.x" }, ... ] }
  21. 21. The Rest: Considerations • Identity management Use Identity Broker (e.g. KeyCloak): Users, Authn, Autzn, SSO, RBAC, Federation, ... • Backup and disaster recovery K8s metadata + app data/volumes: full cluster recovery or copy Docker image management Docker image registry (e.g. Nexus, Artifactory, Docker Hub); image scanning; air-gapped or isolated environment: image registries proxying and caching, “system” images
  22. 22. Q&A
  23. 23. Oleg Chunikhin Chief Technology Officer oleg@kublr.com @olgch Kublr | kublr.com @kublr Thank you!

×