Cloud Native User Group: Prometheus Day 2
•
10 gefällt mir•1,314 views
提到 Kubernetes 要如何監測時，標準答案就是 Prometheus 裝下去就對了，但在擁有多個 K8S Cluster 時，組態如何管理，Metric 如何保管，AlertManager 如何設定…等該怎麼做比較好？此分享為 smalltown 透過 Prometheus Operator 整合 Thanos 來監控多個 K8S Cluster 的經驗談，期許大家聽完此分享可以縮短摸索的時間，更輕鬆的使用 Prometheus
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Cloud Native User Group: Prometheus Day 2
Ähnlich wie Cloud Native User Group: Prometheus Day 2 (20)
Mehr von smalltown
Mehr von smalltown (16)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Cloud Native User Group: Prometheus Day 2
- 2. HELLO!I am Tom Tsai aks smalltown - MaiCoin/AMIS SRE - Taipei HashiCorp User Group Organizer - AWS User Group Taiwan Staff 2
- 5. Outline ▫ Prometheus Basic ▫ Kubernetes Operator ▫ Prometheus Operator ▫ Thanos ▫ Internal/External Monitoring 5
- 7. Prometheus Introduction 7 Exporter Prometheus Alertmanager
- 8. Install Prometheus is Easy ▫ CM Tools: Ansible, Chef, Puppet, SaltStack ▫ Kubernetes Addons ▫ DigitalOcean ▫ ... 8
- 9. Maintain Prometheus is Bothering (1/2) 9 ▫ Problems Which Encounter Within K8S - Configure Target for Many Services - Configure Many Alert Rules - Grafana Dashboard Management
- 10. Maintain Prometheus is Bothering (2/2) 10 ▫ Problem Which Encounter Everywhere - How to Achieve HA - Reliable Historical Data - Downsampling
- 11. Exclude All Problems Today! 11
- 13. Manage Production Service 13 Stateless Stateful Setup Scale Out/Up Backup Upgrade Operation
- 14. K8S Operator 14
- 15. What is Kubernetes Operator ▫ An Operator is a Software with Specific Application Knowledge ▫ Extends the Kubernetes through Custom Controller/Resource ▫ Enable Users to Create, Configure, and Manage Applications 15
- 16. 16 What is Kubernetes Operator An Operator is a Domain Specific Controller
- 17. 17 Operation of Stateless Service kubectl ReplicaSet app=web, env=prod Pod app=web env=prod scale up desired=3 count=1 kubectl ReplicaSet app=web, env=prod desired=3 count=3 Pod app=web env=prod
- 18. Pod app=web env=prod 18 Operation of Stateful Service (Operator) backup kubectl etcd Operator app=3.0.13 size=3 backup-tool interval=300s
- 20. Maintain Prometheus is Bothering 20 ▫ Problems Which Encounter Within K8S - Configure Target for Many Services - Configure Many Alert Rules - Grafana Dashboard Management
- 21. Prometheus Operator Overview 21 K8S Service (Exporter) Prometheus Alertmanager GrafanaK8S CRD: ServiceMonitor PrometheusRule
- 22. 22 K8S CRD Within Prometheus Operator (1/2) ▫ Prometheus - Manage Prometheus StatefulSet - Prometheus Configuration Reloader - Prometheus Rules Reloader ▫ Alertmanager - Manage AlertManager StatefulSet - Configuration Reloader
- 23. 23 K8S CRD Within Prometheus Operator (2/2) ▫ ServiceMonitor - 1 ServiceMonitor Mapping to 1 Exporter Service - All ServiceMonitor Compose Prometheus Configuration ▫ PrometheusRule - 1 PrometheusRule Mapping to 1 Rule File
- 24. 24 How to Monitor A New Service ▫ Precondition - Prometheus Install/Manage By Operator ▫ Add New Exporter and Related Service ▫ Add New ServiceMonitor/PrometheusRule ▫ Use Helm to Deploy, and The Changes Will be Reloaded Automatically
- 25. 25 How to Add A New Grafana Dashboard ▫ Cook Dashboard ▫ Export Dashboard Json File ▫ Use Helm to Deploy to Related K8S ConfigMap ▫ Changes Trigger Grafana Watcher Sidecar to Reload Automatically
- 26. Maintain Prometheus is Bothering (1/2) 26 ▫ Problems Which Encounter Within K8S - Configure Target for Many Services - Configure Many Alert Rules - Grafana Dashboard Management ▫ Everything Uses Helm to Deploy and Store in Version Control
- 27. Thanos
- 28. Maintain Prometheus is Bothering 28 ▫ Problem Which Encounter Everywhere - How to Achieve HA - Reliable Historical Data - Downsampling
- 29. Thanos Main Components ▫ Sidecar ▫ Querier ▫ Store ▫ Compactor 29
- 30. 30 Global View Disk Sidecar Prometheus Disk Sidecar Prometheus Deduplicate Marge Querier ... Thanos Store API Prometheus HTTP API
- 31. 31 Unlimited Retention (1/2) Disk SidecarPrometheus Blocks Blocks Object Storage Blocks
- 32. 32 Unlimited Retention (2/2) Object Storage Blocks Querier Store Cache Blocks Thanos Store API
- 33. 33 Compaction & Downsampling Object Storage Blocks Compactor Disk Blocks Blocks Blocks
- 34. 34 Thanos Overview Object Storage Compactor Store Querier Disk Sidecar Prometheus Disk Sidecar Prometheus ...5 1 1 2 3 4
- 35. Maintain Prometheus is Bothering 35 ▫ Problem Which Encounter Everywhere - How to Achieve HA - Reliable Historical Data - Downsampling
- 39. A Stone's Throw Away ▫ Who Monitor Your Monitor (Prometheus) ▫ Multi-Region Monitoring ▫ Some More Complicated Monitor ▫ ... 39
- 40. 40 The 80/20 Rule ▫ Pay Some Money to Use External Monitor to Avoid Blind Spot ▫ E.g. Site 24X7, Monitis...etc
- 41. 41 THANKS! Any questions? You can find me at ▫ facebook.com/smalltown0110