Kubenetes monitoring basic Heapster, influxdb, prometheus, cAdvior Kubenetes dashboard

1. Kubernetes
#2. Monitoring
조대협 (http://bcho.tistory.com)

2. Agenda
● Kubernetes Monitoring Concept
● Kubernetes Monitoring Architecture
● Kubernetes Dashboard
● Google Stackdriver
● Datadog

3. Kubernetes
Monitoring Concept

4. Kubernetes monitoring stack
1. Host-level metrics and event (CPU,Memory,Disk etc)
2. Container(Docker/Pod) level metrics
(CPU,Memory,Disk)
3. Application metric (APM etc)
4. Kubernetes level - events, by label
Image source : https://www.datadoghq.com/blog/how-to-collect-and-graph-
kubernetes-metrics/

5. K8u monitoring solutions market (2017)
Source : https://thenewstack.io/5-tools-monitoring-kubernetes-scale-production/

6. Kubernetes
Monitoring
Architecture

7. Kubernetes resource monitoring
● cAdvisor collect metrics from nodes and pods and
send it into Kubelet
● Kublet sends all of pods in nodes information to
Heapster.
● Heapster runs as a Pod in cluster
● Heapster stores all of info into storage backend
(Google Stack driver, influx DB)
● It can be visualized by using Grafana
Source : https://www.datadoghq.com/blog/how-to-collect-and-graph-
kubernetes-metrics/

8. Components
cAdvisor
● Open source container resource usage and
performance analysis agent
● Auto discovers all containers in the machine
● Collect CPU,memory,file system and network
usage statics for containers
● Overall machine usage (node) by analyzing the
‘root’ container on the machine
InfluxDB
● Open source time series DB
Grafana
● Dashboard visualization tools

9. Components
cAdvisor application metric (Alpha)
● https://github.com/google/cadvisor/blob/mast
er/docs/application_metrics.md
● cAdvisor can also be configured to collect application
metrics. A container can expose application metrics through
multiple ways
○ on a status page,
○ through structured info like prometheus,
○ or have a separate API for fetching stats.
● Endpoint (Location to collect metrics
from)
● Name of metric
● Type (Counter, Gauge, ...)
● Data Type (int, float)
● Units (kbps, seconds, count)
● Polling Frequency
● Regexps (Regular expressions to
specify which metrics to collect and
how to parse them)
Step 2. Passing configuration to cAdvisor
Step 1. Create configuration

10. Kubernetes Logging
● It supports logging with ELK or Google
stack driver
● ELK
○ The fluentd-elasticsearch pods gather logs from
each node and send them to the elasticsearch-
logging pods
○ These Elasticsearch pods store the logs and
expose them via a REST API.
○ The kibana-logging pod provides a web UI for
reading the logs stored in Elasticsearch, and is
part of a service named kibana-logging
Image source : https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/

11. Kubernetes
dashboard

12. Dashboard
Dashboard is Web-UI based kubernetes interface
● Overview of application running on cluster
● Monitoring resources
● Create or modifying Kubernetes resources
(such as deployments, jobs, daemonset etc)
For example, you can scale a Deployment, initiate
a rolling update, restart a pod or deploy new
applications using a deploy wizard.
Dashboard is not deployed in default. To deploy dashboard run script
% kubectl create -f
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

13. Connect to dashboard
https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above
● For dev
○ kubectl proxy
○ http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-
dashboard:/proxy/
● In production

14. Create sample user
● Guide : https://github.com/kubernetes/dashboard/wiki/Creating-
sample-user

15. Logs

16. Shell
It can also run bash shell thru dashboard

17. Datadog for
Kubernetes

18. Reference
● GKE monitoring with datadog
https://www.datadoghq.com/blog/monitor-google-kubernetes-engine/