Deploy Prometheus - Grafana and EFK stack on Kubic k8s Clusters

1. Deploy Monitoring and
Centralized Logging
with Prometheus + Grafana
and EFK Stack
on Kubic k8s clusters
Syah Dwi Prihatmoko / sdmoko
sdmoko@opensuse.org

2. Syah Dwi Prihatmoko
Moko
@sdmoko

3. openSUSE-ID Member
GNU Linux Bogor Activist
Cloud Engineer at Btech
FOSS Contributor and Enthusiast

4. In this talks,
we'll learn how to deploy monitoring and logging with
Prometheus - Grafana and EFK Stack in Kubic k8s clusters.
The example is to monitoring and logging pods in k8s
clusters.

5. Kubic?

6. Deploy k8s on Kubic
openSUSE MicroOS Kubic came with kubeadm to make it easy to
bootstrapping a kubernetes cluster. Just run this command to init a k8s
cluster.
kubeadm init --pod-network-cidr=10.244.0.0/16
After this you need to configure user to talk to cluster by running
mkdir -p ~/.kube
cp -i /etc/kubernetes/admin.conf ~/.kube/config

7. Deploy k8s on Kubic
The next step is to set up the network plugin. Kubic recommended to use
flannel. Run the command to set up flannel.
kubectl apply -f /usr/share/k8s-yaml/flannel/kube-flannel.yaml
After this you can join the worker node, to get the join command execute
this command on master node.
kubeadm token create --print-join-command
https://blog.sdmoko.net/create-k8s-cluster-with-kubic.html

8. Monitoring
Vs
Logging

9. EFK Stack
Elasticsearch, Fluentd and Kibana

10. EFK Deployment Guide can be
found on my blog.
https://blog.sdmoko.net/efk-cluster
-on-kubernetes.html

11. Deploy EFK Stack
The YAML file available at https://github.com/sdmoko/EFK-kubernetes
First create a storage class
kubectl apply -f 00-storageClass.yaml
After this we need to create PV.
kubectl apply -f 01-pv-0.yaml
kubectl apply -f 02-pv-1.yaml
kubectl apply -f 03-pv-2.yaml

12. Deploy EFK Stack
Then we need to create a namespaces for a logging system.
kubectl apply -f 04-namespaces.yaml
After this create a service for elasticsearch cluster, so the Fluentd and
Kibana can use it.
kubectl apply -f 05-elasticsearch_svc.yaml
Then create a cluster elasticsearch which use the volume that we create
before.
kubectl apply -f 06-elasticsearch_statefulset.yaml

13. Deploy EFK Stack
Then deploy Kibana and Fluentd to make it a complete logging stack.
Where we can visualize the data.
kubectl apply -f 07-kibana.yaml
kubectl apply -f 08-fluentd.yaml

14. Kibana Dashboard

15. Prometheus + Grafana

16. Prometheus + Grafana

17. Prometheus + Grafana deployment guide can
be found on my blog.
https://blog.sdmoko.net/deploy-prometheus-gr
afana-on-kubernetes.html

18. Deploy Prometheus + Grafana
The yaml for this deployment can be found here
https://github.com/sdmoko/k8s-prometheus-grafana
First, we need to deploy prometheus as a metric storage (TSDB).
To deploy prometheus we need to add a RBAC for prometheus.
kubectl apply -f prometheus/00-prometheus-rbac.yaml
Then deploy a configmap for prometheus config.
kubectl apply -f prometheus/01-prometheus-configmap.yaml

19. Deploy Prometheus + Grafana
Then we can create a rules that will trigger the alertmanager.
kubectl apply -f prometheus/02-prometheus-rules.yaml
Then we can create a storage for prometheus store the data.
kubectl apply -f prometheus/03-prometheus-storage.yaml
Then we can create a deployment and service for prometheus.
kubectl apply -f prometheus/prometheus-deployment.yaml
kubectl apply -f prometheus/prometheus-service.yaml

20. Deploy Prometheus + Grafana
Then we deploy kube-metrics server to expose container and pod metrics
other than those exposed by cadvisor on the nodes.
kubectl apply -f kube-state-metrics/kube-state-metrics.yaml
Then we can deploy and provide service for grafana dashboard to visualize
the monitoring data.
kubectl apply -f grafana/grafana-deployment.yaml
kubectl apply -f grafana/grafana-service.yaml

21. Deploy Prometheus + Grafana
Then we open the grafana service from browser, and then add prometheus
data source and add a dashboard. For config of the grafana data source
use:
Name : DS_Prometheus
Type : Prometheus
URL : http://prometheus-service:8080

22. Grafana Dashboard

23. Questions ?

24. Reference
https://github.com/sdmoko/EFK-kubernetes
https://github.com/sdmoko/k8s-prometheus-grafana
https://medium.com/faun/production-grade-kubernetes-monitoring-using-prometheus-78144b835b60
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-o
n-kubernetes

25. Terima Kasih.
Thank You.