Kubernetes is the new cool in 2023. Many database installations are on Kubernetes now. And this creates challenges for Support engineers because traditional monitoring and diagnostic tools work differently on bare hardware and Kubernetes. In this session, I will focus on differences in methods we use to collect metrics, describe challenges that Percona Support hits when working with database installations on Kubernetes, and discuss how we resolve them. This talk will cover all database technologies we support: MySQL, MongoDB, and PostgreSQL. Presented at Percona Live 2023

1. Database in Kubernetes:
Diagnostics and Monitoring
Sveta Smirnova
Principal Support Engineering Coordinator

2. • MySQL Support Engineer
• Author
MySQL Troubleshooting
MySQL Cookbook, 4th Edition
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova

3. Database Performance Influencers
3
© Copyright 2023 Percona ® LLC. All rights reserved

4. • Standard database tools: EXPLAIN
• Interactive tools
• Log files
Query Performance Diagnostics
4
© Copyright 2023 Percona ® LLC. All rights reserved

5. • Interactive tools
• Same as on bare metal
Concurrency Diagnostics
5
© Copyright 2023 Percona ® LLC. All rights reserved

6. • Managed by Kubernetes
• Limits, set by Kubernetes
• DBA does not have access to tools,
available on bare hardware
Hardware
6
© Copyright 2023 Percona ® LLC. All rights reserved

7. • Supported
PMM
7
© Copyright 2023 Percona ® LLC. All rights reserved

8. • Supported
• All Percona operators are shipped with
PMM client
PMM
8
© Copyright 2023 Percona ® LLC. All rights reserved

9. • Supported
• All Percona operators are shipped with
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
PMM
9
© Copyright 2023 Percona ® LLC. All rights reserved

10. • Supported
• All Percona operators are shipped with
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
• Add pmm.enabled=true into
deploy/cr.yaml, configure PMM server
host and API key, then apply
PMM
10
© Copyright 2023 Percona ® LLC. All rights reserved

11. Database Performance Troubleshooting
11
© Copyright 2023 Percona ® LLC. All rights reserved

12. Database Performance Troubleshooting
12
© Copyright 2023 Percona ® LLC. All rights reserved

13. • Global buffers to cache "hot" data
• InnoDB Buffer Pool
• WiredTiger Cache
• Shared buffers in PostgreSQL
• Operation and thread-specific buffers
• "Internal" buffers not under user control
How Memory Used by Databases?
13
© Copyright 2023 Percona ® LLC. All rights reserved

14. • Physically available for nodes
• Limited by the Operator
resources:
requests:
memory: 32G
...
# limits:
# memory: 32G
Memory Limits
14
© Copyright 2023 Percona ® LLC. All rights reserved

15. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
Memory Diagnostics
15
© Copyright 2023 Percona ® LLC. All rights reserved

16. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
• Database-specific instruments
• Performance Schema in MySQL
Memory Diagnostics
16
© Copyright 2023 Percona ® LLC. All rights reserved

17. • Table data
• Indexes
• Log files
• Temporary data
How Disk Used by Databases?
17
© Copyright 2023 Percona ® LLC. All rights reserved

18. volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 2G
Disk Limitations
18
© Copyright 2023 Percona ® LLC. All rights reserved

19. • kubectl get persistentvolumeclaims
Disk Diagnostics
19
© Copyright 2023 Percona ® LLC. All rights reserved

20. • kubectl get persistentvolumeclaims
• kubectl describe nodes
Disk Diagnostics
20
© Copyright 2023 Percona ® LLC. All rights reserved

21. • kubectl get persistentvolumeclaims
• kubectl describe nodes
• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
Disk Diagnostics
21
© Copyright 2023 Percona ® LLC. All rights reserved

22. • kubectl get persistentvolumeclaims
• kubectl describe nodes
• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
• Counters in the database
Disk Diagnostics
22
© Copyright 2023 Percona ® LLC. All rights reserved

23. • Handle connections
• Threads in MySQL, MongoDB
• Sub-processes in PostgreSQL
• Perform calculations
• Perform IO operations
How CPU Used by Databases?
23
© Copyright 2023 Percona ® LLC. All rights reserved

24. resources:
requests:
cpu: "16"
limits:
cpu: "32"
CPU Limits
24
© Copyright 2023 Percona ® LLC. All rights reserved

25. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
CPU Diagnostics
25
© Copyright 2023 Percona ® LLC. All rights reserved

26. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
• Counters in the database
CPU Diagnostics
26
© Copyright 2023 Percona ® LLC. All rights reserved

27. • Communication between server and
client
• Regular client
• Application
• Replicas
• Synchronous nodes
How Databases Use Network?
27
© Copyright 2023 Percona ® LLC. All rights reserved

28. • kubectl describe nodes
Network Diagnostics
28
© Copyright 2023 Percona ® LLC. All rights reserved

29. • kubectl describe nodes
• Logs and statuses of the namespace
kube-system and your CNI
Network Diagnostics
29
© Copyright 2023 Percona ® LLC. All rights reserved

30. • kubectl describe nodes
• Logs and statuses of the namespace
kube-system and your CNI
• Latency in databases
Network Diagnostics
30
© Copyright 2023 Percona ® LLC. All rights reserved

31. • By default, log files are gone after pod
failure
• Critical in such situations as database
crash
Log Files Rotation
31
© Copyright 2023 Percona ® LLC. All rights reserved

32. • By default, log files are gone after pod
failure
• Critical in such situations as database
crash
• Solution
• PXC Operator: separate pod to store logs
• Other operators: K8-recommended
solutions such as Grafana Loki
Log Files Rotation
32
© Copyright 2023 Percona ® LLC. All rights reserved

33. • Created for Support team to automate
K8 troubleshooting collections
• Owned by Support team
• In active progress
• New featurs each version
pt-k8s-debug-collector
33
© Copyright 2023 Percona ® LLC. All rights reserved

34. pt-stalk
• Default for bare
metal
• Bare hardware
• Such tools as
• iostat
• vmstat
• dmesg
• top
pt-k8s-debug-collector
• Default for K8
• kubectl
• Such commands
as
• get pods
• get
storageclasses
• get replicasets
• get logs
Batch Data Collection
34
© Copyright 2023 Percona ® LLC. All rights reserved

35. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
Features in pt-k8s-debug-collector
35
© Copyright 2023 Percona ® LLC. All rights reserved

36. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
Features in pt-k8s-debug-collector
36
© Copyright 2023 Percona ® LLC. All rights reserved

37. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
• kubectl logs
Features in pt-k8s-debug-collector
37
© Copyright 2023 Percona ® LLC. All rights reserved

38. • pt-stalk
• More kubectl commands
• Extended MongoDB support
• Nicer output format for summaries
Future of the pt-k8s-debug-collector
38
© Copyright 2023 Percona ® LLC. All rights reserved

39. • Mind limits in cr.yaml
• Work with K8 administrators for setting
up policies, allowing you to view
hardware usage
• Setup Loki or similar solution to save the
log files after restart
• Enable online database
instrumentation
• Use PMM or similar monitoring solution
• Plan in advance
Conclusion
39
© Copyright 2023 Percona ® LLC. All rights reserved

40. Percona K8 Operators
PMM Demo
pt-k8s-debug-collector
pt-stalk
pg_gather
Grafana Loki
Tools Mentioned
40
© Copyright 2023 Percona ® LLC. All rights reserved

41. Thank you!
/svetsmirnova /svetasmirnova
/svetsmirnova /SvetaSmirnova