Percona Live 2016 presentation on best practices to monitor large scale MySQL deployments.

1. Monitoring MySQL at SCALE

2. Who We Are
Ilan Rabinovitch
Dir. Technical Community
Datadog
Ovais Tariq
Storage SRE
Uber
(formerly at Lithium & Percona)

3. Agenda
1. About Lithium and MySQL
2. Background: Monitoring Challenges in a Dynamic World
3. Theory: Monitoring 101
4. Practical: Triaging a Real Incident at Lithium

4. About Lithium Technologies
Lithium’s platform helps brands connect,
engage and understand their customers

5. MySQL Architecture / Data Flow
•Multi-Tenant SaaS applications
•Typical Master-slave replication setup
•MySQL running
○ On bare metal
○ In AWS public cloud
○ In OpenStack

7. Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis
DevOps Day LA

8. Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis
DevOps Day LA

10. You’re in the cloud and it's everything
you dreamed of!
Autoscaling Infinite StorageManaged
Databases
Container
Orchestration
Private Clouds

12. Collecting data is cheap;
not having it when you
need it can be expensive

14. Instrument all the things!

15. Operational Complexity Increases with..
• Number of things to measure
• Velocity of change

16. How much we measure?
1 instance
• 10 metrics from CloudWatch
1 operating system (e.g., Linux)
• 100 metrics
MySQL Instance
• 350~ metrics

17. 460
metrics per host
46,000
100
instances

25. •Earlier - typical Nagios and Cacti setup
•Static config and lack of context
•No correlation between alerts and
graphs
•No self-service for developers
•In-house tooling has high cost

29. When to let a sleeping
engineer lie?

31. Recurse until you find root cause

32. • Query Time
• Queries Per Second
Data Sources
• Performance Schema
• MySQL Status Variables

33. • Query Time
• Queries Per Second
Sources:
• Performance Schema

34. • Disk Space Usage
• Threads_connected
• Threads_running
• Connection_errors_ internal
• Aborted_connects
• Connection_errors_ max_connections
Sources:
● Server Status Variables

35. • Configuration Change
• Code Deployment
• Service Started / Stopped
• MySQL Upgrades
• Failovers
• etc

45. Change in workload without an increase in
workload affected the schema ‘groupecasino’
• Workload characteristics change to make it more CPU bound
• No increase in IO activity
• Increase in number of read operations
• No change in types of read operations
• Similar number of range queries reading more rows

47. Monitoring 101: Alerting
https://www.datadoghq.com/blog/monitoring-101-alerting/
Monitoring 101: Collecting the Right Data
https://www.datadoghq.com/blog/monitoring-101-collecting-data/
Monitoring 101: Investigating performance issues
https://www.datadoghq.com/blog/monitoring-101-investigation/
Monitoring MySQL Performance Metrics
https://www.datadoghq.com/blog/monitoring-mysql-performance-metrics/
Collecting MySQL Metrics
https://www.datadoghq.com/blog/collecting-mysql-statistics-and-metrics/