MyDBOPS Team has presented on Oracle MySQL user Camp ( 29-07-2016 ). This presentation is about Grafana and Prometheus for MySQL alerting and Dashboard setup.

1. Modern MySQL Monitoring
and Dashboards
Karthik P R
&
Vinoth Kanna R S

2. Agenda
• Monitoring & Dashboards
• What DBA’s Expects in Monitoring ?
• Issues With Current Monitoring.
• Time series databases ! What’s New
• Prometheus as datastore.
• Grafana
• Architecture of monitoring.
• Integration and alerting.
• Real Time examples with MySQL.

3. About MyDBOPS
• MySQL Consulting and Support firm.
• Remote DBA Support
• 24*7 Support and MySQL Monitoring
• Expert MySQL Solutions
• MySQL Cluster Support
• Certified MySQL DBA’s for Support.
• Supporting multiple clients across APAC & US Time Zones.

4. About Presenters
Karthik.P.R
Lead MySQL Consultant, MyDBOPS
6+ years of experience in handling huge data sets in MySQL and its forks.
Prior to that worked at Yahoo! scaling huge data in MySQL. Expertise on
distributed systems in MySQL and MySQL solution provider. Active
MySQL blogger and Speaker too.
Vinoth Kanna RS
Lead MySQL Consultant, My dbops
4+ years experience in handling highly available, mission critical database
systems on MySQL and It's forks. Worked in Percona as a RDBA in multi
client projects before Joining MyDBOPS. Expertise includes proven
records on MySQL Performance Tuning and Scaling.

5. Monitoring & Dashboards
• System without Monitoring is like
flying plane high with blindfolded
eyes.
• Dashboard is like an
Instrumentation in your cockpit.
Make sure you are flying at optimal
condition.
• Several Monitoring solutions are
available in market ( Enterprise /
Open Source ).

6. What DBA’s expect in Monitoring ?
• Ability to visualize the db statistics.
• Integrate with communication channels like ‘slack’, ‘skype’, ‘IM’,’pagerduty’
• Easy sharing of graphs.
• Quick load time and Easy switching of time frames.
• Easy comparison of same stats across multiple hosts.
• Aggregation of related stats on single graphs.
• Display related graphs on a single dashboard.
• Trend Analysis

7. Issues with Present Dashboard Solutions
• Slow response.
• Lack of upgrade in UI design and implementation.
• Consume more disk space.
• More data to process, Hard to maintain retention.
• Not well suited in handling time series data.
• Lack of Integration.
• Not Mobile Friendly.

8. Time Series Database
• Specially developed to store and process high volume of incoming key
value data pairs indexed by time.
Eg: periodic data collected from several hundred heat sensors,
monitoring stats collected from cloud servers (db, linux, etc )
• Easy to filter based on arbitrary patterns defined by the today, yesterday,
day of the week, etc

9. Prometheus
• First choice for storing Server and DB statistics
• Self managed data retention.
• Better Instrumentation.
• Faster retrieval of data using LevelDB indexes.
• In Built , Configurable alert manager.
• Integration with communication channels.
• Can be used as a data store for monitoring dashboards.

10. Grafana
• Faster and beautiful Dashboards.
• More Granular Control.
• Quicker Range Selection.
• Multi Datastore Support ( Prometheus/InfluxDB/ ES / CW )
• Secure Auth ( LDAP, Google, Github OAuth )
• Rich and Custom templates.
• Simple App Integration ( Percona App )
• Easy Sharing.
• Boot strapped UI

11. Architecture of Monitoring.

12. Architecture of Monitoring.
• Node Exporter (9100)
• MySQL Exporter (9104)
• Prometheus (9090)
• Grafana Web UI (3000)
• YAML Config
Grants Required By MySQL Exporter:
GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'prom'@'localhost';
GRANT SELECT ON performance_schema.* TO 'prom'@'localhost';
#cat prometheus.yml
global:
scrape_interval: 5s
scrape_configs:
- job_name: linux
target_groups:
- targets: ['192.168.0.1:9100']
labels:
alias: db1
- job_name: mysql
target_groups:
- targets: ['192.168.0.1:9104']
labels:
alias: db1

13. Integration and Alerting
Prometheus
Alert Integration Architecture

14. Integration and Alerting
# cat alert.rules
ALERT linux_load_high
IF node_load1 > 5
FOR 5m
ANNOTATIONS {description="{labels} Load has constantly exceeded 5 over the past 5 Minutes.",
severity="warning", summary="Linux Load is at 5"}
Reference The Alert Rules On Prometheus Config
rule_files:
- /etc/prom/alert.rules

15. Integration and Alerting
# cat /etc/prom/alertmanager.conf
global:
slack_api_url: 'https://hooks.slack.com/services/########'
route:
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#mydbops_alerts'

16. Integration and Alerting

17. Real Time Examples With MySQL
CPU Usage in VM

18. Real Time Examples With MySQL
CPU Steal in VM

19. Real Time Examples With MySQL
TroubleShooting MySQL Handlers
Index merge intersect caused abnormal “read_next” . The optimizer switch was
disabled to reduce this excessive read operations.
set global optimizer_switch=’index_merge_intersection=off’;

20. Demo
GRAFANA DASHBOARDS
for MySQL

21. Reach Us !
www.mydbops.com
mysqlsupport@mydbops.com
080 - 41010410
Queries ?

22. Thank You !!!