Weitere ähnliche Inhalte
Ähnlich wie ROMA NOVIKOV, BAQ, "Prometheus + grafana based monitoring" (20)
Kürzlich hochgeladen (20)
ROMA NOVIKOV, BAQ, "Prometheus + grafana based monitoring"
- 1. © 2017 Percona1
Roma Novikov
Prometheus + Grafana based Monitoring
Director of Platform Engineering
BAQ CONFERENCE 2019
24 March, 2019
- 2. © 2017 Percona2
About me
Roma Novikov
▪Percona - Director of Platform Engineering (2+ years)
▪Since 2001: Web developer -> Lead/Architect -> Manager -> CTO
▪Interests
• web, highload, monitoring and observability.
- 12. © 2017 Percona12
Why do you need The Monitoring?
Before: Get to know stakeholders and their needs
- 29. © 2017 Percona29
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”
● What about data from “yesterday”?
- 30. © 2017 Percona30
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”
● What about data from “yesterday”?
● What's going on now with another part of the system? How it's
affected?
- 32. © 2017 Percona32
Methods to use - U.S.E.?
USE
▪ For every resource, check utilization, saturation, and errors.
• utilization: the average time that the resource was busy servicing work
• saturation: the degree to which the resource has extra work which it can't service,
often queued
• errors: the count of error events
Brendan D. Gregg : http://www.brendangregg.com/usemethod.html
- 33. © 2017 Percona33
What to use - R.E.D.?
RED
▪ The RED Method defines the three key metrics you should measure for every
microservice in your architecture. Those metrics are:
• (Request) Rate - the number of requests, per second, you services are serving.
• (Request) Errors - the number of failed requests per second.
• (Request) Duration - distributions of the amount of time each request takes.
Tom Wilkie https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/
- 38. © 2017 Percona38
What/Why should I use? - Commercial
● “embedded” From your provider
○ AWS
○ DigitalOcean monitoring
○ ...
- 39. © 2017 Percona39
What/Why should I use? - Commercial
● “embedded” From your provider
○ AWS
○ DigitalOcean monitoring
○ ...
● Limitations
○ features
○ coverage
- 41. © 2017 Percona41
What/Why should I use? - Commercial
● SAAS
○ VividCortex
○ DataDog
○ NewRelic
○ ….
● Limitations
○ $
○ you send data outside your network
- 43. © 2017 Percona43
What/Why should I use? - Open Source
● OpenSource
○ Zabbix
○ Cacti
○ Nagios
○ ...
● Limitations
○ Open Source VS Free
○ Visualization
- 56. © 2017 Percona56
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node
- 57. © 2017 Percona57
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node
● Collection via pull over HTTP
- 58. © 2017 Percona58
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node
● Collection via pull over HTTP
● Simple targets discovery (or static configuration)
- 65. © 2017 Percona65
Prometheus - Exporter
● Usage
○ 150 + exporters mentioned on the website
○ reality - much more
● How to Extend?
○ just include instrumentation lib into your project
■ Go
■ Java or Scala
■ Python
■ Ruby…..
- 68. © 2017 Percona68
Prometheus - Exposition Format
● Exporter format
○ Simple
○ Text-based
● Example : http://localhost:port/metrics
○ # HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
- 70. © 2017 Percona70
Grafana
● The analytics platform for all your metrics
○ Grafana allows you to query, visualize, alert on and understand your
metrics no matter where they are stored. Create, explore, and share
dashboards with your team and foster a data driven culture.
https://grafana.com/grafana
- 76. © 2017 Percona76
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications
- 77. © 2017 Percona77
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications
● Annotations
- 78. © 2017 Percona78
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications
● Annotations
● You can show it to Manager. There is no shame
- 81. © 2017 Percona81
Make them work together - Default
1. Start Exporter
2. Configure Exporter
3. Start Prometheus
4. Configure Prometheus
5. Edit config and Add Exporter details
6. Reload Prometheus Config
7. Start Grafana
8. Set up Grafana and Data source
9. Create Dashboard
- 83. © 2017 Percona83
Make them work together - simplified
1. install PMM Client
2. Install PMM Server (docker)
3. pmm-admin add mysql <MYSQL_PARETERS>
4. Done
- 89. © 2017 Percona89
What is PMM and Why it was created
▪Our own
▪On premise / host your own
▪Free and Open source
- 94. © 2017 Percona94
PMM Dashboards
● 40+ Dashboards (DB and Host metrics)
● Created and used by Percona’s Experts
- 96. © 2017 Percona96
Technical details
Distribution methods
• Docker (server)
• AMI (server)
• OVA (server)
• Linux packages (client)
- 97. © 2017 Percona97
Technical details
Distribution methods
• Docker (server)
• AMI (server)
• OVA (server)
• Linux packages (client)
What is supported
▪MySQL Databases (MySQL, MariaDB, Percona Server, AWS/Aurora)
▪NoSQL Databases(MongoDB, Percona Server for MongoDB)
▪OS (Linux,..)
- 106. © 2017 Percona106
How to Extend?
● Create your Dashboards
● Add existed exporters or write your own
○ Add them as External exporter with one simple command
○ pmm-admin add external:service job_name [instance]
--service-port=PORT_NUMBER
- 107. © 2017 Percona107
Join Us
● DevOps/QA Engineer
● Database QA Engineer
● Golang and K8s SE
● Junior MySQL DBA
// All Remote