Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Performance Monitoring
for Docker environments
Anomaly detection
Monitoring tools
Live demo
About me
@coscale
www.coscale.com
@spolfliet
stijn.polfliet@coscale.com
Performance Monitoring
for Docker environments
Anomaly detection
Anomaly: definition
Static alerts
TODO : more realistic business examples
!
!
!
?
seasonality
correlations
changing or dynamic
environment
Static alert limitations
Challenges
Machine learning perspective: unsupervised learning problem
? evaluation criteria ?
Challenges
statistical significance relevance
⇏
Simple technique: 3- rule
Holt-Winters
● seasonal exponential smoothing
● works quite well on ‘laboratory
data’
● calculation of prediction interval...
Sliding window approach
model
evaluation
of new
data
Local outlier factor
Existing instance based machine learning technique (lazy,
~kNN)
Based on concept of local density
loc...
Local outlier factor, no strong model assumption
spinning
process
Local outlier factor, no free lunch
Scaling: comparing apples and oranges
scale ⇒ distance ⇒ density ⇒ LOF-score
Autoscali...
Load balance detector
Compare multiple signals (mean + variance) in load-balanced environment
Performance Monitoring
for Docker environments
Monitoring Docker
Orchestrator (services, volumes, replication controllers, …)
Resource usage (cpu, memory, disk, network, ...)
Container in...
Docker Stats API
$ docker stats
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
1285939c1fd3 0.07% 796 KiB / 64 ...
docker run 
--volume=/:/rootfs:ro 
--volume=/var/run:/var/run:rw 
--volume=/sys:/sys:ro 
--volume=/var/lib/docker/:/var/li...
agent runs in 1 container
container resource usage
no container internals
no application monitoring
15$ / month / server
S...
agent runs in 1 container or on host
container resource usage
basic application monitoring
15$ / month / server
Datadog
da...
kernel module captures system calls
container resource usage
basic application monitoring
20$ / month / server
Sysdig
sysd...
deep application monitoring
too heavyweight for lightweight container environments
100+$ / month / server
APM vendors
Lightweight agent
• Server metrics from OS
• Container and cluster metrics from Kubernetes and Docker APIs
• Application m...
CoScale architecture
DEMO
Questions?
or contact me at stijn.
polfliet@coscale.com
@spolfliet
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup
Nächste SlideShare
Wird geladen in …5
×

Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup

815 Aufrufe

Veröffentlicht am

Performance monitoring for Docker
Challenges around Docker monitoring - Anomaly detection - CoScale demo
For more info about how to use CoScale Docker monitoring, some reading material here: http://www.coscale.com/blog/how-to-monitor-docker-containers-with-coscale and http://www.coscale.com/blog/how-to-monitor-your-kubernetes-cluster

A summary of CoScale Docker performance monitoring can be found here: http://www.coscale.com/docker-monitoring

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup

  1. 1. Performance Monitoring for Docker environments Anomaly detection Monitoring tools Live demo
  2. 2. About me @coscale www.coscale.com @spolfliet stijn.polfliet@coscale.com
  3. 3. Performance Monitoring for Docker environments Anomaly detection
  4. 4. Anomaly: definition
  5. 5. Static alerts TODO : more realistic business examples ! ! !
  6. 6. ? seasonality correlations changing or dynamic environment Static alert limitations
  7. 7. Challenges Machine learning perspective: unsupervised learning problem ? evaluation criteria ?
  8. 8. Challenges statistical significance relevance ⇏
  9. 9. Simple technique: 3- rule
  10. 10. Holt-Winters ● seasonal exponential smoothing ● works quite well on ‘laboratory data’ ● calculation of prediction intervals relies on normal distribution after removal of seasonality ● => on our real world seasonal data generates too many false positives
  11. 11. Sliding window approach model evaluation of new data
  12. 12. Local outlier factor Existing instance based machine learning technique (lazy, ~kNN) Based on concept of local density local outlier factor(A) = density at point A average density of kNN of point A LOF >> 1 ⇒ outlier en.wikipedia.org/wiki/Local_outlier_factor
  13. 13. Local outlier factor, no strong model assumption spinning process
  14. 14. Local outlier factor, no free lunch Scaling: comparing apples and oranges scale ⇒ distance ⇒ density ⇒ LOF-score Autoscaling? (Mahalanobis distance) => enlarges dimensions with low variance “Curse of dimensionality” dimensionality reduction preprocessing (e.g. PCA), but don’t throw away the anomalies with the bathwater Choosing cross-sections of data to analyze together, e.g. different metric on same container same metric on different containers
  15. 15. Load balance detector Compare multiple signals (mean + variance) in load-balanced environment
  16. 16. Performance Monitoring for Docker environments Monitoring Docker
  17. 17. Orchestrator (services, volumes, replication controllers, …) Resource usage (cpu, memory, disk, network, ...) Container internals (application, database, caching, etc.) Lightweight monitoring for lightweight microservices environment What to monitor?
  18. 18. Docker Stats API $ docker stats CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O 1285939c1fd3 0.07% 796 KiB / 64 MiB 1.21% 788 B / 648 B 3.568 MB / 512 KB 9c76f7834ae2 0.07% 2.746 MiB / 64 MiB 4.29% 1.266 KB / 648 B 12.4 MB / 0 B d1ea048f04e4 0.03% 4.583 MiB / 64 MiB 6.30% 2.854 KB / 648 B 27.7 MB / 0 B Docker API
  19. 19. docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest open http://<your-hostname>:8080/ CAdvisor
  20. 20. agent runs in 1 container container resource usage no container internals no application monitoring 15$ / month / server Scout scoutapp.com
  21. 21. agent runs in 1 container or on host container resource usage basic application monitoring 15$ / month / server Datadog datadoghq.com
  22. 22. kernel module captures system calls container resource usage basic application monitoring 20$ / month / server Sysdig sysdig.com
  23. 23. deep application monitoring too heavyweight for lightweight container environments 100+$ / month / server APM vendors
  24. 24. Lightweight agent • Server metrics from OS • Container and cluster metrics from Kubernetes and Docker APIs • Application metrics from log files and management interfaces • Business & custom metrics from various sources Contextual events • Container lifecycle • Deployments & software releases • Infrastructure changes • Custom events CoScale approach
  25. 25. CoScale architecture
  26. 26. DEMO
  27. 27. Questions? or contact me at stijn. polfliet@coscale.com @spolfliet

×