ZMON is OS monitoring in the cloud used by Zalando for their operations. It provides: 1) Flexible checks and alerts written in Python that integrate via REST APIs and OAuth. 2) Team dashboards and alert inheritance for great team collaboration. 3) Fast scaling metrics using Redis, KairosDB and Grafana for displaying historical data. 4) Notifications via email and mobile apps along with full authentication of all endpoints.

1. ZMON - OS monitoring in the cloud
Atmosphere 2016 | Krakow 17.5.2016 | jan.mussler@zalando.de | @JanMussler

2. 15 countries
3 fulfillment centers
18+ million active customers
3.0+ billion € revenue
135+ million visits per month
1.000+ employees in tech
Europe's Leading Fashion Platform
Visit us: tech.zalando.com

3. Zalando’s Technology History

4. RADICAL AGILITY
AUTONOMY

5. ➊ One AWS account per Team
➋ Deployment with Docker
➌ Managed SSH Access
➍ REST/OAuth 2.0 mandatory
➎ Traceability of changes
IN A NUTSHELL
STUPS

6. AWS
DEPLOYMENT
Senza CLI
Deploy Tool
Pier One
Docker Registry
docker pull
docker push
Taupage
AMI

7. Internet
*.abc.example.org *.xyz.example.org
Team ABC Team XYZ
ISOLATED AWS ACCOUNTS
EC2EC2
ELBELB
EC2

8. ZMON

9. Flexible and extendable: Checks & Alerts in Python
Integrate: REST APIs, OAUTH2, AWS Auto Discovery
Fully configurable via UI / API: no restarts required!
Great for teams: team dashboards, alerts inheritance
Fast/scaling metrics: Redis, KairosDB + Grafana2
Hackweek 2015 - iOS app and Android app ;-)
ZMON - High Lights ;-)

14. Display historic data using Grafana 2

15. Notifications plus iOS and Android App
E-Mail

16. Full authentication for all endpoints
OAUTH2 login flow (e.g. via Github login)
“TV Tokens” for “read-only” dashboard login
Grafana 2 bundled and API implemented
● ZMON stores dashboards incl. tags/stars
● KairosDB proxy
● ElasticSearch proxy (in progress)
ZMON Controller -> UI + REST API

17. Example

18. Tokeninfo (GO)Tokeninfo (GO)
Provider (Java)
Provider (Java)
Tokeninfo (GO)Tokeninfo (GO)
C* Nodes
C* Nodes
C* Nodes
C* Nodes
Plan B Deployment - Multi Region Setup (JWT issue/verification)
C* NodesProvider (Java)ELB
Tokeninfo (Go)ELB
C* NodesProvider (Java)ELB
Tokeninfo (Go)ELB

19. Will create “entities” to describe deployment
ELBs, ASGs, Application, instances,...
Crawls AWS API every 60 sec to update
ZMON AWS Agent - Auto Discovery

20. ➜ ~ zmon entities get "planb-tokeninfo-cd44-oFM6x[aws:999:eu-west-1]"
id: planb-tokeninfo-cd44-oFM6x[aws:999:eu-west-1]
type: instance
application_id: planb-tokeninfo
host: 172.31.169.6
infrastructure_account: aws:999
instance_type: c4.xlarge
ip: 172.31.169.6
ports: { '9020': 9020, '9021': 9021 }
region: eu-west-1
source: registry.opensource.zalan.do/stups/planb-tokeninfo:cd44
stack_name: planb-tokeninfo-eu-west-1
stack_version: cd44
Example Instance Entity

21. Instance Metrics
● Memory usage
● Disk space usage
● CPU usage
● Application logs
● Application metrics
Monitoring Plan-B instances on AWS
Scalyr Agent
Log shipping
Prometheus
Node Agent
:9100/metrics
Taupage AMI (Ubuntu base)
Application Container
Go / Spring Boot / Cassandra
Docker run time
:8080 -> app
:7979 -> metrics

22. Jolokia Request Example

23. Check Results

24. Alert on application metrics

25. HTTP requests reading JSON application metrics
Read JMX data via Jolokia/HTTP for Cassandra
Read Prometheus Node data for EC2 metrics
CloudWatch() queries for ELB metrics
Scalyr API queries for application logs
Check commands used so far

26. Annotated Metric Data in Grafana

27. Annotated Metric Data in Grafana

28. Entities

29. ● hosts, databases, applications, instances ...
● generic key value object
● 10000+ entities in our deployment
Entities
{
"id": "node01:8080",
"type": "instance",
"host": "node01",
"ports": {"8080":8080,"8181":8181},
"application_id": "zmon",
"application_version": "0.1.0",
"dc":"dc1"
}
Entity "node01:8080"

30. Entity Service (part of controller)
id: localhost:5432
type: postgres
host: localhost
port: 5432
shards:
local_zmon_db: "localhost:5432/local_zmon_db"
local-postgres.yaml
Integrated easy-to-use entity store with REST API
Build your own discovery agent (K8S, …)
>zmon entities push local-postgres.yaml

31. Checks

32. ● select subset of entities
● executes Python expression
○ powerful using eval with custom context
○ Builtins: HTTP, PostgreSQL, MySQL, CloudWatch,
Redis, SNMP/NRPE, tcp, SOAP, Scalyr, ES, ...
● returns "value" object
○ Quickly, every check returned "dicts"
Checks

33. REST API to update or use web front end
zmon check-definitions update select-1-check.yaml
Managing checks
name: "Select 1"
owning_team: "Team ZMON"
command: |
sql().execute("select 1 as a").results()
entities:
- type: postgresql
interval: 15
description: "Test connection to PostgreSQL"
select-1-check.yaml

35. Trial Run - Quick feedback and easier development

36. Alerts

37. ● Executes using a check’s value, bound to single check
● Defines team and responsible team
● Allows inheritance from other alert
● Evaluates Python expression yielding True/False
● No "WARNING" state, no "UNKNOWN" state
● Priorities(color) and tags
Alerts

40. Downtimes
● Set or schedule downtimes using the UI
● Use API to automate downtimes, e.g. in deployment tool

41. Anyone can add alerts to checks
Alerts are owned by team
Monitor application boundaries/dependencies
Make use of inheritance to customize
Sharing and reuse of alerts and checks

42. Deployment

43. Workers
(Python)
Workers
(Python)
ZMON Core + UI + KairosDB
Scheduler
(jvm)
Redis
Worker
(Python)
KairosDB
(Java)
Controller
(Java)
PostgreSQL
Queue/State
CLI
(Python)
Check/Alert definition
Entity data
Cassandra
Frontend
(AngularJS)
Metric Cache

44. ZMON in AWS / Multi DC Setup
*.foo.example.org *.bar.example.org
Team "Foo" Team "Bar"
EC2
Instance
EC2
InstanceEC2
Instance
EC2
Instance
ZMON
Appliance
ZMON
ApplianceEC2
Instance
EC2
Instance
ZMON
Data Service
ELB ELB

45. ● Scheduler supports queue filters by entity
○ e.g. {"dc":"dc1"} vs {"dc":"dc2"} queue filters
● Scheduler can apply base filter
○ only handles entities with {"dc":"dc1"}
● Worker can report home using:
○ Redis (we use this across DCs)
○ HTTPS (AWS->DC)
Multi DC / Zone deployment possible

47. Micro
Services

48. Expose your data / Convention on key names/structure
{
"zmon.response.200.GET.checks.all-active-check-definitions.count": 10,
"zmon.response.200.GET.checks.all-active-check-definitions.fifteenMinuteRate": 0.18071,
"zmon.response.200.GET.checks.all-active-check-definitions.fiveMinuteRate": 0.15181,
"zmon.response.200.GET.checks.all-active-check-definitions.oneMinuteRate": 0.10512,
"zmon.response.200.GET.checks.all-active-check-definitions.75thPercentile": 1173,
"zmon.response.200.GET.checks.all-active-check-definitions.95thPercentile": 1233,
"zmon.response.200.GET.checks.all-active-check-definitions.999thPercentile": 1282,
"zmon.response.200.GET.checks.all-active-check-definitions.99thPercentile": 1282,
"zmon.response.200.GET.checks.all-active-check-definitions.max": 1282,
"zmon.response.200.GET.checks.all-active-check-definitions.median": 1161,
"zmon.response.200.GET.checks.all-active-check-definitions.min": 1114
}

49. Application metrics

50. Continued ...

51. Spring boot (extending metrics)
https://github.com/zalando/zmon-actuator
Python (Swagger first on Flask)
https://github.com/zalando/connexion
Clojure (Swagger first)
https://github.com/zalando-stups/friboo/
Example libraries and framework support ...

52. Demo:
https://demo.zmon.io
ZMON on Github:
https://github.com/zalando/zmon
Documentation:
https://docs.zmon.io
Zalando Tech:
https://tech.zalando.com