“‘With microservices every outage is like a murder mystery’ is a common complaint. But it doesn’t have to be! This talk gives an overview on how to monitor distributed applications and how common monitoring APIs can help with this effort. We dive into: - Logs: Why structured logs should be the new norm of logging and how to centralize them. - Metrics: This includes both application metrics (using REST APIs or JMX for example) as well as system metrics, which are more similar to the classical ‘top’ output. - Traces: The Open Tracing Foundation is basically the golden standard of tracing at the moment. We use Sleuth to show how to trace requests through a distributed system and Zipkin to display how long each call takes. This combination of logs, metrics, and traces is also called the holy trinity of monitoring.”

1. Logs, Metrics, Traces
TheAPISideofMonitoring
Philipp Krenn @xeraa

7. Infrastructure | Developer Advocate

8. Disclaimer
I build highly monitored Hello World
apps

10. http://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

11. Logs

12. 2017-10-10 09:34:16.895 INFO [frontend,cf4dc621b927b96e,cf4dc621b927b96e,true] 25308 ---
[nio-8080-exec-8]
net.xeraa.frontend.FrontendController:
X-Span-Export=true, X-B3-SpanId=cf4dc621b927b96e, X-B3-TraceId=cf4dc621b927b96e, name=Peter
Calling something good

13. Nobody wants to parse

14. {
"X-Span-Export":"true",
"X-B3-SpanId":"cf4dc621b927b96e",
"name":"Peter",
"X-B3-TraceId":"cf4dc621b927b96e",
"@timestamp":"2017-10-10T09:34:16.895+00:00",
"severity":"INFO",
"service":"frontend",
"trace":"cf4dc621b927b96e",
"span":"cf4dc621b927b96e",
"parent":"",
"exportable":"true",
"pid":"25308",
"thread":"http-nio-8080-exec-8",
"class":"net.xeraa.frontend.FrontendController",
"mdc":"X-Span-Export=true, X-B3-SpanId=cf4dc621b927b96e, name=Peter, X-B3-TraceId=cf4dc621b927b96e",
"rest":"Calling something good"
}

15. Metrics

17. HTTP

18. JMX

19. Traces

20. 2017-03-29 08:08:35.090 INFO
[frontend,69a851ed700bf1fa,69a851ed700bf1fa,true]
23575 --- [p-nio-80-exec-2]
n.x.MicroserviceMonitoringApplication :
Calling home
2017-03-29 08:08:36.149 INFO
[frontend,69a851ed700bf1fa,10b1f0a0e2de1253,true]
23575 --- [p-nio-80-exec-3]
n.x.MicroserviceMonitoringApplication :
You called home

21. Application: frontend
Trace ID: 69a851ed700bf1fa
Span ID: 10b1f0a0e2de1253
Send to Zipkin: true

22. Zipkin Chrome Extension
https://github.com/openzipkin/zipkin-browser-extension

23. Conclusion

25. Logs
Metrics
Traces

26. More
Uptime, network, system & application metrics, security,
auditing,...

27. Your Turn
https://kibana.xeraa.wtf
https://github.com/xeraa/microservice-monitoring

28. Questions?
Philipp Krenn @xeraa
PS: Stickers