In the cloud native era systems are getting ever more dynamic and complex. With containers and microservices architecture, monitoring and troubleshooting systems is more challenging than ever before. The open source community has risen up to the challenge and delivered solutions that fit modern environments. Open source projects such as Prometheus and the ELK Stack have gathered massive adoption with developers and DevOps engineers, who also carry this skillset between companies and grow the adoption. New open standards, such as OpenMetrics, OpenTracing and OpenTelemetry, are emerging to converge the industry and prevent vendor lock-in. In this talk I will talk about observability, the recommended open source tools and standards, and how to combine them to help you achieve effective observability in your environment.

1. 1
Open Source for
Better Observability
@horovits
Dotan Horovits

2. 2
Quiz: what’s that?

3. Dotan Horovits
@horovits
• Principal Developer Advocate at
• 20 years in the hi-tech industry
• Developer, architect, product
• Co-organizer of DevOpsDays & CNCF Tel Aviv
• Run OpenObservability Talks podcast

4. 4
• What’s observability
• Role of open source in observability
• The leading open source tools
• OpenTelemetry and the unified vision
Agenda
Useful links ahead 📷

5. 5
Monitoring Cloud Native Systems is Hard

6. 6
Monitoring
Cloud Native
Systems
is Hard

7. 7
Cloud Native Monitoring Requires
Observability
“... loosely coupled systems that are resilient,
manageable, and observable.”
in Control Theory:
“Observability is a measure of how well internal states of a
system can be inferred from knowledge of its external outputs.”
Metrics
Logs Traces
Detect
Diagnose Isolate &
improve
Observability
Cloud Native Definition

8. 8
Cloud Native Monitoring Requires
Observability
“... loosely coupled systems that are resilient,
manageable, and observable.”
Metrics
Logs Traces
Detect
Diagnose Isolate &
improve
Observability
In plain English for software systems:
"capability to ask and answer questions based on telemetry data"
Cloud Native Definition
InsideBIGDATA: https://bit.ly/o11y-data-analytics

9. The “What”, the “Why”, the “Where”
Metrics, Logs, Traces

10. 10
Metrics — the “What”
• Detect issues, tell what happened
• Numerical (counters, gauges, histograms)
• Time series storage and query
• Typically combined with alerting on events

11. 11
Logs — the “Why”
• Diagnose issues, tell why it happened
• Textual
• Parse, index, store, query, analyze

12. 12
Traces — the “Where”
• Isolate issues, tell where it happened
• Improve performance
• Construct the trace based on causality

13. 13
3 Pillars of Observability
The ability to ask and answer questions about our system
• Metrics — the “What”
• Logs — the “Why”
• Traces — the “Where”

14. Role in Observability
Open Source

15. 15
By 2025, 70% of new cloud-native
application monitoring will use
open-source instrumentation, rather than
vendor-specific agents for improved
interoperability.
Open Source is the preferred choice
Open source monitoring tools in
use at nearly 60% of surveyed
organizations
Dec 2, 2020
April, 2021
Sep 2, 2020
The most commonly adopted
tools are open source

16. 16
There’s no consolidation
● >¼ use 10-20 tools
● Greatest challenge with observability:
Engineers and teams using
multiple tools (50%)
Cloud Native Observability
Microsurvey 2022
https://bit.ly/cncf-o11y-2022

17. 17
Relicensing is changing OSS landscape
Is “vendor owned open source” an oxymoron? https://horovits.medium.com/b5486a4de1c6
Source: opensource.google

18. 18
Open Source in Observability
• OSS is key
• Tool sprawl
• Relicensed OSS

19. OpenWhat??
https://horovits.medium.com/e9e48

20. 20
Metrics — the “What”
Open Source
* Update Apr 2021: relicensed from Apache2 to AGPLv3 by Grafana Labs.
*

21. 21
Logs — the “Why”
Open Source
* Update Feb 2021: relicensed from Apache2 to non-OSS (SSPL) by Elastic BV.
*
*
**
** Update Apr 2021: relicensed from Apache2 to AGPLv3 by Grafana Labs.

22. 22
Traces — the “Where”
Open Source

23. 23
Observability is more than
Logs+Metrics+Traces
Metrics
Logs Traces
Detect
Diagnose Isolate &
improve
Observability
Unstructured and ad-hoc
Structured, machine-formatted, correlated
Reactive monitoring
Proactive data analysis and insights
Logging/metrics/tracing silos
Unified data collection and analytics

24. 24
How do we generate and capture
Telemetry?
API
SDK Daemon
Collector
Protocol
Agent

25. 25
OpenTelemetry
“OpenTelemetry is an observability framework - software and
tools that assist in generating and capturing telemetry data
from cloud-native software.”
Across Traces, Metrics, Logs

26. 26
A unified set of vendor-agnostic APIs, SDKs and tools
for generating and collecting telemetry data, and then
exporting it to a variety of analysis tools.
Source: https://logz.io/learn/opentelemetry-guide/

27. 27
Modern systems need to be
observable
Open source is key to
Observability
Summary
Open source for better observability
Tool sprawl and data silos
are inhibitors
Single pane of glass to
unleash data analytics

28. 28
Open source for better observability:
https://horovits.medium.com/8c65b5630561
Observability as a data analytics problem:
https://bit.ly/o11y-data-analytics
Cloud Native Observability Microsurvey 2022
https://bit.ly/cncf-o11y-2022
Is “vendor owned open source” an oxymoron?
https://horovits.medium.com/b5486a4de1c6
Quick Dictionary to Open<X> Projects
https://horovits.medium.com/e9e48f0113de
OpenTelemetry Guide:
https://bit.ly/otel-kubecon
OpenObservability Talks:
https://anchor.fm/openobservability
Questions &
Resources
@horovits

29. 29
Thanks
Dotan Horovits
@horovits