The document discusses Grafana Labs' open source observability tools including Prometheus, Grafana, Loki, and Tempo. It provides an overview of each tool's capabilities and Grafana Labs' contributions to them. It also describes how Loki provides highly scalable and cost-effective log aggregation through its storage model and query processing.

1. The Power of Metrics, Logs &
Traces with Open Source

2. ● Loves running
● Baking sourdough bread
https: //www.linkedin.com/in/emil-andreas-siemes-a793926/
Emil Siemes
Principal Solutions Engineer

3. 1000+ Employees across
40+ countries
(50+ in Germany)
1M+
Instances across
LGTM Cloud and
OSS
13M+ Users across OSS
and Cloud Free tier
Founded in 2014

4. We help our customers Build and Operate
resilient systems,
to provide reliable services to their
customers.

5. Investing in the core of observability

6. Prometheus
Largest contributor - employs
44% of Prometheus
maintainers
Open source is at the of what we do
GRAFANA
Vizualize, Correlate & Alert
on Multi-sourced Data
Grafana LOKI
Highly Scalable Log
Aggregation
Grafana MIMIR
Scalable & Performant
backend for all
time-series metrics
Graphite
Leading contributor on
Graphite
Grafana TEMPO
Highly Scalable &
Cost Effective Tracing
OpenTelemetry
Large contributor - employs
maintainters; first class
support in Grafana
K6
Modern load testing
built for developers
Grafana On-Call
Oncall management
system with simple
workflow integrated with
your telemetry

7. Prometheus
Largest contributor - employs
44% of Prometheus
maintainers
Open source is at the of what we do
GRAFANA
Vizualize, Correlate & Alert
on Multi-sourced Data
Grafana LOKI
Highly Scalable Log
Aggregation
Grafana MIMIR
Scalable & Performant
backend for all
time-series metrics
Graphite
Leading contributor on
Graphite
Grafana TEMPO
Highly Scalable &
Cost Effective Tracing
OpenTelemetry
Large contributor - employs
maintainters; first class
support in Grafana
K6
Modern load testing
built for developers
Grafana On-Call
Oncall management
system with simple
workflow integrated with
your telemetry

8. Deployment Flexibility
Open-source | Enterprise | Grafana Cloud
Big tent
Flexibility and openness without limits to integrate 3rd
party data
Observability by open source
A modern, complete and highly modulable platform fully
built on OSS
A Unique Philosophy : Regain Control Over Your
Observability Strategy

9. LGTM: A prometheus inspired O11y stack
Tempo
Backend compatible
with OpenTelemtry,
inexpensive at scale,
100% sampling is
possible
Loki
Low TCO, Highly
scalable and easy to
operate logging
system
Mimir
Highly scalable, highly
available multi-tenant,
durable storage
drop-in replacement
for Prometheus

10. Correlation by design to reduce MTTR
Exemplars
Labels & service discovery
{service="foo", region="eu-west",
instance="node-123"}
Auto-generated
metrics
Logs for trace
TraceID in logs
Logs to metrics
extraction
S3

11. Frontend application
observability

12. Frontend
Backend
infrastructure

13. An open source web SDK for
frontend application observability

15. Frontend Application Observability
in Grafana Cloud
Fully managed offering for capturing, visualizing,
and analyzing frontend application telemetry data within
Grafana Cloud, powered by Grafana Faro
PRIVATE BETA available today for Grafana Cloud customers

16. Getting started with Grafana Faro is super simple
Install the Grafana Faro Web SDK:
Import it into your app:
npm install @grafana/faro-web-sdk
import { initializeFaro } from '@grafana/faro-web-sdk'
initializeFaro({
url: '<http://my-endpoint>',
apiKey: '<api_key>',
app: { name: 'my-app' },
});

18. Loki

19. Out-of-order ingestion
Simplified deployment
Promtail Kafka support
Regex performance improvements
Parallelized binary ops in queries
Improved tail latencies with request hedging
Promtail Cloudflare, Docker, and Graylog support
Query multiple tenants at once
Faster instant queries
Filter & delete certain log lines
July 2022
April 2022
Nov. 2021
Loki v2.4.0
Loki v2.5.0
Loki v2.6.0
Loki v2.7.0
Coming soon

20. 50%
Less CPU
4x
Faster queries
COMING SOON

21. Node #1
Node #2 Loki
Promtail
Promtail
Grafana
logcli
AlertManager
REST
API
Object
Storage
Overview
Node #n
Promtail

22. @160014567432
Timestamp Value
22346
Labels/Selectors
key-value pairs
{app=”nginx”, env=”dev”}
Metric name
http_request
The Prometheus storage model

23. 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”}
Timestamp
with nanosecond precision Content log line
Labels/Selectors
key-value pairs
indexed unindexed
GET /about 1034 Debug “page not found”
Object
Storage
Object
Storage
Aka “the index” Aka “chunks”
The Loki storage model

24. ● Log any and all formats
● Smaller indexes
● Cheaper to run
● Fast queries
● Cut and slice your logs in dynamic ways
Grafana Loki
Query time processing
VS
● Decide on log formats
● Larger indexes
● More expensive to run
● Faster queries
● Restricted to format chosen at ingestion time
Traditional solutions
Ingest time processing
The better tradeoff

25. 10TB 200MB
Log Data Index
Think of it more like a table of contents than an index.
Loki does not index the contents of the logs. Instead, entries are
grouped into streams and indexed with Prometheus-style labels.
Efficient log storage

26. 1PB 80TB
Raw Logs Label match Brute force search -
heavily parallelized
Time frame
1TB 120GB+/s
Fast queries
Think of it as massively distributed grep.

27. Per Second A day A month Active tenants
700MB 43TB 1.25PB 4500
170Cores 1TB RAM
27
Large cluster in Grafana Cloud

28. {cluster=”us-central1”, job=~”dev/loki-.*”} |= “trace_id=7ca877dbda” |~ "SeriesStore.*"
Label matchers
● = contains string.
● != does not contain string.
● =~ matches regular expression.
● !~ does not match regular expression.
Filter expressions
● |= contains string.
● != does not contain string.
● |~ matches regular expression.
● !~ does not match regular expression.
LogQL: Log queries
Parser expressions
Formatter expressions

29. sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
Range vector aggregations
● rate() / bytes_rate()
● count_over_time() / bytes_over_time() /
absent_over_time()
● sum_over_time() / avg_over_time() /
min_over_time() / max_over_time () *
Instant vector aggregations
● sum() / min() / max() / avg()
● stddev() / stdvar()
● count()
● topk() / bottomk()
* Unwrapped range aggregations
LogQL: Metric queries

30. Grafana Agent
Promtail
● Targets discovery for Kubernetes, Syslog, files and more
● Automatically attach labels to your log lines
● Advanced pipeline mechanism for parsing, transforming and
filtering your logs
● Build and expose custom metrics from your logs data
But Loki is open.
logstash Lambda
Loki makes log collection easy

31. Single Binary
- Testing
- Small installations
without HA or
horizontal scaling
needs
SSD or
Microservices
- Horizontal scalability
- Separate Read/Write
paths
- Large installations
Grafana Cloud
● Batteries included:
○ Multi-tenant
○ Automatic scaling
and upgrades
○ RBAC
○ Per tenant QoS
Enterprise Logs
● Self hosted
● Supported by Grafana
Labs
● Security out of the box
● Part of the GES stack
31
How to run Loki

32. Thank you!