My talks on OpenInfra Asia 2021

1. Introduction to Open Telemetry
-Your Universal Telemetry Library-
Tonny Adhi Sabastian
(tonny.sabastian@gojek.com)
Open Infrastructure Day Asia - September 11th, 2021

2. About Me
● Currently : Senior DevOps Engineer at MAPAN (GoTo Group) - https://www.mapan.id (2018 - Now)
○ We’re hiring, please visit https://www.career.mapan.id/
○ Send your CV to mapan-recruitment@gojek.com
○ Interest in our DevOps engineering ? PM me via mail
● Former Lead Engineer for University of Indonesia - Data Center (2012 - 2018) - https://ui.ac.id
● Former Lecturer at Faculty of Computer Science , University of Indonesia (2014 - 2018)
○ Teaching Subject : Distributed System , System Programming, Modern IT Infrastructure
○ Research Subject : IoT , Deep Packet Inspection, Aeronautical Telecommunication Network, Linux System Performance
● Co-founding two startup during 2012 - 2018 , (Rumio - AR Platform & Peentar - IoT Platform) - now defunct
● Former volunteer system administrator for kambing.ui.ac.id - one of Indonesia largest F/OSS repository ; now defunct (2009 -
2018)
● Contact : tonny.sabastian@gojek.com / tonny@segmentationfault.xyz

3. Agenda
● Observability 101
● Meet OpenTelemetry
● Instrumenting with OpenTelemetry - Tracing with Go
Example -
● OpenTelemetry @ MAPAN

4. Observability 101

5. The Observability Triad
Do you know what your codes are doing on 2AM when your payment service is down ?
Monitoring
Tracing
Logging

6. Metrics - It’s About The Number
● Each metric reports a number
● The monitoring system tracks
those numbers over time
● Identify anomalous behavior and
wake someone up
● Recognizes trends for your
infrastructure scaling
● Four Golden Signals - ( Latency -
Traffic - Saturation - Errors )

7. Tracing - Finding Your Bottleneck
● Uniquely identify Requests and
Spans
● Spans carry contextual metadata
● Runtime reports start/end
timestamps for each span
● Span Aggregator find out how
long everything took

8. Logging - How to debug your production at 2AM
● Convey appropriate urgency
● Provide enough information
● Not overwhelming
● Structured for searching not reading

9. Meet OpenTelemetry

10. OpenTelemetry
● OpenCensus + OpenTracing =
OpenTelemetry
● Provides a unified set of open
standards API, SDK, and middleware
for generating and exporting Trace ,
Metrics, (and logging - incoming)
● Enabling quick bootstrap of
observability into your system
● Contributor from various major
vendor and wide community support
source : https://opentelemetry.io

11. OpenTelemetry Architecture (1)
● API : Used to generate telemetry data
● SDK : Implementation of API in various
languages
○ Tracer : responsible for creating and
tracking span - distributed tracing
○ Meter : responsible for creating and
tracking metrics
● OpenTelemetry Proto : protobuf
implementation on semantic convention
and OpenTelemetry Protocol (OTLP)
source :
https://blog.newrelic.com/product-news/what-is-opentelemetry/

12. OpenTelemetry Architecture (2)
● Collector : middleware service on receiving,
processing, and exporting telemetry data to
various telemetry backend (ex : jaeger,
prometheus, datadog, etc). It’s acting as
telemetry pipeline
● Collector composed of following components :
○ receivers : receive telemetry data
○ processors : telemetry data
additional processing
○ exporters : send telemetry data to
final backend
● Collector can be deployed as Agent or Gateway
source :
https://blog.newrelic.com/product-news/what-is-opentelemetry/

13. Instrumenting with OpenTelemetry

14. Tracing API Concepts
● TracerProvider is the entry point of the API. It’s providing access to
Tracers
○ TracerProvider register global configuration of exporter, tracing
attributes and sampler
● Tracer is the class responsible for creating Spans
● Spans is the API to trace an operation within your code

15. Tracing SDK Concepts
● Sampler , controls the number of traces capture and send to backend
○ Sampling Decision Values [ DROP | RECORD_ONLY |
RECORD_AND_SAMPLE ]
○ SDK Built In Samplers [ AlwaysOn | AlwaysOff |
TraceIDRatioBased ]
● SpanProcessor , hook for span start and stop invocation
○ SDK Built In Processor [ ExportFormatConverter | Batcher ]
○ Only invoked when isRecording is true
● SpanExporter, defines interfaces for specific backend exporters (ex :
Jaeger, OTLP, Lighstep, Datadog)

16. Go Instrumentation - Set Dependencies
OpenTelemetry Go Dependencies

17. Go Instrumentation - Create Tracer
● Set SpanExporter using OpenTelemetry
Protocol via otlpgrpc
● Set initial SpanAttributes, such as your
ServiceName
● Deploy TracerProvider with our established
SpanExporter, Sampler and initial
SpanAttributes

18. Go Instrumentation - Adding Span
● Create a Tracer and bind it to a
TracerProvider
● Bind the Tracer to a Span
● Set additional SpanAttributes. Perhaps you
want to trace your http request
● Track an event on your function. Add a
SpanEvent to our Span

19. Go Instrumentation - Trace from your HTTP Router
● Create http handler function with http
request path as its span context
● Bind this function with your http router

20. Go Instrumentation - Spawn your Tracer
● Create a context and bind it to your Tracer
● Bind your http handler function to your http
router. This will create your root span

21. Go Instrumentation - Span Result on Jaeger

22. Go Instrumentation - OpenTelemetry @ MAPAN
● Wrap OpenTelemetry into our Observability
Library
● Wrap our Observability Library into our HTTP
Server Library
● Simple Observability Bootstraping via server
start options

23. Demo

24. Instrumentation Demo
● Demo Repository in Golang https://github.com/tonnyadhi/distributed-tracing-example
● Contains two services
○ Weather Service, fetch weather report from backend and print it in JSON
○ OWM Service, fetch weather report from OpenWeathermap, simplify it and
Sent it to Weather Service
● Distributed Tracing is set between two services using OpenTelemetry HTTP Traces
● Span are sent to OpenTelemetry Collector. The collector will export the span to Jaeger
and Sentry

25. Thank You