Slides of my talk at DE-CIX TechMeeting 2018 at 2018-06-06 in Frankfurt. This is the handout version with all links and features

1. Open source tools for optimizing
your peering infrastructure
@ DE-CIX TechMeeting 2018-06-06
by Daniel Czerwonk

2. • Software / Network Engineer at Mauve Mailorder Software
• Head of Network Freifunk Essen e.V.
• AS44821 (Mauve), AS206356 (Freifunk Essen e.V.),
AS202739 (routing-rocks)
• birdwatcher and bio-routing contributor
• Twitter: @dan_nrw
• Github: https://github.com/czerwonk
• LinkedIn: https://www.linkedin.com/in/czerwonk/
Who is this guy? About me…

3. Our journey starts late 2016
A new networking setup is about to
be build

4. But before that:
Let’s talk about monitoring…

5. • Very small operations team
• Freifunk Essen should be even less ops demanding
• Identify trends/anomalies early
• Capacity planing (beware of retention)
• Source for alerting
• Start point for traffic engineering, etc.
• Source to build post mortem on (in case of outage)
• Dashboard to give a quick overview when needed
Why is monitoring important for me?

6. So, let’s build a monitoring system…

7. • Prometheus to collect metrics
• Grafana to visualize metrics
• Alertmanager with Pushover integration for alerting
• Everything Ansible managed
What I wanted…
+ +

8. • Bird routing daemon
• JunOS running on a few EX series switches
• Host metrics from bare metal software router machines (statistics, resources)
• External network latencies (RIPE ATLAS, etc.)
What I wanted to scrape?

9. What I found…

10. In 2016…
Metric Solution Problem
bird no exporter available
JunOS snmp_exporter
complex configuration,
bad performance
Host metrics node_exporter
Network latencies
blackbox_exporter with
external probe VMs
bad coverage,
only one request per scrape

11. • Official Prometheus project
• On Linux hosts (e.g. Routers)
• Network interface metrics
• Resource consumption: CPU load, RAM usage, Disk space
• Interrupts / context switches
• License: Apache 2.0
• Source: https://github.com/prometheus/node_exporter
node_exporter

12. At least we got the host metrics covered.
And the rest?
I had to solve that…

13. So I started to write some
exporters…

14. • Performance is key feature
• Need for concurrent processing
• Single binary / no dependencies
• Easy installation via go get …
• Existing client API for Prometheus
• Love writing code in golang in my spare time
Which programming language?
I chose golang:

15. atlas_exporter
RIPE ATLAS
Milestones to an exporter suite
bird_exporter
Bird 1.x
2016 20182017
RIPE LABS
article
Support for
bird 2.x
Replaced SNMP
by SSH
junos_exporter
Juniper JunOS
using SNMP
ping_exporter
ICMP probing
mikrotik-exporter
RouterOS

16. • Started late 2016
• Communicates with bird via socket
• Bird 1.x and 2.x supported
• Protocols: BGP, OSPFv2, OSPFv3, Kernel, Static, Device, Direct
• License: MIT
• Source: https://github.com/czerwonk/bird_exporter
bird_exporter

17. bird_exporter
bird_protocol_prefix_import_count{proto=~"BGP|OSPFv3",ip_version="6"}
count(bird_protocol_up{proto=“BGP"} == 1)

18. • BGP session state metrics
• BGP message counts (received, sent, withdrawn, etc.)
• Prefix counts for all supported protocols (imported, exported, filtered, etc.)
• OSPFv2/OSPFv3 neighbour counts
• Protocol uptime
bird_exporter - Features

19. • Started early 2018
• Replacement for RRD based smokeping
• Concerning ICMP also replacement for blackbox_exporter since lack of loss
detection
• Based on go-ping by Digineo: https://github.com/digineo/go-ping
• License: MIT
• Source: https://github.com/czerwonk/ping_exporter
ping_exporter

20. ping_exporter
ping_rtt_mean_ms{ip_version="6"}
ping_loss_percent{ip_version="4"}

21. • Sends and aggregates multiple ICMP ECHO requests
• Roundtrip metrics (current, best, worst)
• Simple way to detect loss
• Supports multiple targets
• DNS refresh ensures the correct IP is measured when DNS is changed
• Only ICMP support at the moment
• Warning: ICMP is not user traffic so keep that in mind when trying to interpret these
metrics
ping_exporter - Features

22. • Started early 2017
• Metrics by requesting measurement results from RIPE ATLAS
• Useful to get an outside view from different other networks
• License: LGPL3 (since the binding used is under this license)
• Source: https://github.com/czerwonk/atlas_exporter
• More info:
https://labs.ripe.net/Members/daniel_czerwonk/using-ripe-atlas-measurement-
results-in-prometheus-with-atlas_exporter
atlas_exporter

23. atlas_exporter
avg(atlas_ping_avg_latency{ip_version="4"}) by (asn)
avg(atlas_traceroute_hops{ip_version="4"}) by (asn)

24. • Ping (success, min/max/avg latency, dups, size)
• Traceroute (success, hop count, rtt)
• NTP (delay, derivation, ntp version)
• DNS (succress, rtt)
• HTTP (return code, rtt, http version, header size, body size)
• SSL Certificates (alert, rtt)
atlas_exporter - Features

25. • Started late 2017
• snmp_exporter did not perform as required
• First implementation using a simple set of SNMP OIDs
• Early 2018: reimplementation using SSH and XML RPC representation
• Alternative to Junipers OpenNTI since telemetry is only supported on newer
versions of JunOS and hardware
• License: MIT
• Source: https://github.com/czerwonk/junos_exporter
junos_exporter

26. • Interfaces (bytes transmitted/received, errors, drops)
• Routes (per table, by protocol)
• Alarms (count)
• BGP (message count, prefix counts per peer, session state)
• OSPFv2, OSPFv3 (number of neighbours)
• Interface diagnostics (optical signals)
• ISIS (number of adjacencies, total number of routers)
• Environment (temperatures)
• Routing engine statistics
junos_exporter - Features

27. • Contribution to existing project
• Only interface and resource metrics at this point
• Added several other features
• License: BSD3
• Source: https://github.com/nshttpd/mikrotik-exporter
mikrotik-exporter

28. • Interface metrics (RX bytes, TX bytes, drops, errors, etc.)
• BGP session states
• BGP message counts (updates, withdraws)
• DHCP leases
• DHCPv6 bindings
• Optical diagnostics
• IPv4/IPv6 pool counts
• System resources (memory, CPU load, etc.)
• Prefix counts per protocol (in RIB)
mikrotik-exporter - Features

29. Dashboard examples
How to combine several exporters?

30. Mauve Network Overview

31. Mauve Routing

32. Alerting
When and how?

33. How to alert?
What the SRE book has taught us:
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html

34. How to alert? A few examples…
Port saturation:
Upstream session down:

35. Thank you for your attention.
Special thanks to all people contributed to my projects!