Time Series metrics can be an important part of a comprehensive monitoring solution. Betfair will present a talk on their experiences running OpenTSDB and a new open source tool called OpenTSP, designed to streamline the process of gathering and delivering system metrics quickly and reliably to multiple endpoints so that you can use any of your favourite tools to analyse the stream. Video: https://www.youtube.com/watch?v=a2zBTD-pveY Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London Follow DOXLON on twitter http://www.twitter.com/doxlon

1. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
SHOW ME
THE
METRICS James Brooks

2. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Born in 1999
UK, Romania, Portugal, Ireland, Malta, Gibraltar, USA
Engineering Blog: www.betsandbits.com
700+ Engineers and Growing
Exchange / Sportsbook /Games
~750k active users
~70M daily transactions
~2.5Bn daily API calls
~100k/s Monitoring Points
E2E P95 transaction times <4ms
One of UK’s most successful .coms with ~£2Bn Market Cap
76

3. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
OpenTSDB &
OpenTSP at Betfair
77

4. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 78
What is OpenTSDB?

5. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 79
What is OpenTSDB?
A database optimised for
storing and querying Time
Series Metrics
os.filesystem.size 1413978032 912234340
host=fe-108 cluster=bf_c mount=/ type=Free
Name Timestamp Value
metadata

6. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 80
OpenTSDB Architecture
Hadoop
• TSDB Read
• TSDB Write
Frontend
Server
• TSDB Read
• TSDB Write
Frontend
Server
• TSDB Read
• TSDB Write
Frontend
Server
LoadBalancer
Metrics
Queries

7. TSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 81
OpenTSDB Architecture
Metrics
Queries

8. Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 82
OpenTSDB Architecture (ctd)
TSDB

9. Source :http://mo.nitor.me/hadoop-hbase-opentsdb/

10. Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 84
OpenTSDB Architecture (ctd)
Metrilyx
TSDB

11. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 85

12. Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 86
OpenTSDB Architecture (ctd)
Metrilyx
TSDB Visualiser

13. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 87

14. tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 88
OpenTSDB Architecture (ctd)
TSDB
tcollector
collect-os
collect-jmx
tcollector
collect-os
collect-jmx

15. • TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 89

16. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 90

17. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 91

18. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 92

19. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 93

20. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 94

21. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 95

22. • TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
• Monitoring and Alerting
• Limitations
• Load
• Recency
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 96

23. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 97

24. • TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
• Monitoring and Alerting
• Limitations
• Load
• Recency
• TSP
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 98

25. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 99
OpenTSP

26. tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 100
OpenTSP
TSDB
tcollector
collect-os
collect-jmx
forwarder
collect-os
collect-jmx

27. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 101
OpenTSP Architecture: Forwarder
TAG
FILTER
collect-statse

28. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 102
OpenTSP Architecture: Statse
application collect-statse
count
mean
median
p95
p99
forwarder
collect-statse
count
mean
median
p95
p99
forwarder
application collect-statse
count
mean
median
p95
p99
forwarder
application

29. poller
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 103
OpenTSP Architecture: Poller
collect-snmp
collect-f5
collect-netscaler
switches
routers
firewalls
load balancers
storage

30. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 104
OpenTSP Architecture: Aggregator
forwarder
forwarder
poller
aggregator
collect-site

31. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 105
OpenTSP Architecture: Controller
controller
collectors aggregator

32. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 106
[james@ohlookahostname~]$1CONTROL_HOST=tspconfig.inf.betfair1tspDforwarder1Dt1
{1
11111111"Filter":1[1
1111111111111111{1
111111111111111111111111"Match":1[1
11111111111111111111111111111111"",1
11111111111111111111111111111111"host",1
11111111111111111111111111111111"^$"1
111111111111111111111111],1
111111111111111111111111"Set":1[1
11111111111111111111111111111111"",1
11111111111111111111111111111111"host",1
11111111111111111111111111111111“ohlookahostname.inf.betfair"1
111111111111111111111111]1
1111111111111111}1
11111111],1
11111111"LogPath":1"/var/log/tsp/forwarder.log",1
11111111"Relay":1{1
1111111111111111"aggregator":1{1
111111111111111111111111"DropRepeats":1false,1
111111111111111111111111"Host":1“tspaggregator.inf.betfair",1
111111111111111111111111"MaxConnsPerHost":11,1
111111111111111111111111"OnQueueFull":1"Drop"1
1111111111111111},1
1111111111111111"tsd":1{1
111111111111111111111111"DropRepeats":1true,1
111111111111111111111111"Host":1"tsdb.inf.betfair",1
111111111111111111111111"MaxConnsPerHost":11,1
111111111111111111111111"OnQueueFull":1"Drop"1
1111111111111111}1
11111111},1
11111111"CollectPath":1"/etc/tsp/collect.d"1
}1

33. Visualise
r
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 107
TSP at Betfaircontroller
collectors TSDB Visualiser
tsdb2mysql mysqld
feedhealth
Riemann
future?
aggregator
aggregator

34. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 108
Open Source
• tsp-forwarder
• collect-statse
• tsp-poller
• collect-f5
• collect-netscaler
• collect-snmp
• tsp-aggregator
• tsp-controller
Apache Licence v2
http://github.com/betfair/opentsp

35. james.brooks@betfair.com
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 109