OpenTSDB is an open source time series database that allows for storing and querying of large amounts of metrics data. It is written in Java and uses HBase for distributed storage. Data is stored by metric ID and timestamp as the row key in HBase. Tags are stored at the end of the row key. The speaker discusses their experiences using OpenTSDB at Gutefrage.net to store hundreds of millions of data points with thousands of time series. They discuss challenges with the UI and time resolution and look forward to improvements in OpenTSDB 2.

1. openTSDB - Metrics for a
distributed world
Oliver Hankeln / gutefrage.net
@mydalon

2. Who am I?
Senior Engineer - Data and Infrastructure at
gutefrage.net GmbH
Was doing software development before
DevOps advocate

3. Who is Gutefrage.net?
Germany‘s biggest Q&A platform
#1 German site (mobile) about 5M Unique Users
#3 German site (desktop) about 17M Unique Users
> 4 Mio PI/day
Part of the Holtzbrinck group
Running several platforms (Gutefrage.net,
Helpster.de, Cosmiq, Comprano, ...)

4. What you will get
Why we chose openTSDB
What is openTSDB?
How does openTSDB store the data?
Our experiences
Some advice

5. Why we chose
openTSDB

6. We were looking at some
options
Munin Graphite openTSDB Ganglia
Scales well no sort of yes yes
Keeps all
data
no no yes no
Creating
metrics
easy easy easy easy

7. We have a winner!
Munin Graphite openTSDB Ganglia
Scales well no sort of yes yes
Keeps all
data
no no yes no
Creating
metrics
easy easy easy easy
Bingo!

8. Separation of concerns
$
unzip|strip|touch|finger|grep|mount|fsck|more|yes|fsck|fsck|fsck|umo
unt|sleep

9. The ecosystem
App feeds metrics in via RabbitMQ
We base Icinga checks on the metrics
We evaluate etsy Skyline for anomaly
detection
We deploy sensors via chef

10. openTSDB
Written at StumbleUpon but OpenSource
Uses HBase (which is based on HDFS) as a
storage
Distributed system (multiple TSDs)

11. The big picture
HBase
TSD
TSD
TSD
TSDUI
API
tcollector
This is really a
cluster

12. Putting data into
openTSDB
$ telnet tsd01.acme.com 4242
put proc.load.avg5min 1382536472 23.2 host=db01.acme.com

13. It gets even better
tcollector is a python script that runs your
collectors
handles network connection, starts your
collectors at set intervals
does basic process management
adds host tag, does deduplication

14. A simple tcollector script
#!/usr/bin/php
<?php#Cast a die$die = rand(1,6);echo "roll.a.d6 " . tim

15. What was that HDFS
again?
HDFS is a distributed filesystem suitable for
Petabytes of data on thousands of machines.
Runs on commodity hardware
Takes care of redundancy
Used by e.g. Facebook, Spotify, eBay,...

16. Okay... and HBase?
HBase is a NoSQL database / data store on
top of HDFS
Modeled after Google‘s BigTable
Built for big tables (billions of rows, millions of
columns)
Automatic sharding by row key

17. How openTSDB stores
the data

18. Keys are key!
Data is sharded across regions based on their
row key
You query data based on the row key
You can query row key ranges (say e.g. A...D)
So: think about key design

19. Take 1
Row key format: timestamp, metric id

20. Take 1
Row key format: timestamp, metric id
1382536472, 5 17
Server A
Server B

21. Take 1
Row key format: timestamp, metric id
1382536472, 5 17
1382536472, 6 24
Server A
Server B

22. Take 1
Row key format: timestamp, metric id
1382536472, 5 17
1382536472, 6 24
1382536472, 8 12
1382536473, 5 134
1382536473, 6 10
1382536473, 8 99
Server A
Server B

23. Take 1
Row key format: timestamp, metric id
1382536472, 5 17
1382536472, 6 24
1382536472, 8 12
1382536473, 5 134
1382536473, 6 10
1382536473, 8 99
1382536474, 5 12
1382536474, 6 42
Server A
Server B

24. Solution: Swap
timestamp and metric id
Row key format: metric id, timestamp
5, 1382536472 17
6, 1382536472 24
8, 1382536472 12
5, 1382536473 134
6, 1382536473 10
8, 1382536473 99
5, 1382536474 12
6, 1382536474 42
Server A
Server B

25. Solution: Swap
timestamp and metric id
Row key format: metric id, timestamp
5, 1382536472 17
6, 1382536472 24
8, 1382536472 12
5, 1382536473 134
6, 1382536473 10
8, 1382536473 99
5, 1382536474 12
6, 1382536474 42
Server A
Server B

26. Take 2
Metric ID first, then timestamp
Searching through many rows is slower than
searching through viewer rows. (Obviously)
So: Put multiple data points into one row

27. Take 2 continued
5, 1382608800
+23 +35 +94 +142
17 1 23 42
5, 1382612400
+13 +25 +88 +89
3 44 12 2

28. Take 2 continued
5, 1382608800
+23 +35 +94 +142
17 1 23 42
5, 1382612400
+13 +25 +88 +89
3 44 12 2
Row key

29. Take 2 continued
5, 1382608800
+23 +35 +94 +142
17 1 23 42
5, 1382612400
+13 +25 +88 +89
3 44 12 2
Row key
Cell Name

30. Take 2 continued
5, 1382608800
+23 +35 +94 +142
17 1 23 42
5, 1382612400
+13 +25 +88 +89
3 44 12 2
Row key
Cell Name Data point

31. Where are the tags
stored?
They are put at the end of the row key
Both metric names and metric values are
represented by IDs

32. The Row Key
3 Bytes - metric ID
4 Bytes - timestamp (rounded down to the
hour)
3 Bytes tag ID
3 Bytes tag value ID
Total: 7 Bytes + 6 Bytes * Number of tags

33. Let‘s look at some
graphs

34. Our experiences

35. What works well
We store about 200M data points in several
thousand time series with no issues
tcollector is decoupling measurement from
storage
Creating new metrics is really easy

36. Challenges
The UI is seriously lacking
no annotation support out of the box
Only 1s time resolution (and only 1
value/s/time series)

37. salvation is coming
OpenTSDB 2 is around the corner
millisecond precision
annotations and meta data
improved API

38. Friendly advice
Pick a naming scheme and stick to it
Use tags wisely (not more than 6 or 7 tags per
data point)
Use tcollector
wait for openTSDB 2 ;-)

39. Questions?
Please contact me:
oliver.hankeln@gutefrage.net
@mydalon
I‘ll upload the slides and tweet about it