Presentation at Search Meetup Munich: January Meetup

1. Elasticsearch as a time series
database
Felix Barnsteiner | iSYS Software

2. Felix Barnsteiner
● Author of the blog “Elasticsearch
as a Time Series Data Store”
● Project Lead stagemonitor
○ Open Source performance
monitoring
● iSYS Software GmbH
○ custom software
○ jstage.de
■ E-Commerce Platform, Consulting
and Services
○ We’re hiring!

3. Agenda
● Time Series What?
● Why Elasticsearch?
● Data Model
● Elasticsearch Mapping Tuning
● Index Management

5. Time Series what?
Time
t

6. Time Series what?
Series
t

7. Time Series what?
Series
t

8. Time Series what?
t
CPUUtilisation%

9. Time Series what?
t
Revenue/minute

10. Why Elasticsearch?
Decision Process

11. In search for a new TSDB
● New database for stagemonitor
● Replacing Graphite
○ Installation can be tricky
○ Scaling issues
○ Memory hungry
○ No support for special chars
■ GET /index.html
■ GET_|index:html
● Good things about Graphite
○ Grafana
○ Functions
○ Community/Tools/Collectors

12. Stagemonitor
● Open Source Java Performance Monitoring
○ Mainly Web Applications
● Monitoring of clustered environments
● For Dev, QA and Ops

13. In Browser Widget
●

14. Request Analysis
● Stores Request Traces directly to
Elasticsearch
○ Response Time
○ Status Code
○ Browser/Device/OS/User Agent Type
○ Exceptions
● No Logstash/Log parsing needed
● Kibana dashboard

15. Metrics
● Response Times
● Real User Monitoring
● JVM
● Operating System
● EhCache
● Logging
● JDBC

16. TSDB Requirements
● Easy to install
● Scalability
● Variety of functions
○ derivative
○ max/min/std
○ percentiles
○ ...
● Great visualization support
● Maturity

17. Many Options
● InfluxDB
● OpenTSDB
● Elasticsearch
● KairosDB
● Prometheus
● Druid

18. Don’t Match Requirements
● OpenTSDB
○ Not easy to install
○ Hadoop and Hbase
● KairosDB
○ Not easy to install
○ Cassandra + KairosDB + Config
● Prometheus
○ No Clustering
● Druid
○ No visualization tool

19. Easy to install
● InfluxDB
○ rpm, deb, homebrew
● Elasticsearch
○ rpm, deb, download & run
○ Runs anywhere
○ Stagemonitor already requires Elasticsearch

20. Scaleability
● InfluxDB
○ CERN Presentation*
● Elasticsearch
* http://cds.cern.ch/record/2011172/files/LHCb-TALK-2015-060.pdf
*

21. Maturity
● InfluxDB
○ Many core changes
○ Clustering in alpha state (max 3 nodes)
○ New Storage Engine in beta
● Elasticsearch
○ Yes!

22. Functions
● InfluxDB
○ docs.influxdata.com/influxdb/v0.
9/query_language/functions/
● Elasticsearch
○ www.elastic.
co/guide/en/elasticsearch/reference/current/se
arch-aggregations.html
○ Hold-Winters: predictions based on seasonal
patterns

23. Visualization Support
● InfluxDB
○ Grafana
○ Chronograf
● Elasticsearch
○ Kibana
○ Grafana
○ Timelion

24. Elasticsearch Benefits
● Large community
● Proven to scale
● Redundancy - replicas
● Backups - snapshot and restore
● Easy to install and manage
● Already needed for stagemonitor
○ Only one instead of two data bases to install
● Marvel uses ES as a TSDB
● CERN says

25. Stagemonitor Outputs
● Elasticsearch
○ Default
○ Preconfigured Dashboards
● InfluxDB
● Graphite

26. Data Model

27. How do Timers work?
● Stagemonitor uses Dropwizard Metrics
● Not one report per request
○ Not scaleable
○ Sampling
● Response times are aggregated in memory
● Stores 1000 representative response times
● Computes metrics like percentiles, min/max,
stdev, throughput/rate
● Reports each 60 seconds (configurable)

28. metics_store_benchmark.my_hostname.local.timer_1.count 10
metics_store_benchmark.my_hostname.local.timer_1.mean 123
metics_store_benchmark.my_hostname.local.timer_1.min 12
metics_store_benchmark.my_hostname.local.timer_1.max 1234
metics_store_benchmark.my_hostname.local.timer_1.p50 124
…
Doesn’t really fit into Elasticsearch’s data model
How do I filter by a certain host? Regex??
Classic Data Model - Timer
Application Host Instance Name Value

29. Regex??

30. Data Model 2.0
● Tags and values
● Self descriptive
● Add tags
(dimensions)
afterwards
○ Without changing
identity
○ Backwards
compatible

31. Data Model - Timer
● Metric Timer name
○ http_time
○ jdbc_time

32. Data Model - Timer
● Metric Timer name
○ http_time
○ jdbc_time

33. Data Model - Timer
● Global tags
○ Application
○ Host
○ Instance
○ Region
○ Datacenter
○ Availability Zone
○ ...

34. Data Model - Timer
● Metrics
○ mean/min/max
○ percentiles
○ throughput, count

35. Elasticsearch Mapping Tuning

36. Mapping Template

37. Mapping
● _all: contains all values of all other fields
● _source: contains original JSON
● Saves disk space
● BUT…
● JSON is not visible
● Data can only be retrieved via aggregations

38. Mapping
● Don’t analyze tags (like application, host, ...)
○ No full text search/stemming needed
○ Only filter by exact values
● doc_values: true
○ Reduces heap usage (and OOMEs) at the cost of
disk space
○ Needed for aggregations (un-inverted index)
○ Default in 2.0

39. Mapping
● index: no for metric values
○ Not searchable by value
○ Saves disk space
● Using integers and floats instead of longs and
doubles
○ Saves disk space
○ OK for our use case
○ May not be OK for yours

40. Benchmark
● ~23 M datapoints
● A week’s worth of stagemonitor metric data
● when reporting every minute
-> ~500MB
InfluxDB: ~360MB (0.9.4.1)

41. Index Management

42. One Index Per Day
● Logstash like index format
● stagemonitor-metrics-2016.01.18
● Only relevant indices have to be queried
● Easier/more efficient to delete
● Mapping can be changed every day
○ Number of shards

43. Hot/Cold Architecture
● Recent data is queried more often
● Beefy nodes hold recent data (hot)
○ More CPU, RAM
○ SSDs
● Cheaper nodes hold older data (cold)
○ Less CPU, RAM
○ HDDs

44. Hot/Cold Architecture
Beefy Node(s) Cheap Node(s)
elasticsearch.yml
node.box_type: hot
elasticsearch.yml
node.box_type: cold
Index
Today
Index
Yesterday
Index 2
days ago
... ... ...

45. Shard allocation filtering
● New Indices are allocated to hot nodes
{
"template": "stagemonitor-metrics-*",
"settings": {
"index": {
"routing.allocation.require.box_type": "hot"
}
}
…
}

46. Shard allocation filtering
● As indices grow old, they are moved to cold
nodes
PUT /stagemonitor-requests-*,
-stagemonitor-requests-2016.01.18,
-stagemonitor-requests-2016.01.17,
-stagemonitor-requests-2016.01.16/_settings
{
index.routing.allocation.require.box_type=cold
}

47. Hot/Cold Architecture
Beefy Node(s) Cheap Node(s)
elasticsearch.yml
node.box_type: hot
elasticsearch.yml
node.box_type: cold
New
Index
Old
Index
Old
Index

48. Force Merge (optimize)
● Improves resource use
● Improves cluster recovery speed and restarts
of node
● Requires a lot of CPU and disk resources
while running
● Should be performed on cold nodes
○ They usually aren’t doing much
○ Hot nodes usually are busy due to
indexing/querying

49. Deleting indices
● Limit storage requirements
● Only keep metrics for one year

50. Lifecycle
New Old
Very
Old
Cheap Node
Optimize
Beefy Node Delete
Archive

51. Tools
● Curator
○ Command line tool by Elastic
○ Execute 3 jobs as cron
● Stagemonitor
○ Automatically handled
○ How many days on hot nodes
○ Configurable delete delay

52. Questions?
Blog: elastic.co/blog/elasticsearch-as-a-time-
series-data-store
Stagemonitor: stagemonitor.org
Twitter: @felix_b, @stagemonitor
@felix_b