1. Elasticsearch for reporting
analytics on communities
Elasticsearch Meetup - 23 April 2014
Marc Harrison

2. Lithium makes software that helps brands
better connect with their customers
Our social software helps companies respond on social networks and build
trusted content on a community they own.

3. Empower brands to distill terabytes of daily
data into understanding participation
▪ fast
▪ flexible
▪ scalable
What
products/services
are generating the
most conversations?
Who is authoring
content that
generates the most
kudos/likes?
Are customer posts
getting timely
replies?
What types of
content does this
audience segment
look for?

4. Lithium Social Intelligence (LSI)

5. Cluster specs
▪ One of our clusters – elastic search 1.0
• 7+ billion documents/4.4+ TB and growing fast!
• 21 nodes (3 masters, 2 clients, 16 data)

6. Lessons learned
▪ Bulk loading
▪ Faceting

7. Bulk initial load / rebuild of data
Hadoop
mysql stream
Transform/
route
…
JSON Elasticsearch

8. Bulk loading
▪ Make sure ingest logic is robust
• Idempotent for bulk reply - ‘_id’
• Include revision based on processor/time
• Check cluster/index status to make sure ready to ingest
▪ Know the cache and thread pool sizes
• Bulk – fixed - # of processors - queue size 50
• Handle back off and retry
▪ How many docs?
• Like capacity - test with data –
• number of shards
• index.refresh_interval: 30s
• indices.memory.index_buffer_size: 5%
• indices.memory.*
• index.translog.*

9. Search - time series pattern for scale

10. Faceting
▪ Don't forget about memory!
• Strings - not_analyzed
• Numbers long vs int, double vs float, etc
• Do you need seconds/minutes when faceting?
• fielddata format - doc_values (1.0)
• Admin API’s allow checking field data size + evictions
• indices.cache.filter.size: 15%
• indices.fielddata.cache.size: 45%

11. Faceting II
▪ Accuracy
• shard_size
• Number of shards
• Cardinality
• Routing
▪ Great custom plugin
framework
• Uniques
• Array faceting

12. Impact
▪ Order of magnitude improvement
▪ Developers able to focus on improving insights
▪ community + elasticsearch + hadoop + horton works =
exciting

13. Select settings (data center)
• bootstrap.mlockall: true
• cluster.routing.allocation.disk.threshold_enabled: true
• http.compression: true
• transport.tcp.compress: true
• gateway.recover_after_data_nodes: 13
• gateway.recover_after_master_nodes: 2
• gateway.recover_after_time: 3m
• gateway.expected_nodes: 17
• indices.memory.index_buffer_size: 5%
• indices.cache.filter.size: 15%
• indices.fielddata.cache.size: 45%
• index.store.type: mmapfs
• index.translog.flush_threshold_ops: 10000
• action.auto_create_index: false
• action.disable_delete_all_indices: true
• cluster.routing.allocation.node_initial_primaries_recoveries: 4
• cluster.routing.allocation.node_concurrent_recoveries: 15
• indices.recovery.max_bytes_per_sec: 100mb
• indices.recovery.concurrent_streams: 5
• discovery.zen.minimum_master_nodes: 2
• index.search.slowlog.threshold.query.warn: 5s
• index.search.slowlog.threshold.query.info: 1s
• index.indexing.slowlog.threshold.index.warn: 5s
• plugin.mandatory: lithium-unique-facets

14. Questions?