The document discusses Lucidworks Fusion, a log analytics platform that combines Apache Solr, Logstash, and Kibana. It describes how Fusion uses a time-based partitioning scheme to index logs into daily collections with hourly shards for query performance. It also discusses using transient collections to handle high volume indexing into multiple shards to avoid bottlenecks. The document provides details on schema design considerations, moving old data to cheaper storage, and GC tuning for Solr deployments handling large-scale log analytics.

2. GC Tuning
Call for Papers open through May 8
lucenerevolution.org
October 13-16 Ÿ Austin, TX

3. • Time-based partitioning scheme
• Transient collections for high-volume indexing
• Schema design considerations
• Q&A
Webinar: Fast Log Indexing with Fusion / Solr

4. Large-scale log analytics
• Index billions of log events per day, near real-time
• Analyze logs over time: today, recent, past week,
past 30 days, …
• Easy to use dashboards to visualize common
questions and allow for ad hoc analysis
• Ability to scale linearly as business grows …
with sub-linear growth in costs!
• Easy to setup, easy to manage, easy to use

5. • Combines the power of open
source Solr, Logstash, Kibana, and
more to deliver a feature rich UI for
search and data visualization.
• Gives user the power to perform ad
hoc search and analysis on massive
amounts of multi-structured and
time series data stored in Solr.
• Users can quickly transform results
into sharable visualizations and
dashboards.
Lucidworks Fusion Dashboard

6. Why Fusion / Solr for Log Analytics?
• Speed
• Proven scalability
• Community
• Cost effective

7. Fusion Collection
A collection is a distributed index defined by:
• configuration stored in ZooKeeper (schema.xml, solrconfig.xml, …)
• one or more shards: documents are distributed across N partitions of the index
• document routing strategy: how documents get assigned to shards
• replication factor: how many copies of each document in the collection
• replica placement strategy: rack awareness, etc (see SOLR-6620)
Sharding increases parallelism during indexing and query execution
Replication enables load-balancing and fault-tolerance
https://twitter.com/bretthoerner/status/476830302430437376

8. Fusion
Log Analytics
Dashboard
logs_feb26
(daily collection)
logs_feb25
(daily collection)
logs_feb01
(daily collection)
h00
(shard)
h22
(shard)
h23
(shard)
h00
(shard)
h22
(shard)
h23
(shard)
Add replicas
to support higher
query volume &
fault-tolerance
recent_logs
(colllection alias)
Use a collection
alias to make multiple
collections look like a
single collection; minimize
exposure to partitioning
strategy in client layer
Every daily collection has 24 shards (h00-h23), each covering 1-hour blocks of log messages
Time-based Data Partitioning Scheme

9. Time-based Data Partitioning Benefits
• Optimizing read performance for write-once, read-many-times
type data
• Recent data stays “hot” in Solr memory-based caches and OS
cache
• Queries can be directed to specific partitions based on time
range filters
Show me errors occurring in my database layer in the past 10 minutes
Visualize activity for the past 2 hours …
• Easy to drop or move older data to more cost-effective hardware
• Easier to scale-out by adding more nodes, no shard-splitting,
rebalancing, or reindexing
But …

10. Time-based Data Partitioning Limitations
• Indexing into a single hourly shard cannot keep up with write-volume
• 15-20K docs/sec per shard is normal
• Not unusual to need an avg. of 50K docs/sec with ability to scale to 120K
• Shard being written to would be a hotspot that would hurt query performance
• Partitioning details exposed to indexing applications

11. logs_feb26
(daily collection)
logs_feb25
(daily collection)
logs_feb01
(daily collection)
h00
(shard)
h22
(shard)
h23
(shard)
recent_logs
(colllection alias)
Time-based Data Partitioning with Transient Collection
shard4
shard3
shard2
shard1
Latest 1-hour shard
gets built from
merging shards at
time bucket boundary
logs_feb26_h23
(transient collection)
current_log_writer
(colllection alias)
log indexing app
(logstash, flume)
current_log_writer collection alias
rolls over to a new transient collection
every hour; the shards in the transient
collection are merged into the 1-hour
shard and added to the daily collection

12. Transient Collection Highlights
• Hash-based document routing across all shards
• Scale up the number of shards to meet peak write-volume reqts.
• Indexing applications write to an alias, e.g. “current_log_writer”
• Rollover to a new transient collection every hour
• Use Solr merge index support to create a single, immutable hourly shard
• Read alias includes current transient collection (to query very recent data)
• Achieved >120K docs per second with replication in a cluster with 20 servers

13. Direct Updates to Leaders
server-sideclient-side

14. logs_feb26
(daily collection)
logs_feb25
(daily collection)
h00
(shard)
h22
(shard)
h23
(shard)
recent_logs
(colllection alias)
Merge Transient Collection into Daily Collection
current_log_writer
(colllection alias)
log indexing app
(logstash, flume)
shard4
shard3
shard2
shard1
logs_feb26_h23
(transient collection)
logs_feb27_h00
(transient collection)
h23
(shard)
2015-02-27 00:00:00
Solr merge

15. Nuts and Bolts
• Most features are provided out-of-the-box with Fusion
- Connectors
- Banana Dashboard
- SolrCloud & Solr Scale Toolkit
- Collection Management
- Monitoring, security, etc …
• Transient collection rollover
- Custom Solr UpdateRequestProcessor
- Creates new transient collection when needed
- Updates collection aliases (reads and writes)
- Tricky in a distributed cluster, uses ZooKeeper to coordinate
• Merge transient shards into single hourly shard
- Custom Python script
- Pulls shards from transient collection to a single host (throttled)
- Runs Solr merge to create 1 index from many shards
- Activates new hourly shard & removes old transient
- Adds replicas for new hourly shard

16. Moving older data to less expensive storage
• SSD preferred for recent data, older data can be on “spinning rust”
• Move older data off SSD to slower disks
- ADDREPLICA API to add new replica on another host
- Solr 5+ supports throttling replication transfer speed
- or, simply scp –l 500 … (-l for limit copy at 0.5 Mbit/s)

17. Schema Design
• At large scale, disk (SSD) and memory are still expensive!
• Balance flexibility with performance and efficiency (disk usage)
• Fast unique IDs
Use UUID v1 if possible: http://johannburkard.de/software/uuid/
http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html
• DocValues for fields that you facet and sort on frequently
• Trie-based date for range queries
• Catch all text field for improved recall

18. DocValues
• Faster sorting and faceting, both of which are used extensively in log analytics
• Column-oriented view of a field in a Solr index (uninverted)
• Disk & OS cache vs. Java Heap
“… we tuned everything about the default codec to keep all the data structures on disk but still be fast.”
Robert Muir, Lucene/Solr Committer, LUCENE-5761
• Helps reduce pressure on Java GC

19. GC Tuning
• Dashboards put a lot of pressure on the Java GC
• Stop-the-world GC pauses can lead to ZooKeeper session expiration (which is always bad)
• Good success with 32gb max heap (-Xmx32g) but have to tune it …
• MMapDirectory relies on sufficient memory available to the OS cache (off-heap)
• Enable verbose GC logging (even in prod) so you can troubleshoot issues:
-­‐verbose:gc
–Xloggc:gc.log
-­‐XX:+PrintHeapAtGC
-­‐XX:+PrintGCDetails
-­‐XX:+PrintGCDateStamps
-­‐XX:+PrintGCCause
-­‐XX:+PrintTenuringDistribution
-­‐XX:+PrintGCApplicationStoppedTime

20. GC Tuning
-­‐XX:CMSTriggerPermRatio=80
-­‐XX:CMSFullGCsBeforeCompaction=1
-­‐XX:+ParallelRefProcEnabled
-­‐XX:+CMSParallelRemarkEnabled
-­‐XX:CMSMaxAbortablePrecleanTime=6000
-­‐XX:CMSInitiatingOccupancyFraction=50
-­‐XX:+UseCMSInitiatingOccupancyOnly
-­‐XX:PretenureSizeThreshold=128m
-­‐XX:+CMSScavengeBeforeRemark
-­‐XX:ParallelGCThreads=9
-­‐XX:ConcGCThreads=9
-­‐XX:+UseParNewGC
-­‐XX:+UseConcMarkSweepGC
-­‐XX:MaxTenuringThreshold=12
-­‐XX:TargetSurvivorRatio=90
-­‐XX:SurvivorRatio=6
-­‐XX:NewRatio=5
-­‐Xmx32g
-­‐Xms16g
-­‐Xss256k

21. Thank you.
Download Fusion: http://lucidworks.com/fusion/
Webinar recording will be available soon.
Questions?