Tubular Labs - Using Elastic to Search Over 2.5B Videos
•
0 likes•651 views
Presentation given by Stas Starcha at the Elasticsearch meetup on 9/13/16.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Tubular Labs - Using Elastic to Search Over 2.5B Videos
Similar to Tubular Labs - Using Elastic to Search Over 2.5B Videos (20)
Recently uploaded
Recently uploaded (20)
Tubular Labs - Using Elastic to Search Over 2.5B Videos
- 1. Using Elastic to search over 2.5B videos
- 2. Talk structure ● 4 steps to make user experience great again ● 4 patterns to simplify architecture and reduce costs © 2016 Tubular Labs 2
- 3. Data size ● 2.5B documents ● AVG doc size 2Kb, 4Tb total size ● 200M daily updates (~8% of the index) ● Constant indexing rate of 3k/s with spikes ● Querying rate 1-3 r/s (low concurrency) © 2016 Tubular Labs 3
- 4. Hardware ● 52 x c3.4xlarge ● 128 shards ● 16 cores per node ● ~3 shards per node ● 832 cores, 16Tb SSD, 1.5Tb RAM © 2016 Tubular Labs 4 ● 26 x c3.8xlarge ● 416 shards ● 32 cores per node ● 16 shards per node ● 832 cores, 16Tb SSD, 1.5Tb RAM Before After (25% bigger)
- 5. Indexing
- 6. Optimize indexing ● Using bulk API • 1Mb per batch (500 docs), should be 5k docs/s • Recommended 5-15Mb ● Increasing refresh interval • From 1 to 30 seconds ● Monitoring bulk.rejected • Increased bulk.queueSize from 50 to 2000 © 2016 Tubular Labs 6
- 7. Searching
- 8. Product view © 2016 Tubular Labs 8 Summary Search results Term aggregations
- 9. Before optimization © 2016 Tubular Labs 9
- 10. Goal © 2016 Tubular Labs 10 • Slow queries • From 15 to 5 seconds for 95th • Seeking for 3x improvement Problem Goal
- 11. Understand hardware utilization © 2016 Tubular Labs 11 • Run the heaviest query • No bottlenecks (CPU, disk IO, network) • Thread pool search.size 25 • Max search.active is 3
- 12. CPU utilization © 2016 Tubular Labs 12 • Know • Your • Concurrency
- 13. Benchmarking # of shards © 2016 Tubular Labs 13 On a single 32 cores node
- 14. More CPU per request results © 2016 Tubular Labs 14 15s to 7.5s
- 15. Search & Aggregations © 2016 Tubular Labs 15 • Searching and sorting is fast • 8 term aggregations are slow
- 16. Aggregation impact © 2016 Tubular Labs 16
- 17. Check facet usage © 2016 Tubular Labs 17 ● Talk to your product manager ● Low product usage ● Remove networks and claims aggregations ● Replace facets with filters
- 18. Removing two aggregations results © 2016 Tubular Labs 18 15s to 5.3s
- 19. Cardinality © 2016 Tubular Labs 19 ● Reduce cardinality ● Going from 200M to 5M (channels to creators) ● Reducing # of topics from 5M to 500
- 20. Reducing cardinality results © 2016 Tubular Labs 20 15s to 4.4s
- 21. Split query and aggregations © 2016 Tubular Labs 21 ● Searching and aggregating separately ● Using shard-level query cache ● Showing results in UI asynchronously
- 22. Split query and aggregations results © 2016 Tubular Labs 22 15s to 4.0s
- 23. Performance gain © 2016 Tubular Labs 23 ● From 15 to 4 seconds (<5 seconds) ● Overall improvement 3.7x ● What about costs?
- 25. Part 2. Goals © 2016 Tubular Labs 25 ● Reduce costs ● Improve reliability ● Simplify architecture ● Reduce variability in latency
- 26. Current flow © 2016 Tubular Labs 26 ● Too many dependencies ● Expensive intermediate storage
- 27. Denormalization © 2016 Tubular Labs 27 ● 90% of data is shared ● No extra calls from frontend
- 28. Partial updates with Update API (experimental) © 2016 Tubular Labs 28
- 29. “Partial” updates with parent-child relations (experimental) © 2016 Tubular Labs 29
- 30. Split data by hot/full (idea for future) © 2016 Tubular Labs 30 ● Cheaper hardware on full ● Shard allocation filtering
- 31. Thank you