This document outlines 10 steps to optimize a search solution using Apache Solr. It describes common problems encountered with initial search implementations using MySQL alone, and how moving to Solr helped address them. Key steps included implementing Solr for full-text search, separating read and write operations, adding slaves for increased query capacity, implementing multiple deployment groups, and offloading filtering, sorting and aggregation to Solr. Garbage collection tuning also improved performance.

1. SOLR Power FTW
10 EZ Steps to SOLR Domination
Alex Pinkin
@apinkin

2. Alex Pinkin
● Director of Engineering,
Data and Infrastructure,
Bazaarvoice
@apinkin

3. Bazaarvoice
● SaaS company
powering user generated content
on thousands of web sites
● 5+ billion page views per month
● 300+ billion impressions

4. SOLR Stats
Documents 350 MM
Index size 330 GB
AVG QPS 3,400
MAX QPS 12,500
AVG Response time 7 ms
95P Response time 14 ms

5. Initial State - back in 2007...
● SoR: MySQL
● Search: MySQL
● Numerous indexes
● Numerous aggregates

6. Problem 1
FULL TEXT SEARCH

7. Solution 1: FT Search
● SOLR Master + Hot Backup
● Single index

8. Problem 2
● Reads overwhelming master
● No R/W isolation

9. Solution 2
● R/W separation

10. Problem 3
● QPS Growing

11. Solution 3
● Add more slaves

12. Problem 4
● Number of Clients is growing

13. Solution 4
● Multiple Deployment Groups

14. Problem 5
● MySQL still in trouble: Too many Indexes, Aggregates

15. Solution 5
● Offload filtering and sorting to SOLR
● Add more slaves
● Add more RAM

16. Problem 6
● Query latency is getting worse

17. Solution 6
● Multiple cores
● Core management tools
● Transparent data migration

18. Problem 7
MySQL is still in trouble
● Too many aggregate tables
● Error-prone aggregation logic

19. Solution 7
● Off-load simple aggregates to SOLR
● Use Faceting

20. Problem 8
● SOLR is GC'ing itself to death
● 99p Latency = :-(

21. Solution 8
● GC tuning
○ Concurrent Low Pause Collector
○ Parallel Young Generation Collector
○ Incremental mode

22. Problem 9
● Disaster Recovery (Multi-DC)

23. Solution 9
● Cross-DC replication

24. Problem 10
● Schema Changes...
"ALTER TABLE OF DOOM" @josh_wills

25. Solution 10
● Bittorrent
for the rescue!

26. Remaining Challenges
● Elasticity
● HA: failover
● Schema changes

27. Questions
@apinkin