Presented by James Atherton, Search Team Lead, 7digital A usage/case study, describing our journey as we implemented Lucene/Solr, the lessons we learned along the way and where we hope to go in the future.How we implemented our instant search/search suggest. How we handle trying to index 400 million tracks and metadata for over 40 countries, comprising over 300GB of data, and about 70GB of indexes. Finally where we hope to go in the future.

1. Implementing Search with Solr at 7digital
James Atherton
Content Discovery Team Lead

2. Implementing Search
with Solr
James Atherton
Content Discovery Lead
@mr_road

3. Who is 7digital?
Online digital content provider
Covering over 47 territories
Online music store: www.7digital.com
API: api.7digital.com
We power a number of music services:
Samsung
Blackberry
Turntable.fm
Pure

4. Where we came from...
SQL Searches
SELECT *
FROM <table>
WHERE name LIKE '<search_term>%';
This was SLOW and BAD!!

5. Wrapped Solr in an API

6. Old Architecture
API
DB

7. Domain Objects
Artist Documents
Release Documents (e.g. album or single)
Track Documents

8. First Attempt - 2011
• Artists and Releases
• Solr 1.4
• 17 stores
• ~40GB
• Dropped DIH as it had issues

9. 2011 Architecture
HTTP
API
Search
API Solr
DB
Solr
Tracks
Artists
Releases

10. 2012
• Added Tracks Core
• Solr 3.5
• 47 stores
• ~400GB
• More than 430 M docs
• Didn't revisit DIH

11. Current Architecture
HTTP
API
Search
API
Artist/Release
Solrs
Track Solrs
Track Solrs
Track Solrs
Track Solrs
Artist/Release
Solrs

12. Things Learnt
We should have split by <X>; for us Shops.

13. Beware Inflection Points
Data size: 400GB != 40GB * 10
Throughput: 600 rpm IS NOT 4 * 150 rpm

14. What we want in our servers?
RAM ?
Fast Disks?
CPUs?
Virtual?
Bare Metal?

15. Optimize really...?

16. Cache Warming/First search?

17. Testing
Test ingestion/data import, then test again
Your data is not as clean as you think
Load test early and often
We need to be better at this still

18. Logs
Logging is worth its weight in gold
But don't get weighed down

19. Monitoring
We use statsd/graphite and NewRelic:

20. Visualise Indexing
Which territory's data has been indexed?

21. Instant Search

22. Magic Deploys
We recently adopted CFEngine, it is awesome!!

23. The Future
HTTP
API
Search
API
Artist Solrs
Track
Solrs
Track
Solrs
Track
Solrs
Release
Solrs
Track Solrs
Solr Cloud, in
the Cloud??

24. Questions
?

25. Resources
https://github.com/etsy/statsd/
https://github.com/7digital
http://d3js.org/

26. James Atherton
@mr_road
@7digital