Presented by Marc Sturlese, Architect, Backend engineer, Trovit In this talk I will explain how we combine a mixed architecture using Hadoop for batch indexing and Storm, HBase and Zookeeper to keep our indexes updated in near real time.Will talk about why we didn't choose just a default Solr Cloud and it's real time feature (mainly to avoid hitting merges while serving queries on the slaves) and the advantages and complexities of having a mixed architecture. Both parts of the infrastucture and how they are coordinated will be explained with details.Finally will mention future lines, how we plan to use Lucene real time feature.

1. Batch Indexing & Near Real Time,
keeping things fast
Marc Sturlese
Software engineer @ Trovit

2. About me...
• Marc Sturlese – @sturlese
• Software engineer @Trovit. R&D focused
• Responsible for search and scalability

3. Agenda
• Who we are
• Batch architecture. Hadoop & Hive
• Near real time architecture. Storm & stuff
• Putting it all together
• Alternatives and Future directions
• Questions

4. Who we are
Trovit, a search engine for classifieds

5. Who we are

6. Batch Layer
• Hadoop based
• Documents are crunched by a pipeline of MR
jobs
• Hive to save stats of each phase

7. Batch Layer
Pipeline overview
Incoming data
Deployment
Lucene Indexes
Ad Processor Diff Matching Expiration Deduplication Indexing
t – 1
External Data
Hive Stats
Hadoop Cluster

8. Batch Layer
The good things!
• Index always built from scratch. Small number of
big segments
• Multicast deployment allows to send indexes to
all slaves at the same time.
• Backups convenient on HDFS

9. Batch Layer
That was cool but...
• Not even close to real time
• Crunch documents in batch means to wait until
all is processed. This can take a few hours
• We want to show the user fresher results!

10. Near real time Layer
Storm and stuff to the rescue

11. Near real time Layer
Storm properties
• Distributed real time computation system
• Fault tolerance
• Horizontal scalability
• Low latency
• Reliability

12. Near real time Layer
Storm in action
Slave
Slave
Solr prod replicas
Slave
XML feed
XML feed
Kafka partition
Kafka partition
Storm topologySources
Kafka spout
Kafka spout
XML spout Doc Manager bolt Indexer bolt
SHUFFLE
GROUPING GROUPING
FIELD

13. Near real time Layer
Storm in action
• Spouts just read and send
• Doc Manager Bolt processes and classifies
• Indexer Bolt adds documents to Solr
• Replicated logic with different implementation
• Careful not to overload Solr slaves...

14. Near real time Layer
Storm in action

15. Near real time Layer
Storm in action. But...

16. Near real time Layer
Storm in action. But...
• Now Solr has to handle user queries and storm
inserts
• Field grouping on Indexer Bolt for politeness
• Small bulks to reduce insert requests
• Committing on many cores, same host, same
time can be painful

17. Near real time Layer
Storm in action - Committing
Indexer Bolt Cars US
Real state UK R1 Cars US R1 Cars US R2 Jobs BR R1 Jobs BR R2 Real state ES R1
Indexer Bolt Jobs BR
ZooKeeper Locker
Slave 1 Slave 2 Slave N
. . .

18. Near real time Layer
Storm in action
• Adding documents now is fast
• Keep number of segments small
• Avoid merges on big segments
• Just add new docs (no deletes or updates)

19. Mixed Architecture
Putting it all together
15
Slave
Slave
Solr prod replicas
Slave
XML feed
XML feed
Kafka partition
Kafka partition
Storm topologySources
Hbase doc info
Bulk add
Exists?
MR Pipeline
zk

20. Mixed Architecture
Swapping indexes
• NRT docs might not be contained in the new
batch index (even fresher than the “being built”
batch index)
• This can lead to inconsistencies...

21. Mixed Architecture
Swapping indexes. Time jumps!

22. Mixed Architecture
Swapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexer
Batch indexer

23. Mixed Architecture
Swapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexer
Batch indexer

24. Mixed Architecture
Swapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexer
Batch indexer
NRT t+1
NRT t+2

25. Mixed Architecture
Swapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexer
Batch indexer
NRT t+1
NRT t+2

26. Mixed Architecture
Swapping indexes
• NRT indexed docs must be stored in a
temporary storage
• Fetch missing docs from the storage and add
them before the next deploy
• This avoids time jumps

27. Mixed Architecture
Storm and Hadoop
• Near real time inserts, low latency
• Hadoop handles deletes and updates. No rush
on those
• No merges on big segments so optimal query
response times
• Tolerant to human errors
• Temporary lost of accuracy on the NRT layer

28. Alternatives
SolrCloud - Why not?
• Good for the vast majority of use cases
• Incremental inserts/updates/deletes oriented.
Pay segment merges per real time
• Need to deploy full indexes fast (faster that rsync
or http replication)
• Now full deploy easier with aliases

29. Future lines
Lucene real time feature
• Allows to see docs in the index before they are
committed
• Good but not a must right now for the use case
• Very easy to integrate on the current
architecture

30. ??

31. Thanks for your attention!
Marc Sturlese
marc@trovit.com
Lucene/Solr Revolution 2013, San Diego, May 1 2013

32. CONFERENCE PARTY
The Tipsy Crow: 770 5th Ave
Starts after Stump The Chump
Your conference badge gets
you in the door
TOMORROW
Breakfast starts at 7:30
Keynotes start at 8:30