My talk at Apache Big Data NA 2016

1. Managing a large scale
SolrCloud cluster via APIs
Anshum Gupta
Sr. Software Engineer, IBM Watson
Apache Lucene/Solr Committer & PMC member

2. About me
• Anshum Gupta, Apache Lucene/Solr committer and PMC member,
IBM Watson Search team.
• Interested in search and related stuff.
• Apache Lucene since 2006 and Solr since 2010.
© 2014 International Business Machines Corporation 2

3. Apache Solr is the most widely-used search solution
on the planet.
Solr has tens of thousands of applications in
production.
You use everyday.
8,000,000+
Total downloads
Solr is both established
and growing.
250,000+
Monthly downloads
2,500+
Open Solr jobs and the largest community
of developers.

4. A little bit of history
• SolrCloud
• Introduced in 4.0
• Not Solr in the Cloud but set of distributed features for Solr
• No APIs to manage distributed clusters
• We’ve come a long way since then
© 2014 International Business Machines Corporation 4

5. 01
SolrCloud - Physical Architecture
ZooKeep
er
Node 1 Node 2
Load
Balancer
Client
Client
Client
Client
Client
Client
Client
Client
Client
Lots
Of
Interaction
Coins by Creative Stall from the Noun Project

6. SolrCloud Terminology
• Node: JVM process that hosts the Solr web application
• Collection: Logic index distributed across multiple nodes
• Shard: Logical slice of a collection. Each shard has a name, hash range
(optional), leader, and replicationFactor.
• Replica: Physical Solr index that can accept updates or queries. Implemented
as a Solr core.
• ReplicationFactor: Number of copies of each shard.
• Leader: One replica per shard which assumes special duties like versioning.
They are elected using ZooKeeper.
• Overseer: Special node that manages cluster level operations. Also elected
using Zookeeper.
© 2014 International Business Machines Corporation 6

7. CONFIDENTIAL
Exploring the APIs - Getting Started
© 2014 International Business Machines Corporation 7

8. Creating a collection
• HTTP API
• Response
© 2014 International Business Machines Corporation 8
http://localhost:8983/solr/admin/collections?action=CREATE&name=foo&numShard
s=2&replicationFactor=1
{"responseHeader":{
"status":0,
"QTime":1848},
"success":{
"":{"responseHeader":{
"status":0,
"QTime":1741},
"core":"foo_shard1_replica1"}}}

9. Add a replica
• HTTP API
• Response
© 2014 International Business Machines Corporation 9
http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=fo
o&shard=shard1
{"responseHeader":{
"status":0,
"QTime":1156},
"success":{
"":{"responseHeader":{
"status":0,
"QTime":1147},
"core":"foo_shard1_replica4"}}}
+

10. We added more than what we need - Delete replica
• HTTP API
• Response
© 2014 International Business Machines Corporation 10
http://localhost:8983/solr/admin/collections?action=DELETEREPLICA&collection
=foo&shard=shard1&replica=core_node3
{"responseHeader":{
"status":0,
"QTime":111},
"success":{
"":{
"responseHeader":{
"status":0,
"QTime":2}}}}

11. CONFIDENTIAL
Exploring the APIs - Rule based scaling out
© 2014 International Business Machines Corporation 11

12. Rule based Replica Assignment
• Use it during creation of:
• Collection
• Shard
• Replica
• Rules based on:
• Hostname
• Free disk space
• Number of cores
• Node name
• System property
• Other Metrics
© 2014 International Business Machines Corporation 12

13. Rule based Replica Assignment
• How is a rule defined?
• When creating a new replica, apply to each node.
• Supported Operators
© 2014 International Business Machines Corporation 13
f(shard, replica, tag)
default, > , < , !

14. Rule based Replica Assignment - Examples
• Example #1 - for any shard, limit number of replicas to 2 on any node
• Example #2 - For any replica, limit number of cores to 5 on a node
• Example #3 - Do not create any cores on host 198.21.2.19
© 2014 International Business Machines Corporation 14
shard:*,replica:<2,node:*
replica:*,cores:<5
host:!198.21.2.19

15. CONFIDENTIAL
Adjusting to growth - Moving data
© 2014 International Business Machines Corporation 15

16. Divide and Rule? - Let’s split a shard
• HTTP API
• Response
© 2014 International Business Machines Corporation 16
http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=ba
r&shard=shard1
Too Verbose to Share!

17. Another way to move data - Migrate
• HTTP API
• Response
• What happens?
• Move documents to target collection
• Handle incoming updates and have temporary routing for original collection to
send updates to target
© 2014 International Business Machines Corporation 17
http://localhost:8983/solr/admin/collections?action=MIGRATE&collection=foo&s
plit.key=a!&target.collection=bar
Too Verbose to fit in here!

18. CONFIDENTIAL
Monitoring your Cluster
© 2014 International Business Machines Corporation 18

19. LIST Collections - Let’s see what we have
• HTTP API
• Response
© 2014 International Business Machines Corporation 19
http://localhost:8983/solr/admin/collections?action=LIST
{"responseHeader":{
"status":0,
"QTime":0},
"collections":["a",
"bar",
"foo"]}

20. CLUSTERSTATUS : Don’t we like details?
© 2014 International Business Machines Corporation 20
http://localhost:8983/solr/admin/
collections?action=CLUSTERSTATUS
{"cluster":{
"collections":{
"foo":{
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node1":{
"core":"foo_shard1_replica1",
"base_url":"http://192.168.1.125:8983/solr",
"node_name":"192.168.1.125:8983_solr",
"state":"active",
"leader":"true"},
"core_node2":{
"core":"foo_shard1_replica2",
"base_url":"http://192.168.1.125:8983/solr",
"node_name":"192.168.1.125:8983_solr",
"state":"active"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"100",
"autoAddReplicas":"false",
"znodeVersion":14,
"configName":"a"}},
"live_nodes":["192.168.1.125:8983_solr"]}}
• HTTP API

21. OVERSEERSTATUS - More than a sneak peek
© 2014 International Business Machines Corporation 21
http://localhost:8983/solr/admin/collections?action=OVERSEERSTATUS
• HTTP API
• Response - Verbose. Actually, more than that!
• Stats for:
• Errors
• Time taken
• Much more

22. CONFIDENTIAL
Managing your Cluster
© 2014 International Business Machines Corporation 22

23. Set a specific node to be the Overseer
© 2014 International Business Machines Corporation 23
http://localhost:8983/solr/admin/collections?action=ADDROLE&role=overseer&node
=192.168.1.125:8983_solr
• ADD Role
• Response - Nothing substantial but a 200 on success
• Supported role:
• Only Overseer for now
• Remember - It’s only a preference

24. Setting a preferred leader
© 2014 International Business Machines Corporation 24
http://localhost:8983/solr/admin/collections?action=ADDREPLICAPROP&shard=shard
1&collection=foo&replica=core_node2&property=preferredLeader&property.value=tr
ue
• Using ADDREPLICAPROP
• Sets ‘preferredLeader’ to ‘true’. The only pre-defined property.

25. Setting preferred leaders for the entire collection
© 2014 International Business Machines Corporation 25
http://localhost:8983/solr/admin/collections?action=BALANCESHARDUNIQUE&collect
ion=foo&property=preferredLeader
• Using BALANCESHARDUNIQUE
• Sets ‘preferredLeader’ to ‘true’ evenly across the collection

26. Rebalance the leaders
© 2014 International Business Machines Corporation 26
http://localhost:8983/solr/admin/collections?action=REBALANCELEADERS&collecti
on=foo
• Using REBALANCELEADERS
• This allows preferredLeaders to take up leadership for shards.

27. I lost my leader! Where do I go now?
© 2014 International Business Machines Corporation 27
http://localhost:8983/solr/admin/collections?action=FORCELEADER&collection=fo
o&shard=shard1
• Force a leader election using FORCELEADER API
• This can ONLY be used when the shard has no leader.

28. CONFIDENTIAL
Recipe #1 - Shard Splitting
© 2014 International Business Machines Corporation 28

29. Splitting a shard - It’s more than just the single API call
• Monitor disk-size
• CLUSTERSTATUS & Core STATUS
• SPLITSHARD
• Make sure there’s enough spare disk space
• Add one more replica
• Force leader election (DELETEREPLICA)
• Delete old INACTIVE shard (Parent)
© 2014 International Business Machines Corporation 29

30. CONFIDENTIAL
Recipe #2 - Ensuring high availability
© 2014 International Business Machines Corporation 30

31. High availability through a monitoring application
• Auto-add replica
• Monitor cluster status - CLUSTERSTATUS API
• ADD Replica using replica placement strategy
• Remove replicas when there are too many
© 2014 International Business Machines Corporation 31

32. CONFIDENTIAL
Recipe #3 - Migrating Cluster Infrastructure
© 2014 International Business Machines Corporation 32

33. Migrating cluster infrastructure
• CLUSTERSTATUS, and OVERSEERSTATUS
• Overseer, Leaders, etc.
• Move the Overseer first OR last
• Rebalance leaders
• Force leader elections if required*
© 2014 International Business Machines Corporation 33
*The assumption is you know what you’re doing!

34. CONFIDENTIAL
What’s more?
© 2014 International Business Machines Corporation 34

35. There’s a lot more, a single session is too less!
• Version 2 APIs
• SolrJ support for all the APIs - Easy to tool in Java
• Security
• Cross Data Center Support
© 2014 International Business Machines Corporation 35

36. Coming Soon - SolrCloud Backup and Restore!
• Committed just a few days ago
• Back up and restore of
• Collections
• ZK metadata
• Configs
• Release with 6.1 (and may be 5.6)
© 2014 International Business Machines Corporation 36

37. http://www.twitter.com/anshumgupta
http://www.linkedin.com/in/anshumgupta/
anshum@apache.org
Connect @