1. Curator
The Netflix ZooKeeper Client Library
Jordan Zimmerman
Senior Platform Engineer
Netflix, Inc.
jzimmerman@netflix.com
@rangalt

2. Agenda

3. Agenda
• Background
• Overview of Curator
• The Recipes
• Some Low-Level Details
• Q&A

4. Background

5. What’s wrong with this
code?
ZooKeeper client = new ZooKeeper(...);
client.create(“/foo”, data, ...);

6. ZooKeeper Surprise

7. ZooKeeper Surprise
• Almost no ZK client call is safe
• You cannot assume success
• You must handle exceptions

8. The Recipes Are Hard
Locks
Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock. These can be
implemented using ZooKeeeper. As with priority queues, first define a lock node.
Note
There now exists a Lock implementation in ZooKeeper recipes directory. This is distributed with the release -- src/recipes/lock directory of the release artifact.
Clients wishing to obtain a lock do the following:
1. Call create( ) with a pathname of "_locknode_/guid-lock-" and the sequence and ephemeral flags set. The guid is needed in case the
create() result is missed. See the note below.
2. Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect).
3. If the pathname created in step 1 has the lowest sequence number suffix, the client has the lock and the client exits the protocol.
4. The client calls exists( ) with the watch flag set on the path in the lock directory with the next lowest sequence number.
5. if exists( ) returns false, go to step 2. Otherwise, wait for a notification for the pathname from the previous step before going to step 2.
The unlock protocol is very simple: clients wishing to release a lock simply delete the node they created in step 1.
Here are a few things to notice:
• The removal of a node will only cause one client to wake up since each node is watched by exactly one client. In this way, you avoid the
herd effect.
• There is no polling or timeouts.
• Because of the way you implement locking, it is easy to see the amount of lock contention, break locks, debug locking problems, etc.
Recoverable Errors and the GUID
• If a recoverable error occurs calling create() the client should call getChildren() and check for a node containing the guid used in the
path name. This handles the case (noted above) of the create() succeeding on the server but the server crashing before returning the
name of the new node.

9. Even the Distribution
Has Issues
from org.apache.zookeeper.recipes.lock.WriteLock
if (id == null) {
long sessionId = zookeeper.getSessionId();
String prefix = "x-" + sessionId + "-";
// lets try look up the current ID if we failed
// in the middle of creating the znode
findPrefixInChildren(prefix, zookeeper, dir);
idName = new ZNodeName(id);
}

10. Even the Distribution
Has Issues
from org.apache.zookeeper.recipes.lock.WriteLock
if (id == null) {
long sessionId = zookeeper.getSessionId();
String prefix = "x-" + sessionId + "-";
// lets try look up the current ID if we failed
// in the middle of creating the znode
findPrefixInChildren(prefix, zookeeper, dir);
idName = new ZNodeName(id);
}
Bad handling of Ephemeral-Sequential issue!

11. What About ZKClient?
• Unclear if it’s still being supported
Eleven open issues (back to 10/1/2009)
• README:
“+ TBD”
• No docs
• Little or no retries
• Design problems:
• All exceptions converted to RuntimeException
• Recipes/management code highly coupled
• Lots of foreground synchronization
• Small number of tests
• ... etc ...
• ...

14. Introducing Curator

15. Introducing Curator
Curator n ˈkyo͝orˌātər: a keeper or custodian of a
museum or other collection - A ZooKeeper
Keeper
Three components:
Client - A replacement/wrapper for the bundled ZooKeeper class
Framework - A high-level API that greatly simplifies using
ZooKeeper
Recipes - Implementations of some of the common ZooKeeper
"recipes" built on top of the Curator Framework

16. Overview of Curator

17. The Curator Stack
• Client
• Framework
• Recipes
• Extensions

18. The Curator Stack
• Client
• Framework
• Recipes Curator Recipes
Curator Framework
• Extensions Curator Client
ZooKeeper

20. Curator is a platform
for writing ZooKeeper
Recipes

21. Curator Client
manages the
ZooKeeper
Connection

22. Curator Recipes
Curator Framework
Curator Client
ZooKeeper
Curator Client
manages the
ZooKeeper
Connection

23. Curator Framework
uses retry for all
operations and provides
a friendlier API

24. Curator Recipes
Curator Framework
Curator Client
ZooKeeper
Curator Framework
uses retry for all
operations and provides
a friendlier API

25. Curator Recipes:
implementations of all
recipes listed on the
ZK website (and more)

26. Curator Recipes
Curator Framework
Curator Client
ZooKeeper
Curator Recipes:
implementations of all
recipes listed on the
ZK website (and more)

27. The Recipes

28. The Recipes
• Leader Selector
• Distributed Locks
• Queues
• Barriers
• Counters
• Atomics
• ...

29. CuratorFramework
Instance
CuratorFrameworkFactory.newClient(...)
---------------------
CuratorFrameworkFactory.builder()
.connectString(“...”)
...
.build()
Usually injected as a singleton

30. Must Be Started
client.start();
// client is now ready for use

31. Leader Selector
By far the most common usage of
ZooKeeper
Distributed lock with a notification
mechanism

32. Sample

33. public class CleanupLeader implements
LeaderSelectorListener
{
...
@Override
public void takeLeadership(CuratorFramework client)
throws Exception
{
while ( !Thread.currentThread().isInterrupted() )
{
sleepUntilNextPeriod();
doPeriodicCleanup();
}
}
}
...
LeaderSelector leaderSelector =
new LeaderSelector(client, path, new CleanupLeader());
leaderSelector.start();

34. Distributed Locks
• InterProcessMutex
• InterProcessReadWriteLock
• InterProcessMultiLock
• InterProcessSemaphore

35. Distributed Locks
• InterProcessMutex
• InterProcessReadWriteLock
• InterProcessMultiLock
• InterProcessSemaphore
Very similar to JDK locks

36. Sample

37. InterProcessMutex mutex =
new InterProcessMutex(client, lockPath);
mutex.acquire();
try
{
// do work in critical section
}
finally
{
mutex.release();
}

38. Low-Level Details

39. public void process(WatchedEvent event)
{
boolean wasConnected = isConnected.get();
boolean newIsConnected = wasConnected;
if ( event.getType() == Watcher.Event.EventType.None )
{
newIsConnected = (event.getState() == Event.KeeperState.SyncConnected);
if ( event.getState() == Event.KeeperState.Expired )
{
handleExpiredSession();
}
}
if ( newIsConnected != wasConnected )
{
isConnected.set(newIsConnected);
connectionStartMs = System.currentTimeMillis();
}
...
}

40. public static boolean shouldRetry(int rc)
{
return (rc == KeeperException.Code.CONNECTIONLOSS.intValue()) ||
(rc == KeeperException.Code.OPERATIONTIMEOUT.intValue()) ||
(rc == KeeperException.Code.SESSIONMOVED.intValue()) ||
(rc == KeeperException.Code.SESSIONEXPIRED.intValue());
}
public void takeException(Exception exception) throws Exception
{
boolean rethrow = true;
if ( isRetryException(exception) )
{
if ( retryPolicy.allowRetry(retryCount++, System.currentTimeMillis() - startTimeMs) )
{
rethrow = false;
}
}
if ( rethrow )
{
throw exception;
}
}

41. byte[] responseData = RetryLoop.callWithRetry
(
client.getZookeeperClient(),
new Callable<byte[]>()
{
@Override
public byte[] call() throws Exception
{
byte[] responseData;
responseData = client.getZooKeeper().getData(path,
...);
}
return responseData;
}
}
);
return responseData;

42. client.withProtectedEphemeralSequential()

43. final AtomicBoolean firstTime = new AtomicBoolean(true);
String returnPath = RetryLoop.callWithRetry
(
client.getZookeeperClient(),
new Callable<String>()
{
@Override
public String call() throws Exception
{
...
String createdPath = null;
if ( !firstTime.get() && doProtectedEphemeralSequential )
{
createdPath = findProtectedNodeInForeground(localPath);
}
...
}
}
);

44. public interface ConnectionStateListener
{
public void stateChanged(CuratorFramework
client, ConnectionState newState);
}
public enum ConnectionState
{
SUSPENDED,
RECONNECTED,
LOST
}

45. if ( e instanceof KeeperException.ConnectionLossException )
{
connectionStateManager.addStateChange(ConnectionState.LOST);
}
private void validateConnection(CuratorEvent curatorEvent)
{
if ( curatorEvent.getType() == CuratorEventType.WATCHED )
{
if ( curatorEvent.getWatchedEvent().getState() ==
Watcher.Event.KeeperState.Disconnected )
{
connectionStateManager.addStateChange(ConnectionState.SUSPENDED);
internalSync(this, "/", null);
}
else if ( curatorEvent.getWatchedEvent().getState() ==
Watcher.Event.KeeperState.Expired )
{
connectionStateManager.addStateChange(ConnectionState.LOST);
}
else if ( curatorEvent.getWatchedEvent().getState() ==
Watcher.Event.KeeperState.SyncConnected )
{
connectionStateManager.addStateChange(ConnectionState.RECONNECTED);
}
}
}

46. Testing Utilities

47. • TestingServer: manages an internally
running ZooKeeper server
// Create the server using a random port
public TestingServer()
• TestingCluster: manages an internally
running ensemble of ZooKeeper servers.
// Creates an ensemble comprised of n servers.
// Each server will use a temp directory and
// random ports
public TestingCluster(int instanceQty)

48. Extensions
• Discovery
• Discovery REST Server
• Exhibitor
• ???

49. Extensions
• Discovery
• Discovery REST Server
• Exhibitor Curator Recipes
• ???
Curator Framework
Extensions
Curator Client
ZooKeeper

50. Exhibitor
Sneak Peak

51. Exhibitor
Sneak Peak

52. Exhibitor
Sneak Peak

53. Exhibitor
Sneak Peak

54. Exhibitor
Sneak Peak
March or April 2012
Open Source on Github

57. Netflix Github

58. Netflix Github
Netflix’s home for Open Source

59. Maven Central

60. Maven Central
Binaries pushed to Maven Central
<dependency>
<groupId>com.netflix.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>1.1.0</version>
</dependency>

61. Much%younger%–%much%thinner0
Jordan Zimmerman
jzimmerman@netflix.com
@randgalt

62. Q&A
Much%younger%–%much%thinner0
Jordan Zimmerman
jzimmerman@netflix.com
@randgalt