SlideShare ist ein Scribd-Unternehmen logo
Curator
The Netflix ZooKeeper Client Library




                           Jordan Zimmerman
                                   Senior Platform Engineer
                                                 Netflix, Inc.
                                 jzimmerman@netflix.com
                                                  @rangalt
Agenda
Agenda
• Background
• Overview of Curator
• The Recipes
• Some Low-Level Details
• Q&A
Background
What’s wrong with this
        code?


ZooKeeper       client = new ZooKeeper(...);

client.create(“/foo”, data, ...);
ZooKeeper Surprise
ZooKeeper Surprise
• Almost no ZK client call is safe
• You cannot assume success
• You must handle exceptions
The Recipes Are Hard
       Locks
Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock. These can be
implemented using ZooKeeeper. As with priority queues, first define a lock node.

Note
There now exists a Lock implementation in ZooKeeper recipes directory. This is distributed with the release -- src/recipes/lock directory of the release artifact.

Clients wishing to obtain a lock do the following:

  1.  Call create( ) with a pathname of "_locknode_/guid-lock-" and the sequence and ephemeral flags set. The guid is needed in case the
      create() result is missed. See the note below.
  2. Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect).
  3. If the pathname created in step 1 has the lowest sequence number suffix, the client has the lock and the client exits the protocol.
  4. The client calls exists( ) with the watch flag set on the path in the lock directory with the next lowest sequence number.
  5. if exists( ) returns false, go to step 2. Otherwise, wait for a notification for the pathname from the previous step before going to step 2.
The unlock protocol is very simple: clients wishing to release a lock simply delete the node they created in step 1.

Here are a few things to notice:

  •    The removal of a node will only cause one client to wake up since each node is watched by exactly one client. In this way, you avoid the
       herd effect.
  •    There is no polling or timeouts.
  •    Because of the way you implement locking, it is easy to see the amount of lock contention, break locks, debug locking problems, etc.
Recoverable Errors and the GUID
  •    If a recoverable error occurs calling create() the client should call getChildren() and check for a node containing the guid used in the
       path name. This handles the case (noted above) of the create() succeeding on the server but the server crashing before returning the
       name of the new node.
Even the Distribution
         Has Issues
from org.apache.zookeeper.recipes.lock.WriteLock
if (id == null) {
    long sessionId = zookeeper.getSessionId();
    String prefix = "x-" + sessionId + "-";
    // lets try look up the current ID if we failed
    // in the middle of creating the znode
    findPrefixInChildren(prefix, zookeeper, dir);
    idName = new ZNodeName(id);
}
Even the Distribution
           Has Issues
  from org.apache.zookeeper.recipes.lock.WriteLock
  if (id == null) {
      long sessionId = zookeeper.getSessionId();
      String prefix = "x-" + sessionId + "-";
      // lets try look up the current ID if we failed
      // in the middle of creating the znode
      findPrefixInChildren(prefix, zookeeper, dir);
      idName = new ZNodeName(id);
  }


Bad handling of Ephemeral-Sequential issue!
What About ZKClient?
•   Unclear if it’s still being supported
    Eleven open issues (back to 10/1/2009)
•   README:
    “+ TBD”
•   No docs
•   Little or no retries
•   Design problems:
          •   All exceptions converted to RuntimeException
          •   Recipes/management code highly coupled
          •   Lots of foreground synchronization
          •   Small number of tests
          •   ... etc ...
•   ...
Curator intro
Curator intro
Introducing Curator
Introducing Curator
Curator n ˈkyo͝orˌātər: a keeper or custodian of a
museum or other collection - A ZooKeeper
Keeper
Three components:
  Client - A replacement/wrapper for the bundled ZooKeeper class

  Framework - A high-level API that greatly simplifies using
  ZooKeeper

  Recipes - Implementations of some of the common ZooKeeper
  "recipes" built on top of the Curator Framework
Overview of Curator
The Curator Stack
• Client
• Framework
• Recipes
• Extensions
The Curator Stack
• Client
• Framework
• Recipes        Curator Recipes


                Curator Framework



• Extensions      Curator Client

                    ZooKeeper
Curator intro
Curator is a platform
for writing ZooKeeper
        Recipes
Curator Client
 manages the
 ZooKeeper
 Connection
Curator Recipes


                 Curator Framework

                   Curator Client

                     ZooKeeper




Curator Client
 manages the
 ZooKeeper
 Connection
Curator Framework
   uses retry for all
operations and provides
    a friendlier API
Curator Recipes


                   Curator Framework

                     Curator Client

                       ZooKeeper




 Curator Framework
   uses retry for all
operations and provides
    a friendlier API
Curator Recipes:
implementations of all
 recipes listed on the
ZK website (and more)
Curator Recipes


                  Curator Framework

                    Curator Client

                      ZooKeeper




   Curator Recipes:
implementations of all
 recipes listed on the
ZK website (and more)
The Recipes
The Recipes
• Leader Selector
• Distributed Locks
• Queues
• Barriers
• Counters
• Atomics
• ...
CuratorFramework
               Instance
CuratorFrameworkFactory.newClient(...)
              ---------------------
CuratorFrameworkFactory.builder()
   .connectString(“...”)
   ...
   .build()




         Usually injected as a singleton
Must Be Started

client.start();


// client is now ready for use
Leader Selector
By far the most common usage of
ZooKeeper


       Distributed lock with a notification
       mechanism
Sample
public class CleanupLeader implements
         LeaderSelectorListener
  {
      ...
      @Override
      public void takeLeadership(CuratorFramework client)
            throws Exception
      {
         while ( !Thread.currentThread().isInterrupted() )
         {
           sleepUntilNextPeriod();
           doPeriodicCleanup();
         }
      }
  }




...
LeaderSelector leaderSelector =
    new LeaderSelector(client, path, new CleanupLeader());
leaderSelector.start();
Distributed Locks
• InterProcessMutex
• InterProcessReadWriteLock
• InterProcessMultiLock
• InterProcessSemaphore
Distributed Locks
• InterProcessMutex
• InterProcessReadWriteLock
• InterProcessMultiLock
• InterProcessSemaphore

            Very similar to JDK locks
Sample
InterProcessMutex mutex =
    new InterProcessMutex(client, lockPath);

mutex.acquire();
try
{
    // do work in critical section
}
finally
{
    mutex.release();
}
Low-Level Details
public void process(WatchedEvent event)
{
    boolean wasConnected = isConnected.get();
    boolean newIsConnected = wasConnected;
    if ( event.getType() == Watcher.Event.EventType.None )
    {
        newIsConnected = (event.getState() == Event.KeeperState.SyncConnected);
        if ( event.getState() == Event.KeeperState.Expired )
        {
            handleExpiredSession();
        }
    }

    if ( newIsConnected != wasConnected )
    {
        isConnected.set(newIsConnected);
        connectionStartMs = System.currentTimeMillis();
    }

     ...
}
public static boolean      shouldRetry(int rc)
{
    return (rc == KeeperException.Code.CONNECTIONLOSS.intValue()) ||
        (rc == KeeperException.Code.OPERATIONTIMEOUT.intValue()) ||
        (rc == KeeperException.Code.SESSIONMOVED.intValue()) ||
        (rc == KeeperException.Code.SESSIONEXPIRED.intValue());
}




public void         takeException(Exception exception) throws Exception
{
    boolean     rethrow = true;
    if ( isRetryException(exception) )
    {
        if ( retryPolicy.allowRetry(retryCount++, System.currentTimeMillis() - startTimeMs) )
        {
            rethrow = false;
        }
    }

    if ( rethrow )
    {
        throw exception;
    }
}
byte[]      responseData = RetryLoop.callWithRetry
(
    client.getZookeeperClient(),
    new Callable<byte[]>()
    {
        @Override
        public byte[] call() throws Exception
        {
            byte[]      responseData;
            responseData = client.getZooKeeper().getData(path,
               ...);
            }
            return responseData;
        }
    }
);
return responseData;
client.withProtectedEphemeralSequential()
final AtomicBoolean     firstTime = new AtomicBoolean(true);
String                  returnPath = RetryLoop.callWithRetry
(
    client.getZookeeperClient(),
    new Callable<String>()
    {
        @Override
        public String call() throws Exception
        {
           ...

               String createdPath = null;
               if ( !firstTime.get() && doProtectedEphemeralSequential )
               {
                   createdPath = findProtectedNodeInForeground(localPath);
               }
             ...
         }
     }
);
public interface ConnectionStateListener
{
    public void stateChanged(CuratorFramework
        client, ConnectionState newState);
}

public enum ConnectionState
{
    SUSPENDED,
    RECONNECTED,
    LOST
}
if ( e instanceof KeeperException.ConnectionLossException )
  {
      connectionStateManager.addStateChange(ConnectionState.LOST);
  }



private void validateConnection(CuratorEvent curatorEvent)
{
    if ( curatorEvent.getType() == CuratorEventType.WATCHED )
    {
        if ( curatorEvent.getWatchedEvent().getState() ==
          Watcher.Event.KeeperState.Disconnected )
        {
            connectionStateManager.addStateChange(ConnectionState.SUSPENDED);
            internalSync(this, "/", null);
        }
        else if ( curatorEvent.getWatchedEvent().getState() ==
          Watcher.Event.KeeperState.Expired )
        {
            connectionStateManager.addStateChange(ConnectionState.LOST);
        }
        else if ( curatorEvent.getWatchedEvent().getState() ==
          Watcher.Event.KeeperState.SyncConnected )
        {
            connectionStateManager.addStateChange(ConnectionState.RECONNECTED);
        }
    }
}
Testing Utilities
• TestingServer: manages an internally
  running ZooKeeper server
  // Create the server using a random port
  public TestingServer()




• TestingCluster: manages an internally
  running ensemble of ZooKeeper servers.
  // Creates an ensemble comprised of n servers.
  // Each server will use a temp directory and
  // random ports
  public TestingCluster(int instanceQty)
Extensions
• Discovery
• Discovery REST Server
• Exhibitor
• ???
Extensions
• Discovery
• Discovery REST Server
• Exhibitor                Curator Recipes




• ???
                          Curator Framework
                                              Extensions
                            Curator Client

                              ZooKeeper
Exhibitor
  Sneak Peak
Exhibitor
  Sneak Peak
Exhibitor
  Sneak Peak
Exhibitor
  Sneak Peak
Exhibitor
       Sneak Peak




 March or April 2012
Open Source on Github
Curator intro
Curator intro
Netflix Github
Netflix Github




Netflix’s home for Open Source
Maven Central
Maven Central
Binaries pushed to Maven Central

  <dependency>
      <groupId>com.netflix.curator</groupId>
      <artifactId>curator-recipes</artifactId>
      <version>1.1.0</version>
  </dependency>
Much%younger%–%much%thinner0


        Jordan Zimmerman
 jzimmerman@netflix.com
                @randgalt
Q&A


 Much%younger%–%much%thinner0


         Jordan Zimmerman
  jzimmerman@netflix.com
                 @randgalt

Weitere ähnliche Inhalte

Was ist angesagt?

Developing distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka ClusterDeveloping distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka Cluster
Konstantin Tsykulenko
 
Apache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesdayApache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesday
Andrei Savu
 
Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!
Joydeep Banik Roy
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
knowbigdata
 
Zookeeper In Action
Zookeeper In ActionZookeeper In Action
Zookeeper In Action
juvenxu
 
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API ExamplesApache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
Binu George
 
ZooKeeper - wait free protocol for coordinating processes
ZooKeeper - wait free protocol for coordinating processesZooKeeper - wait free protocol for coordinating processes
ZooKeeper - wait free protocol for coordinating processes
Julia Proskurnia
 
Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...
Jimmy Lai
 
Comparing ZooKeeper and Consul
Comparing ZooKeeper and ConsulComparing ZooKeeper and Consul
Comparing ZooKeeper and Consul
Ivan Glushkov
 
Introduction to apache zoo keeper
Introduction to apache zoo keeper Introduction to apache zoo keeper
Introduction to apache zoo keeper
Omid Vahdaty
 
Zookeeper Introduce
Zookeeper IntroduceZookeeper Introduce
Zookeeper Introduce
jhao niu
 
Consul - service discovery and others
Consul - service discovery and othersConsul - service discovery and others
Consul - service discovery and others
Walter Liu
 
REEF: Towards a Big Data Stdlib
REEF: Towards a Big Data StdlibREEF: Towards a Big Data Stdlib
REEF: Towards a Big Data Stdlib
DataWorks Summit
 
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
使用ZooKeeper打造軟體式負載平衡
使用ZooKeeper打造軟體式負載平衡使用ZooKeeper打造軟體式負載平衡
使用ZooKeeper打造軟體式負載平衡
Lawrence Huang
 
Apache Zookeeper
Apache ZookeeperApache Zookeeper
Apache Zookeeper
Nguyen Quang
 
Zoo keeper in the wild
Zoo keeper in the wildZoo keeper in the wild
Zoo keeper in the wild
datamantra
 
Terraform introduction
Terraform introductionTerraform introduction
Terraform introduction
Jason Vance
 
Akka Actor presentation
Akka Actor presentationAkka Actor presentation
Akka Actor presentation
Gene Chang
 
Native container monitoring
Native container monitoringNative container monitoring
Native container monitoring
Rohit Jnagal
 

Was ist angesagt? (20)

Developing distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka ClusterDeveloping distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka Cluster
 
Apache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesdayApache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesday
 
Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
 
Zookeeper In Action
Zookeeper In ActionZookeeper In Action
Zookeeper In Action
 
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API ExamplesApache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API Examples
 
ZooKeeper - wait free protocol for coordinating processes
ZooKeeper - wait free protocol for coordinating processesZooKeeper - wait free protocol for coordinating processes
ZooKeeper - wait free protocol for coordinating processes
 
Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...
 
Comparing ZooKeeper and Consul
Comparing ZooKeeper and ConsulComparing ZooKeeper and Consul
Comparing ZooKeeper and Consul
 
Introduction to apache zoo keeper
Introduction to apache zoo keeper Introduction to apache zoo keeper
Introduction to apache zoo keeper
 
Zookeeper Introduce
Zookeeper IntroduceZookeeper Introduce
Zookeeper Introduce
 
Consul - service discovery and others
Consul - service discovery and othersConsul - service discovery and others
Consul - service discovery and others
 
REEF: Towards a Big Data Stdlib
REEF: Towards a Big Data StdlibREEF: Towards a Big Data Stdlib
REEF: Towards a Big Data Stdlib
 
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
 
使用ZooKeeper打造軟體式負載平衡
使用ZooKeeper打造軟體式負載平衡使用ZooKeeper打造軟體式負載平衡
使用ZooKeeper打造軟體式負載平衡
 
Apache Zookeeper
Apache ZookeeperApache Zookeeper
Apache Zookeeper
 
Zoo keeper in the wild
Zoo keeper in the wildZoo keeper in the wild
Zoo keeper in the wild
 
Terraform introduction
Terraform introductionTerraform introduction
Terraform introduction
 
Akka Actor presentation
Akka Actor presentationAkka Actor presentation
Akka Actor presentation
 
Native container monitoring
Native container monitoringNative container monitoring
Native container monitoring
 

Andere mochten auch

Apache Curator: Past, Present and Future
Apache Curator: Past, Present and FutureApache Curator: Past, Present and Future
Apache Curator: Past, Present and Future
Jordan Zimmerman
 
Exhibitor Introduction
Exhibitor IntroductionExhibitor Introduction
Exhibitor Introduction
Jordan Zimmerman
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
Saurav Haloi
 
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Martin Zapletal
 
Introduction to Kafka and Zookeeper
Introduction to Kafka and ZookeeperIntroduction to Kafka and Zookeeper
Introduction to Kafka and Zookeeper
Rahul Jain
 
Chicago Hadoop Users Group: Enterprise Data Workflows
Chicago Hadoop Users Group: Enterprise Data WorkflowsChicago Hadoop Users Group: Enterprise Data Workflows
Chicago Hadoop Users Group: Enterprise Data Workflows
Paco Nathan
 
Spring 3.1 and MVC Testing Support - 4Developers
Spring 3.1 and MVC Testing Support - 4DevelopersSpring 3.1 and MVC Testing Support - 4Developers
Spring 3.1 and MVC Testing Support - 4Developers
Sam Brannen
 
Reactive Programming With Akka - Lessons Learned
Reactive Programming With Akka - Lessons LearnedReactive Programming With Akka - Lessons Learned
Reactive Programming With Akka - Lessons Learned
Daniel Sawano
 
A Sceptical Guide to Functional Programming
A Sceptical Guide to Functional ProgrammingA Sceptical Guide to Functional Programming
A Sceptical Guide to Functional Programming
Garth Gilmour
 
The no-framework Scala Dependency Injection Framework
The no-framework Scala Dependency Injection FrameworkThe no-framework Scala Dependency Injection Framework
The no-framework Scala Dependency Injection Framework
Adam Warski
 
Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2
Martin Zapletal
 
Effective akka scalaio
Effective akka scalaioEffective akka scalaio
Effective akka scalaio
shinolajla
 
Actor Based Asyncronous IO in Akka
Actor Based Asyncronous IO in AkkaActor Based Asyncronous IO in Akka
Actor Based Asyncronous IO in Akka
drewhk
 
ZooKeeper Futures
ZooKeeper FuturesZooKeeper Futures
ZooKeeper Futures
Cloudera, Inc.
 
ZooKeeper (and other things)
ZooKeeper (and other things)ZooKeeper (and other things)
ZooKeeper (and other things)
Jonathan Halterman
 
Taming Pythons with ZooKeeper
Taming Pythons with ZooKeeperTaming Pythons with ZooKeeper
Taming Pythons with ZooKeeper
Jyrki Pulliainen
 
Zookeeper
ZookeeperZookeeper
Zookeeper
ltsllc
 
Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)
Jyrki Pulliainen
 
Large volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive PlatformLarge volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive Platform
Martin Zapletal
 
Efficient HTTP Apis
Efficient HTTP ApisEfficient HTTP Apis
Efficient HTTP Apis
Adrian Cole
 

Andere mochten auch (20)

Apache Curator: Past, Present and Future
Apache Curator: Past, Present and FutureApache Curator: Past, Present and Future
Apache Curator: Past, Present and Future
 
Exhibitor Introduction
Exhibitor IntroductionExhibitor Introduction
Exhibitor Introduction
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
 
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
 
Introduction to Kafka and Zookeeper
Introduction to Kafka and ZookeeperIntroduction to Kafka and Zookeeper
Introduction to Kafka and Zookeeper
 
Chicago Hadoop Users Group: Enterprise Data Workflows
Chicago Hadoop Users Group: Enterprise Data WorkflowsChicago Hadoop Users Group: Enterprise Data Workflows
Chicago Hadoop Users Group: Enterprise Data Workflows
 
Spring 3.1 and MVC Testing Support - 4Developers
Spring 3.1 and MVC Testing Support - 4DevelopersSpring 3.1 and MVC Testing Support - 4Developers
Spring 3.1 and MVC Testing Support - 4Developers
 
Reactive Programming With Akka - Lessons Learned
Reactive Programming With Akka - Lessons LearnedReactive Programming With Akka - Lessons Learned
Reactive Programming With Akka - Lessons Learned
 
A Sceptical Guide to Functional Programming
A Sceptical Guide to Functional ProgrammingA Sceptical Guide to Functional Programming
A Sceptical Guide to Functional Programming
 
The no-framework Scala Dependency Injection Framework
The no-framework Scala Dependency Injection FrameworkThe no-framework Scala Dependency Injection Framework
The no-framework Scala Dependency Injection Framework
 
Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2
 
Effective akka scalaio
Effective akka scalaioEffective akka scalaio
Effective akka scalaio
 
Actor Based Asyncronous IO in Akka
Actor Based Asyncronous IO in AkkaActor Based Asyncronous IO in Akka
Actor Based Asyncronous IO in Akka
 
ZooKeeper Futures
ZooKeeper FuturesZooKeeper Futures
ZooKeeper Futures
 
ZooKeeper (and other things)
ZooKeeper (and other things)ZooKeeper (and other things)
ZooKeeper (and other things)
 
Taming Pythons with ZooKeeper
Taming Pythons with ZooKeeperTaming Pythons with ZooKeeper
Taming Pythons with ZooKeeper
 
Zookeeper
ZookeeperZookeeper
Zookeeper
 
Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)
 
Large volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive PlatformLarge volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive Platform
 
Efficient HTTP Apis
Efficient HTTP ApisEfficient HTTP Apis
Efficient HTTP Apis
 

Ähnlich wie Curator intro

Native Java with GraalVM
Native Java with GraalVMNative Java with GraalVM
Native Java with GraalVM
Sylvain Wallez
 
(DEV204) Building High-Performance Native Cloud Apps In C++
(DEV204) Building High-Performance Native Cloud Apps In C++(DEV204) Building High-Performance Native Cloud Apps In C++
(DEV204) Building High-Performance Native Cloud Apps In C++
Amazon Web Services
 
JavaOne 2017 - TestContainers: integration testing without the hassle
JavaOne 2017 - TestContainers: integration testing without the hassleJavaOne 2017 - TestContainers: integration testing without the hassle
JavaOne 2017 - TestContainers: integration testing without the hassle
Anton Arhipov
 
Java Concurrency, Memory Model, and Trends
Java Concurrency, Memory Model, and TrendsJava Concurrency, Memory Model, and Trends
Java Concurrency, Memory Model, and Trends
Carol McDonald
 
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
FalafelSoftware
 
Breaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit TestingBreaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit Testing
Steven Smith
 
Performance Test Driven Development with Oracle Coherence
Performance Test Driven Development with Oracle CoherencePerformance Test Driven Development with Oracle Coherence
Performance Test Driven Development with Oracle Coherence
aragozin
 
Construire une application JavaFX 8 avec gradle
Construire une application JavaFX 8 avec gradleConstruire une application JavaFX 8 avec gradle
Construire une application JavaFX 8 avec gradle
Thierry Wasylczenko
 
Ob1k presentation at Java.IL
Ob1k presentation at Java.ILOb1k presentation at Java.IL
Ob1k presentation at Java.IL
Eran Harel
 
Silicon Valley JUG: JVM Mechanics
Silicon Valley JUG: JVM MechanicsSilicon Valley JUG: JVM Mechanics
Silicon Valley JUG: JVM Mechanics
Azul Systems, Inc.
 
Java EE 6 CDI Integrates with Spring & JSF
Java EE 6 CDI Integrates with Spring & JSFJava EE 6 CDI Integrates with Spring & JSF
Java EE 6 CDI Integrates with Spring & JSF
Jiayun Zhou
 
Step-by-step Development of an Application for the Java Card Connected Platform
Step-by-step Development of an Application for the Java Card Connected PlatformStep-by-step Development of an Application for the Java Card Connected Platform
Step-by-step Development of an Application for the Java Card Connected Platform
Eric Vétillard
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
Idan Gazit
 
Virtualizing Java in Java (jug.ru)
Virtualizing Java in Java (jug.ru)Virtualizing Java in Java (jug.ru)
Virtualizing Java in Java (jug.ru)
aragozin
 
Symfony2 from the Trenches
Symfony2 from the TrenchesSymfony2 from the Trenches
Symfony2 from the Trenches
Jonathan Wage
 
Symfony2 - from the trenches
Symfony2 - from the trenchesSymfony2 - from the trenches
Symfony2 - from the trenches
Lukas Smith
 
symfony on action - WebTech 207
symfony on action - WebTech 207symfony on action - WebTech 207
symfony on action - WebTech 207
patter
 
Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.
wellD
 
Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.
Michele Giacobazzi
 
JVM Mechanics: When Does the JVM JIT & Deoptimize?
JVM Mechanics: When Does the JVM JIT & Deoptimize?JVM Mechanics: When Does the JVM JIT & Deoptimize?
JVM Mechanics: When Does the JVM JIT & Deoptimize?
Doug Hawkins
 

Ähnlich wie Curator intro (20)

Native Java with GraalVM
Native Java with GraalVMNative Java with GraalVM
Native Java with GraalVM
 
(DEV204) Building High-Performance Native Cloud Apps In C++
(DEV204) Building High-Performance Native Cloud Apps In C++(DEV204) Building High-Performance Native Cloud Apps In C++
(DEV204) Building High-Performance Native Cloud Apps In C++
 
JavaOne 2017 - TestContainers: integration testing without the hassle
JavaOne 2017 - TestContainers: integration testing without the hassleJavaOne 2017 - TestContainers: integration testing without the hassle
JavaOne 2017 - TestContainers: integration testing without the hassle
 
Java Concurrency, Memory Model, and Trends
Java Concurrency, Memory Model, and TrendsJava Concurrency, Memory Model, and Trends
Java Concurrency, Memory Model, and Trends
 
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
 
Breaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit TestingBreaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit Testing
 
Performance Test Driven Development with Oracle Coherence
Performance Test Driven Development with Oracle CoherencePerformance Test Driven Development with Oracle Coherence
Performance Test Driven Development with Oracle Coherence
 
Construire une application JavaFX 8 avec gradle
Construire une application JavaFX 8 avec gradleConstruire une application JavaFX 8 avec gradle
Construire une application JavaFX 8 avec gradle
 
Ob1k presentation at Java.IL
Ob1k presentation at Java.ILOb1k presentation at Java.IL
Ob1k presentation at Java.IL
 
Silicon Valley JUG: JVM Mechanics
Silicon Valley JUG: JVM MechanicsSilicon Valley JUG: JVM Mechanics
Silicon Valley JUG: JVM Mechanics
 
Java EE 6 CDI Integrates with Spring & JSF
Java EE 6 CDI Integrates with Spring & JSFJava EE 6 CDI Integrates with Spring & JSF
Java EE 6 CDI Integrates with Spring & JSF
 
Step-by-step Development of an Application for the Java Card Connected Platform
Step-by-step Development of an Application for the Java Card Connected PlatformStep-by-step Development of an Application for the Java Card Connected Platform
Step-by-step Development of an Application for the Java Card Connected Platform
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
 
Virtualizing Java in Java (jug.ru)
Virtualizing Java in Java (jug.ru)Virtualizing Java in Java (jug.ru)
Virtualizing Java in Java (jug.ru)
 
Symfony2 from the Trenches
Symfony2 from the TrenchesSymfony2 from the Trenches
Symfony2 from the Trenches
 
Symfony2 - from the trenches
Symfony2 - from the trenchesSymfony2 - from the trenches
Symfony2 - from the trenches
 
symfony on action - WebTech 207
symfony on action - WebTech 207symfony on action - WebTech 207
symfony on action - WebTech 207
 
Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.
 
Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.Repetition is bad, repetition is bad.
Repetition is bad, repetition is bad.
 
JVM Mechanics: When Does the JVM JIT & Deoptimize?
JVM Mechanics: When Does the JVM JIT & Deoptimize?JVM Mechanics: When Does the JVM JIT & Deoptimize?
JVM Mechanics: When Does the JVM JIT & Deoptimize?
 

Kürzlich hochgeladen

Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
Priyanka Aash
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
Priyanka Aash
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
Priyanka Aash
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 

Kürzlich hochgeladen (20)

Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 

Curator intro

  • 1. Curator The Netflix ZooKeeper Client Library Jordan Zimmerman Senior Platform Engineer Netflix, Inc. jzimmerman@netflix.com @rangalt
  • 3. Agenda • Background • Overview of Curator • The Recipes • Some Low-Level Details • Q&A
  • 5. What’s wrong with this code? ZooKeeper client = new ZooKeeper(...); client.create(“/foo”, data, ...);
  • 7. ZooKeeper Surprise • Almost no ZK client call is safe • You cannot assume success • You must handle exceptions
  • 8. The Recipes Are Hard Locks Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock. These can be implemented using ZooKeeeper. As with priority queues, first define a lock node. Note There now exists a Lock implementation in ZooKeeper recipes directory. This is distributed with the release -- src/recipes/lock directory of the release artifact. Clients wishing to obtain a lock do the following: 1. Call create( ) with a pathname of "_locknode_/guid-lock-" and the sequence and ephemeral flags set. The guid is needed in case the create() result is missed. See the note below. 2. Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect). 3. If the pathname created in step 1 has the lowest sequence number suffix, the client has the lock and the client exits the protocol. 4. The client calls exists( ) with the watch flag set on the path in the lock directory with the next lowest sequence number. 5. if exists( ) returns false, go to step 2. Otherwise, wait for a notification for the pathname from the previous step before going to step 2. The unlock protocol is very simple: clients wishing to release a lock simply delete the node they created in step 1. Here are a few things to notice: • The removal of a node will only cause one client to wake up since each node is watched by exactly one client. In this way, you avoid the herd effect. • There is no polling or timeouts. • Because of the way you implement locking, it is easy to see the amount of lock contention, break locks, debug locking problems, etc. Recoverable Errors and the GUID • If a recoverable error occurs calling create() the client should call getChildren() and check for a node containing the guid used in the path name. This handles the case (noted above) of the create() succeeding on the server but the server crashing before returning the name of the new node.
  • 9. Even the Distribution Has Issues from org.apache.zookeeper.recipes.lock.WriteLock if (id == null) { long sessionId = zookeeper.getSessionId(); String prefix = "x-" + sessionId + "-"; // lets try look up the current ID if we failed // in the middle of creating the znode findPrefixInChildren(prefix, zookeeper, dir); idName = new ZNodeName(id); }
  • 10. Even the Distribution Has Issues from org.apache.zookeeper.recipes.lock.WriteLock if (id == null) { long sessionId = zookeeper.getSessionId(); String prefix = "x-" + sessionId + "-"; // lets try look up the current ID if we failed // in the middle of creating the znode findPrefixInChildren(prefix, zookeeper, dir); idName = new ZNodeName(id); } Bad handling of Ephemeral-Sequential issue!
  • 11. What About ZKClient? • Unclear if it’s still being supported Eleven open issues (back to 10/1/2009) • README: “+ TBD” • No docs • Little or no retries • Design problems: • All exceptions converted to RuntimeException • Recipes/management code highly coupled • Lots of foreground synchronization • Small number of tests • ... etc ... • ...
  • 15. Introducing Curator Curator n ˈkyo͝orˌātər: a keeper or custodian of a museum or other collection - A ZooKeeper Keeper Three components: Client - A replacement/wrapper for the bundled ZooKeeper class Framework - A high-level API that greatly simplifies using ZooKeeper Recipes - Implementations of some of the common ZooKeeper "recipes" built on top of the Curator Framework
  • 17. The Curator Stack • Client • Framework • Recipes • Extensions
  • 18. The Curator Stack • Client • Framework • Recipes Curator Recipes Curator Framework • Extensions Curator Client ZooKeeper
  • 20. Curator is a platform for writing ZooKeeper Recipes
  • 21. Curator Client manages the ZooKeeper Connection
  • 22. Curator Recipes Curator Framework Curator Client ZooKeeper Curator Client manages the ZooKeeper Connection
  • 23. Curator Framework uses retry for all operations and provides a friendlier API
  • 24. Curator Recipes Curator Framework Curator Client ZooKeeper Curator Framework uses retry for all operations and provides a friendlier API
  • 25. Curator Recipes: implementations of all recipes listed on the ZK website (and more)
  • 26. Curator Recipes Curator Framework Curator Client ZooKeeper Curator Recipes: implementations of all recipes listed on the ZK website (and more)
  • 28. The Recipes • Leader Selector • Distributed Locks • Queues • Barriers • Counters • Atomics • ...
  • 29. CuratorFramework Instance CuratorFrameworkFactory.newClient(...) --------------------- CuratorFrameworkFactory.builder() .connectString(“...”) ... .build() Usually injected as a singleton
  • 30. Must Be Started client.start(); // client is now ready for use
  • 31. Leader Selector By far the most common usage of ZooKeeper Distributed lock with a notification mechanism
  • 33. public class CleanupLeader implements LeaderSelectorListener { ... @Override public void takeLeadership(CuratorFramework client) throws Exception { while ( !Thread.currentThread().isInterrupted() ) { sleepUntilNextPeriod(); doPeriodicCleanup(); } } } ... LeaderSelector leaderSelector = new LeaderSelector(client, path, new CleanupLeader()); leaderSelector.start();
  • 34. Distributed Locks • InterProcessMutex • InterProcessReadWriteLock • InterProcessMultiLock • InterProcessSemaphore
  • 35. Distributed Locks • InterProcessMutex • InterProcessReadWriteLock • InterProcessMultiLock • InterProcessSemaphore Very similar to JDK locks
  • 37. InterProcessMutex mutex = new InterProcessMutex(client, lockPath); mutex.acquire(); try { // do work in critical section } finally { mutex.release(); }
  • 39. public void process(WatchedEvent event) { boolean wasConnected = isConnected.get(); boolean newIsConnected = wasConnected; if ( event.getType() == Watcher.Event.EventType.None ) { newIsConnected = (event.getState() == Event.KeeperState.SyncConnected); if ( event.getState() == Event.KeeperState.Expired ) { handleExpiredSession(); } } if ( newIsConnected != wasConnected ) { isConnected.set(newIsConnected); connectionStartMs = System.currentTimeMillis(); } ... }
  • 40. public static boolean shouldRetry(int rc) { return (rc == KeeperException.Code.CONNECTIONLOSS.intValue()) || (rc == KeeperException.Code.OPERATIONTIMEOUT.intValue()) || (rc == KeeperException.Code.SESSIONMOVED.intValue()) || (rc == KeeperException.Code.SESSIONEXPIRED.intValue()); } public void takeException(Exception exception) throws Exception { boolean rethrow = true; if ( isRetryException(exception) ) { if ( retryPolicy.allowRetry(retryCount++, System.currentTimeMillis() - startTimeMs) ) { rethrow = false; } } if ( rethrow ) { throw exception; } }
  • 41. byte[] responseData = RetryLoop.callWithRetry ( client.getZookeeperClient(), new Callable<byte[]>() { @Override public byte[] call() throws Exception { byte[] responseData; responseData = client.getZooKeeper().getData(path, ...); } return responseData; } } ); return responseData;
  • 43. final AtomicBoolean firstTime = new AtomicBoolean(true); String returnPath = RetryLoop.callWithRetry ( client.getZookeeperClient(), new Callable<String>() { @Override public String call() throws Exception { ... String createdPath = null; if ( !firstTime.get() && doProtectedEphemeralSequential ) { createdPath = findProtectedNodeInForeground(localPath); } ... } } );
  • 44. public interface ConnectionStateListener { public void stateChanged(CuratorFramework client, ConnectionState newState); } public enum ConnectionState { SUSPENDED, RECONNECTED, LOST }
  • 45. if ( e instanceof KeeperException.ConnectionLossException ) { connectionStateManager.addStateChange(ConnectionState.LOST); } private void validateConnection(CuratorEvent curatorEvent) { if ( curatorEvent.getType() == CuratorEventType.WATCHED ) { if ( curatorEvent.getWatchedEvent().getState() == Watcher.Event.KeeperState.Disconnected ) { connectionStateManager.addStateChange(ConnectionState.SUSPENDED); internalSync(this, "/", null); } else if ( curatorEvent.getWatchedEvent().getState() == Watcher.Event.KeeperState.Expired ) { connectionStateManager.addStateChange(ConnectionState.LOST); } else if ( curatorEvent.getWatchedEvent().getState() == Watcher.Event.KeeperState.SyncConnected ) { connectionStateManager.addStateChange(ConnectionState.RECONNECTED); } } }
  • 47. • TestingServer: manages an internally running ZooKeeper server // Create the server using a random port public TestingServer() • TestingCluster: manages an internally running ensemble of ZooKeeper servers. // Creates an ensemble comprised of n servers. // Each server will use a temp directory and // random ports public TestingCluster(int instanceQty)
  • 48. Extensions • Discovery • Discovery REST Server • Exhibitor • ???
  • 49. Extensions • Discovery • Discovery REST Server • Exhibitor Curator Recipes • ??? Curator Framework Extensions Curator Client ZooKeeper
  • 54. Exhibitor Sneak Peak March or April 2012 Open Source on Github
  • 60. Maven Central Binaries pushed to Maven Central <dependency> <groupId>com.netflix.curator</groupId> <artifactId>curator-recipes</artifactId> <version>1.1.0</version> </dependency>
  • 61. Much%younger%–%much%thinner0 Jordan Zimmerman jzimmerman@netflix.com @randgalt
  • 62. Q&A Much%younger%–%much%thinner0 Jordan Zimmerman jzimmerman@netflix.com @randgalt

Hinweis der Redaktion

  1. \n
  2. * Background - ZK issues, the need for a wrapper, etc. - mention that you can go more in depth on this\n* Why Curator was written, etc.\n* Low-level - details of the client/framework. Error handling, assumptions, etc.\n* Mention that this will be very technical - lots of code\n
  3. * Background - ZK issues, the need for a wrapper, etc. - mention that you can go more in depth on this\n* Why Curator was written, etc.\n* Low-level - details of the client/framework. Error handling, assumptions, etc.\n* Mention that this will be very technical - lots of code\n
  4. * Background - ZK issues, the need for a wrapper, etc. - mention that you can go more in depth on this\n* Why Curator was written, etc.\n* Low-level - details of the client/framework. Error handling, assumptions, etc.\n* Mention that this will be very technical - lots of code\n
  5. * Background - ZK issues, the need for a wrapper, etc. - mention that you can go more in depth on this\n* Why Curator was written, etc.\n* Low-level - details of the client/framework. Error handling, assumptions, etc.\n* Mention that this will be very technical - lots of code\n
  6. * Background - ZK issues, the need for a wrapper, etc. - mention that you can go more in depth on this\n* Why Curator was written, etc.\n* Low-level - details of the client/framework. Error handling, assumptions, etc.\n* Mention that this will be very technical - lots of code\n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. Mention that you contributed part on recoverable errors\n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. Becomes a persistent, unchanging handle to the ZK ensemble\n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. Kishore Gopalakrishna from Linked-in\n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n