Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Zeromq anatomy & jeromq

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 19 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Zeromq anatomy & jeromq (20)

Anzeige

Aktuellste (20)

Zeromq anatomy & jeromq

  1. 1. ZeroMQ Anatomy & JeroMQ
  2. 2. ZeroMQ MQ 가 아니무니다.. TCP on Steroids
  3. 3. ZMQ Socket ROUTER DEALER XPUB XSUB PUSH PULLREP REQ PUB SUB
  4. 4. ZeroMQ Anatomy R w
  5. 5. io_thread_t ZMQ Socket Mailbox Signaler ypipe_t <cmd_t> Session Socket Session Socket poller r fd fd ypipe_t <msg_t> inpipe ypipe_t <msg_t> outpipe ZMQ Socket ZMQ Socket ZMQ Socket ZMQ Socket SessionSessionSession Context Mailbox r
  6. 6. Msg union { ... struct { unsigned char data [29]; unsigned char size; unsigned char type; unsigned char flags; } vsm; struct { content_t *content; unsigned char unused [29 + 1 - sizeof (content_t*)]; unsigned char type; unsigned char flags; } lmsg; DATA S T F T F S DATA vsm lmsg
  7. 7. writer thread reader thread YQueue yqueue_t <msg_t, 256> yqueue_t <command_t, 16> struct chunk_t { T values [N]; chunk_t *prev; chunk_t *next; }; atomic_ptr_t <chunk_t> spare_chunk ... begin back • Only single thread can read from begin_chunk • Only single thread can write to yqueue_t <msg_t> to back chunk • Only single thread can write to yqueue_t <command_t> at a time
  8. 8. YPipe atomic_ptr_t <T> c Reader Writer T* r T* w T* f bool flush () { if ( c.cas ( w, f ) != w ) { c.set ( f ) ; w = f ; return false ; // reader is sleeping } w = f ; return true ; } bool check_read () { if ( queue.front () != r ) return true ; r = c.cas ( queue.front () , NULL ) ; if ( queue.front () == r ) return false ; // nothing to read return true; } read() => queue<T>.pop write() => queue<T>.push r w f
  9. 9. Signaler socketpair ( AF_UNIX, SOCK_STREAM, 0, sv[2] ) int w = sv[0]; // writer fd int r = sv[1]; // reader fd, Used at IOThread.poll Reader recv () { :: recv ( r , dummy[1] ); } wait ( timeout ) { poll ( r , timeout ); } Writer send () { :: send ( w, dummy[1] ); }
  10. 10. Mailbox ypipe_t <command_t, 16> cpipe signaler_t signaler Reader Writer recv (timeout) { signaler.wait (timeout) return cpipe.read () } send (cmd) { lock () cpipe.write (cmd) cpipe.flush () unlock () signaler.send () }
  11. 11. ZMQ Socket ZMQ Socket mailbox_t signaler_ typipe_t <cmd_t> Session Socket Session Socket r fd fd ypipe_t <msg_t> inpipe ypipe_t <msg_t> outpipe • Session is create when connect / accept •Worker thread owns ZMQ socket • IOThread owns Sessions • recv msg from socket and en-queue inpipe • recv msg from outpipe and send to socket • zsocket.send en-queue to outpipe • zsocket.recv de-queue from inpipe if any
  12. 12. main/worker IOThread io_thread ZMQ Socket mailbox Session Socket poller ZMQ Socket ZMQ Socket ZMQ Socket ZMQ Socket Session mailbox zsocket.write(msg) outpipe.write(msg) activate_read session.read_activated engine.activate_out engine.out_event SessionSessionSession Stream Engine engine.in_event outpipe.write(msg) activate_read zsocket.read_activated msg = inpipe.read
  13. 13. in_event () { // read from socket size = 8192 // *** buffer = decoder . get_buffer (size) ::recv (buffer, size) decoder . process_buffer(buffer) session . flush() } out_event () { // write to socket size = 8192 // *** buffer = encoder . get_data (size) ::send (buffer, size) } msg = parse (buffer) session.write (msg) Stream Engine msg = session.read () buffer.fill (msg)
  14. 14. Summary inline lock-free well-defined threading model less system calls • malloc / free • bulk socket recv / send • balance copy overhead
  15. 15. JeroMQ - jeromq.org Hand-made Compatible with ZeroMQ-3 tcp:// inproc:// ipc:// pgm:// same dev experience 95%~ performance
  16. 16. JeroMQ - diff Java NIO SelectableChannel (Direct) ByteBuffer Signaler Pipe.SourceChannel / Pipe.SinkChannel Mixed with counter YQueue - circular linked list Plain Socket Proxy Persistence - Kafka alike MappedByteBuffer ZeroCopy
  17. 17. Future Zero Persistence Queue http://github.com/miniway/zper Faster than Kafka 50% Less code Configurable topology Java 7 - AsynchronousChannel
  18. 18. Conclusion Why reinvent the wheel “The only limitation is your imagination and sobriety”
  19. 19. Q/A

×