Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 71 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie ZeroMQ with NodeJS (20)

Aktuellste (20)

Anzeige

ZeroMQ with NodeJS

  1. 1. ØMQ @ NodeJS ØMQ: Small Pieces for Scalable Software Fernando D. Alonso
  2. 2. 1. Facts about ØMQ [ ]
  3. 3. Facts about ØMQ Low level messaging library ○ It is written in C++. ○ A thin layer between application and transport layers. ○ Message-passing using in-memory queues. ○ Concise API which talks over the ZMTP protocol. ○ Works on top of TCP, IPC, in-memory, and PGM/EPGM communication protocols.
  4. 4. ○ Philosophy: horizontal scalability to reduce points of failure. ○ Network topology aware. Point to point communication reduce latencies. ○ Communication is reduced to a few patterns, which combined are powerful. ○ No need to have a multitenant service. Facts about ØMQ A broker is not mandatory
  5. 5. Facts about ØMQ Messaged oriented communication ○ Messages as first-class citizen. No need to fight with framing or buffering. ○ You can send multipart messages. ○ Messages are atomic. ○ Asynchronous messaging. A single background I/O thread does the job for ØMQ sockets. "A ØMQ message is a discrete unit of data passed between applications or components of the same application. ØMQ messages have no internal structure and from the point of view of ØMQ itself they are considered to be opaque binary data."
  6. 6. Facts about ØMQ Fast for development ○ Interchangeable transports: e.g, scaling from one IPC server to a bunch of TCP. ○ Automatic reconnections. ○ Multicore made easy. ○ Provides zmq_poll to read BSD sockets.
  7. 7. ○ It doesn't provide message persistence. ○ It doesn't provide data serialization. ○ It doesn't provide data compression. ○ It doesn't provide message encryption. ○ It doesn't provide security mechanisms. Not in the next version of the ZMTP protocol ! (*) ZTMP 3.0 draft protocol: http://hintjens.com/blog:39 Facts about ØMQ Aims for simplicity
  8. 8. Facts about ØMQ Platforms & Languages ○ Bindings for 30+ languages including NodeJS, Ruby, Python, Java, C, C++, Scala, Erlang, Perl, PHP, .NET. ○ Linux & Windows. ○ Raspberry PI. ○ Android OS.
  9. 9. 2. Install [ , ]
  10. 10. Install On Linux from source sudo apt-get install build-essential libtool autoconf automake uuid-dev wget http://download.zeromq.org/zeromq-3.2.3.tar.gz tar xvzf zeromq-3.2.3.tar.gz ./configure make sudo make install sudo ldconfig NOTES: (*) Check http://download.zeromq.org/ for more versions. (**) Avoid 3.0.x, 3.1.x, 3.2.0, 3.2.1 versions. They lack backwards compatibility. (***) Check ./configure --help for more options on build. (****) ./configure --with-pgm
  11. 11. Install On Linux using packages Libzmq 3.2: ● Debian Sid: sudo apt-get install libzmq3-dev Libzmq 2.2: ● Debian Wheezy, Jessie, Sid: sudo apt-get install libzmq-dev ● Debian Squeeze backport: http://packages.debian.org/squeeze- backports/libzmq1
  12. 12. Install On Mac brew install zmq
  13. 13. Install NodeJS bindings NPM Packages: ● zmq (https://github.com/JustinTulloss/zeromq.node) [RECOMMENDED] ● zmq-stream (https://github.com/Schoonology/zmq-stream) npm install zmq node > var zmq = require('zmq');
  14. 14. Install Bindings in other languages Ruby: https://github.com/chuckremes/ffi-rzmq [RECOMMENDED] Python: https://github.com/zeromq/pyzmq Erlang: https://github.com/zeromq/erlzmq2 Java JZMQ (uses JNI): https://github.com/zeromq/jzmq Java JeroMQ: https://github.com/zeromq/jeromq Scala ZeroMQ: https://github.com/mDialog/scala-zeromq Scala ZeroMQ (official): https://github.com/valotrading/zeromq-scala-binding ...
  15. 15. 3. Sockets
  16. 16. Sockets Creation (*) http://api.zeromq.org/3-2:zmq-socket ØMQ C API NodeJS ZMQ req ZMQ_REQ rep ZMQ_REP dealer ZMQ_DEALER router ZMQ_ROUTER push ZMQ_PUSH pull ZMQ_PULL ØMQ C APINodeJS ZMQ pub ZMQ_PUB sub ZMQ_SUB xsub ZMQ_XSUB xpub ZMQ_XPUB var sock = zmq.socket ("<SOCKET_TYPE>"); zmq_socket(context, <SOCKET_TYPE>); Request Reply Dealer (async Request) Router (async Reply) Push Pull Publisher Subscriber XSubscriber XPublisher
  17. 17. Sockets Bind & Connect sock.bind("tcp://10.0.0.1:5555", function(err) { .. }); sock.connect("tcp://10.0.0.1:5555"); sock.close(); (*) http://api.zeromq.org/3-2:zmq-bind (**) http://api.zeromq.org/3-2:zmq-connect (***) http://api.zeromq.org/3-2:zmq-close ØMQ C API zmq_bind(socket, "tcp://10.0.0.1:5555"); zmq_connect(socket, "tcp://10.0.0.1: 5555"); zmq_close(socket); NodeJS ZMQ Bind Connect Close
  18. 18. Sockets Bind & Connect sock.bindSync("tcp://10.0.0.1:5555"); // This line gets executed once the socket // is binded, but it breaks on error. NodeJS ZMQ BindSync
  19. 19. Sockets Transports TCP ● usage: tcp://<address>:<port> sock.bind("tcp://192.168.1.100:5555"); sock.bind("tcp://eth0:5555"); // binds to the first eth0 interface sock.bind("tcp://*:5555"); // binds to all interfaces (*) http://api.zeromq.org/3-2:zmq-tcp ØMQ C API zmq_bind(socket, "tcp://192.168.1.100: 5555"); zmq_bind(socket, "tcp://eth0:5555"); zmq_bind(socket, "tcp://*:5555"); NodeJS ZMQ
  20. 20. Sockets Transports Inter-Process Communication (IPC) ● usage: ipc://<address> ● Requires RW permissions on the specified path. (*) http://api.zeromq.org/3-2:zmq-ipc sock.bind("ipc:///tmp/mysocket"); ØMQ C API zmq_bind(socket, "ipc:///tmp/mysocket"); NodeJS ZMQ
  21. 21. Sockets Transports In-process Communication (in-memory) ● usage: inproc://<address> ● Requires to bind before connect. ● Max. address name length is 256. (*) http://api.zeromq.org/3-2:zmq-inproc (**) Buggy on node-zmq package. sock.bind("inproc://queue"); ØMQ C API zmq_bind(socket, "ipc://queue"); NodeJS ZMQ
  22. 22. Sockets Transports PGM (multicast) ● usage: pgm://<address>;<multicast_address>:<port> ● Requires sudo privileges to access raw IP sockets. ● Needs ØMQ built with PGM extension (./configure --with-pgm). (*) http://api.zeromq.org/3-2:zmq-pgm sock.bind( "pgm://192.168.1.100;239.192.1.1:3055" ); ØMQ C API zmq_bind(socket, "pgm://192.168.1.100; 239.192.1.1:3055"); NodeJS ZMQ
  23. 23. Sockets Transports Encapsulated PGM (multicast) ● usage: epgm://<address>;<multicast_address>:<port> ● Needs ØMQ built with PGM extension (./configure --with-pgm). (*) http://api.zeromq.org/3-2:zmq-pgm sock.bind( "epgm://192.168.1.100;239.192.1.1:3055" ); ØMQ C API zmq_bind(socket, "epgm://192.168.1.100; 239.192.1.1:3055"); NodeJS ZMQ
  24. 24. Sockets Send & Receive Send sock.send("My message"); sock.on("message", function(msg) { console.log(msg.toString()); }); (*) http://api.zeromq.org/3-2:zmq-send (**) http://api.zeromq.org/3-2:zmq-recv Receive sock.on("message", function(msg) { console.log("Received " + msg); }); // <msg> is a Buffer object.
  25. 25. Sockets Multipart messages ØMQ Queue zmq_send(socket, "Como", 4, ZMQ_SNDMORE); Messages are atomic A ØMQ message is composed of 1 or more message parts. zmq_send(socket, "andas", 5, 0); 4 ComoPart 1 5 andasPart 2 ØMQ ensures that peers receive either all message parts of a message or none at all. The total number of message parts is unlimited except by available memory.
  26. 26. Sockets Multipart messages sock.send("First part", zmq.ZMQ_SNDMORE); sock.send("Second part"); sock.send(["First part", "Second part"]); Send var r = sock.send("Como", zmq. ZMQ_SNDMORE); console.log(r); Check the outgoing buffer and socket state: running this will output -> > { type: 'req', _zmq: { state: 0, onReady: [Function] }, _outgoing: [ [ <Buffer 43 6f 6d 6f>, 2 ] ], _shouldFlush: true, _events: { message: [Function] } }
  27. 27. Sockets Multipart messages sock.on("message", function(first_part, second_part) { console.log(first_part.toString()); console.log(second_part.toString()); }); sock.on("message", function() { for(var key in arguments) { console.log("Part " + key + ": " + arguments[key]); }; }); Receive sock.on("message", function() { var messages = Array.prototype.slice.call(arguments); messages.forEach(function(msg) { console.log("Part:" + msg); }); });
  28. 28. Sockets Batched messaging ØMQ Queue 3 Msg "ØMQ batches messages in opportunistic manner. It sends all the messages available at the moment in one go. Latency of subsequent messages will be improved because sending single batch to the card is faster then sending lot of small messages." sock.send("Part A", zmq.ZMQ_SNDMORE); sock.send("Msg"); sock.send("Part B"); 6 Part A 6 Part B ØMQ Socket send send send <write busy> <write ready> Client Thread ØMQ I/O Thread send batch <write busy>
  29. 29. Patterns Plugging multiple transports Example var zmq = require('zmq'), pub = zmq.socket('pub'); pub.bindSync('tcp://127.0.0.1:5555'); pub.bindSync('ipc:///tmp/zmq.sock'); setInterval(function() { pub.send("I am polyglot!"); }, 1000); sub = zmq.socket('sub'); sub.connect('ipc:///tmp/zmq.sock'); sub.subscribe(''); sub.on('message', function(msg) { console.log("Received: " + msg); }); var zmq = require('zmq'), sub = zmq.socket(sub'); sub.connect('tcp://127.0.0.1:5555'); sub.subscribe(''); sub.on('message', function(msg) { console.log(Received: ' + msg); });
  30. 30. Sockets Setting Socket Options (*) http://api.zeromq.org/3-2:zmq-setsockopt Setsockopt ● usage (NodeJS ZMQ API): sock.setsockopt(<option>, <value>); or sock.<option> = <value>; sock.identity = 'monitor-1'; sock.setsockopt('identity', 'monitor-1'); Identity: // value can be any string up to 255 length. // identity is required on sockets // connecting to a Router.
  31. 31. Sockets Setting Socket Options Setsockopt subscriber.subscribe('MUSIC'); subscriber.setsockopt('subscribe', 'MUSIC'); Subscribe: // sets a message filter for subscriber sockets. subscriber.unsubscribe('MUSIC'); susbscriber.setsockopt('subscribe', 'POP'); Unsubscribe: // sets off a message filter for subscriber sockets.
  32. 32. 4. Patterns [ ]
  33. 33. Patterns Request / Reply Synchronous task distribution Ø REP #1 Ø REP # 2 Ø REQ < round robin > ... < synchronous > Used when each message needs to be matched with a response. Handles only one message at time. Strict send-recv cycle.
  34. 34. Patterns Request / Reply Ø REP #1 Ø REP # 2Ø REQ send recv (wait response) send recv (dont wait response) recv (dont wait response)
  35. 35. Patterns Request / Reply Ø REP #1 Ø REP # 2Ø REQ send recv recv send send -> REP #1 is down FAILURE REQ #1 will not recover !
  36. 36. Patterns Request / Reply Basic Request / Reply var zmq = require('zmq'); reply = zmq.socket('rep'); reply.bind('tcp://127.0.0.1:5555', function(err) { if (err) throw err; } ); reply.on('message', function(msg) { console.log('Received: ' + msg); reply.send('Pong!'); }); var zmq = require('zmq'); request = zmq.socket('req'); request.connect('tcp://127.0.0.1:5555'); request.send('Ping'); request.on('message', function(msg) { console.log('Response: ' + msg); });
  37. 37. Patterns Request / Reply Dealer socket message delivery Messages have to be multipart, consisting on: an empty delimiter header, followed by the content body parts. sock.send("Body"); Outgoing Queue 0 6 Body sock.send(new Buffer([]), zmq. ZMQ_SNDMORE); send send
  38. 38. Patterns Request / Reply Dealer socket message delivery Outgoing messages are round-robined among all connected peers. However, sending depends on the availability of the receiver. sock.send( ["", "Msg"]); Outgoing Queue 3 Msg 6 Part A Peer Socket A Peer Socket B sock.send( ["", "Bla"); sock.send( ["", "Msg2"]); send send send To A To B To A send <write ready> <write busy> <write ready> 0 0 4 Msg2 0
  39. 39. Patterns Request / Reply Dealer socket message delivery Reconnections are handled automatically: messages are asynchronously delivered. Response time does not block sending more messages. Outgoing Queue Peer Socket A Peer Socket B send -> Up again sock.send( ["", "Msg"]); 3 Msg sock.send( ["", "Msg2"]); send send To A To A 0 4 Msg2 0
  40. 40. Patterns Request / Reply Ø REP #1 Ø REP # 2Ø DEALER enqueue < Msg A > send < Msg B > -> REP #1 is down enqueue < Msg C > send < Msg A, Msg C > send < Msg D > Dealer socket handles peer reconnection automatically. It will send messages that queued for that peer once it has established connection.
  41. 41. Patterns Request / Reply > Response from: 5001 > Response from: 5001 > Response from: 5002 > Response from: 5001 > Response from: 5001 > Response from: 5002 > ... Dealer example var dealer = zmq.socket('dealer'); dealer.connect('tcp://127.0.0.1:5001'); dealer.connect('tcp://127.0.0.1:5002'); setInterval(function() { dealer.send(["", "Msg"]); }, 1000); dealer.on('message', function(h, msg) { console.log('Response from: ' + msg); }); running this 'might' output ->
  42. 42. Patterns Request / Reply Router socket messaging Messages have to be multipart, consisting on: an identity frame, an empty delimiter frame, followed by the content body parts. This applies to both incoming and outgoing messages. sock.send("Body"); Outgoing Queue 0 6 Body sock.send("", zmq. ZMQ_SNDMORE); send send sock.send( "worker-1", zmq. ZMQ_SNDMORE); 8 worker-1 send
  43. 43. Patterns Request / Reply Router is asynchronous Incoming messages are fair-queued among all connected and available peers. Peer Socket A Peer Socket B ØMQ Router Incoming Queue Socketsend send send
  44. 44. Patterns Request / Reply Router is asynchronous Receiving depends on the availability of the sender. Peer Socket A Peer Socket B ØMQ Router Incoming Queue Socketsend send
  45. 45. Patterns Push / Pull Unidirectional data distribution The Push / Pull pattern fits well on pipelined communication, i.e., when no response is required for a given message. Ø Ø ... ØØ
  46. 46. Patterns Push / Pull Binding the Push socket Outgoing messages are round-robined among all connected and available peers. push.send("Msg"); push.send("Part B"); Outgoing Queue Peer Socket A Peer Socket B push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Msg2"); 3 Msg 6 Part A 6 Part B 4 Msg2 send send send send
  47. 47. Patterns Push / Pull Binding the Push socket Outgoing messages are round-robined among all connected and available peers. Outgoing Queue Peer Socket A Peer Socket B -> Down3 Msg 6 Part A 6 Part B 4 Msg2 send send send push.send("Msg"); push.send("Part B"); push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Msg2"); send send send send
  48. 48. Patterns Push / Pull Binding the Push socket Disconnections and reconnections are handled automatically. Outgoing Queue Peer Socket A Peer Socket B 4 More 6 Part 1 6 Part 2 5 Other send send send push.send("More"); send push.send("Part 2"); push.send("Part 1", zmq.ZMQ_SNDMORE); send send push.send("Other"); send
  49. 49. Patterns Push / Pull Parallel task distribution Ø PULL #1 Ø PULL # 2 Ø PUSH < round robin > ...
  50. 50. Patterns Push / Pull Passive agents / Monitor Ø PULLØ PUSH ... Ø PUSH
  51. 51. Patterns Push / Pull Binding the Pull socket Push's outgoing messages are round-robined among all connected peers. push.send("Msg"); push.send("Part B"); Outgoing Queue Peer Socket A Peer Socket B push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Msg2"); send send send send 3 Msg 6 Part A 6 Part B 4 Msg2 To A To B To A
  52. 52. Patterns Push / Pull Binding the Pull socket Push's outgoing messages are round-robined among all connected peers. push.send("Msg"); push.send("Part B"); Outgoing Queue Peer Socket A Peer Socket B push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Msg2"); send send send send send send send 3 Msg 6 Part A 6 Part B 4 Msg2 To A To B To A
  53. 53. Patterns Push / Pull Binding the Pull socket Disconnections and reconnections are handled automatically. Outgoing Queue Peer Socket A Peer Socket B send send -> Down4 More 6 Part 1 6 Part 2 5 Other To A To A To B push.send("More"); send push.send("Part 2"); push.send("Part 1", zmq.ZMQ_SNDMORE); send send push.send("Other"); send
  54. 54. Patterns Push / Pull Binding the Pull socket Sending is asynchronous, it depends on the availability of the receiver. Outgoing Queue Peer Socket A Peer Socket B send -> Up again 5 OtherTo B push.send("More"); send push.send("Part 2"); push.send("Part 1", zmq.ZMQ_SNDMORE); send send push.send("Other"); send
  55. 55. Patterns Push / Pull Ø PULLØ PUSH #1 ... Ø PUSH #2 Active agents / collector pattern
  56. 56. Patterns Publish / Subscribe Broadcast data distribution Ø SUB #1 Ø PUB #1 ... Ø SUB #2 Ø SUB #2
  57. 57. Message distribution Patterns Publish / Subscribe sock.send("Msg"); Outgoing Queue 3 Msg Subscriber A Subscriber B send send Outgoing messages are send to each connected and available peers. Subscriber C
  58. 58. Message distribution Patterns Publish / Subscribe sock.send("Msg"); Outgoing Queue 3 Msg Subscriber A Subscriber B send send Outgoing messages are send to each connected and available peers. Subscriber C 5 Othersock.send("Other"); send
  59. 59. Subscribers can register to a specific topic. Data filtering Patterns Publish / Subscribe ØMQ Publisher Subscriber A Subscriber B Subscriber C Outgoing Queue subscribe AR subscribe VE Socket subscribe VE
  60. 60. Subscribers can register to a specific topic. Data filtering Patterns Publish / Subscribe ØMQ Publisher Subscriber A Subscriber B Subscriber C Outgoing Queue subscribe AR subscribe VE Socket subscribe VE sock.send("AR news"); send 7 AR news send
  61. 61. Subscribers can register to a specific topic. Data filtering Patterns Publish / Subscribe ØMQ Publisher Subscriber A Subscriber B Subscriber C Outgoing Queue subscribe AR subscribe VE Socket subscribe VE sock.send("VE news"); send 7 VE news sendsend
  62. 62. Patterns Publish / Subscribe Data filtering example pub = zmq.socket('pub'); pub.bindSync("tcp://10.0.0.12:3055"); count = 0 setInterval(function() { pub.send("TEST " + count++); }, 1000); sub = zmq.socket('sub'); sub.connect("tcp://10.0.0.12:3055"); sub.subscribe("TEST"); sub.on("message", function(msg) { console.log("Received: " + msg); }); // older messages won't be // received > TEST 6 > TEST 7 > TEST 8 > TEST 9 > TEST 10 running this 'might' output ->
  63. 63. Patterns Publish / Subscribe Multicast example var zmq = require('zmq'), pub = zmq.socket('pub'); pub.bind( "epgm://10.0.0.12;239.192.1.1:3055", function(err) { if(err) throw err; } ) setInterval(function() { pub.send("From pid: " + process.pid); }, 1000); var zmq = require('zmq'), sub = zmq.socket('sub'); sub.connect( "epgm://10.0.0.13;239.192.1.1:3055" ); sub.subscribe(""); sub.on('message', function(msg) { console.log("Received " + msg); }); (*) See running demo at http://youtu.be/NQrH0SATPk0
  64. 64. "XSUB and XPUB are exactly like SUB and PUB except they expose subscriptions as special messages." Data distribution proxy Patterns XSUB / XPUB XSUB/XPUB Proxy Subscriber A Subscriber B Subscriber C XPUB Socket XSUB Socket Publisher A Publisher B Publisher C subscribe PY forward subscription
  65. 65. Messages are forwarded by the proxy to the registered subscribers. Data distribution proxy Patterns XSUB / XPUB XSUB/XPUB Proxy Subscriber A Subscriber B Subscriber C XPUB Socket XSUB Socket Publisher A Publisher B Publisher C forward message send "PY<msg>" send
  66. 66. 5. Useful links [ ]
  67. 67. Useful Links Workshop examples http://github.com/krakatoa/node_zmq_workshop
  68. 68. Useful Links The Guide http://zguide.zeromq.org/page:all
  69. 69. Useful Links Pieter Hintjens personal blog http://hintjens.com/
  70. 70. Useful Links API References http://api.zeromq.org/3-2:zmq-socket http://api.zeromq.org/3-2:zmq-setsockopt
  71. 71. Preguntas ? fedario@gmail.com github.com/krakatoa Fernando Dario Alonso Gracias !

×