Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Distributed app development node.js + zeroMQ = awwyeah
ZeroMQ from a noob’s     viewpoint
RUBEN TAN• NodeJS enthusiast• Author of eazy  (https://github.com/soggie/eazy)• Dota 2-er
DISCLAIMER• I AM NOT A ZEROMQ EXPERT• (not even remotely close)• Case study by consulting a successful  implementer of ZMQ...
Objective• 1,000,000,000 ad impressions per month  (on average)• 80ms maximum response time for each ad• Good luck, Apache
Webserver Choices• Netty - Java can actually be fast  https://netty.io• Gevent - For Python lovers  http://gevent.org/• No...
So... which one?• All 3, but that’s not the point• Speed isn’t the most important factor• Reliability under load is more i...
Welcome to thedistributed app   ecosystem
Definitions, definitions• Distributed: distributed across multiple  geological “clouds”• Apps: each app handles a small scop...
Distributed• Route 53 -> HAProxy -> Webserver ->  Apps• Scale webservers by creating new instances  behind HAProxy• Scale ...
Distributed                   Route 53 (DNS)                  HAProxy “LB layer”Webserver            Webserver            ...
Apps• Each app should perform ONE single  domain of function• Write the app like you’re writing an API• The simpler the ap...
User app• List of exposed API:  •   create new user  •   edit user  •   delete user  •   merge user accounts  •   authenti...
Ecosystem• Apps communicate using JSON-RPC• Apps connect to each other using ZMQ• “Controller” scripts and monit handles  ...
Quick & dirty ZeroMQ• ZMQ = socket library• Types of socket: •   inproc •   IPC •   TCP •   pgm •   epgm
Socket Types• Push-pull• Req-rep• Pub-sub (not covered)• Dealer-router (not covered)• Xreq-xrep (not covered)
Push-pull• Push socket pushes data to pull socket(s)• Pull socket pulls data from push socket(s)       Push               ...
Push-pull• Use cases: •   Sending email: apps pushes email data to email app •   Logging: apps pushes messages to a loggin...
Push-pull• Push socket can connect to multiple pull  sockets (fan-out)• Pull sockets can listen on multiple push  sockets ...
Advanced Push-Pull• Example: Log file processing• Log file needs to be parsed• Push app (ventilator) parses the log file• Pus...
Advanced Push-pull                           Worker                           Worker             push                   pu...
How do we scale?
ADD MOREWORKERS
Req-rep• Req socket sends data to rep socket• Rep socket receives data from req socket• Rep socket replies to req socket• ...
Req-rep• Use cases: •   Inter-app communications: apps that has dependencies between     each other uses a REQ-REP pattern...
Req-rep REQ                   REPSocket                Socket          req REQ                   REPSocket                ...
How to scale?
ADD MORE REP APPS
All together now...
Sample ZMQ code: 01   var zmq = require(zmq), 02     push = zmq.socket(push), 03     pullA = zmq.socket(pull), 04     pull...
Notes• Each app can define and use multiple ZMQ  sockets• You can organize your app’s internals using  inproc ZMQ sockets (...
Libraries• NodeJS ZMQ binding  (https://github.com/JustinTulloss/zeromq.node)• ZMQ Guide  (http://zguide.zeromq.org/page:a...
- the end -
Nächste SlideShare
Wird geladen in …5
×

Distributed app development with nodejs and zeromq

16.977 Aufrufe

Veröffentlicht am

Veröffentlicht in: Technologie, Bildung
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Distributed app development with nodejs and zeromq

  1. 1. Distributed app development node.js + zeroMQ = awwyeah
  2. 2. ZeroMQ from a noob’s viewpoint
  3. 3. RUBEN TAN• NodeJS enthusiast• Author of eazy (https://github.com/soggie/eazy)• Dota 2-er
  4. 4. DISCLAIMER• I AM NOT A ZEROMQ EXPERT• (not even remotely close)• Case study by consulting a successful implementer of ZMQ in an adserver
  5. 5. Objective• 1,000,000,000 ad impressions per month (on average)• 80ms maximum response time for each ad• Good luck, Apache
  6. 6. Webserver Choices• Netty - Java can actually be fast https://netty.io• Gevent - For Python lovers http://gevent.org/• Node.js - The reason why you’re here http://nodejs.org/
  7. 7. So... which one?• All 3, but that’s not the point• Speed isn’t the most important factor• Reliability under load is more important than pure performance• 1 missed impression = 1 potential disaster (imagine BMW ads showing up in ThePirateBay)• (but secretly, I prefer node.js)
  8. 8. Welcome to thedistributed app ecosystem
  9. 9. Definitions, definitions• Distributed: distributed across multiple geological “clouds”• Apps: each app handles a small scope of responsibility• Ecosystem: apps talk to apps via a standardized protocol
  10. 10. Distributed• Route 53 -> HAProxy -> Webserver -> Apps• Scale webservers by creating new instances behind HAProxy• Scale apps by creating new instances in their own clouds
  11. 11. Distributed Route 53 (DNS) HAProxy “LB layer”Webserver Webserver Webserver App cloud App cloud (EU) (NA)
  12. 12. Apps• Each app should perform ONE single domain of function• Write the app like you’re writing an API• The simpler the app, the better• App can be written in any language
  13. 13. User app• List of exposed API: • create new user • edit user • delete user • merge user accounts • authenticate user
  14. 14. Ecosystem• Apps communicate using JSON-RPC• Apps connect to each other using ZMQ• “Controller” scripts and monit handles lifecycle
  15. 15. Quick & dirty ZeroMQ• ZMQ = socket library• Types of socket: • inproc • IPC • TCP • pgm • epgm
  16. 16. Socket Types• Push-pull• Req-rep• Pub-sub (not covered)• Dealer-router (not covered)• Xreq-xrep (not covered)
  17. 17. Push-pull• Push socket pushes data to pull socket(s)• Pull socket pulls data from push socket(s) Push Pull Socket Socket
  18. 18. Push-pull• Use cases: • Sending email: apps pushes email data to email app • Logging: apps pushes messages to a logging app • Data crunching: apps pushes partials to processing app(s) to work on • Analytics: apps pushes tracking data to analytics app to process
  19. 19. Push-pull• Push socket can connect to multiple pull sockets (fan-out)• Pull sockets can listen on multiple push sockets (fan-in)• Best part: ZeroMQ handles the load balancing (usually via a LRU)
  20. 20. Advanced Push-Pull• Example: Log file processing• Log file needs to be parsed• Push app (ventilator) parses the log file• Push app continuously fires partials to pull apps (workers)• Workers push into another pull app (collector)
  21. 21. Advanced Push-pull Worker Worker push pushVentilator Collector pull pull Worker Worker
  22. 22. How do we scale?
  23. 23. ADD MOREWORKERS
  24. 24. Req-rep• Req socket sends data to rep socket• Rep socket receives data from req socket• Rep socket replies to req socket• NOTE: Req socket always expects replies (remember that girl that you waited 5 years for?)
  25. 25. Req-rep• Use cases: • Inter-app communications: apps that has dependencies between each other uses a REQ-REP pattern • Caching: apps req cached items from cache app, cache app rep with cached item • API: 3rd party developers issue req to your API app, which filters the request and replies with the results
  26. 26. Req-rep REQ REPSocket Socket req REQ REPSocket Socket rep REQ REPSocket Socket
  27. 27. How to scale?
  28. 28. ADD MORE REP APPS
  29. 29. All together now...
  30. 30. Sample ZMQ code: 01 var zmq = require(zmq), 02 push = zmq.socket(push), 03 pullA = zmq.socket(pull), 04 pullB = zmq.socket(pull); 05 06 pullA.on(message, function (msg) { 07 console.log(pullA received + msg); 08 }); 09 10 pullB.on(message, function (msg) { 11 console.log(pullB received + msg); 12 }); 13 14 pullA.bindSync(inproc://nodehack-rocks); 15 pullB.bindSync(inproc://oh-la-la); 16 17 push.connect(inproc://nodehack-rocks); 18 push.connect(inproc://oh-la-la); 19 20 for (var i = 0; i < 10; i++) { 21 push.send(this is message + i); 22 }
  31. 31. Notes• Each app can define and use multiple ZMQ sockets• You can organize your app’s internals using inproc ZMQ sockets (better MVC)• Controller scripts are important• There are advanced patterns for reliability (read the ZMQ guide)
  32. 32. Libraries• NodeJS ZMQ binding (https://github.com/JustinTulloss/zeromq.node)• ZMQ Guide (http://zguide.zeromq.org/page:all)• Eazy (https://github.com/soggie/eazy)
  33. 33. - the end -

×