Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Distributed app development with nodejs and zeromq

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 33 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie Distributed app development with nodejs and zeromq (20)

Anzeige

Aktuellste (20)

Distributed app development with nodejs and zeromq

  1. 1. Distributed app development node.js + zeroMQ = awwyeah
  2. 2. ZeroMQ from a noob’s viewpoint
  3. 3. RUBEN TAN • NodeJS enthusiast • Author of eazy (https://github.com/soggie/eazy) • Dota 2-er
  4. 4. DISCLAIMER • I AM NOT A ZEROMQ EXPERT • (not even remotely close) • Case study by consulting a successful implementer of ZMQ in an adserver
  5. 5. Objective • 1,000,000,000 ad impressions per month (on average) • 80ms maximum response time for each ad • Good luck, Apache
  6. 6. Webserver Choices • Netty - Java can actually be fast https://netty.io • Gevent - For Python lovers http://gevent.org/ • Node.js - The reason why you’re here http://nodejs.org/
  7. 7. So... which one? • All 3, but that’s not the point • Speed isn’t the most important factor • Reliability under load is more important than pure performance • 1 missed impression = 1 potential disaster (imagine BMW ads showing up in ThePirateBay) • (but secretly, I prefer node.js)
  8. 8. Welcome to the distributed app ecosystem
  9. 9. Definitions, definitions • Distributed: distributed across multiple geological “clouds” • Apps: each app handles a small scope of responsibility • Ecosystem: apps talk to apps via a standardized protocol
  10. 10. Distributed • Route 53 -> HAProxy -> Webserver -> Apps • Scale webservers by creating new instances behind HAProxy • Scale apps by creating new instances in their own clouds
  11. 11. Distributed Route 53 (DNS) HAProxy “LB layer” Webserver Webserver Webserver App cloud App cloud (EU) (NA)
  12. 12. Apps • Each app should perform ONE single domain of function • Write the app like you’re writing an API • The simpler the app, the better • App can be written in any language
  13. 13. User app • List of exposed API: • create new user • edit user • delete user • merge user accounts • authenticate user
  14. 14. Ecosystem • Apps communicate using JSON-RPC • Apps connect to each other using ZMQ • “Controller” scripts and monit handles lifecycle
  15. 15. Quick & dirty ZeroMQ • ZMQ = socket library • Types of socket: • inproc • IPC • TCP • pgm • epgm
  16. 16. Socket Types • Push-pull • Req-rep • Pub-sub (not covered) • Dealer-router (not covered) • Xreq-xrep (not covered)
  17. 17. Push-pull • Push socket pushes data to pull socket(s) • Pull socket pulls data from push socket(s) Push Pull Socket Socket
  18. 18. Push-pull • Use cases: • Sending email: apps pushes email data to email app • Logging: apps pushes messages to a logging app • Data crunching: apps pushes partials to processing app(s) to work on • Analytics: apps pushes tracking data to analytics app to process
  19. 19. Push-pull • Push socket can connect to multiple pull sockets (fan-out) • Pull sockets can listen on multiple push sockets (fan-in) • Best part: ZeroMQ handles the load balancing (usually via a LRU)
  20. 20. Advanced Push-Pull • Example: Log file processing • Log file needs to be parsed • Push app (ventilator) parses the log file • Push app continuously fires partials to pull apps (workers) • Workers push into another pull app (collector)
  21. 21. Advanced Push-pull Worker Worker push push Ventilator Collector pull pull Worker Worker
  22. 22. How do we scale?
  23. 23. ADD MORE WORKERS
  24. 24. Req-rep • Req socket sends data to rep socket • Rep socket receives data from req socket • Rep socket replies to req socket • NOTE: Req socket always expects replies (remember that girl that you waited 5 years for?)
  25. 25. Req-rep • Use cases: • Inter-app communications: apps that has dependencies between each other uses a REQ-REP pattern • Caching: apps req cached items from cache app, cache app rep with cached item • API: 3rd party developers issue req to your API app, which filters the request and replies with the results
  26. 26. Req-rep REQ REP Socket Socket req REQ REP Socket Socket rep REQ REP Socket Socket
  27. 27. How to scale?
  28. 28. ADD MORE REP APPS
  29. 29. All together now...
  30. 30. Sample ZMQ code: 01 var zmq = require('zmq'), 02 push = zmq.socket('push'), 03 pullA = zmq.socket('pull'), 04 pullB = zmq.socket('pull'); 05 06 pullA.on('message', function (msg) { 07 console.log('pullA received ' + msg); 08 }); 09 10 pullB.on('message', function (msg) { 11 console.log('pullB received ' + msg); 12 }); 13 14 pullA.bindSync('inproc://nodehack-rocks'); 15 pullB.bindSync('inproc://oh-la-la'); 16 17 push.connect('inproc://nodehack-rocks'); 18 push.connect('inproc://oh-la-la'); 19 20 for (var i = 0; i < 10; i++) { 21 push.send('this is message ' + i); 22 }
  31. 31. Notes • Each app can define and use multiple ZMQ sockets • You can organize your app’s internals using inproc ZMQ sockets (better MVC) • Controller scripts are important • There are advanced patterns for reliability (read the ZMQ guide)
  32. 32. Libraries • NodeJS ZMQ binding (https://github.com/JustinTulloss/zeromq.node) • ZMQ Guide (http://zguide.zeromq.org/page:all) • Eazy (https://github.com/soggie/eazy)
  33. 33. - the end -

Hinweis der Redaktion

  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

×