Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Message:Passing - lpw 2012

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 150 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie Message:Passing - lpw 2012 (20)

Weitere von Tomas Doran (20)

Anzeige

Message:Passing - lpw 2012

  1. 1. Messaging, interoperability and log aggregation - a new framework Tomas Doran (t0m) <bobtfish@bobtfish.net>
  2. 2. Sponsored by • state51 • Pb of mogilefs, 100+ boxes. • > 4 million tracks on-demand via API • > 400 reqs/s per server, >1Gb peak from backhaul • Suretec VOIP Systems • UK voice over IP provider • Extensive API, including WebHooks for notifications • TIM Group • “Alpha capture” applications • Java / Scala / Clojure / ruby / puppet / python / perl
  3. 3. What? • This talk is about my new perl library: Message::Passing
  4. 4. Why?
  5. 5. Why? • I’d better stop, and explain a specific problem.
  6. 6. Why? • I’d better stop, and explain a specific problem. • The solution that grew out of this is more generic.
  7. 7. Why? • I’d better stop, and explain a specific problem. • The solution that grew out of this is more generic. • But it illustrates my concerns and design choices well.
  8. 8. Why? • I’d better stop, and explain a specific problem. • The solution that grew out of this is more generic. • But it illustrates my concerns and design choices well. • And everyone likes a story, right?
  9. 9. Once upon a time... • I was bored of tailing log files across dozens of servers
  10. 10. Once upon a time... • I was bored of tailing log files across dozens of servers • splunk was amazing, but unaffordable
  11. 11. Logstash
  12. 12. Centralised logging
  13. 13. Centralised logging • Syslog isn’t good enough
  14. 14. Centralised logging • Syslog isn’t good enough • UDP is lossy, TCP not much better
  15. 15. Centralised logging • Syslog isn’t good enough • UDP is lossy, TCP not much better • Limited fields
  16. 16. Centralised logging • Syslog isn’t good enough • UDP is lossy, TCP not much better • Limited fields • No structure to actual message
  17. 17. Centralised logging • Syslog isn’t good enough • UDP is lossy, TCP not much better • Limited fields • No structure to actual message • RFC3164 - “This document describes the observed behaviour of the syslog protocol”
  18. 18. Centralised logging • Syslog isn’t good enough • Structured app logging
  19. 19. Centralised logging • Syslog isn’t good enough • Structured app logging • We want to log data, rather than text from our application
  20. 20. Centralised logging • Syslog isn’t good enough • Structured app logging • We want to log data, rather than text from our application • E.g. HTTP request - vhost, path, time to generate, N db queries etc..
  21. 21. Centralised logging • Syslog isn’t good enough • Structured app logging
  22. 22. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure
  23. 23. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure
  24. 24. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure • Cases we do not control (e.g. apache)
  25. 25. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure • Cases we do not control (e.g. apache) • SO MANY DATE FORMATS. ARGHH!!
  26. 26. Apache [27/Jun/2012:23:57:03 +0000]
  27. 27. Apache
  28. 28. ElasticSearch [2012-06-26 02:08:26,879]
  29. 29. ElasticSearch
  30. 30. RabbitMQ 26-Jun-2012::16:18:30
  31. 31. RabbitMQ
  32. 32. MongoDB Thu Jun 28 01:02:29
  33. 33. MongoDB
  34. 34. Syslog Jun 28 00:17:26
  35. 35. Syslog
  36. 36. .Net ‘tick’ 634763158360000000 100 ns from 1st Jan 1AD (Except those that are from 3rd Jan)
  37. 37. .Net ‘tick’
  38. 38. MySQL 120404 12:31:04
  39. 39. MySQL
  40. 40. Aaaaaaannnnyyyway... • Please use ISO8601 • or epochseconds • or epochmicroseconds • In UTC!
  41. 41. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure • Publish logs as JSON to a message queue
  42. 42. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure • Publish logs as JSON to a message queue • JSON is fast, and widely supported
  43. 43. Centralised logging • Syslog isn’t good enough • Structured app logging • Post-process log files to re-structure • Publish logs as JSON to a message queue • JSON is fast, and widely supported • Great for arbitrary structured data!
  44. 44. Message queue
  45. 45. Message queue • Flattens load spikes!
  46. 46. Message queue • Flattens load spikes! • Only have to keep up with average message volume, not peak volume.
  47. 47. Message queue • Flattens load spikes! • Only have to keep up with average message volume, not peak volume. • Logs are bursty! (Peak rate 1000x average.)
  48. 48. Message queue • Flattens load spikes! • Only have to keep up with average message volume, not peak volume. • Logs are bursty! (Peak rate 1000x average.) • Easy to scale - just add more consumers
  49. 49. Message queue • Flattens load spikes! • Only have to keep up with average message volume, not peak volume. • Logs are bursty! (Peak rate 1000x average.) • Easy to scale - just add more consumers • Allows smart routing
  50. 50. Message queue • Flattens load spikes! • Only have to keep up with average message volume, not peak volume. • Logs are bursty! (Peak rate 1000x average.) • Easy to scale - just add more consumers • Allows smart routing • Great as a common integration point.
  51. 51. elasticsearch
  52. 52. elasticsearch • Just tip JSON documents into it
  53. 53. elasticsearch • Just tip JSON documents into it • Figures out type for each field, indexes appropriately.
  54. 54. elasticsearch • Just tip JSON documents into it • Figures out type for each field, indexes appropriately. • Free sharding and replication
  55. 55. elasticsearch • Just tip JSON documents into it • Figures out type for each field, indexes appropriately. • Free sharding and replication • Histograms!
  56. 56. Logstash In JRuby, by Jordan Sissel Input Simple: Filter Output Flexible Extensible Plays well with others Nice web interface
  57. 57. Logstash
  58. 58. Logstash IS MASSIVE
  59. 59. 440Mb IDLE!
  60. 60. Logstash on each host is totally out...
  61. 61. Logstash on each host is totally out... • Running it on elasticsearch servers which are already dedicated to this is fine..
  62. 62. Logstash on each host is totally out... • Running it on elasticsearch servers which are already dedicated to this is fine.. • I’d still like to reuse all of it’s parsing
  63. 63. Logstash on each host is totally out... • Running it on elasticsearch servers which are already dedicated to this is fine.. • I’d still like to reuse all of it’s parsing • How about I just log to AMQP from my app?
  64. 64. Logstash on each host is totally out... • Running it on elasticsearch servers which are already dedicated to this is fine.. • I’d still like to reuse all of it’s parsing • How about I just log to AMQP from my app? • Doooom!
  65. 65. ZeroMQ has the correct semantics
  66. 66. ZeroMQ has the correct semantics • Pub/Sub sockets
  67. 67. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking
  68. 68. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed)
  69. 69. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable
  70. 70. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable • Arbitrary message size
  71. 71. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable • Arbitrary message size • IO done in a background thread
  72. 72. On host log collector • ZeroMQ SUB socket • App logs - pre structured • Syslog listener • Forward rsyslogd • Log file tailer • Ship to AMQP
  73. 73. On host log collector
  74. 74. This talk • Is about my new library: Message::Passing • The clue is in the name...
  75. 75. This talk • Is about my new library: Message::Passing • The clue is in the name... • Hopefully really simple
  76. 76. This talk • Is about my new library: Message::Passing • The clue is in the name... • Hopefully really simple • Maybe even useful!
  77. 77. This talk • Is about my new library: Message::Passing • The clue is in the name... • Hopefully really simple • Maybe even useful! • Definitely small - you can replace / rewrite it easily.
  78. 78. Lets make it generic! • So, I wanted a log shipper
  79. 79. Lets make it generic! • So, I wanted a log shipper • I ended up with a framework for messaging interoperability
  80. 80. Lets make it generic! • So, I wanted a log shipper • I ended up with a framework for messaging interoperability • Whoops!
  81. 81. Lets make it generic! • So, I wanted a log shipper • I ended up with a framework for messaging interoperability • Whoops! • Got sick of writing scripts..
  82. 82. Does this actually work? • YES - In production at four sites for me.
  83. 83. Does this actually work? • YES - In production at four sites for me. • Some of the adaptors are partially complete
  84. 84. Does this actually work? • YES - In production at four sites for me. • Some of the adaptors are partially complete • Dumber than logstash - no multiple threads/cores
  85. 85. Does this actually work? • YES - In production at four sites for me. • Some of the adaptors are partially complete • Dumber than logstash - no multiple threads/cores • ZeroMQ is insanely fast
  86. 86. Other people are using it in production! Two people I know of already writing have already written adaptors!
  87. 87. Events - my model for message passing
  88. 88. Events - my model for message passing • a hash {}
  89. 89. Events - my model for message passing • a hash {} • Output consumes events: • method consume ($event) { ...
  90. 90. Events - my model for message passing • a hash {} • Output consumes events: • method consume ($event) { ... • Input produces events: • has output_to => (..
  91. 91. Events - my model for message passing • a hash {} • Output consumes events: • method consume ($event) { ... • Input produces events: • has output_to => (.. • Filter does both
  92. 92. Simplifying assumption $self->output_to->consume($message)
  93. 93. Events
  94. 94. That’s it.
  95. 95. That’s it. • No, really - that’s all the complexity you have to care about!
  96. 96. That’s it. • No, really - that’s all the complexity you have to care about! • Except for the complexity introduced by the inputs and outputs you use.
  97. 97. That’s it. • No, really - that’s all the complexity you have to care about! • Except for the complexity introduced by the inputs and outputs you use. • Unified attribute names / reconnection model, etc.. This helps, somewhat..
  98. 98. Inputs and outputs • ZeroMQ In / Out • AMQP (RabbitMQ) In / Out • STOMP (ActiveMQ) In / Out • elasticsearch Out • Redis PubSub In/Out • Syslog In • MongoDB Out • Collectd In/Out • HTTP POST (“WebHooks”) Out • UDP packets In/Out (e.g. statsd)
  99. 99. DSL • Building more complex chains easy! • Multiple inputs • Multiple outputs • Multiple independent chains
  100. 100. CLI • 1 Input • 1 Output • 1 Filter (default Null) • For simple use, or testing.
  101. 101. CLI • Encode / Decode step is just a Filter • JSON by default • Supply command line, or config file • Daemon features
  102. 102. The dist: Message::Passing • Core dist supplies CLI, DSL, roles for reuse.
  103. 103. The dist: Message::Passing • Core dist supplies CLI, DSL, roles for reuse. • Adaptors for most protocols in other modules.
  104. 104. The dist: Message::Passing • Core dist supplies CLI, DSL, roles for reuse. • Adaptors for most protocols in other modules. • Moo based - small footprint, can be fatpacked (no XS dependencies).
  105. 105. The dist: Message::Passing • Core dist supplies CLI, DSL, roles for reuse. • Adaptors for most protocols in other modules. • Moo based - small footprint, can be fatpacked (no XS dependencies). • Moose compatible.
  106. 106. Example? message-pass --input STDIN --output STDOUT {} {}
  107. 107. Less trivial example message-pass --input ZeroMQ --input_options ‘{“socket_bind”:”tcp://*:5222”}’ --output STDOUT message-pass --output ZeroMQ --output_options ‘{“connect”:”tcp://127.0.0.1:5222”}’ --input STDIN
  108. 108. Across the network:
  109. 109. Multiple subscribers
  110. 110. Jenga?
  111. 111. Jenga: message-pass --input STDIN --output STOMP --output_options '{"destination":"/queue/foo","hostname":"localhost", "port":"6163", "username":"guest", "password":"guest"}' message-pass --input STOMP --output Redis --input_options '{"destination":"/queue/ foo", "hostname":"localhost","port":"6163","username":"guest","password":"guest"}' --output_options '{"topic":"foo","hostname":"127.0.0.1","port":"6379"}' message-pass --input Redis --output AMQP --input_options '{"topics": ["foo"],"hostname":"127.0.0.1","port":"6379"}' --output_options '{"hostname":"127.0.0.1","username":"guest","password":"guest", "exchange_name":"foo"}' message-pass --input AMQP --output STDOUT --input_options '{"hostname":"127.0.0.1", "username":"guest", "password":"guest", "exchange_name":"foo","queue_name":"foo"}'
  112. 112. Jenga!
  113. 113. Example 4? • The last example wasn’t silly enough!
  114. 114. Example 4? • The last example wasn’t silly enough! • How could I top that?
  115. 115. Example 4? • The last example wasn’t silly enough! • How could I top that? • Plan - Re-invent mongrel2
  116. 116. Example 4? • The last example wasn’t silly enough! • How could I top that? • Plan - Re-invent mongrel2 • Badly
  117. 117. PSGI • PSGI $env is basically just a hash.
  118. 118. PSGI • PSGI $env is basically just a hash. • (With a little fiddling), you can serialize it as JSON
  119. 119. PSGI
  120. 120. PSGI • PSGI $env is basically just a hash.
  121. 121. PSGI • PSGI $env is basically just a hash. • (With a little fiddling), you can serialize it as JSON
  122. 122. PSGI • PSGI $env is basically just a hash. • (With a little fiddling), you can serialize it as JSON • PSGI response is just an array.
  123. 123. PSGI • PSGI $env is basically just a hash. • (With a little fiddling), you can serialize it as JSON • PSGI response is just an array. • Ignore streaming responses!
  124. 124. Demo? plackup -E production -s Twiggy -MPlack::App::Message::Passing -e'Plack::App::Message::Passing->new(return_address => "tcp://127.0.0.1:5555", send_address => "tcp://127.0.0.1:5556")->to_app' plackup -E production -s Message::Passing testapp.psgi --host 127.0.0.1 --port 5556
  125. 125. PUSH socket does fan out between multiple handlers. Reply to address embedded in request Run multiple ‘handler’ processes. Hot restarts, hot add / remove workers
  126. 126. Other applications • Anywhere an asynchronous event stream is useful! • Monitoring • Metrics transport • Queued jobs - worker pool
  127. 127. Other applications (Web stuff) • User activity (ajax ‘what are your users doing’) • WebSockets / MXHR • HTTP Push notifications - “WebHooks”
  128. 128. WebHooks • HTTP PUSH notification • E.g. Paypal IPN • Shopify API
  129. 129. What about logstash? • Use my lightweight code on end nodes. • Use logstash for parsing/filtering on the dedicated hardware (elasticsearch boxes) • Filter to change my hashes to logstash compatible hashes • For use with MooseX::Storage and/or Log::Message::Structured
  130. 130. Interoperating - a real example
  131. 131. Interoperating - a real example • Log JSON events out of apps (in multiple languages) to ZMQ
  132. 132. Interoperating - a real example • Log JSON events out of apps (in multiple languages) to ZMQ • Collect and munge with Message::Passing script ‘logcollector’
  133. 133. Interoperating - a real example • Log JSON events out of apps (in multiple languages) to ZMQ • Collect and munge with Message::Passing script ‘logcollector’ • Send to central logstash
  134. 134. Interoperating - a real example • Log JSON events out of apps (in multiple languages) to ZMQ • Collect and munge with Message::Passing script ‘logcollector’ • Send to central logstash • Send onto statsd to aggregate
  135. 135. Interoperating - a real example • Log JSON events out of apps (in multiple languages) to ZMQ • Collect and munge with Message::Passing script ‘logcollector’ • Send to central logstash • Send onto statsd to aggregate • Graphs in graphite
  136. 136. Standard log message
  137. 137. Standard event message
  138. 138. TimedWebRequest • A standard event • Page generation time, URI, HTTP status
  139. 139. statsd
  140. 140. statsd • Rolls up counters and timers into metrics
  141. 141. statsd • Rolls up counters and timers into metrics • One bucket per stat, emits values every 10 seconds
  142. 142. statsd • Rolls up counters and timers into metrics • One bucket per stat, emits values every 10 seconds • Counters: Request rate, HTTP status rate
  143. 143. statsd • Rolls up counters and timers into metrics • One bucket per stat, emits values every 10 seconds • Counters: Request rate, HTTP status rate • Timers: Total page time, mean page time, min/max page times
  144. 144. statsd
  145. 145. statsd
  146. 146. Code • https://metacpan.org/module/ Message::Passing • https://github.com/suretec/Message-Passing • #message-passing on irc.perl.org • Examples: git://github.com/2941747.git

Hinweis der Redaktion

  • \n
  • Mention state51 are hiring in London\nMention Tim Group are hiring in London/Boston.\n
  • But, before I talk about perl at you, I&amp;#x2019;m going to go off on a tangent..\n
  • I wrote code. And writing code is never something to be proud of; at least if your code looks like mine it isn&amp;#x2019;t... So I&amp;#x2019;d better justify this hubris somehow..\n
  • I wrote code. And writing code is never something to be proud of; at least if your code looks like mine it isn&amp;#x2019;t... So I&amp;#x2019;d better justify this hubris somehow..\n
  • I wrote code. And writing code is never something to be proud of; at least if your code looks like mine it isn&amp;#x2019;t... So I&amp;#x2019;d better justify this hubris somehow..\n
  • I wrote code. And writing code is never something to be proud of; at least if your code looks like mine it isn&amp;#x2019;t... So I&amp;#x2019;d better justify this hubris somehow..\n
  • \n
  • \n
  • Isn&amp;#x2019;t he cute? And woody!\nWho knows what this is?\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • MooseX::Storage!\nThis isn&amp;#x2019;t mandatory - you can just log plain hashes if you&amp;#x2019;re concerned about performance.\nSPOT THE TYPO\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • No, really - JSON::XS is lightning fast\n
  • No, really - JSON::XS is lightning fast\n
  • No, really - JSON::XS is lightning fast\n
  • No, really - JSON::XS is lightning fast\n
  • No, really - JSON::XS is lightning fast\n
  • No, really - JSON::XS is lightning fast\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • Most message queues have bindings in most languages.. So by abstracting message routing out of your application, and passing JSON hashes - you are suddenly nicely cross language!\n
  • \n
  • \n
  • \n
  • \n
  • Very simple model - input (pluggable), filtering (pluggable by type) in C, output (pluggable)\nLots of backends - AMQP and elasticsearch + syslog and many others\nPre-built parser library for various line based log formats\nComes with web app for searches.. Everything I need!\n
  • And it has an active community.\nThis is the alternate viewer app..\n
  • Lets take a simple case here - I&amp;#x2019;ll shove my apache logs from N servers into elasticsearch\nI run a logstash on each host (writer), and one on each elasticsearch server (reader)..\n\n
  • First problem...\n
  • Well then, I&amp;#x2019;m not going to be running this on the end nodes.\n
  • Has a whole library of pre-built parsers for common log formats.\nAlso, as noted, it&amp;#x2019;s faster, and notably it&amp;#x2019;s multi-threaded, so it&amp;#x2019;ll use multiple cores..\n
  • Has a whole library of pre-built parsers for common log formats.\nAlso, as noted, it&amp;#x2019;s faster, and notably it&amp;#x2019;s multi-threaded, so it&amp;#x2019;ll use multiple cores..\n
  • Has a whole library of pre-built parsers for common log formats.\nAlso, as noted, it&amp;#x2019;s faster, and notably it&amp;#x2019;s multi-threaded, so it&amp;#x2019;ll use multiple cores..\n
  • Has a whole library of pre-built parsers for common log formats.\nAlso, as noted, it&amp;#x2019;s faster, and notably it&amp;#x2019;s multi-threaded, so it&amp;#x2019;ll use multiple cores..\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
  • Yes, this could still be &amp;#x2018;a script&amp;#x2019;, in fact I did that at first...\nBut I now have 3 protocols, who&amp;#x2019;s to say I won&amp;#x2019;t want a 4th..\n\n
  • Note the fact that we have a cluster of ES servers here.\nAnd we have two log indexers. You can cluster RabbitMQ also.\nHighly reliable solution (against machine failure). Highly scaleable solution (just add ES servers)\nWe use RabbitMQ as this also allows someone to tap a part of the log stream, could just use ZMQ throughout.\n
  • At the same time, I want something that can be used for real work (i.e. not just a toy)\n
  • At the same time, I want something that can be used for real work (i.e. not just a toy)\n
  • At the same time, I want something that can be used for real work (i.e. not just a toy)\n
  • At the same time, I want something that can be used for real work (i.e. not just a toy)\n
  • At the same time, I want something that can be used for real work (i.e. not just a toy)\n
  • I had a log shipper script. A long indexer script. An alerting (nagios) script. An irc notification script.\n
  • I had a log shipper script. A long indexer script. An alerting (nagios) script. An irc notification script.\n
  • I had a log shipper script. A long indexer script. An alerting (nagios) script. An irc notification script.\n
  • I had a log shipper script. A long indexer script. An alerting (nagios) script. An irc notification script.\n
  • By insanely fast, I mean I can generate, encode as JSON, send, receive, decode as JSON over 25k messages a second. On this 3 year old macbook..\n
  • By insanely fast, I mean I can generate, encode as JSON, send, receive, decode as JSON over 25k messages a second. On this 3 year old macbook..\n
  • By insanely fast, I mean I can generate, encode as JSON, send, receive, decode as JSON over 25k messages a second. On this 3 year old macbook..\n
  • By insanely fast, I mean I can generate, encode as JSON, send, receive, decode as JSON over 25k messages a second. On this 3 year old macbook..\n
  • \n
  • Filters are just a combination of input and output\n
  • Filters are just a combination of input and output\n
  • Filters are just a combination of input and output\n
  • Filters are just a combination of input and output\n
  • So the input has an output, that output always has a consume method...\nTADA!\n
  • You can build a &amp;#x201C;chain&amp;#x201D; of events. This can work either way around.\nThe input can be a log file, the output can be a message queue (publisher)\nInput can be a message queue, output can be a log file (consumer)\n
  • The docs still suck, sorry - I have tried ;)\n
  • The docs still suck, sorry - I have tried ;)\n
  • The docs still suck, sorry - I have tried ;)\n
  • All of these are on CPAN already.\n
  • DSL - Domain specific language.\nTry to make writing scripts really simple.\n
  • But you shouldn&amp;#x2019;t have to write ANY code to play around.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Demo1\nSimple demo of the CLI in one process (STDOUT/STDIN)\n
  • Demo1\nSimple demo of the CLI in one process (STDOUT/STDIN)\n
  • Demo1\nSimple demo of the CLI in one process (STDOUT/STDIN)\n
  • Less simple demo - lets actually pass messages between two processes.\nArrows indicate message flow. ZeroMQ is a lightning bolt as it&amp;#x2019;s not quite so trivial..\n
  • Demo PUBSUB and round robin..\n
  • So, lets play Jenga with message queues!\n
  • \n
  • I would have added ZeroMQ. Except then the diagram doesn&amp;#x2019;t fit on the page.\nI&amp;#x2019;ll leave this as an exercise for the reader!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • I&amp;#x2019;ll talk a very little more about webhooks\n
  • Error stream\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

×