how we built a firehose with NodeJS and ZeroMQ

1. Firehose
Building a real-time data stream with NodeJS
Luca Grulla@Forward
London, 2/8/2011

2. Introduction
Or what we wanted to build

3. We like to have a (near) real-time view
of the data stream

4. Resilient

5. Scalable

6. Accurate
(but not necessarily precise)

8. Verse and Chorus
Or how we built it

9. Stack
• NodeJS (0.4.9)
• CoffeeScript
• ZeroMQ

10. Architecture

11. Interlude
Or joys and pitfalls of building a tail
library for node

12. Living the asynchronous dream

13. Living the asynchronous dream
• Error manifests itself only at high throughput
(7MB/secs)
• The few implementations you can find around
are wrong (including Log.io – we are planning
to contribute back)

14. Breathing in an asynchronous way

15. Learnings
• You need to think asynchronous to write
asynchronous code
• Dispatch internal events if you need to keep
control of the flow
• “Internal queues” to scale(up to a certain
limit)

16. Performance
• Beyond 5MB we start to fill the internal queue
• We are limited by IO, not from CPU/memory
• Consistent throughput

17. Questions ?

18. References
• npm install tail
• https://github.com/forward/node-tail