7. –Wikipedia Pipeline_(Software)
“A pipeline consists of a chain of processing
elements arranged so that the output of each
element is the input of the next...
The information that flows in these pipelines is
often a stream of records, bytes or bits, and the
elements of a pipeline may be called filters; this is
also called the pipes and filters design
pattern.”
8. -Enterprise Integration Patterns
(Gregor Hope & Bobby Woolf)
“Use the Pipes and Filters architectural style
to divide a larger processing task into a sequence
of smaller, independent processing steps (filters)
that are connected by channels (pipes).”
16. • Pump
- Component that feeding a pipeline with data
• Filter
- Component that transforming data
• Sink
- Component that collects data and terminates a
pipeline
31. Glad
• RabbitMQ cluster
• Object state represented as pipes
• Akka fault tolerance with queue acknowledgement
• Scale up - Akka router
• Scale out - Independent server nodes consuming
the same queues
36. Graphviz
• Short for Graph Visualisation Software
• Open source tools initiated by AT&T Labs
Research
• Graphviz consists of a graph description language
named the DOT language
38. Pineapple
• Based on experience from KHS
• Akka & Jocote
• Wire up actors from model
• Parsed from DOT file
• Created by ModelBuilder
• Discovery API (Spring, JSR 330 or your own)
• Pipes & Filters API
42. Next
• Connection recovery
• Support for
• Ack/nack
• Backout queues (dead letters)
• Using actor selection as url
• Cron scheduling (ConsumeType)
• JMX
• Enabling/disabling filters, pumps and sinks
• Show model
• Monitor UI
43. Apache Flink
• Apache Flink 0.10.0 - Nov 2015
• Ugli Jocote 1.6.0.0 - Nov 2015
• Looks good and has momentum
44. I been thinking of
• A Jocote Flink Connector
• An option for choosing between Jocote
and Flink in Pineapple
• A Camel-Jocote-Driver