SlideShare ist ein Scribd-Unternehmen logo
1 von 46
Downloaden Sie, um offline zu lesen
Pipes & Filters
Architectural
Pattern
Fredrik Kivi
About Me
Have you ever end up with a situation
there you doing multiple entity updates
of the status to reflect your application
lifecycle?
The Database As Queue
Anti-Pattern
-Mike Hadlow
”When all you have is a hammer,
every problem looks like a nail”
The

Pipes & Filters 

Architectural Pattern
–Wikipedia Pipeline_(Software)
“A pipeline consists of a chain of processing
elements arranged so that the output of each
element is the input of the next...
The information that flows in these pipelines is
often a stream of records, bytes or bits, and the
elements of a pipeline may be called filters; this is
also called the pipes and filters design
pattern.”
-Enterprise Integration Patterns

(Gregor Hope & Bobby Woolf)
“Use the Pipes and Filters architectural style
to divide a larger processing task into a sequence
of smaller, independent processing steps (filters)
that are connected by channels (pipes).”
Related patterns
• Collection Pipeline
• Java Stream API
• Chain of Responsibility (CoR)
• Servlet Filter
History
• 1972-1974 Unix Pipelines, Pipes & Filters
• 1994-1996 Pipes & Filters Architectural Pattern
• PLoP (Pattern Languages of Programs Conf.)
• Papers
• Books
Terminology
• Pump

- Component that feeding a pipeline with data
• Filter

- Component that transforming data
• Sink 

- Component that collects data and terminates a
pipeline
PIPE PROPERTIES
• Connections
• Streams
• Channels
• Represent object status
FILTER PROPERTIES
• Transform
• Component
• Concurrent
• Process
• Independent, parallel, decomposed processing
Filter implementation
Akka Actors
• Independent, parallel, decomposed processing
• Fault tolerant
• Scala up and out
• Akka & Apache Camel?
• Not in my taste ;)
Pipe implementation
JMS Hello World
Jocote
@Folksam 2012-2016
KHS
• Pumps - Jocote (IBM MQ)
• Pipes - Jocote (RabbitMQ)
• Filters - Akka Actors
• Sinks - LPR
• Scheduling - Akka Scheduler, Quartz
• Enabling/disabling - Akka Cluster Event
Retrospective
Glad
• RabbitMQ cluster
• Object state represented as pipes
• Akka fault tolerance with queue acknowledgement
• Scale up - Akka router
• Scale out - Independent server nodes consuming
the same queues
Sad
• No orchestration
• Hard to follow asynchronous calls as development
What if
Graphviz
• Short for Graph Visualisation Software
• Open source tools initiated by AT&T Labs
Research
• Graphviz consists of a graph description language
named the DOT language
TOOLS
• Eclipse GEF
• www.webgraphviz.com
Pineapple
• Based on experience from KHS
• Akka & Jocote
• Wire up actors from model
• Parsed from DOT file
• Created by ModelBuilder
• Discovery API (Spring, JSR 330 or your own)
• Pipes & Filters API
D E M O
Next
• Connection recovery
• Support for
• Ack/nack
• Backout queues (dead letters)
• Using actor selection as url
• Cron scheduling (ConsumeType)
• JMX
• Enabling/disabling filters, pumps and sinks
• Show model
• Monitor UI
Apache Flink
• Apache Flink 0.10.0 - Nov 2015
• Ugli Jocote 1.6.0.0 - Nov 2015
• Looks good and has momentum
I been thinking of
• A Jocote Flink Connector
• An option for choosing between Jocote

and Flink in Pineapple
• A Camel-Jocote-Driver
• https://github.com/ugli/jocote
• https://github.com/ugli/pineapple
• fredrik.kivi@ugli.se
Pipes & Filters Architectural Pattern

Weitere ähnliche Inhalte

Was ist angesagt?

Design patterns ppt
Design patterns pptDesign patterns ppt
Design patterns ppt
Aman Jain
 
Software Requirements in Software Engineering SE5
Software Requirements in Software Engineering SE5Software Requirements in Software Engineering SE5
Software Requirements in Software Engineering SE5
koolkampus
 
Uml deployment diagram
Uml deployment diagramUml deployment diagram
Uml deployment diagram
Asraa Batool
 

Was ist angesagt? (20)

formal verification
formal verificationformal verification
formal verification
 
Design patterns ppt
Design patterns pptDesign patterns ppt
Design patterns ppt
 
UML
UMLUML
UML
 
Introduction to UML
Introduction to UMLIntroduction to UML
Introduction to UML
 
Design Patterns
Design PatternsDesign Patterns
Design Patterns
 
White box ppt
White box pptWhite box ppt
White box ppt
 
Software Engineering : Requirement Analysis & Specification
Software Engineering : Requirement Analysis & SpecificationSoftware Engineering : Requirement Analysis & Specification
Software Engineering : Requirement Analysis & Specification
 
System and network administration network services
System and network administration network servicesSystem and network administration network services
System and network administration network services
 
Design Pattern in Software Engineering
Design Pattern in Software EngineeringDesign Pattern in Software Engineering
Design Pattern in Software Engineering
 
Software development life cycle (SDLC)
Software development life cycle (SDLC)Software development life cycle (SDLC)
Software development life cycle (SDLC)
 
Class diagrams
Class diagramsClass diagrams
Class diagrams
 
4b use-case analysis
4b use-case analysis4b use-case analysis
4b use-case analysis
 
Software Engineering - chp4- design patterns
Software Engineering - chp4- design patternsSoftware Engineering - chp4- design patterns
Software Engineering - chp4- design patterns
 
Software Requirements in Software Engineering SE5
Software Requirements in Software Engineering SE5Software Requirements in Software Engineering SE5
Software Requirements in Software Engineering SE5
 
Waterfall model of Software Engineering
Waterfall model of Software EngineeringWaterfall model of Software Engineering
Waterfall model of Software Engineering
 
Goals Of Software Design - The main goals
Goals Of Software Design - The main goalsGoals Of Software Design - The main goals
Goals Of Software Design - The main goals
 
Lecture 12 requirements modeling - (system analysis)
Lecture 12   requirements modeling - (system analysis)Lecture 12   requirements modeling - (system analysis)
Lecture 12 requirements modeling - (system analysis)
 
Introduction to Network and System Administration
Introduction to Network and System AdministrationIntroduction to Network and System Administration
Introduction to Network and System Administration
 
Uml deployment diagram
Uml deployment diagramUml deployment diagram
Uml deployment diagram
 
SE_Lec 05_System Modelling and Context Model
SE_Lec 05_System Modelling and Context ModelSE_Lec 05_System Modelling and Context Model
SE_Lec 05_System Modelling and Context Model
 

Ähnlich wie Pipes & Filters Architectural Pattern

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
Legacy Typesafe (now Lightbend)
 
End-to-end Data Governance with Apache Avro and Atlas
End-to-end Data Governance with Apache Avro and AtlasEnd-to-end Data Governance with Apache Avro and Atlas
End-to-end Data Governance with Apache Avro and Atlas
DataWorks Summit
 
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Kai Wähner
 
Building a Business Logic Translation Engine with Spark Streaming for Communi...
Building a Business Logic Translation Engine with Spark Streaming for Communi...Building a Business Logic Translation Engine with Spark Streaming for Communi...
Building a Business Logic Translation Engine with Spark Streaming for Communi...
Spark Summit
 
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Kai Wähner
 

Ähnlich wie Pipes & Filters Architectural Pattern (20)

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
 
KSQL Intro
KSQL IntroKSQL Intro
KSQL Intro
 
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
 
Stacktician - CloudStack Collab Conference 2014
Stacktician - CloudStack Collab Conference 2014Stacktician - CloudStack Collab Conference 2014
Stacktician - CloudStack Collab Conference 2014
 
Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL
Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQLSteps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL
Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL
 
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
 
End-to-end Data Governance with Apache Avro and Atlas
End-to-end Data Governance with Apache Avro and AtlasEnd-to-end Data Governance with Apache Avro and Atlas
End-to-end Data Governance with Apache Avro and Atlas
 
Edward King SPEDDEXES 2014
Edward King SPEDDEXES 2014Edward King SPEDDEXES 2014
Edward King SPEDDEXES 2014
 
DEVNET-1106 Upcoming Services in OpenStack
DEVNET-1106	Upcoming Services in OpenStackDEVNET-1106	Upcoming Services in OpenStack
DEVNET-1106 Upcoming Services in OpenStack
 
A machine learning and data science pipeline for real companies
A machine learning and data science pipeline for real companiesA machine learning and data science pipeline for real companies
A machine learning and data science pipeline for real companies
 
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
 
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
 
Building a Business Logic Translation Engine with Spark Streaming for Communi...
Building a Business Logic Translation Engine with Spark Streaming for Communi...Building a Business Logic Translation Engine with Spark Streaming for Communi...
Building a Business Logic Translation Engine with Spark Streaming for Communi...
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)
 
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoStreaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
 
Ruby to Scala in 9 weeks
Ruby to Scala in 9 weeksRuby to Scala in 9 weeks
Ruby to Scala in 9 weeks
 
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
 
Reactive Streams - László van den Hoek
Reactive Streams - László van den HoekReactive Streams - László van den Hoek
Reactive Streams - László van den Hoek
 
A Practical Guide To End-to-End Tracing In Event Driven Architectures
A Practical Guide To End-to-End Tracing In Event Driven ArchitecturesA Practical Guide To End-to-End Tracing In Event Driven Architectures
A Practical Guide To End-to-End Tracing In Event Driven Architectures
 
Getting Deep on Orchestration - Nickoloff - DockerCon16
Getting Deep on Orchestration - Nickoloff - DockerCon16Getting Deep on Orchestration - Nickoloff - DockerCon16
Getting Deep on Orchestration - Nickoloff - DockerCon16
 

Kürzlich hochgeladen

Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts ServiceCall Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
amitlee9823
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
ZurliaSoop
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
MarinCaroMartnezBerg
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
amitlee9823
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 

Kürzlich hochgeladen (20)

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort ServiceBDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
 
Halmar dropshipping via API with DroFx
Halmar  dropshipping  via API with DroFxHalmar  dropshipping  via API with DroFx
Halmar dropshipping via API with DroFx
 
BabyOno dropshipping via API with DroFx.pptx
BabyOno dropshipping via API with DroFx.pptxBabyOno dropshipping via API with DroFx.pptx
BabyOno dropshipping via API with DroFx.pptx
 
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts ServiceCall Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
 
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
 
Capstone Project on IBM Data Analytics Program
Capstone Project on IBM Data Analytics ProgramCapstone Project on IBM Data Analytics Program
Capstone Project on IBM Data Analytics Program
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Sampling (random) method and Non random.ppt
Sampling (random) method and Non random.pptSampling (random) method and Non random.ppt
Sampling (random) method and Non random.ppt
 
Week-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionWeek-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interaction
 
Ravak dropshipping via API with DroFx.pptx
Ravak dropshipping via API with DroFx.pptxRavak dropshipping via API with DroFx.pptx
Ravak dropshipping via API with DroFx.pptx
 
BigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptxBigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptx
 
Invezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signalsInvezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signals
 
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 

Pipes & Filters Architectural Pattern