SlideShare a Scribd company logo
1 of 18
Download to read offline
  Essentially, programs that can be created by
themselves.
  We’re twisting the term a little for streams
  We will be defining streaming constructs using
streaming constructs
  By streams I mean beams which are just streams and
pipes for compute as opposed to IO
Any info? Please contact #nodeland police at @cianomaidin on twitter…
  Readable Streams
  IO / throughput oriented
  Events – data, end, error, close
  Methods – pause, resume, end, destroy

  Writeable Streams
  IO / throughput oriented
  Events - drain, pause, resume, error, close
  Methods – write, end, destroy
  IO Oriented
  Finite, not infinite
  Asynchronous
  Lazy
  Assumes in transit data, not in memory forms
  Great for IO. Not efficient for compute
  Sources
  Where data pours in

  Sinks
  Where results pop out

  Operators
 
 
 
 

Filters – Drop or pass on data based on a UDF
Transforms – Mutate data based on a UDF
Branch with Filters
Combine with Sinks
  Compute Oriented
  Infinite, not finite
  Synchronous
  Eager
  Assumes in memory form, not encoded or wire data
  Convenient for compute. Not good for IO
  Eager: a.pipe(operation).pipe(z).pipe(a);
  Implies:
RangeError: Maximum call stack size exceeded
  npm install beam
 
 
 

Compute oriented streams
Branch, Combine, Filter, Transform data
Extend with goto and modules.

  npm install eep
 
 
 
 
 

Beam will be integrated into eep soon
Eep provides aggregate windows
Slide, Tumble, Monotonic, Periodic
Stats Library
Ordering now supported

  npm install nodesapiens
 

Robosapiens V1 for node.js

  Code on github
 

https://github.com/darach

More Related Content

Viewers also liked

成功學與未來學Jpg
成功學與未來學Jpg成功學與未來學Jpg
成功學與未來學Jpgeddy lin
 
FunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - StreamsFunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - Streamsdarach
 
Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014darach
 
Evening At Soulfood Books
Evening At Soulfood BooksEvening At Soulfood Books
Evening At Soulfood BooksCraig Larsen
 
EFL Munich - February 2013 - "Conversational Big Data with Erlang"
EFL Munich - February 2013 - "Conversational Big Data with Erlang"EFL Munich - February 2013 - "Conversational Big Data with Erlang"
EFL Munich - February 2013 - "Conversational Big Data with Erlang"darach
 
Streamy, Pipy, Analyticy
Streamy, Pipy, AnalyticyStreamy, Pipy, Analyticy
Streamy, Pipy, Analyticydarach
 
QCON London 2013
QCON London 2013QCON London 2013
QCON London 2013darach
 
Big Data, Mob Scale.
Big Data, Mob Scale.Big Data, Mob Scale.
Big Data, Mob Scale.darach
 
Deconstructing Lambda
Deconstructing LambdaDeconstructing Lambda
Deconstructing Lambdadarach
 
Data distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsData distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsdarach
 
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011darach
 
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...mandalina landy
 
The Normal Probability Distribution
The Normal Probability DistributionThe Normal Probability Distribution
The Normal Probability Distributionmandalina landy
 

Viewers also liked (14)

成功學與未來學Jpg
成功學與未來學Jpg成功學與未來學Jpg
成功學與未來學Jpg
 
Angeline
AngelineAngeline
Angeline
 
FunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - StreamsFunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - Streams
 
Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014
 
Evening At Soulfood Books
Evening At Soulfood BooksEvening At Soulfood Books
Evening At Soulfood Books
 
EFL Munich - February 2013 - "Conversational Big Data with Erlang"
EFL Munich - February 2013 - "Conversational Big Data with Erlang"EFL Munich - February 2013 - "Conversational Big Data with Erlang"
EFL Munich - February 2013 - "Conversational Big Data with Erlang"
 
Streamy, Pipy, Analyticy
Streamy, Pipy, AnalyticyStreamy, Pipy, Analyticy
Streamy, Pipy, Analyticy
 
QCON London 2013
QCON London 2013QCON London 2013
QCON London 2013
 
Big Data, Mob Scale.
Big Data, Mob Scale.Big Data, Mob Scale.
Big Data, Mob Scale.
 
Deconstructing Lambda
Deconstructing LambdaDeconstructing Lambda
Deconstructing Lambda
 
Data distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsData distribution in the cloud with Node.js
Data distribution in the cloud with Node.js
 
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
 
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...
KAJIAN TERHADAP AMALAN DAN TINGKAHLAKU PENGGUNA TERHADAP PENGGUNAAN LESTARI D...
 
The Normal Probability Distribution
The Normal Probability DistributionThe Normal Probability Distribution
The Normal Probability Distribution
 

Similar to Meta Programming with Streams and Pipes

Data Types Unit-5 IGCSE ICT
Data Types Unit-5 IGCSE ICTData Types Unit-5 IGCSE ICT
Data Types Unit-5 IGCSE ICTmkhisalg
 
Tools part 4
Tools part 4Tools part 4
Tools part 4MR Z
 
Output subsystem interfacing
Output subsystem interfacingOutput subsystem interfacing
Output subsystem interfacingmanojmanoj218596
 
"Taming the Dragon": Data Flow Programming with Zenoh-Flow
"Taming the Dragon": Data Flow Programming with Zenoh-Flow"Taming the Dragon": Data Flow Programming with Zenoh-Flow
"Taming the Dragon": Data Flow Programming with Zenoh-FlowZettaScaleTechnology
 
Streams in node js
Streams in node jsStreams in node js
Streams in node jsKushal Likhi
 
Exp2003 Int Ppt 01
Exp2003 Int Ppt 01Exp2003 Int Ppt 01
Exp2003 Int Ppt 01lonetree
 
Harry Potter and Enormous Data (Pavlo Baron)
Harry Potter and Enormous Data (Pavlo Baron)Harry Potter and Enormous Data (Pavlo Baron)
Harry Potter and Enormous Data (Pavlo Baron)Pavlo Baron
 
HSC Communication systems
HSC Communication systems HSC Communication systems
HSC Communication systems greg robertson
 
Transmitting And Receiving ( Sim And Han).Pptw.Quiz
Transmitting And Receiving ( Sim And Han).Pptw.QuizTransmitting And Receiving ( Sim And Han).Pptw.Quiz
Transmitting And Receiving ( Sim And Han).Pptw.Quizmfridolfson
 
Network visibility and control using industry standard sFlow telemetry
Network visibility and control using industry standard sFlow telemetryNetwork visibility and control using industry standard sFlow telemetry
Network visibility and control using industry standard sFlow telemetrypphaal
 
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...Getting Started with the NodeMCU- Getting started with Internet of Things (Io...
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...AkshetPatel
 
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB
 
Traffic analysis for Planning, Peering and Security by Julie Liu
Traffic analysis for Planning, Peering and Security by Julie LiuTraffic analysis for Planning, Peering and Security by Julie Liu
Traffic analysis for Planning, Peering and Security by Julie LiuMyNOG
 
Tracing Micro Services with OpenTracing
Tracing Micro Services with OpenTracingTracing Micro Services with OpenTracing
Tracing Micro Services with OpenTracingHemant Kumar
 
Deeplearning for industries | Data to Production
Deeplearning for industries | Data to ProductionDeeplearning for industries | Data to Production
Deeplearning for industries | Data to ProductionRahul Kumar
 
Cambridge IGCSE ICT Revision Notes
Cambridge IGCSE ICT Revision Notes Cambridge IGCSE ICT Revision Notes
Cambridge IGCSE ICT Revision Notes Yash Bhatt
 

Similar to Meta Programming with Streams and Pipes (20)

Unit 5
Unit 5Unit 5
Unit 5
 
Data Types Unit-5 IGCSE ICT
Data Types Unit-5 IGCSE ICTData Types Unit-5 IGCSE ICT
Data Types Unit-5 IGCSE ICT
 
Data Communication Basics
Data Communication BasicsData Communication Basics
Data Communication Basics
 
Tools part 4
Tools part 4Tools part 4
Tools part 4
 
Output subsystem interfacing
Output subsystem interfacingOutput subsystem interfacing
Output subsystem interfacing
 
"Taming the Dragon": Data Flow Programming with Zenoh-Flow
"Taming the Dragon": Data Flow Programming with Zenoh-Flow"Taming the Dragon": Data Flow Programming with Zenoh-Flow
"Taming the Dragon": Data Flow Programming with Zenoh-Flow
 
Streams in node js
Streams in node jsStreams in node js
Streams in node js
 
Exp2003 Int Ppt 01
Exp2003 Int Ppt 01Exp2003 Int Ppt 01
Exp2003 Int Ppt 01
 
Harry Potter and Enormous Data (Pavlo Baron)
Harry Potter and Enormous Data (Pavlo Baron)Harry Potter and Enormous Data (Pavlo Baron)
Harry Potter and Enormous Data (Pavlo Baron)
 
HSC Communication systems
HSC Communication systems HSC Communication systems
HSC Communication systems
 
Transmitting And Receiving ( Sim And Han).Pptw.Quiz
Transmitting And Receiving ( Sim And Han).Pptw.QuizTransmitting And Receiving ( Sim And Han).Pptw.Quiz
Transmitting And Receiving ( Sim And Han).Pptw.Quiz
 
Network visibility and control using industry standard sFlow telemetry
Network visibility and control using industry standard sFlow telemetryNetwork visibility and control using industry standard sFlow telemetry
Network visibility and control using industry standard sFlow telemetry
 
Stream Processing Overview
Stream Processing OverviewStream Processing Overview
Stream Processing Overview
 
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...Getting Started with the NodeMCU- Getting started with Internet of Things (Io...
Getting Started with the NodeMCU- Getting started with Internet of Things (Io...
 
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
 
Traffic analysis for Planning, Peering and Security by Julie Liu
Traffic analysis for Planning, Peering and Security by Julie LiuTraffic analysis for Planning, Peering and Security by Julie Liu
Traffic analysis for Planning, Peering and Security by Julie Liu
 
Tracing Micro Services with OpenTracing
Tracing Micro Services with OpenTracingTracing Micro Services with OpenTracing
Tracing Micro Services with OpenTracing
 
Deeplearning for industries | Data to Production
Deeplearning for industries | Data to ProductionDeeplearning for industries | Data to Production
Deeplearning for industries | Data to Production
 
Cisco Activity
Cisco ActivityCisco Activity
Cisco Activity
 
Cambridge IGCSE ICT Revision Notes
Cambridge IGCSE ICT Revision Notes Cambridge IGCSE ICT Revision Notes
Cambridge IGCSE ICT Revision Notes
 

Recently uploaded

🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024The Digital Insurer
 
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Principled Technologies
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 

Recently uploaded (20)

🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 

Meta Programming with Streams and Pipes

  • 1.
  • 2.   Essentially, programs that can be created by themselves.   We’re twisting the term a little for streams   We will be defining streaming constructs using streaming constructs   By streams I mean beams which are just streams and pipes for compute as opposed to IO
  • 3. Any info? Please contact #nodeland police at @cianomaidin on twitter…
  • 4.   Readable Streams   IO / throughput oriented   Events – data, end, error, close   Methods – pause, resume, end, destroy   Writeable Streams   IO / throughput oriented   Events - drain, pause, resume, error, close   Methods – write, end, destroy
  • 5.   IO Oriented   Finite, not infinite   Asynchronous   Lazy   Assumes in transit data, not in memory forms   Great for IO. Not efficient for compute
  • 6.
  • 7.
  • 8.   Sources   Where data pours in   Sinks   Where results pop out   Operators         Filters – Drop or pass on data based on a UDF Transforms – Mutate data based on a UDF Branch with Filters Combine with Sinks
  • 9.   Compute Oriented   Infinite, not finite   Synchronous   Eager   Assumes in memory form, not encoded or wire data   Convenient for compute. Not good for IO
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.   Eager: a.pipe(operation).pipe(z).pipe(a);   Implies: RangeError: Maximum call stack size exceeded
  • 16.
  • 17.
  • 18.   npm install beam       Compute oriented streams Branch, Combine, Filter, Transform data Extend with goto and modules.   npm install eep           Beam will be integrated into eep soon Eep provides aggregate windows Slide, Tumble, Monotonic, Periodic Stats Library Ordering now supported   npm install nodesapiens   Robosapiens V1 for node.js   Code on github   https://github.com/darach