TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
Meta Programming with Streams and Pipes
1.
2. Essentially, programs that can be created by
themselves.
We’re twisting the term a little for streams
We will be defining streaming constructs using
streaming constructs
By streams I mean beams which are just streams and
pipes for compute as opposed to IO
3. Any info? Please contact #nodeland police at @cianomaidin on twitter…
5. IO Oriented
Finite, not infinite
Asynchronous
Lazy
Assumes in transit data, not in memory forms
Great for IO. Not efficient for compute
6.
7.
8. Sources
Where data pours in
Sinks
Where results pop out
Operators
Filters – Drop or pass on data based on a UDF
Transforms – Mutate data based on a UDF
Branch with Filters
Combine with Sinks
9. Compute Oriented
Infinite, not finite
Synchronous
Eager
Assumes in memory form, not encoded or wire data
Convenient for compute. Not good for IO