** Video of this talk is here: https://youtu.be/MQGXrrhGUTw ** The first talk of the Meetup on the 11th of April 2017, hosted by weeronline.nl in their Amsterdam offices. Streams are everywhere! Akka Streams help us model streaming processes using a very descriptive DSL and optimising resource usage.

1. Akka Streams at Weeronline

2. A little info about myself
Tech Lead /
Software Architect
@ticofab

3. Streams, streams everywhere!
Many processes can be modelled as a stream.

4. Streams, streams everywhere!
Many processes can be modelled as a stream.

5. Streams, streams everywhere!
Many processes can be modelled as a stream.

7. What is a stream?
A potentially unbounded stream of data that flows in
a sequential fashion from a Producer to a Consumer.
• Challenge 1: the consumer cannot predict how much
data it will need to handle
• Challenge 2: the speed of production and
consumption can be different

8. What is Backpressure?
Two entities exchange items.
100 op/sec 10 op/sec
FAST SLOW
buffer
overflow!
Increase buffer size
Drop + resend (TCP)
…

9. What is Backpressure?
The receiver requests items that he knows he can handle.

10. Advantages of applying back pressure
Sender can do other things while there are no requests
No overflowing buffers (the website will be slower but will stay up)
Less network traffic
…

11. What is Backpressure?
You “apply backpressure” when the receiver is able to say
to the sender
“SLOW DOWN DUDE”

12. What is Backpressure?

13. What is Backpressure? (extra)
TOP DEFINITION
Pressure
A name for weed. A South Florida term.
"Yo, you got da pressure"
"Im rollin up da pressure"

14. The Reactive Streams standard
• “Reactive Streams” is a standardisation of dynamic
back pressure.
• Defines a minimal interface to make different systems
communicate with built-in back pressure.
• Applies to both runtime environments (JVM) &
network protocols.
http://www.reactive-streams.org

15. The Reactive Streams standard
You don’t need to know the internal semantics
(unless you want to provide an implementation)

16. Akka Streams
Reactive Streams interface
Akka Streams implementation
Akka Streams API

17. Akka Streams
• A producer (Source)
• A consumer (Sink)
• An optional transformation (Flow)
Every stream is composed by:

18. Basic Sources
Source(List(1, 2, 3))
Source.fromFuture(futureResult)
Source.single("one single element”)
Source.tick(initialDelay, interval, item)

19. Basic Sinks
Sink.ignore
Sink.foreach(println)
Sink.fold[Int, Int](0)(_ + _)

20. The simplest stream (no Flow)
Source(1 to 10)
.to(Sink.foreach(println))
.run()
Source(1 to 10)
.runWith(Sink.foreach(println))
Source(1 to 10)
.runForeach(println)

21. The Simplest Stream (with Flow)
Source(1 to 100)
val multiplier = Flow[Int].map(_ * 2)
.runForeach(println).via(multiplier)

22. Source.tick(0.second, 1.second, randomString())
.via( )
.runForeach(println)
Akka Streams - More complex graphs
Broadcast
Append
“Fabio” to
string
Append
“Reactive”
to String
Evaluate
string
length
kindaUselessFlow
Print string

23. Akka Streams - More complex graphs
val myFlow: Flow[String, Int, NotUsed] = Flow.fromGraph(
GraphDSL.create() {
implicit builder =>
val fabioAppender = builder.add(Flow[String].map(s => s + "Fabio"))
val reactiveAppender = builder.add(Flow[String].map(s => s + "Reactive"))
val lengthFlow = builder.add(Flow[String].map(s => s.length))
val printSink: Sink[String, Future[Done]] = Sink.foreach(println)
val bcast = builder.add(Broadcast[String](2))
bcast ~> fabioAppender ~> lengthFlow
bcast ~> reactiveAppender ~> printSink
FlowShape(bcast.in, lengthFlow.out)
})

24. Akka Streams - More complex graphs
bcast ~> fabioAppender ~> lengthFlow
bcast ~> reactiveAppender ~> printSink
FlowShape(bcast.in, lengthFlow.out)
Broadcast
Append
“Fabio” to
string
Append
“Reactive”
to String
Evaluate
string
length
Print string

25. Data processing at Weeronline
KNMI
Royal Dutch
Meteorological
Institute

26. First approach: Actors
Processing
Supervisor
Filtering
Supervisor
Imaging
Supervisor
Output
Supervisor
Filter
Task 2
Filtering
Task 1
… Image sub
team
supervisor
Imaging sub
team 2
supervisor

27. First approach: Actors
• Difficult to control resources (read: OOM exception)
• LOTS of messages, difficult to track and interpret logs
• Cascading timeouts

28. Second approach: Streams
KNMI
Data
Partition
Conversion
Conversion
Matrix
evaluation
Matrix
evaluation
Scan Flow
Forecast Flow
Merge Complete

29. Second approach: Streams
Broadcast
GIF
GeoTIFF 1 Collection
Upload
Zip Complete
Another
Matrix
GeoTIFF 2
Broadcast
Upload
Upload
CSV 1
CSV 2
Upload
Upload
Group
Group
Zip
Group
Collection

30. Advantages of Akka Streams
Average RAM usage down 40%
Back pressure is built in
Powerful native shapes (grouping, fan in/out..)
Debugging is a lot easier
Easier to handle failures (where is the problem?)
Reasonings about it is easier (overview!)

31. Lesson learned by others
https://tech.zalando.com/blog/comparing-akka-streams-actors-and-plain-futures
https://softwaremill.com/replacing-akka-actors-with-akka-streams/
https://sap1ens.com/blog/2015/10/25/actors-streams-and-futures-in-akka/

32. Questions?
@ticofab
Thanks!