Flow-based programming with Elixir

1. Flow-based
programming
with Elixir
3rd Kyiv Elixir meetup, October 1, 2016

2. Hello!
I am Anton Mishchuk
Ruby developer at Matic
Elixir fan
ESpec’s author and maintainer
Github: antonmi
2

3. Introduction
◈ Elixir new awesome feature - GenStage
◈ Elixir Flow-Based Programming (Peter C Marks)
◈ Flow-Based Programming, 2nd Edition
(J. Paul Morrison)
3

4. What I’d tell you
◈ “Conventional programming”
◈ Flow-Based Programming
◈ “Telegram problem” example
◈ Elixir GenStage
◈ “Telegram problem” redesign
◈ Data-flow routing with GenStage
4

5. 1.
Software design in
conventional programming
5
Warning! Too general philosophy!

6. Micro level
6
◈ Basic components:
○ Objects in OOP comprising both
information and behaviour
○ Functions and data-structures in FP
◈ Hierarchical organization
◈ Sequential communication or
evaluation

7. DSLs and Frameworks
as “coordination languages”
◈ Define common abstractions
◈ Initialize your components
◈ Use components in right order
7

8. Visualization
8

9. Concurrency
& parallelism
9

10. ◈ Hierarchic structure of program
◈ Code is procedural and sequential
◈ Visualisation is more about structure
◈ Concurrency is not native
What is wrong
with conventional programming?
10

11. Basic concepts
2.
Flow-Based Programming
11

12. “FBP - a programming paradigm that defines
applications as networks of "black box" processes,
which exchange data across predefined connections by
message passing, where the connections are specified
externally to the processes. These black box processes
can be reconnected endlessly to form different
applications without having to be changed internally.
https://en.wikipedia.org/wiki/Flow-based_programming
12

13. FBP
diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
Processes, ports, connections
13

14. “... whereas the conventional approaches to
programming start with process and view data as
secondary, business applications are usually
designed starting with data and viewing process as
secondary – processes are just the way data is
created, manipulated and destroyed.
J. Paul Morrison, Flow-Based Programming, 2nd Edition
14

15. Express a problem in
terms of transforms on
streams of data
15

16. Soft Drink Bottling Factory
◈ Independent well-defined components
◈ Clean interfaces
◈ Simple to reconfigure
◈ Minimizes side-effects
◈ Designer can sit at one “station”, or can
follow an item through system
http://www.jpaulmorrison.com/fbp/FBPnew.ppt 16

17. FPB characteristics
◈ Asynchronous processes communicating
via streams of data packets
◈ Data packets with a lifetime of their own
◈ Definition of connections external to
components
◈ Consistent view from macro to micro
http://www.jpaulmorrison.com/fbp/FBPnew.ppt 17

18. Native parallelism
18

19. FBP is about
“coordination
language”
19

20. Everything new is actually well-forgotten old
◈ 1971, IBM, basic concepts
◈ 1975, Bank of Montreal, on-line banking
system, still works!
◈ 1994, Flow-Based Programming, 1st Ed
◈ 2010, Flow-Based Programming, 2st Ed
◈ 2012, NoFlo project by Henri Bergius,
FBP in JavaScript
◈ JavaFBP, C#FBP, CppFBP (C++ and Lua)
20

21. What about
Erlang/Elixir?
21

22. FBP example with Elixir
3.
Telegram problem
22

23. Telegram problem
A program which accepts lines of text and
generates output lines of a different
predefined length, without splitting any of
the words in the text.
The program accepts an input stream of
lines of some length and produce an
output stream of lines of another length.
23

24. Problem FBP design using GenServers
ReadSeq Recompose WriteSeqDecompose
Lines Words Lines
◈ Each component is a GenServer.
◈ Each component implements only
one function - “run” which transforms
input to output
◈ Communication is asynchronous
◈ Code is here:
https://github.com/antonmi/kyiv_meetup_3
24

25. ReadSeq (naive implementation)
25

26. Decompose (naive implementation)
26

27. Recompose (naive implementation)
27

28. WriteSeq (naive implementation)
28

29. The main Telegram module
(naive implementation)
29

30. What is cool in this simple implementation
◈ Each component do its own part of work
◈ The only API is “init” and “run”
◈ Components works in parallel
◈ Components are under supervisor
30

31. What is wrong with the implementation
◈ Components are not independent (each
of them knows who is the next)
◈ Flow is controlled internally by each of
the component
◈ There is no back-pressure mechanism
(when some component lags it can be
overflowed by data)
31

32. Experimental feature.
4.
Elixir GenStage
32

33. GenStage abstractions
◈ GenStage is build on top of GenServer
◈ Each of the stage can be producer,
consumer or both
producer
producer
consumer
consumer
producer
consumer
33

34. GenStage communication
◈ Consumer subscribes to producer
◈ Consumer asks for data (sends a demand)
◈ Producer sends data
producer consumer
Subscribe
Ask
Events
34

35. GenStage API
◈ init(state) which must return:
○ {:producer, state}
○ {:producer_consumer, state}
○ {:consumer, state}
◈ handle_demand(events, from, state)
○ {:noreply, events, new_state}
◈ handle_events(events, from, state)
○ {:noreply, events, new_state}
35

36. 5.
Telegram problem
redesign
36

37. ReadSeq GenStage
37

38. Decompose GenStage
38

39. Recompose GenStage
39

40. WriteSeq GenStage
40

41. Main Telegram module
41

42. It is awesome!
◈ Each component is independent!
◈ Data flow is coordinated externally!
◈ There is a back-pressure!
42

43. 5.
Data-flow routing
with GenStage
43

44. GenStage Dispatchers
◈ GenStage.DemandDispatcher
○ sends events to the highest demand
◈ GenStage.BroadcastDispatcher
○ accumulates demand from all
consumers before broadcasting events
to all of them
◈ GenStage.PartitionDispatcher
○ sends events according to partitions
44

45. Duplicate file example
45
ReadSeq
WriteSeq
Broadcast
WriteSeq
Code is here:
https://github.com/antonmi/kyiv_meetup_3

46. The only new component is Broadcast
46

47. Split file example
47
Code is here:
https://github.com/antonmi/kyiv_meetup_3
ReadSeq
WriteSeq
Split
WriteSeq

48. The only new component is Split
48

49. 6.
Conclusion
49

50. Erlang VM
+
Elixir GenStage
=
FBP paradigm
50

51. Thanks!