Elixir Club 5

1. Flowex - Flow-Based
Programming with
Elixir GenStage
Elixir Club 5, Kyiv, January 28, 2017

2. - Ruby developer at Matic Insurance
- Elixir fan
- Author and maintainer of
ESpec BDD test framework
github: antonmi
Hello!
I am Anton Mishchuk
2

3. The problem
◎ There are some problems in
“conventional programing”:
- code is procedural and sequential
- parallelism is not native
- hierarchical structure of program
- visualisation is more about structure
◎ We need alternative approaches for
building our programs to make software
better!
3

4. My previous FBP talk
4
http://www.slideshare.net/AntonMishchuk
/flowbased-programming-with-elixir

5. The goal of the talk
◎ Discuss a special case of Flow-Based
Programming: “Railway FBP” (R-FBP)
◎ Present Flowex library which helps to
create R-FBP abstractions
◎ Show that Elixir GenStage is not only about
data-processing but about software design
5

6. What will be about
◎ Flow-Based Programming (FBP)
◎ Railway Oriented Programming (ROP)
◎ Railway FBP
◎ Flowex library
Lots of pictures!
6

7. 1.
Flow-Based
Programming
Basic concepts
7

8. “
FBP - a programming paradigm
that defines applications as
networks of "black box" processes,
which exchange data across
predefined connections by
message passing
J. Paul Morrison, Flow-Based Programming, 2nd Edition
8

9. FBP diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
9

10. Express a problem in terms of
transforms on streams of data
http://www.jpaulmorrison.com/fbp/FBPnew.ppt
10

11. Design pros
◎ Independent and reusable components
◎ Clean interfaces
◎ Simple to reconfigure
◎ Minimizes side-effects
◎ Designer can sit at one “station”, or can
follow an item through system
http://www.jpaulmorrison.com/fbp/FBPnew.ppt
11

12. “Native parallelism”
12

13. 2.
Railway Oriented
Programming
Design pattern
13

14. Express a problem in terms of
sequence of functions calls
request
|> validate_request
|> get_user
|> update_db_from_request
|> send_email
|> return_http_message
14

15. Define common interface for
successful and error cases
◎ For example use:
{:ok, "data"} or {:error, "Failure reason"}
◎ Each function returns {:ok, "data"} or
{:error, "Failure reason"}
◎ If function is called with {:error, "Failure
reason"} it just bypasses it
15

16. Define interface to bypass errors
http://fsharpforfunandprofit.com/rop/
16

17. So the program is a railway
from input to output
http://fsharpforfunandprofit.com/rop/
17

18. Elixir Plug is a good example!
◎ Each “plug” function receives and returns a
“connection structure” %Plug.Conn{}
◎ %Plug.Conn{} contains:
assigns, cookies, halted, host, method,
params, resp_body, resp_cookies, status,
and many other attributes
18

19. 3.
Railway FBP
ROP + FBP
19

20. Let’s transform each part of
“Railway” into FBP component!
20

21. Place each function into separate process
validate send_emailupdate_db
That is the idea!
21

22. Why Elixir?
22

23. GenStage
◎ Easiest way to implement a chain of
communicating processes
◎ Back-pressure mechanism
producer consumer
Subscribe
Ask
Events
23

24. Metaprogramming
◎ Simple way for sharing functionality
◎ Ability to create expressive DSL
24

25. 4.
Flowex
Railway FBP
25

26. Consider an example!
26

27. Calculate (number + 1) * 2 - 3
27

28. Note, the functions are cool!
◎ They have the same interface
◎ They operate with predefined struct
We can join them into pipeline:
28

29. Let’s add some Flowex magic!
29

30. … and rename the module to FunPipeline
30

31. “use Flowex.Pipeline”
◎ Adds ‘pipe’ macro which allows to mark
functions “to be placed” into separate
Genstage
◎ Defines ‘start’ and ‘stop’ functions for
creating and destroying “Flowex pipelines”
◎ Defines ‘run’ function to perform
calculations
31

32. Start FunPipeline!
32

33. FunPipeline instance
add_one minus_threemult_by_twoproducer consumer
Supervisor
Elixir GenStages
33

34. %Flowex.Pipeline{} struct
◎ module - the name of the module
◎ in_name - unique name of 'producer'
◎ out_name - unique name of 'consumer'
◎ sup_pid - pid of the pipeline supervisor
34

35. Run calculations via “run”
Using FunPipeline.run/2 function
Note, ‘a’, ‘b’ and ‘c’ were set!
35

36. Run calculations using Flowex.Client
Using Flowex.Client module
36

37. How it works in details
add_one
minus_three
mult_by_two
producer
consumer
Flowex.Client
%FunPipeline{number: 2}
%Flowex.IP{
struct: %FunPipeline{number: 2},
...}
number: 2
number: 2
number: 3
number: 6
number: 3
number: 3
self()
%FunPipeline{number: 3}
37

38. What is sync and what is async?
Client
Asynchronous
communicationself()
Synchronous
communication
38

39. Is there no parallelism?
39

40. There can be a lot of clients!
Client
Client
Client
40

41. Bottlenecks.
What if there is very slow
process in the pipeline?
41

42. Just small changes in the code
42

43. And you get this
add_one
minus_three
mult_by_twoproducer consumer
mult_by_two
minus_three
mult_by_two
43

44. Reusable components with
“module pipelines”
44

45. Module must implement just two functions
Like Elixir Plug module
45

46. Pipeline module
46

47. Conclusion
47

48. Why it is cool!
48

49. It is easy to understand
◎ Pipelines explicitly define a
structure of data will be processed
◎ Pipelines explicitly define a
way the data will come
◎ Pipelines explicitly define a
parallel executors structure
49

50. It is easy to maintain and reuse
◎ There is a predefined set of working
processes (components) in a program
◎ Each component is isolated
◎ Pipelines can (and should) reuse
components
50

51. It is about controlled parallelism
◎ One can controll number of clients
supplying data to a pipeline
◎ One can control the number of processes
available for each component
51

52. Thank you! Questions?
52
You can star the project here:
https://github.com/antonmi/flowex