1.
The Benefits of Orchestration
in Distributed Systems
@NiallDeehan

2.
Who I am?
Niall Deehan

3.
Distributed systems
Challenges of
asynchronicity
Distributed
Transactions
Communication
is complex

4.
Do A
Do B
All or
nothing
+
try {
tx.begin();
doA();
doB();
tx.commit();
} catch (Exception e) {
tx.rollback();
}
@Transactional
public void createCustomer(Customer cust) {
// ...
}
Or simply:
Once upon a time:

5.
A
C
I
D
Atomicity
Consistency
Isolation
Durability

6.
Distributed systems

7.
Distributed systems

8.
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Failure will happen.
Accept it!
But keep it local!
Be resilient.

9.
Communication
is complex

10.
Let‘s start with a simple example
Credit
Card
Payment
REST

11.
Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV1.java

12.
It‘s a blocking request.
It doesn‘t scale

13.
Starbucks does not use two phase commit
Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html
Photo by John Ingle

14.
That means
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Local
ACID
Local
ACID
1 (micro-)service
1 aggregate
1 program
1 resource
Violates „I“
of ACID

15.
Youmightknowthisfrom:
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Photo by Gerhard51, available under Creative Commons CC0 1.0 license.

16.
It is impossible to
differentiate certain
failure scenarios:
Independant of
communication style!
Service
Provider
Client

17.
Network problems
Credit
Card
Payment
charge

18.
Let‘s start with a simple example
Credit
Card
Payment
REST

19.
Circuit
Breaker
Photo by CITYEDV, available under Creative Commons CC0 1.0 license.

20.
Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV2.java

21.
Fail fast
is important

22.
Fail fast
is important
but not enough!

23.
Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

24.
Photo by Tookapic, available under Creative Commons CC0 1.0 license.

25.
Translation :
„There was an error
while sending your
boarding pass“

26.
Check-in
Web-UI
Bernd
Current situation

27.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Bernd

28.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Bernd

29.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Circuit
breaker
Bernd

30.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Bernd

31.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Bernd

32.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Stateful
Retry
Bernd

33.
Translation:
It‘s your problem
now.

34.
We are having some technical
difficulties and cannot present you
your boarding pass right away.
But we do actively retry ourselves, so
lean back, relax and we will send it
on time.

35.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Possible situation – much better!
Bernd

36.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Possible situation – much better!
Stateful
Retry
Bernd

37.
Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Stateful
Retry
Possible situation – much better!
The failure
never leaves
this scope!
Bernd

38.
Persist thing
(Entity, Document, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Complex, proprietary,
heavyweight, slow,
don‘t scale,
developer adverse
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

39.
Warning:
Contains Opinion

40.
Workflow engines,
state machines
It is
relevant
in modern
architectures

41.
What about open source…?

42.
CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

43.
What about purpose built and open source…

44.
CADENCE
Workflow engines,
state machines

45.
CADENCE
also at scale
Workflow engines,
state machines

46.
CADENCE
for todays demo
Workflow engines,
state machines

47.
Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV3.java

48.
Challenges of
asynchronicity

49.
Payment
Now you have a state machine!
Credit
Card
REST

50.
has to implement
Retry
has to implement
Idempotency
Client Service Provider

51.
We are processing your payment.
Do not leave this page.
And for god sake – do not reload!
It is a business
problem
anyway!

52.
We are processing your payment.
Do not leave this page.
And for god sake – do not reload!
It is a business
problem
anyway!
We are currently processing your request.
Don‘t worry, it will happen safely –
even if you loose connection.
Feel free to reload this page any time!

53.
Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

54.
Distributed
systems

55.
It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

56.
Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

57.
Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

58.
Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Workflows live within
service boundaries

59.
„The customer wants a synchronous response“
Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
„Eh – no!“

60.
generateBoardingPass
HTTP 200 OK
HTTP 202 ACCEPTED
Check-in
A synchronous response is possible in the
happy case, otherwise it is switched to
asynchronous processing.

61.
Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV4.java

62.
Challenges of
asynchronicity

63.
Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Asynchronous communication
You need to
monitor
timeouts

64.
Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Remember…
The failure
never leaves
this scope!

65.
Workflow…

66.
Workflow…

67.
BPMN
Business Process
Model and Notation
ISO Standard

68.
BPMN
Executable
and mature
Easy to
understand*
Widespread
*(for Biz and Dev)

69.
Graphical models?

70.
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

71.
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

72.
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

73.
Living documentation for long-running behaviour

74.
Visual HTML reports for test cases

75.
Proper
Operations
Visibility + Context

76.
Compare to e.g. Step Functions

77.
Imagine more complex stuff

78.
has to implement
Timeout, Retry
has to implement
Idempotency
Client Service Provider

79.
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

80.
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

81.
Avoids thundering herds
Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.

82.
Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.
https://aws.amazon.com/de/message/5467D2/

83.
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

84.
Distributed
Transactions
Distributed
Transactions

85.
Compensation – the classical example
Saga
book
hotel
book
car
book
flight
cancel
hotel
cancel
car
1. 2. 3.
5.6.
In case of failure
trigger compensations
book
trip

86.
2 alterntive approaches: choreography & orchestration

87.
Event-driven choreography
Hotel
Flight
Car
Trip
Trip
booked
Flight
booked
Trip
requested
Hotel
booked
Car
booked
Request
trip

88.
Event-driven choreography
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
Perform undo
(cancel car booking)
Perform undo
(cancel hotel)

89.
Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!

90.
Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
You have to adjust all services and redeploy at the same time!
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!

91.
What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License

92.
Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Book
flight

93.
Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Book
flight
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!
You have to adjust one service and redeploy only this one!

94.
Describe orchestration with BPMN
Trip
Trip
booked
Request
trip

95.
The workflow is part of the service
Trip

96.
The workflow is part of the service
Trip
Payment

97.
Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV6.java

98.
has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service Provider

99.
has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service ProviderDon‘t forget
about state

100.
# Be aware of complexity of distributed systems
# Know strategies and tools to handle it
e.g. Circuit breaker (Hystrix)
Workflow engine for stateful retry, waiting, timeout
and compensation (Camunda)

101.
Thank you!