Slides from my talk at QCon London on 5th of March 2019. More information can be found here: https://berndruecker.io/complex-event-flows-in-distributed-systems/ Abstract: Event-driven architectures enable nicely decoupled microservices and are fundamental for decentral data management. However, using peer-to-peer event chains to implement complex end-to-end logic crossing service boundaries can accidentally increase coupling. Extracting such business logic into dedicated services reduces coupling and allows to keep sight of larger-scale flows - without violating bounded contexts, harming service autonomy or introducing god services. Service boundaries get clearer and service APIs get smarter by focusing on their potentially long running nature. I will demonstrate how the new generation of lightweight and highly-scalable state machines ease the implementation of long running services. Based on my real-life experiences, I will share how to handle complex logic and flows which require proper reactions on failures, timeouts and compensating actions and provide guidance backed by code examples to illustrate alternative approaches.

1. Complex event flows in
distributed systems
@berndruecker

2. 3 common hypotheses I check today:
# Events decrease coupling
# Orchestration needs to be avoided
# Workflow engines are painful
@berndruecker

3. Warning:
Contains Opinion

4. Berlin, Germany
mail@berndruecker.io
@berndruecker
Bernd Ruecker
Co-founder and
Chief Technologist of
Camunda

6. Simplified example:
dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0
license. https://www.flickr.com/photos/0xf2/29873149904/
@berndruecker

7. Three steps…
@berndruecker

8. Who is involved? Some bounded contexts…
Checkout
Payment
Inventory
Shipment
@berndruecker

9. (Micro-)services
Checkout
Payment
Inventory
Shipment
@berndruecker

10. Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
Dedicated Development Teams
@berndruecker

11. Events decrease coupling
@berndruecker

12. Example
Checkout
Payment
Inventory
Shipment
The button blinks if we can
ship within 24 hours
@berndruecker

13. Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
The button blinks if we can
ship within 24 hours
@berndruecker

14. Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
Good
Stored
Read
Model
Good
Fetched
The button blinks if we can
ship within 24 hours
*Events are facts about what happened (in the past)
@berndruecker

15. Order
Placed
Payment
Received
Goods
Fetched
Notification
Checkout
Payment
Inventory
Shipment
Event-driven architecture
@berndruecker

16. Events can decrease coupling*
*e.g. decentral data-management, read models,
extract cross-cutting aspects
@berndruecker

17. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

18. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

19. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

20. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

21. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

22. Monitoring Workflows Across Microservices
https://www.infoq.com/articles/monitor-workflow-collaborating-microservices
@berndruecker

23. Typical approachesDistributed Tracing
Data Lake / Event Monitoring
Process Mining
Process Tracking
@berndruecker

24. Stefan Tilkov: Microservice Patterns & Antipatterns - MicroXchg 2018
@berndruecker

25. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment
@berndruecker

26. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped
@berndruecker

27. Photo by born1945, available under Creative Commons BY 2.0 license.
@berndruecker

28. What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
@berndruecker

29. „Challenges?“
Source:
Microservices orchestration survey,
July 2018, 354 responses
https://camunda.com/microservices-orchestration-survey-results-2018/
@berndruecker

30. Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs to happen in the future
Payment
received
Order
placed
Retrieve
payment
@berndruecker

31. Order
It is about where to decide about the coupling!
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
Order decides
. to listen to the event
. to issue the command
@berndruecker

32. Order
It is about where to decide about the coupling!
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
It can still be messaging!
@berndruecker

33. Commands help to avoid (complex)
peer-to-peer event chains
@berndruecker

34. Orchestration needs to be avoided
@berndruecker

35. Smart ESB-like middleware
Checkout
Payment
Inventory
Shipment
Order
Order
placed
Payment
received
Good
fetched
Good
shipped
@berndruecker

36. Dumb pipes
Checkout
Payment
Inventory
Shipment
Order
Smart endpoints
and dumb pipes
Martin Fowler
@berndruecker

37. Danger of god services?
Checkout
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann
Payment
Inventory
Shipment
@berndruecker

38. Danger of god services?
Checkout
Payment
Inventory
Shipment
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann
@berndruecker

39. A god service is only created
by bad API design!
@berndruecker

40. Example
Order Payment
Retrieve
Payment
@berndruecker

41. Example
Order Payment
Credit
Card
Retrieve
Payment
@berndruecker

42. Example
Order Payment
Credit
Card
Retrieve
Payment
Rejected
@berndruecker

43. Example
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected
@berndruecker

44. Example
Order Payment
Client of dumb endpoints easily become a god services.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected
@berndruecker

45. Payment
failed
Who is responsible to deal with problems?
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received
@berndruecker

46. Payment
failed
Long running services
Order Payment
Credit
Card
Retrieve
Payment
Rejected
Payment
received
Smart endpoints are
potentially long-running
@berndruecker

47. Persist thing
(Entity, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State
@berndruecker

48. Workflow engines are painful
Complex, proprietary, heavyweight, central, developer adverse, …
@berndruecker

49. Avoid the wrong tools!
Death by properties panel
Low-code is great!
(You can get rid
of your developers!)
Complex, proprietary, heavyweight, central, developer adverse, …
@berndruecker

50. Workflow engines,
state machines
It is
relevant
in modern
architectures
@berndruecker

51. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines
@berndruecker

52. CADENCE
Workflow engines,
state machines
@berndruecker

53. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“
@berndruecker

54. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)
@berndruecker

55. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker

56. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker

57. BPMN
Business Process
Model and Notation
ISO Standard
@berndruecker

58. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…
@berndruecker

59. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through
@berndruecker

60. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances
@berndruecker

61. Payment
Now you have a state machine!
@berndruecker

62. Payment
Easy to handle time
@berndruecker

63. Distributed
systems
@berndruecker

64. Example with synchronous communication
REST
Order Payment
Credit
Card
@berndruecker

65. Example with synchronous communication
REST
Order Payment
Credit
Card
@berndruecker

66. Example with synchronous communication
REST
Order Payment
Credit
Card
@berndruecker

67. Example with synchronous communication
REST
Order Payment
Credit
Card
Stateful
Retry
@berndruecker

68. Works also for asynchronous communication
Order Payment
Credit
Card
Monitor
Timeouts
@berndruecker

69. Distributed
systems
@berndruecker

70. It is impossible to
differentiate certain
failure scenarios:
Independant of
communication style!
Service
Provider
Client
@berndruecker

71. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST
@berndruecker

72. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST
The service can be
long running.
You get a better API and less gods
@berndruecker

73. Workflows live inside service boundaries
@berndruecker

74. No BPM(N) monoliths
https://blog.bernd-ruecker.com/avoiding-the-bpm-monolith-when-using-bounded-contexts-d86be6308d8
@berndruecker

75. Pat Helland
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce
@berndruecker

76. Pat Helland
Grown-Ups Don’t Use
Distributed Transactions
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce
@berndruecker

77. Distributed transactions using compensation *
Compensation
@berndruecker

78. Homework:
Try to do this purely event-driven!
Send to: mail@berndruecker.io
@berndruecker

79. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
improve
communication
improve
communication
Ops
@berndruecker

80. Visual HTML reports for test cases
@berndruecker

81. Living documentation for long-running behaviour
@berndruecker

82. Proper
Operations
Visibility + Context
@berndruecker

83. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Operate with visibility
and context
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops
@berndruecker

84. Monitoring Workflows Across Microservices
https://www.infoq.com/articles/monitor-workflow-collaborating-microservices
@berndruecker

85. Tracking
Checkout Inventory
Payment Shipment
Event Bus
Workflow
Engine
https://www.confluent.io/kafka-summit-sf18/the_big_picture @berndruecker

86. Journey towards more orchestration
@berndruecker

87. Vodafone, Liongate & WDW
Presented at CamundaCOn Berlin 2018

88. Before mapping processes
explicitly with BPMN, the truth was
buried in the code and nobody
knew what was going on.
Jimmy Floyd, 24 Hour Fitnesse
„
@berndruecker

89. …
It addresses one of the core issues in a distributed
microservices architecture—where is the source of
truth for the coordinated interaction of the
entire system?
…
the system we are replacing uses a complex peer-
to-peer choreography that requires reasoning
across multiple codebases to understand.
https://medium.com/@sitapati/node-js-client-for-zeebe-microservices-orchestration-engine-72287e4c7d94
Josh Wulf
Credit Sense
@berndruecker

90. Lightweight workflow engines are
great – don‘t DIY*
*e.g. enabling potentially long-running services, solving hard
developer problems, can run decentralized
@berndruecker

91. Sales-Order & Order-Fulfillment
via Camunda
for every order worldwide
(Q2 2017: 22,2 Mio)
@berndruecker

92. Code, code, code…
@berndruecker

93. Event-driven example
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/berndruecker/flowing-retail/
Human
Tasks
H2 H2
@berndruecker

94. # Events decrease coupling: sometimes
read-models, but no complex peer-to-peer event chains!
# Orchestration needs to be avoided: sometimes
no ESB, smart endpoints/dumb pipes, balance orchestration and choreography
# Workflow engines are painful: some of them
lightweight engines are easy to use and can run decentralized,
they solve hard developer problems, don‘t DIY
@berndruecker

95. Thank you!
@berndruecker

96. mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-
workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/