Process automation technology progressed massively over the last years and can handle modern scalability and resiliency requirements. It is not the dinosaur some people think it is! Now there are super interesting combinations between Kafka and a workflow engine possible, such as starting or influencing process instances upon certain events in a stream, orchestrating microservices by writing commands as Kafka records, or using Kafka to collect information about decentralized process executions. In this talk, I will briefly walk you through these possibilities presenting real-life use cases and some sample code using open source tools.

1. Apache Kafka Meets
Workflow Engines
Bernd Ruecker
Co-Founder and Chief Technologist
@Camunda
@berndruecker

2. Workflow engines.
Really?
https://www.flickr.com/photos/dullhunk/28251201308

3. There are pretty amazing use cases
@berndruecker
Order Management in Telco
5,000 process instances/s
Money Transfer
200 process instances/s
Perseverance Mars Rover
Nasa

4. Customer onboarding example
@berndruecker

5. Customer onboarding example
@berndruecker

6. Scale
Onboarding
you might
think of
Real-life
onboarding use
case in banking
Money transfer
or trading
Process Instances
per second
@berndruecker

7. Making phone
calls
Scale
Telco
Onboarding
Process Instances
per second
@berndruecker

8. A process
Steps
In a certain
sequence
Which might be
long running
@berndruecker
Time-controlled
events

9. A workflow engine
Or more scalable:
@berndruecker

10. A workflow engine
Or more scalable:
@berndruecker
Camunda
Platform
Camunda
Cloud
Two workflow engines:
- RDBMS / traditional & on-prem
- Event-Sourcing / cloud-native

11. https://blog.bernd-ruecker.com/zeebe-
io-a-horizontally-scalable-distributed-
workflow-engine-45788a90d549
“Zeebe” is the workflow engine offered
within Camunda Cloud

12. Workflow engines at scale
https://camunda.com/blog/2019/08/zeebe-horizontal-scalability/
@berndruecker

13. Show me code!
https://github.com/berndruecker/kafka-camunda-spring-simple

14. Technical
Example
Java Spring Boot
Application
Camunda Cloud
Confluent Cloud
Kafka Topic
Zeebe
Worker
Via subscription
Publish record
Kafka
Listener
Via subscription
Correlate to process instance
https://github.com/berndruecker/kafka-camunda-spring-simple
@berndruecker

15. Kafka Connect

16. Why?
Some use cases…
@berndruecker

17. Microservices
Checkout
Payment
Inventory
Shipment
@berndruecker

18. Order
Placed
Payment
Received
Goods
Shipped
Notification
Event-driven & Reactive
@berndruecker
Checkout
Payment
Inventory
Shipment

19. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

20. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

21. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

22. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

23. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

24. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment
@berndruecker

25. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped
@berndruecker

26. What we wanted
Photo by Lijian Zhang, under Creative Commons SA 2.0 License and Wikimedia Commons / CC BY-SA 4.0
@berndruecker
Choreography

27. Why is it so tempting?
Service
A
Event Bus
A
B
Service
B
C
Service
C
D
E
F
Service
D
G
Service
…
Service
…
Service …
Service
…
Service …
Service …
Service
…
@berndruecker

28. Why is it so tempting?
Service
A
Event Bus
A
B
Service
B
C
Service
C
D
E
F
Service
D
G
Service
…
Service
…
Service …
Service
…
Service …
Service …
Service
…
Adding is easy!
You can „buy“ a shorter
initial time-to-value
by choreography.
It yields in technical debt.
@berndruecker

29. Order
Fulfillment
Decide about responsibility
Checkout
Payment
Inventory
Shipment
Payment
received
Order
placed
Retrieve
payment
@berndruecker

30. Order
Fulfillment
Decide about responsibility
Checkout
Payment
Inventory
Shipment
Payment
received
Order
placed
Retrieve
payment
@berndruecker
This is
choreography
This is
orchestration

31. Order
Fulfillment
Decide about responsibility
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
It can still be Apache Kafka!
@berndruecker

32. ?
Event Command
Message Record Event
Fact,
happened in the past,
immutable
Intend,
Want s.th. to happen,
The intention itself is a fact
@berndruecker

33. ?
Event Command
Message Record Event
@berndruecker

34. Commands in disguise
The Customer Needs To Be
Sent A Message To Confirm
Address Change
Event
Send
Message
Wording of
Sender
Wording of
recipient

35. Checkout Order Payment
Event-driven:
Decision to couple is on the receiving side
Command-driven
Decision to couple is on the sending side
Direction of dependency
Retrieve
Payment
Order
placed
Payment
received
Direction of dependency
@berndruecker

36. Order
Checkout
Payment
Inventory
Shipment
Stateful orchestration
This orchestration
requires state
@berndruecker

37. Order
Checkout
Payment
Inventory
Shipment
@berndruecker

38. Now it is easy to change the process flow
@berndruecker

39. Some code?
https://github.com/berndruecker/flowing-retail/tree/master/kafka
@berndruecker

40. Customer created
Sam Newman: Building Microservices
@berndruecker

41. Mix orchestration and choreography
Orchestration
Orchestration
Orchestration
Choreography
@berndruecker

42. Another use case: Transactions
Service
Kafka
RDMS
1. Receive
4. Send additional events
2. Business Logic
3. Send
response
@berndruecker

43. Another use case example:
Vehicle Maintenance

44. Action
Insights
Measurements
generates Leads to
oil pressure is
80 psi
oil pressure is
critically high
Schedule
maintenance
Call driver to
stop and inspect
…
@berndruecker

45. Action
Insights
Measurements
generates Leads to
oil pressure is
80 psi
oil pressure is
critically high
Schedule
maintenance
Kafka Topic
Measures
Kafka Topic
Insights
Camunda Workflow
Maintenance
Stateful
Stream
Processor
Decision
Workflows
Event Streams
@berndruecker
Custom
Connector

46. https://blog.bernd-ruecker.com/
zeebe-loves-kafka-d82516030f99
@berndruecker

47. @berndruecker
Kafka
Camunda
Produce
Record
Consume
Records
Apache Kafka

48. Summary
• A workflow engine is complementary to Apache Kafka
• The technical integration is easy
• There are tools that scale
• Typical use cases
• Microservices orchestration (balanced with choreography)
• Business transactions
• Actions resulting from streams
@berndruecker

49. Thank you!
@berndruecker

50. Want to learn more?
https://ProcessAutomationBook.com/
Free electronic version available
Featuring code examples:

51. mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-
workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/

52. Berlin, Germany
http://berndruecker.io/
mail@berndruecker.io
@berndruecker
Bernd Ruecker
Co-founder and
Chief Technologist of
Camunda