Digital transformation change things dramatically in enterprise IT so does it with Integration solutions like an ESB. Why an ESB is not a good solution in Microservice architectures and why it's best to look for alternative approaches - often referred to as hybrid integration platforms (HIP) striving for Self Service Integration. The Simple Workflows (SWF) of AWS can be a first step. Learn about the Why and How we do it at Scout24.

1. Will trade an ESB for an agile
Integration solution in the
Cloud
Berlin Expert Days,
16. September 2016, Urania Berlin
@KayLerch | Engineering Manager | Immobilien Scout

2. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

3. Competitive forces and what they do with Enterprise IT
Competitive
forcesMarket
IT of an E-commerce business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
Continuous DeploymentProduct
BEDCon 2016 | @KayLerch

4. “Two speed IT” parts heavily depend on each other not only for the good
Competitive
forcesMarket
A digital business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Continuous Deployment
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
Organizational
clash
Technical
clash
„Two Speed IT“
Product
BEDCon 2016 | @KayLerch

5. Overcome technical burden with Integration strategy (SOA?)
Competitive
forcesMarket
A digital business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Continuous Deployment
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
„Two Speed IT“
Product
BEDCon 2016 | @KayLerch
Organizational
clash
Technical
clash
ICC
ESB

6. Well, not … Integration projects tend to be traditional (often) for good reason
Competitive
forcesMarket
A digital business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Continuous Deployment
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
„Two Speed IT“
Product
BEDCon 2016 | @KayLerch
Organizational
clash
Technical
clash
ICC
ESB

7. An integration strategy should follow a “bimodal” approach where so-
called hybrid integration platforms (HIP) strive for Self Service Integration
Competitive
forcesMarket
A digital business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Continuous Deployment
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
„Two Speed IT“
Product
ISG
HIP
BEDCon 2016 | @KayLerch

8. The digital ecosystem brings a lot of new potential but also threat
Mobile
IoT
User Data
Cloud Social Networks Digital marketplace
Digital ecosystem
discruptive
BEDCon 2016 | @KayLerch

9. Businesses need to adopt and integrate these potential
Mobile
IoT
User Data
Cloud Social Networks Digital marketplace
Digital ecosystem
discruptive
BEDCon 2016 | @KayLerch

10. An HIP got its name from being the gateway for those new stuff
Mobile
IoT
User Data
Cloud Social Networks Digital marketplace
Digital ecosystem
discruptive
BEDCon 2016 | @KayLerch

11. HIP is a concept whereas iPaaS, iSaaS are (commercial) solutions in the cloud
Mobile
IoT
User Data
Cloud Social Networks Digital marketplace
Digital ecosystem
discruptive
iPaaS
iSaaS
BEDCon 2016 | @KayLerch

12. Ok, you got your BINGO! Let’s move on in the real world …
BEDCon 2016 | @KayLerch

13. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

14. Given an ESB in charge of syncing business data across the enterprise
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
BEDCon 2016 | @KayLerch

15. This centralized hub is key for processes arching over multiple domains
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
BEDCon 2016 | @KayLerch

16. It encapsulates the burden of integrating system interfaces with brokers
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }
BEDCon 2016 | @KayLerch

17. Teams depend on those brokers in order to change their interfaces
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }
API
API
API
API
BEDCon 2016 | @KayLerch

18. It would be great to push responsibility (integration logic) to the edges
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

19. It would also be great to delegate process ownership (orchestration logic)
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

20. ESB left over for “dirty work” (messaging, tracking, governance, …)
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

21. It should also be in the cloud to not be isolated from the digital ecosystem
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

22. It should also be in the cloud to not be isolated from the digital ecosystem
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

23. Why can’t this thing just be like an HIP?
HIP
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }
BEDCon 2016 | @KayLerch

24. As our company moves to AWS anyway, let’s give SWF a try
• There is PAYG
• AWS ecosystem is huge, it serves all the hip stuff like serverless architecture (Lambda, API Gateway),
IoT, Mobile integration, Messaging, Elastic computing, Container deployments and more
• There‘s a big community around AWS
• SWF (Simple Workflows) is used by NASA for processing data from the Mars-Rover on earth
You got me at „Mars“…
BEDCon 2016 | @KayLerch

25. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy

26. SWF is a workflow engine scheduling tasks for all the workflow participants
BEDCon 2016 | @KayLerch

27. A workflow starter simply kicks off a workflow with some input via API
1
BEDCon 2016 | @KayLerch
{ "childPolicy": "string", "domain": "string", "executionStartToCloseTimeout": "string", "input": "string", "lambdaRole": "string", "tagList": [
"string" ], "taskList": { "name": "string" }, "taskPriority": "string", "taskStartToCloseTimeout": "string", "workflowId": "string", "workflowType":
{ "name": "string", "version": "string" } }
{ "runId": "string" }

28. SWF has no clue what comes next so it schedules a “decision task”
2
BEDCon 2016 | @KayLerch

29. A decider owns the actual workflow logic. It is an application polling for
tasks via API and returns decisions to the SWF engine
3
BEDCon 2016 | @KayLerch
{ "domain": "string", "identity": "string", "maximumPageSize": number, "nextPageToken": "string", "reverseOrder": boolean, "taskList": { "name":
"string" } }
{ "decisions": [ { "cancelTimerDecisionAttributes": { … }, "cancelWorkflowExecutionDecisionAttributes": { … },
"completeWorkflowExecutionDecisionAttributes": { … }, "continueAsNewWorkflowExecutionDecisionAttributes": { … }, "recordMarkerDecisionAttributes": {
… }, "requestCancelActivityTaskDecisionAttributes": { … }, "requestCancelExternalWorkflowExecutionDecisionAttributes": { … },
"scheduleActivityTaskDecisionAttributes": { … }, "scheduleLambdaFunctionDecisionAttributes": { … },
"signalExternalWorkflowExecutionDecisionAttributes": { … }, "startChildWorkflowExecutionDecisionAttributes": { … }, "startTimerDecisionAttributes":
{ … } } ], "executionContext": "string", "taskToken": "string" }

30. SWF fulfills the remotely given decisions –> e.g. it schedules an activity task
4
BEDCon 2016 | @KayLerch

31. An activity worker polls for tasks from a task list, works on it and returns a result
5
BEDCon 2016 | @KayLerch
{ "domain": "string", "identity": "string", "taskList": { "name": "string" } }
{ "result": "string", "taskToken": "string" }
{ "details": "string", "reason": "string", "taskToken": "string" }

32. SWF receives the result – again has no clue how to go on – so it schedules
another decision task
6
BEDCon 2016 | @KayLerch

33. The decider receives the task. It now contains a detailed protocol of what
happened in this workflow-execution. Based on that it gives the next decision(s)
7
BEDCon 2016 | @KayLerch

34. SWF schedules the next task – according to the last decision by the decider.
Activity workers can be serverless Lambda-functions as well
8
BEDCon 2016 | @KayLerch

35. This goes on and on until the decider aka workflow worker decides for
completing the workflow. What again is executed by SWF itself.
BEDCon 2016 | @KayLerch

36. You keep track of your workflow executions in AWS console
BEDCon 2016 | @KayLerch

37. AWS CloudWatch provides performance metrics for your workflows
BEDCon 2016 | @KayLerch

38. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

39. SWF-powered IoT-Photobooth
BEDCon 2016 | @KayLerch
2) Shoot a photo
View progress on breadboard
3) View on Slack
1) Press a button

40. AWS Lambda
AWS SWF
Lambda
Task
Lambda
Upload to S3
& Start
Workflow
Lambda
Update Shadow
Lambda
Task
Lambda
Get S3 Photo and
Post Message to
Slack
Lambda
Task
Lambda
Task
Lambda
Task
Lambda
Shorten Url
with Bitly
Lambda
Update Shadow
Lambda
Update Shadow
AWS EC2
WorkflowWorker
DeciderImpl
Raspberry Pi
AWS IoT
Thing
Shadow
Python Script
GPIO
Rule
AWS S3
Bucket
PutFile
StartWFInvoke
MQTT
Publish
MQTT
Receive
Update
Shadow
Poll Tasks
Shoot
Impulse
Impulse

41. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

42. So let’s use SWF for replacing the ESB in our company.
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }
API
API
API
API
BEDCon 2016 | @KayLerch

43. Client-side worker integrate their services by pulling tasks from SWF
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
wor
ker
wor
ker
wor
ker
wor
ker
Acti
vity
Acti
vity
Acti
vityActi
vity
AWS
SWF
BEDCon 2016 | @KayLerch

44. Workers poll for tasks in task lists associated with declarative activity type
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list
Task
list
Task
list
HTTP
REST
HTTP
REST
HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
wor
ker
wor
ker
BEDCon 2016 | @KayLerch

45. Workers can be Lambda functions as well in charge of the edge team
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
lam
bda
AWS lam
bda
AWS
Event
Trigger
Event
Trigger
BEDCon 2016 | @KayLerch

46. That’s nice, but there’s still a central orchestration controlled by one decider
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
lam
bda
AWS lam
bda
AWS
Event
Trigger
Event
Trigger
BEDCon 2016 | @KayLerch

47. Delegate parts of the orchestration in child workflows reclaims process ownership
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
HTTP
REST
AWS
SWF
wor
ker
lam
bda
AWS
Event
TriggerEvent
Trigger
HTTP
REST
AWS
Acti
vity
Task
list
wor
ker
Sub
WF
Acti
vity
BEDCon 2016 | @KayLerch
Task
list

48. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

49. Reclaim process ownership leads to real end-2-end-autonomy of teams
Sales
BEDCon 2016 | @KayLerch

50. End-2-End autonomy along the whole lifecycle of a worker
Sales
BEDCon 2016 | @KayLerch

51. We call them Microworker as this is what they really are
Sales
BEDCon 2016 | @KayLerch

52. Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

56. Q & A
BEDCon 2016 | @KayLerch
It‘s time to ask questions and give feedback.
Thanks for joining my session ...

57. Backup (SWF at JPL Datacenter of NASA)
Sales
BEDCon 2016 | @KayLerch

58. Backup (SWF-powered processing of images from Mars rover)
Sales
BEDCon 2016 | @KayLerch

59. Backup (SWF-powered contract data distribution at Scout24)
Sales
BEDCon 2016 | @KayLerch

60. Backup (SWF-powered contract data distribution at Scout24)
Sales
BEDCon 2016 | @KayLerch