Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 102 Anzeige

OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan

Herunterladen, um offline zu lesen

Integrating microservices and taming distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and discuss how to avoid them using Open Source orchestration. Communication is complex. With everything being distributed failures are normal, so you need sophisticated failure handling strategies (e.g. stateful retry). A synchronicity requires you to handle timeouts. This is not only about milliseconds, systems get much more resilient when you can wait for minutes, hours or even longer. Distributed transactions cannot simply be delegated to protocols like XA. So, you need to solve the requirement to retain consistency in case of failures. I will not only use slides but demonstrate concrete source code examples available on GitHub.

Integrating microservices and taming distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and discuss how to avoid them using Open Source orchestration. Communication is complex. With everything being distributed failures are normal, so you need sophisticated failure handling strategies (e.g. stateful retry). A synchronicity requires you to handle timeouts. This is not only about milliseconds, systems get much more resilient when you can wait for minutes, hours or even longer. Distributed transactions cannot simply be delegated to protocols like XA. So, you need to solve the requirement to retain consistency in case of failures. I will not only use slides but demonstrate concrete source code examples available on GitHub.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Ähnlich wie OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan (20)

Aktuellste (20)

Anzeige

OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan

  1. 1. The Benefits of Orchestration in Distributed Systems @NiallDeehan
  2. 2. Who I am? Niall Deehan
  3. 3. Distributed systems Challenges of asynchronicity Distributed Transactions Communication is complex
  4. 4. Do A Do B All or nothing + try { tx.begin(); doA(); doB(); tx.commit(); } catch (Exception e) { tx.rollback(); } @Transactional public void createCustomer(Customer cust) { // ... } Or simply: Once upon a time:
  5. 5. A C I D Atomicity Consistency Isolation Durability
  6. 6. Distributed systems
  7. 7. Distributed systems
  8. 8. Some Service Some Service Some Service Some Service Some Service Some Service Some Service Failure will happen. Accept it! But keep it local! Be resilient.
  9. 9. Communication is complex
  10. 10. Let‘s start with a simple example Credit Card Payment REST
  11. 11. Live hacking https://github.com/flowing/flowing-retail/blob/master/payment- rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV1.java
  12. 12. It‘s a blocking request. It doesn‘t scale
  13. 13. Starbucks does not use two phase commit Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle
  14. 14. That means Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Local ACID Local ACID 1 (micro-)service 1 aggregate 1 program 1 resource Violates „I“ of ACID
  15. 15. Youmightknowthisfrom: Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Photo by Gerhard51, available under Creative Commons CC0 1.0 license.
  16. 16. It is impossible to differentiate certain failure scenarios: Independant of communication style! Service Provider Client
  17. 17. Network problems Credit Card Payment charge
  18. 18. Let‘s start with a simple example Credit Card Payment REST
  19. 19. Circuit Breaker Photo by CITYEDV, available under Creative Commons CC0 1.0 license.
  20. 20. Live hacking https://github.com/flowing/flowing-retail/blob/master/payment- rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV2.java
  21. 21. Fail fast is important
  22. 22. Fail fast is important but not enough!
  23. 23. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda
  24. 24. Photo by Tookapic, available under Creative Commons CC0 1.0 license.
  25. 25. Translation : „There was an error while sending your boarding pass“
  26. 26. Check-in Web-UI Bernd Current situation
  27. 27. Check-in Barcode Generator Web-UI Output Mgmt Current situation Bernd
  28. 28. Check-in Barcode Generator Web-UI Output Mgmt Current situation Bernd
  29. 29. Check-in Barcode Generator Web-UI Output Mgmt Current situation Circuit breaker Bernd
  30. 30. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Bernd
  31. 31. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Bernd
  32. 32. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Stateful Retry Bernd
  33. 33. Translation: It‘s your problem now.
  34. 34. We are having some technical difficulties and cannot present you your boarding pass right away. But we do actively retry ourselves, so lean back, relax and we will send it on time.
  35. 35. Check-in Barcode Generator Web-UI Output Mgmt Possible situation – much better! Bernd
  36. 36. Check-in Barcode Generator Web-UI Output Mgmt Possible situation – much better! Stateful Retry Bernd
  37. 37. Check-in Barcode Generator Web-UI Output Mgmt Stateful Retry Possible situation – much better! The failure never leaves this scope! Bernd
  38. 38. Persist thing (Entity, Document, Actor, …) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Complex, proprietary, heavyweight, slow, don‘t scale, developer adverse Scheduling, Versioning, operating, visibility, scalability, … Handling State
  39. 39. Warning: Contains Opinion
  40. 40. Workflow engines, state machines It is relevant in modern architectures
  41. 41. What about open source…?
  42. 42. CADENCE Silicon valley has recognized Workflow engines, state machines
  43. 43. What about purpose built and open source…
  44. 44. CADENCE Workflow engines, state machines
  45. 45. CADENCE also at scale Workflow engines, state machines
  46. 46. CADENCE for todays demo Workflow engines, state machines
  47. 47. Live hacking https://github.com/flowing/flowing-retail/blob/master/payment- rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV3.java
  48. 48. Challenges of asynchronicity
  49. 49. Payment Now you have a state machine! Credit Card REST
  50. 50. has to implement Retry has to implement Idempotency Client Service Provider
  51. 51. We are processing your payment. Do not leave this page. And for god sake – do not reload! It is a business problem anyway!
  52. 52. We are processing your payment. Do not leave this page. And for god sake – do not reload! It is a business problem anyway! We are currently processing your request. Don‘t worry, it will happen safely – even if you loose connection. Feel free to reload this page any time!
  53. 53. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  54. 54. Distributed systems
  55. 55. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
  56. 56. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  57. 57. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  58. 58. Check-in Barcode Generator Web-UI Bernd Output Mgmt Workflows live within service boundaries
  59. 59. „The customer wants a synchronous response“ Check-in Barcode Generator Web-UI Bernd Output Mgmt „Eh – no!“
  60. 60. generateBoardingPass HTTP 200 OK HTTP 202 ACCEPTED Check-in A synchronous response is possible in the happy case, otherwise it is switched to asynchronous processing.
  61. 61. Live hacking https://github.com/flowing/flowing-retail/blob/master/payment- rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV4.java
  62. 62. Challenges of asynchronicity
  63. 63. Check-in Barcode Generator Web-UI Bernd Output Mgmt Asynchronous communication You need to monitor timeouts
  64. 64. Check-in Barcode Generator Web-UI Bernd Output Mgmt Remember… The failure never leaves this scope!
  65. 65. Workflow…
  66. 66. Workflow…
  67. 67. BPMN Business Process Model and Notation ISO Standard
  68. 68. BPMN Executable and mature Easy to understand* Widespread *(for Biz and Dev)
  69. 69. Graphical models?
  70. 70. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  71. 71. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  72. 72. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  73. 73. Living documentation for long-running behaviour
  74. 74. Visual HTML reports for test cases
  75. 75. Proper Operations Visibility + Context
  76. 76. Compare to e.g. Step Functions
  77. 77. Imagine more complex stuff
  78. 78. has to implement Timeout, Retry has to implement Idempotency Client Service Provider
  79. 79. Manigfold architecture options https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  80. 80. Manigfold architecture options https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  81. 81. Avoids thundering herds Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.
  82. 82. Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license. https://aws.amazon.com/de/message/5467D2/
  83. 83. Manigfold architecture options https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  84. 84. Distributed Transactions Distributed Transactions
  85. 85. Compensation – the classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. In case of failure trigger compensations book trip
  86. 86. 2 alterntive approaches: choreography & orchestration
  87. 87. Event-driven choreography Hotel Flight Car Trip Trip booked Flight booked Trip requested Hotel booked Car booked Request trip
  88. 88. Event-driven choreography Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled Perform undo (cancel car booking) Perform undo (cancel hotel)
  89. 89. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
  90. 90. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled You have to adjust all services and redeploy at the same time! We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
  91. 91. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License
  92. 92. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight
  93. 93. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! You have to adjust one service and redeploy only this one!
  94. 94. Describe orchestration with BPMN Trip Trip booked Request trip
  95. 95. The workflow is part of the service Trip
  96. 96. The workflow is part of the service Trip Payment
  97. 97. Live hacking https://github.com/flowing/flowing-retail/blob/master/payment- rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV6.java
  98. 98. has to implement Timeout, Retry, Compensation has to offer Compensation has to implement Idempotency Client Service Provider
  99. 99. has to implement Timeout, Retry, Compensation has to offer Compensation has to implement Idempotency Client Service ProviderDon‘t forget about state
  100. 100. # Be aware of complexity of distributed systems # Know strategies and tools to handle it e.g. Circuit breaker (Hystrix) Workflow engine for stateful retry, waiting, timeout and compensation (Camunda)
  101. 101. Thank you!

×