Our feedback on the creation of a distributed enterprise service bus.

1. Build your own
Service Bus
ALT.NET Paris
October 2013
ABC Arbitrage

2. Olivier Coanet
OMG this guy does not tweet
ABC Arbitrage Asset Management
www.abc-arbitrage.com
Kevin Lovato
@alprema
ALT.NET France
www.altnetfr.org
www.meetup.com/altnetfr

3. Build your own
Service Bus

4. Application 1
Application 2
ESB
Application 3
Copyright Bullshit Consulting ®

8. The tube is a lie

9. Application 1
Application 2
Qpid
Service Bus V1:
broker-based

10. Application
Abc.ServiceBus
Qpid.Client
Qpid
Bus
library
Broker-based
Transport
layer

11. Application 1
Abc.ServiceBus
Qpid.Client
Application 2
Abc.ServiceBus
Qpid.Client
Qpid
+ infrastructure services
Qpid.Client
Abc.ServiceBus
Application 3
ESB

12. Service 1
Abc.ServiceBus
Qpid.Client
Service 2
Abc.ServiceBus
Qpid.Client
Qpid
Qpid.Client
Abc.ServiceBus
Service 3

13. And they were blissfully, blissfully happy
forever and ever

14. And they were blissfully, blissfully happy
forever and ever

15. NEW REQUIREMENTS

16. Higher troughtput

17. Better latency

18. Custom messaging usage was
missing Service Bus features

19. and the stack was kind of

20. version

21. LEFT
DEAD
.NET client

22. Buy or Build?

24. Brokerless transport recipe
• Peer to peer communication protocol

25. Brokerless transport recipe
• Peer directory

26. Brokerless transport recipe
• Optional and parallel persistence

27. Brokerless transport recipe
Chef tip:
• Use interoperable ingredients

28. Brokerless transport recipe
The cherry on the cake:
• Build a gateway for Qpid communication

29. Maybe you want implementation details?

30. P2P communication
Protocol Buffers

31. “ØMQ sockets are the world-saving
superheroes of the networking world”
Pieter Hintjens

32. Service 1
Service 2
Service 3
PUSH
PUSH
PUSH
PULL
Service 4

33. Who do we connect to?

34. Connecting to the directory
Directory
Service 1
Service 2
Service 3

35. Connecting to the directory
Directory
RegisterCommand
+ I listen to HelloEvent
Service 1
Peers list + subscriptions
Service 2
Service 3

36. Connecting to the directory
Directory
New service ID + subscriptions
Service 1
Service 2
Service 3

37. Connecting to the directory
Directory
Service 1
Service 2
HelloEvent
Service 3

38. The directory, our new SPOF

39. Connecting to the directories
Directory 1
Service 1
Directory 2
Service 2
Service 3

40. Connecting to the directories
Directory 1
Directory 2
RegisterCommand
+ I listen to HelloEvent
Service 1
RegisterCommand
+ I listen to HelloEvent
Service 2
Service 3

41. Connecting to the directories
HelloEvent
listeners:
Service 1
HelloEvent
listeners:
Service 2
Directory 1
Service 1
Directory 2
Service 2
Service 3

42. Connecting to the directories
HelloEvent
listeners:
Service 1
HelloEvent
listeners:
Service 2
Directory 1
Directory 2
RegisterCommand
Service 1
Service 2
Peers list + subscriptions
Service 3

43. Connecting to the directories
HelloEvent
listeners:
Service 1
HelloEvent
listeners:
Service 2
Directory 1
Directory 2
HelloEvent
listeners:
Service 2
Service 1
Service 2
Service 3

44. Cassandra to the rescue
Directory 1
Directory 2

45. What if a service goes down?

46. Persisting messages
Service 1
Copy of HelloEvent
Persistence
service
HelloEvent
Stored messages
Stored messages
Stored messages
Service 2

47. Persisting messages
Service 1
Persistence
service
HelloEvent Ack
Service 2

48. Persistence service
• Persistence service acting as frontend
• Cassandra for storage

49. Messages persistence model
Row Key
Abc_Service1
2013-09-03 11:03:52
Serialized message
Abc_Service2
2013-10-03 08:25:38
…
2012-03-09 23:01:00
...
..
…
…

50. Acks persistence model
Row Key
Abc_Service1
2013-09-03 11:03:52
null
Abc_Service2
2013-10-03 08:25:38
…
2012-03-09 23:01:00
...
..
…
…

51. Error recovery
Service 1
Persistence
service
Stored messages
Stored messages
Stored messages
Service 2
Messages sent to Service 2
while it was down

52. Error recovery
• Phase 1 : Missed messages recovery
Service 1
Persistence
service
Missed messages recovery
Service 2
Stored messages
Stored messages
Stored messages

53. Error recovery
• Phase 1 : Missed messages recovery
• Phase 2 : Mixed recovery
Service 1
Persistence
service
Normal message reception
Streaming of the messages
from the persistence
Service 2

54. Error recovery
Service 1
• Phase 1 : Missed messages recovery
• Phase 2 : Mixed recovery
• Phase 3 : Normal phase
Persistence
service
Normal message reception
Service 2

55. Meet our latest SPOF, the persistence

56. But actually, no
• Persistance only required at startup
• Messages are also stored in the ZMQ outbound buffer

57. Migration time !

58. Inter-bus gateway
Service 1
Service 3
Broker
Gateway
Service 4
Zebus
QPID
Service 2

59. Summary
•
•
•
•
Running in production
Migration: 40% done
Gateway under heavy load
Owning the transport layer is double edged

60. Thank you