Talk I gave at LRUG about RabbitMQ, how we're using it at GoCardless and the some tools we built - Hutch and AMQPC Hutch is an opinionated library for RabbitMQ which makes using it for comms in SOA super simple

1. RabbitMQ & Hutch
Pete Hamilton - GoCardless

2. About Me
• Developer at GoCardless
• CompSci Student at ICL
• Github: https://github.com/petehamilton
• Blog: pete-hamilton.co.uk
• Twitter: @peterejhamilton

3. SOA Communication
A (very) brief overview
Awesome Service

4. SOA Communication
A (very) brief overview
Awesomer Services

5. SOA Communication
A (very) brief overview
Awesomer Services

6. SOA Communication
A (very) brief overview
?
Client - Server?
Message Queues?

7. RabbitMQ

8. RabbitMQ
Why?
• Nice Learning Curve
• Reliable
• Active Community
• Polyglot
• Scalable
• Proven itself in battle production

9. RabbitMQ
Why?
• Payments are pretty serious stuff
• Async = Potential problems
• Network Failure?
• Consumer Crashes?
• Duplication/redelivery?
• RabbitMQ helps with confirms,
acknowledgements, mandatory flags

10. RabbitMQ Routing
P X
C1
C2
Q1
Q2

11. Fanout Exchange
RabbitMQ Routing
P X
C1
C2
Q1
Q2

12. Direct Exchange
RabbitMQ Routing
P X
C1
C2
type=direct
firefly
star_trek
big_bang_theory
Q1
Q2

13. Topic Exchange
RabbitMQ Routing
P X
C1
C2
type=topic
tv.scifi.*
tv.comedy.*
C3
tv.#
Q1
Q2
Q3
Sample Message Routing Keys:
- tv.comedy.big_bang_theory
- tv.scifi.firefly
- tv.scifi.star_trek

14. RabbitMQ
• Very Flexible, but...
• Which data structure do you use?
• How do you serialise your data?
• Which fields do you require in your messages?
• How should messages be routed? fan out?
direct? topically?
• How do new nodes set themselves up?
• What form should “Boiler Plate” code take?

15. A system for processing RabbitMQ messages in Ruby

16. • Our Conventions (Warning: Opinions!)
• Topic Exchange Routing
• Serialisation w/ JSON
• Message IDs and Timestamps
• Simple to use
• Configurable
• Specific ruby files or full Rails apps

17. Publisher

18. Consumer

22. https://github.com/gocardless/hutch

23. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes

24. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
S S
M
C1

25. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
S S
M
x C1

26. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
S S
M
x C1

27. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
S S
M
C1

28. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
S S
M
C1
x

29. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
M S
M
C1
x

30. RabbitMQ
Scaling
• Good guide to clustering online
• Potential for issues with multiple nodes
• Clients exist for load testing, but...
• We wanted ++ parallelism
• Built “AMQPC”

31. AMQPC
• Written in Go
• Producer & Consumer modes
• Traffic Modes
• One off burst
• Intermittent burst
• Consistent load

32. AMQPC

33. https://github.com/gocardless/amqpc

34. Useful Links
• Hutch - www.github.com/gocardless/hutch
• AMQPC - www.github.com/gocardless/amqpc
• RabbitMQ - http://www.rabbitmq.com/getstarted.html

35. Questions?