Presentation about RabbitMQ given at SpringOne 2GX

1. Data Migration at Scale with
RabbitMQ and Spring Integration
By Álvaro Videla and Félix López
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

2. Álvaro Videla
• Developer Advocate at Pivotal / RabbitMQ
• Co-Author of RabbitMQ in Action
• Creator of the RabbitMQ Simulator
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/
internals.html
• @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
2

3. Álvaro Videla
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
3
Co-authored
!
RabbitMQ in Action
http://bit.ly/rabbitmq

4. What is RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
4

5. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
5

6. RabbitMQ
• Multi Protocol Messaging Server
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
6

7. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
7

8. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
8

9. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
• Written in Erlang/OTP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
9

10. Multi Protocol
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
10
http://bit.ly/rmq-protocols

11. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
11

12. Polyglot
• Java
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
12

13. Polyglot
• Java
• node.js
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
13

14. Polyglot
• Java
• node.js
• Erlang
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
14

15. Polyglot
• Java
• node.js
• Erlang
• PHP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
15

16. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
16

17. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
• .Net
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
17

18. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
• .Net
• Haskell
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
18

19. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
19
Even COBOL!!!11

20. http://www.rabbitmq.com/download.html
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
20
Unix - Mac - Windows

21. Messaging with RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
21
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator

22. RabbitMQ Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
22

23. http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-
latency-and-bandwidth/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
23

24. RabbitMQ Basic Qos Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
24

25. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
25
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?

26. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
26
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?
Queue length goes to infinity over time.

27. Prevent Unbound Buffers
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?
Queue length goes to infinity over time.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
27
https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-
with-rabbitmq/

28. Recommended Reading
Performance Modeling and
Design of Computer Systems:
Queueing Theory in Action
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
28

29. What you get out of the box
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
29

30. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
30

31. Some RabbitMQ Features
• Persistent Messages
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
31

32. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
32

33. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
33

34. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
34

35. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
• Message Ordering
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
35

36. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
36

37. Some RabbitMQ Features
• Dead Letter Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
37

38. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
38

39. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
39

40. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
40

41. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
41

42. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
• Shovel
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
42

43. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
43

44. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
44
• SSL Connections

45. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
45
• SSL Connections
• User Authentication

46. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
46
• SSL Connections
• User Authentication
• Per resource authorisation

47. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
47
• SSL Connections
• User Authentication
• Per resource authorisation
• Different kinds of users

48. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
48
• SSL Connections
• User Authentication
• Per resource authorisation
• Different kinds of users
• Various authentication mechanisms via plugins (LDAP)

49. Management Interface / REST API
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
49

50. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
50

51. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
51
Flexibility

52. Questions?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
52

53. Learn More. Stay Connected
@old_sound @flopezluis
@springcentral | spring.io/video
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 53