A walk-through of the design and architecture of RabbitMQ - Ayanda Dube

1. AYANDA DUBE ERLANG SOLUTIONS LTD RABBITMQ DESIGN & ARCHITECTURE 12/11/2018 Twitter: dube_aya Github: Ayanda-D Email: ayandaonline@gmail.com

2. QUICK ADDITIONAL ACKNOWLEDGEMENTS ▸ Saskia Kuipers (ESL) ▸ Johan Rhodin and Lovisa Johansson (84codes) ▸ Pieter Humphrey (Pivotal) ▸ Pivotal Engineering Team for all the great work, as always! (Dan Carwin’s team) ▸ Everyone here, Erlang Solutions customers, some who’ve traveled from quite far

3. GOAL: DESIGN & ARCHITECTURE P C ▸ Explain the internal design and architecture of rabbit APP-1 APP-2 ▸ Ever present need for efﬁcient communication

4. QUOTE "A picture is worth a thousand words" - Fred Barnard - Ayanda Dube …. (last night!) "A software system diagram is worth a thousand lines of code”

5. INITIALIZATION: START SCRIPTS

6. INITIALIZATION: START SCRIPTS rabbitmq-server rabbitmq-env rabbitmq-defaults erl RABBITMQ_NODENAME RABBITMQ_NODENAME CONFIG_FILE LOG_BASE MNESIA_BASE ENABLED_PLUGINS_FILE

7. INITIALIZATION: ERLANG NODE RABBITMQ_START_RABBIT

8. INITIALIZATION: RABBIT BOOT MODULE RABBITMQ_BOOT_MODULE = rabbit rabbit:boot( )

9. INITIALIZATION: BOOT rabbit boot! update application configs hipe compile initialise logging log hipe compile result prepare cluster status files upgrade mnesia check cluster consistency Broker Start! rabbitmq.config

10. INITIALIZATION: BROKER START setup plugins start applications + plugins sd notify (“READY”) log broker started run boot steps

11. INITIALIZATION: BOOT STEPS codec_correctness_check rabbit_alarm database (rabbit_mnesia) database_sync (mnesia_sync) ﬁle_handle_cache worker_pool rabbit_registry rabbit_core_metrics rabbit_memory_monitor guid_generator (rabbit_gui) delegate_sup (mnesia_sync) rabbit_node_monitor rabbit_epmd_monitor upgrade_queues recovery empty_db_check direct_client connection_tracking networking notify_cluster background_gc rabbit_core_metrics_gc rabbit_looking_glass internal boot steps kernel_ready external_infrastructure core_initialized routing_ready rabbit_event log_relay pre_boot plugin boot steps

12. INITIALIZATION: INTERNAL BOOT STEPS auth mechanisms exchange types queue mirror modes policies (e.g. validators) queue master locators vhost limits priority queue runtime parameters enabled

13. INITIALIZATION: NODE STATE mnesia5672 PLUGINS rabbit alarm rabbit reader rabbit memory monitor rabbit node monitor recovery file handle cache rabbit registry rabbit Sup worker pool rabbit epmd monitor delegate sup D=16 rabbit_user rabbit_user_permission rabbit_vhost rabbit_queue rabbit_durable_queue rabbit_route

14. AMQP P C ▸ Connection establishment mnesia 5672 PLUGINS rabbit alarm rabbit reader rabbit memory monitor rabbit node monitor recovery file handle cache rabbit registry rabbit Sup worker pool rabbit epmd monitor delegate sup D=16 INITIALIZED APP-1 APP-2

15. AMQP HANDLING: CONNECTION ESTABLISHMENT rabbit channel_0 5671P rabbit reader RANCH TCP connect 5672 rabbit writer rabbit connection sup ranch acceptor proc_lib #TCP Sock# connection.start connection.start_ok [login] connection.tune connection.tune_ok [login success] connection.open connection.open_ok assemble frame assemble frame assemble frame TCP AMQP #AMQP-Connection#

16. AMQP HANDLING: CONNECTION ESTABLISHMENT rabbit reader main_loop receive messge(Socket) handle input handshake <<"AMQP", A, B, C, D, Rest/binary>> Aquire Framing Module {A, B, C, D} {0, 0, 9, 1} {1, 1, 0, 9} {1, 1, 8, 0} rabbit_framing_amqp_0_8 rabbit_framing_amqp_0_9_1 rabbit_framing_amqp_0_9_1 {1, 1, 8, 0} rabbit_framing_amqp_0_8 {ID, 1, 0, 0} rabbit_amqp1_0_reader start_connection AMQP 1.0 infrastructure register connection PID (pg_group) set server properties set auth mechanisms connection.start{ } send on channel 0 rabbit writer send command

17. AMQP HANDLING: CONNECTION ESTABLISHMENT rabbit sup tcp_listener suptcp_listener ranch listener sup ranch conns sup ranch acceptors sup rabbit connection sup Protocol rabbit connection sup rabbit reader rabbit connection helper 10 ranch acceptor ProtocolPid rabbit reader recvloop(Socket) {active, once} mainloop recv handle_input FSM Transport:accept/2 CLIENT CONNECTION REQUESTS RANCH Started on BOOT

18. AMQP HANDLING: NODE STATE P C ▸ Connection establishment ▸ Channel creation ch1 ch2 Ch3 ch1 ch2 ch3

19. AMQP HANDLING: CHANNEL OPEN rabbit channel sup 5672 rabbit channel P rabbit reader channel.open rabbit channel common rabbit channel sup sup channel.open ChPid channel.open rabbit writer channel.open_ok

20. AMQP HANDLING: CHANNEL OPEN rabbit reader receive loop(Socket) {active, once} handle input FSM Channel 0? Y handle method 0 process frame handle AMQP frame N lookup Channel ChPID rabbit_command assembler create new channel undeﬁned ChannelMax? N rabbit_channel_sup_sup start_channel store ChPID (dictionary) {ch_pid, ChPid} {channel, Channel} Y ChannelCount + 1 Protocol decode_method_fields e.g. Protocol = rabbit_framing_amqp_0_9_1 rabbit_channel:do Method handle_cast( Method ) gen_server:cast/2 Method Update Channel State “running” Channel Created Stats ChPID

21. AMQP HANDLING: CHANNEL OPEN rabbit channel sup sup rabbit channel sup rabbit limiter rabbit writer rabbit channel DIRECT NETWORK

22. AMQP: NODE STATE P C ▸ Connection establishment ▸ Channel creation ▸ Create/declare an exchange ch2

23. AMQP HANDLING: EXCHANGE DECLARATION 5672 rabbit channel P rabbit reader exchange.declare rabbit exchange exchange.declare_ok rabbit exchange type exchange.declare_ok direct fanout topic headers custom insert lookup {ok, X} not_found declare

24. AMQP HANDLING: EXCHANGE DECLARATION rabbit channel exchange.declare valid type? Y N configure permitted? N Y lookup X {ok, X} {error, not_found} RETURN DECLARE MNESIAexchange.declare_ok exchange.declare_ok

25. AMQP: NODE STATE ▸ Connection establishment P C ▸ Channel creation ▸ Create/declare an exchange ch2 ▸ Create/declare a queue

26. AMQP HANDLING: QUEUE DECLARATION 5672 rabbit channel P queue.declare rabbit amqqueue queue.declare_ok rabbit queue process queue.declare_ok lookup {ok, Q} not_found queue stats [M, C] delegate stat backing queue queue.declare_ok nowait declare rabbit queue process sup init {new, Q} {ok, Q, 0, 0} BQ:len BQ:init [Q, Durable, AutoDelete, Args, …]

27. AMQP HANDLING: QUEUE DECLARATION rabbit_amqque:declare queue.bind set policy queue master location mirroring queue master location node rabbit queue supervisor Node rabbit prequeue process Queue StartMode Slave Start Queue Master Start Queue Slave Master Q Initialize Queue Index Initiaze Message Store(s) Q ! {init, new} BQ:init initialization Transient Persistent

28. AMQP: NODE STATE ▸ Connection establishment ▸ Channel creation ▸ Create/declare an exchange ▸ Create/declare a queue ▸ Bind queue to exchange P Cch2

29. AMQP HANDLING: QUEUE TO EXCHANGE BINDING 5672 rabbit channel P queue.bind [Q, X] rabbit binding queue.bind_ok read [ _ ] [ ] add add(Q, X) rabbit_route rabbit_route rabbit_durable_route rabbit_semi_durable_route queue.bind_ok

30. AMQP HANDLING: QUEUE TO EXCHANGE BINDING rabbit channel queue.bind format SRC & DST strip “n” and “r” resource names Y write permitted DST? Y N Is default X? N Y read permitted X? Y N RETURN rabbit_binding valid binding? N lookup binding [ _ ] Y [] INSERT BINDING MNESIA queue.bind_ok rabbit writer

31. AMQP: NODE STATE ▸ Connection establishment ▸ Channel creation ▸ Declare an exchange ▸ Declare a queue ▸ Bind queue to exchange ▸ Subscribe Consumer P Cch2

32. AMQP HANDLING: CONSUMING rabbit channelP basic.consume rabbit queue process basic_consume delegaterabbit amqqueue rabbit queue consumers ChPid LimiterPid basic_consume add ChPid rabbit limiter LimiterPid activate priority queue in [ChPid, Consumer] core metrics consumer_created basic.consume_ok

33. OPERATIONS: NODE STATE ▸ Connection establishment P C ▸ Channel creation ▸ Declare an exchange ch2 ▸ Declare a queue ▸ Bind queue to exchange ▸ Subscribe consumer ▸ Publish messages ch2 ch2 ch2

34. AMQP HANDLING: MESSAGE PUBLISH rabbit channelP basic.publish rabbit queue process message route Queue(s) rabbit basic backing queue delivery rabbit exchange delegate #delivery [M, Queues] rabbit amqqueue #delivery{} rabbit queue consumer delivery_attempt is_duplicate? publish QPids

35. AMQP HANDLING: BACKING QUEUE rabbit queue process backing queue rabbit queue index embed? publish rabbit queue msg store msg_store #msg_status{ } Transient Persistent or Internal BQ Queues store[ #msg_status ] Q1 [ALPHAs] Q2 [BETAS & GAMMAS] Q3 [BETAS & GAMMAS] Q4 [ALPHAs] ALPHA BETA GAMMA DELTA RAM MSG POS RAM DISK RAM DISK RAM & DISK DISK DISK #delivery{}

36. AMQP: NODE STATE P Cch2 ch2 ch2 ch2 APP-1 APP-2

37. OPERATIONS: CLUSTERING ▸ Adding “extra nodes” to mnesia conﬁguration ▸ Replication and synchronisation of metadata ▸ Internal internode routing/referencing synch synch ▸ Default distribution port 25672

38. OPERATIONS: HA QUEUES M S1 S2 gm gm gmC synch synch ▸ Backing queue switches to the mirror queue master GM ▸ HA policies dynamically update queue state ▸ Synchronisation over Erlang distribution

39. CONCLUSION: DESIGN & ARCHITECTURE ▸ Not entirely ﬁxed/static, common aspects ▸ Usage varies its internal design ▸ Creation and tearing down of resources (AMQP) ▸ Plugins will alter operation and design ▸ Version releases introduce design and architecture shifts https://github.com/rabbitmq/rabbitmq-server/ https://github.com/rabbitmq/rabbitmq-common/

40. END: THANK YOU! QUESTIONS Twitter: dube_aya Github: Ayanda-D Email: ayandaonline@gmail.com

A walk-through of the design and architecture of RabbitMQ - Ayanda Dube

RabbitMQ Summit

A walk-through of the design and architecture of RabbitMQ - Ayanda Dube