All Things Open 2014 - Day 1
Wednesday, October 22nd, 2014
Gavin Roy
VP of Architecture for AWeber Communications
Back Dev
Messaging Standards and Systems - AMQP & RabbitMQ
Find more by Gavin here: https://speakerdeck.com/gmr
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Messaging Standards and Systems - AMQP & RabbitMQ
1. Messaging Standards and Systems
AMQP & RabbitMQ
Gavin M. Roy
VP of Architecture
AWeber Communications
Twitter: @Crad
All Things Open
October 22, 2014
2. About Me
VP of Architecture
AWeber Communications
Blame me for pika, rabbitpy,
pamqp, and a handful of
RabbitMQ plugins
Blog: https://gavinroy.com
Github: https://github.com/gmr
Book: http://manning.com/roy
7. AMQP 0-9-1
❖ Currently has wider support than AMQP 1.0
❖ Multiple Broker Implementations:
RabbitMQ, Apache Qpid, and SwiftMQ to name a few
❖ Specifies a Model and Protocol
8. Native AMQP 0-8/0-9-1 Clients
C
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python
Ruby
Scala
9. ❖ Exchange
Receives and route messages
❖ Message Queue
Stores messages until they can be
consumed
❖ Binding
Defines the relationship between an Exchange and Queue and provides
routing criteria
Advanced Message Queueing Model
X
Exchange
Binding
Message Queue
10. Routing Keys
❖ Provided when publishing a message
❖ Compared against binding keys by exchanges
❖ Ideally provide context to the message:
❖ Connote the type of the message
❖ Categorize the content in the message
❖ Specify the type of consumer that should receive it
11. Advanced Message Queueing Protocol
❖ Compact, binary frame format wire protocol
❖ Bi-directional RPC
❖ Commands consist of Classes and Methods:
❖ Example Request:
Queue.Declare(name=“foo”)
❖ Response:
Queue.DeclareOk(messages=0, consumers=0)
12. Common AMQP Terms
❖ Broker
A server that implements AMQP
❖ Producer or Publisher
A client application that sends messages to a broker
❖ Consumer
A client application that reads messages from a queue
16. AMQP Messages
❖ Comprised of 3 or more frames:
❖ Method Frame
Basic.Publish, Basic.Deliver, etc
❖ Content Header with body size & message properties
timestamp, message-id, app-id, etc
❖ n Body Frames with the opaque message payload
18. RabbitMQ
❖ Open Source (MPL)
❖ Written in Erlang/OTP
❖ Developed/Maintained by Pivotal
❖ Multi-Protocol
AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP,
HTTP, Web-STOMP & More
❖ Roots in AMQP 0-8/0-9-1
19. Who Uses It?
Agora Games
Chef
Google AdMob
Instagram
MeetMe
Mercado Libre
Mozilla
NASA
New York Times
National Science Foundation
Openstack
Rapportive
Reddit
Soundcloud
(and many more)
20. Why Use RabbitMQ?
❖ Create loosely-coupled applications
❖ Communicate across applications or platforms
❖ Tap into pre-existing message flows for new purposes
❖ Scale-out clustering for growth, throughput, and HA
❖ Federation for WAN latencies and network partitions
❖ Extensible plugin-in architecture
21. RabbitMQ Extensions to AMQP
Authentication Failure
Exchange to Exchange Bindings
Delivery Confirmations
Basic.Nack
Consumer cancellations
Consumer priorities
Dead Letter Exchanges
Alternate Exchanges
Connection blocking
Message CC & BCC Routing
Queue Length Limits
Per Queue Message TTL
Per Message TTL
Queue TTL
Message User ID Validation
Auto-delete exchanges
22. RabbitMQ Clustering
❖ LAN Only
❖ Adds highly-available queues
❖ Is cohesive, publish and consume
from any node
❖ Leverages native Erlang clustering
and communication
❖ Has multiple strategies for dealing
with network partitions
❖ Manually configured via
configuration or command line*
23. RabbitMQ Plugins
auth-backend-amqp
Use AMQP to respond to RabbitMQ
auth requests
auth-backend-http
Use a custom web service to provide
authentication
autocluster-consul
Automatically create clusters using
Consul
influxdb-storage-exchange
Store JSON messages as events in
InfluxDB
presence-exchange
Publishes messages upon binding
changes
rabbitmq-toke
Tokyo Cabinet backing store for queue
messages
rabbitmq-top
Top like view of RabbitMQ processes
in the management UI
sharding
Scale out RabbitMQ with automatic
queue sharding
(and many more)
25. Built-In Exchange Types
❖ Direct
String matching on the routing key
❖ Fanout
No routing key, messages delivered to all bound queues
❖ Topic
Pattern matching in the routing key
❖ Headers
No routing key, value matching in the headers property
26. Topic Exchange Binding Keys
Routing Key: namespace.delimited.keys
#
Receive all messages
namespace.#
Receive all messages in namespace
namespace.delimited.*
Receive all namespace.delimited messages
namespace.*.keys
Receive all namespace messages ending with keys
27. Exchange to Exchange Binding
X
X
X
Queue
Queue
Queue
Queue
Queue
Queue
Messages published into a topic exchange are
routed to two other exchanges and a queue
Messages published into the consistent-hashing
exchange are distributed amongst three queues
Messages published into a direct
exchange routed to two queues
P
29. Dead Letter Exchanges
P X
Reject Queue
Queue C
X
Consumer rejects the
message without requeue
Dead-Letter Exchange
routes the message
30. Exchange Plugins
Consistent Hashing
Distribute messages via hashed value
of routing key
Event
Publishes messages on AMQP events
such as queue creation
Random
Distribute messages across all bound
queues randomly
PostgreSQL LISTEN
Subscribes to and publishes
PostgreSQL notifications
Recent History
Sends the last n messages to any
newly bound queue
Reverse Topic
Allows for routing patterns at publish
time and not via binding
Riak Storage
Stores messages published through
the exchange into Riak
Script Exchange
Calls out to external scripts for
message routing
33. Persisted Messages IO Model
Disk
Queue
Message Queue
Queue
Messages with delivery-mode 2 sent
into RabbitMQ are persisted to disk
Pointers to the message are placed
in the queue data structures
When the message is no longer in
a queue, it is removed from disk
34. HA Queues & Performance
Queue
Queue
Queue
P
RabbitMQ Cluster
A publisher sends to any
node in a RabbitMQ cluster
The RabbitMQ servers in the
cluster synchronize the state
of the message in the queue
The message that was published is put in
the queue and is stored on each server
37. Trending & Monitoring
❖ RabbitMQ Management Plugin provides internal stats
❖ Queue depths, connection counts, throughput, memory
usage, etc
❖ Monitor with common tools such as Nagios or Sensu to
services such as Boundary and NewRelic
❖ Stream based monitoring with Riemann for anomaly
detection
38. Thoughts on Configuration
❖ Use configuration management!
❖ Even for exchanges, queues, and bindings if possible
❖ Helpful for disaster recovery
❖ Use RabbitMQ’s Policies when possible
❖ Exchanges and queues are immutable
❖ Deleting and redeclaring for changes can be
disruptive
44. RPC
C / PC / P
P / C
C / PResponse Queue
X
X
Worker
Queue
Front-end web app
receives a request
Internet
App publishes RPC request
One of the subscribed
worker consumers receives
the RPC request
After processing the request,
the worker publishes the replyApp receives the reply
from the response queue
App replies to client
45. Delayed Messages
WARNING: This pattern is a hack and can cause undesirable situations should the wait queue overwhelm the system!
P X
X
Wait Queue
Ready Queue C
The wait queue is declared with a
message TTL and a dead-letter exchangeMessages that will be delivered
later are routed to a wait queue
When a message times out due to
the TTL, it is published to the DLX
The internally re-published message is
routed to the queue with active consumers