2. Scaling RabbitMQ
to 11
INTERACTIVE POLLS
• http://pollev.com/gmr
• SMS: 22333
2
Gavin M. Roy @crad - Surge 2011
3. myYearbook.com
• #1 Comscore Teen Destination, Top Trafficked Sites in US
• Use of RabbitMQ dates back to 2009
• Multiple clusters for different purposes
• Browser <-> Backend Message Bus
• Messaging, Event Processing
3
Gavin M. Roy @crad - Surge 2011
5. ABOUT AMPQ
• Platform, Vendor Neutral Messaging format
• 0.9.1 finalized in November 2008
• JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,
IONA Technologies, Cisco Systems, Envoy Technologies,
Twist Process Innovations
• Members now include Bank of America, Barclays Banks,
Microsoft Corporation, Novell, VMWare
• 1.0 Final: October 2011
5
Gavin M. Roy @crad - Surge 2011
6. ABOUT RABBITMQ
• Born in 2006, Released in 2007, Acquired in 2010
• Written in Erlang/OTP
• Talks AMQP 0.8 and 0.9.1
• Features include:
• Clustering, Active/Active HA Queues
• Management UI and API
• Plugin Architecture
6
Gavin M. Roy @crad - Surge 2011
7. ABOUT YOU & RABBITMQ
Gavin M. Roy @crad - Surge 2011
11. QUEUES
• Durability: Queue survives reboot
• Auto-Delete: Delete the queue when consumer goes away
• Exclusive: Only one consumer may consume
• x-expires: Auto-Delete after given duration
• x-message-ttl: Auto-discard a message after ttl
11
Gavin M. Roy @crad - Surge 2011
18. FANOUT
• 1:N Message Delivery Pattern
• No Routing Keys Involved
• Queues bound to a fanout exchange get all messages sent to
the exchange
18
Gavin M. Roy @crad - Surge 2011
19. DIRECT
• 1:1 Message Delivery Pattern
• Routing Keys are direct, no wildcarding
• All queues bound to a routing key get the message
19
Gavin M. Roy @crad - Surge 2011
20. TOPIC EXCHANGES
• Pattern Matching in Routing Keys
• Publish one message type to Surge.Sessions, another to
Surge.Mentions
• Period delimiter, * stays within the period scope, # does not
• Example patterns: *.Mentions, Surge.*, #
• Slower than Fanout and Direct
20
Gavin M. Roy @crad - Surge 2011
21. HEADERS EXCHANGES
• Matches on Basic.Properties headers table values to the
arguments table specified when binding a queue
• Use x-match when declaring exchange to specify any or all
headers must match
• Slow implementation in Erlang
21
Gavin M. Roy @crad - Surge 2011
22. PLUGIN EXCHANGES
• Consistent Hash Exchange: Round-robin distribution to queues
• External: RPC Exchange Plugin
• Script: Similar in intent to External
• Riak
• Last-Value Cache
• Recent History Exchange: Keeps last 20 messages routed
• Global Fanout: Sends to all queues, regardless of routing key
22
Gavin M. Roy @crad - Surge 2011
40. BASIC.QOS
• Prefetch Size and Count
• How many messages the
client wants the RabbitMQ
to pre-deliver to the client
• Doesn’t work in no_ack
Gavin M. Roy @crad - Surge 2011
48. VORPAL BUNNY
• Light-weight PHP client for RabbitMQ
• Used the experimental JSON-RPC Channel plugin
• Handles the overhead of the HTTP, JSON-RPC to AMQP
analog
• Fire and forget message publishing
• Can not be transparently load balanced
48
Gavin M. Roy @crad - Surge 2011
59. Scaling RabbitMQ
to 11
FEDERATION
• Exchange Plugin
• Provides ability to route all
messages to a x-federation
exchange to another broker
Gavin M. Roy @crad - Surge 2011
60. SHOVEL PLUGIN
• From Queue on one broker to an
exchange on another broker.
• Effective for cross-cluster data
broadcasting
Gavin M. Roy @crad - Surge 2011
61. TRENDING & MONITORING
• RabbitMQ Message Velocity Rates Gotcha
• Management API Plugin exposes RESTful API including
monitoring data points
• 2.6.1 allows monitoring tag for users to allow data access
without full admin access
• Queue.Declare(passive=True) gets queue depths as well
• Plug in to Graphite, Nagios,or your favorite tool
61
Gavin M. Roy @crad - Surge 2011