SlideShare ist ein Scribd-Unternehmen logo
1 von 63
Downloaden Sie, um offline zu lesen
Scaling RabbitMQ
                                  to 11




Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                                        to 11

    INTERACTIVE POLLS
    • http://pollev.com/gmr

    • SMS: 22333




2

                      Gavin M. Roy @crad - Surge 2011
myYearbook.com
• #1 Comscore Teen Destination, Top Trafficked Sites in US

• Use of RabbitMQ dates back to 2009
• Multiple clusters for different purposes

 • Browser <-> Backend Message Bus

 • Messaging, Event Processing



                                                             3

                    Gavin M. Roy @crad - Surge 2011
DISASTER PORN
     Gavin M. Roy @crad - Surge 2011
ABOUT AMPQ
• Platform, Vendor Neutral Messaging format

• 0.9.1 finalized in November 2008
 • JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,
    IONA Technologies, Cisco Systems, Envoy Technologies,
    Twist Process Innovations

• Members now include Bank of America, Barclays Banks,
   Microsoft Corporation, Novell, VMWare

• 1.0 Final: October 2011
                                                             5

                    Gavin M. Roy @crad - Surge 2011
ABOUT RABBITMQ
•   Born in 2006, Released in 2007, Acquired in 2010

•   Written in Erlang/OTP

•   Talks AMQP 0.8 and 0.9.1

•   Features include:

•    Clustering, Active/Active HA Queues

•    Management UI and API

•    Plugin Architecture

                                                          6

                        Gavin M. Roy @crad - Surge 2011
ABOUT YOU & RABBITMQ
      Gavin M. Roy @crad - Surge 2011
RABBITMQ
& AMQP
CONCEPTS
   Gavin M. Roy @crad - Surge 2011
PUBLISHERS

   Gavin M. Roy @crad - Surge 2011
CONSUMERS

   Gavin M. Roy @crad - Surge 2011
QUEUES
• Durability: Queue survives reboot

• Auto-Delete: Delete the queue when consumer goes away
• Exclusive: Only one consumer may consume

• x-expires: Auto-Delete after given duration

• x-message-ttl: Auto-discard a message after ttl




                                                          11

                    Gavin M. Roy @crad - Surge 2011
MESSAGES
• Have client specified properties:

 • Content type, Encoding, Timestamp, App-Id, User-Id,
    Headers

 • Delivery Mode:

  • 1: Non-Persistent

  • 2: Persistent



                                                         12

                    Gavin M. Roy @crad - Surge 2011
MESSAGE
ROUTING
It’s not just about Queues.




                     Gavin M. Roy @crad - Surge 2011
ROUTING KEY
In addition to the exchange, determines what is done with a
message.



                    Gavin M. Roy @crad - Surge 2011
CC & BCC
More routing options per message, but violates AMQP.




                    Gavin M. Roy @crad - Surge 2011
HEADERS
EXCHANGE
More on this later.




                      Gavin M. Roy @crad - Surge 2011
EXCHANGES

   Gavin M. Roy @crad - Surge 2011
FANOUT
• 1:N Message Delivery Pattern

• No Routing Keys Involved
• Queues bound to a fanout exchange get all messages sent to
   the exchange




                                                               18

                    Gavin M. Roy @crad - Surge 2011
DIRECT
• 1:1 Message Delivery Pattern

• Routing Keys are direct, no wildcarding
• All queues bound to a routing key get the message




                                                      19

                    Gavin M. Roy @crad - Surge 2011
TOPIC EXCHANGES
• Pattern Matching in Routing Keys

• Publish one message type to Surge.Sessions, another to
   Surge.Mentions

• Period delimiter, * stays within the period scope, # does not

• Example patterns: *.Mentions, Surge.*, #

• Slower than Fanout and Direct



                                                                  20

                     Gavin M. Roy @crad - Surge 2011
HEADERS EXCHANGES
• Matches on Basic.Properties headers table values to the
   arguments table specified when binding a queue

• Use x-match when declaring exchange to specify any or all
   headers must match

• Slow implementation in Erlang




                                                              21

                    Gavin M. Roy @crad - Surge 2011
PLUGIN EXCHANGES
•   Consistent Hash Exchange: Round-robin distribution to queues

•   External: RPC Exchange Plugin

•   Script: Similar in intent to External

•   Riak

•   Last-Value Cache

•   Recent History Exchange: Keeps last 20 messages routed

•   Global Fanout: Sends to all queues, regardless of routing key

                                                                    22

                       Gavin M. Roy @crad - Surge 2011
ONE MORE
THING...
Exchange-to-Exchange Bindings




                   Gavin M. Roy @crad - Surge 2011
CLUSTERS

   Gavin M. Roy @crad - Surge 2011
STATS, DISK
AND RAM
NODES
    Gavin M. Roy @crad - Surge 2011
WHERE DOES
THE QUEUE
LIVE?
   Gavin M. Roy @crad - Surge 2011
CROSS-NODE
PUBLISHING
&
CONSUMING
   Gavin M. Roy @crad - Surge 2011
MAX CLUSTER
SIZE
~32 notes is a realistic maximum node count for a cluster




                     Gavin M. Roy @crad - Surge 2011
SCALING ISSUES?




      Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                  to 11




Gavin M. Roy @crad - Surge 2011
WHAT TO WATCH




     Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                     to 11




       RABBITMQ




Y U NO TAKE MESSAGE?




   Gavin M. Roy @crad - Surge 2011
TCP
BACKPRESSURE
MUCH?
    Gavin M. Roy @crad - Surge 2011
MEMORY
MANAGEMENT

  Gavin M. Roy @crad - Surge 2011
[{RABBIT,
 [{VM_MEMORY_HIGH_WATERMARK,
   0.4}]
}].
                       40%
          Gavin M. Roy @crad - Surge 2011
MESSAGE
PERSISTENCE
50-70% performance penalty compared to non-persisted
messages. *   http://www.rabbitmq.com/faq.html#performance-persistent




                              Gavin M. Roy @crad - Surge 2011
CONSUMING
Ack, Nack and Reject vs Basic.Consume(no_ack=True)




                    Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                                                   to 11




Server                                                                        Client
Basic.Deliver
                                                              Basic.[Ack/Nack/Reject]
Basic.[Ack/Nack/Reject]Ok




                            Gavin M. Roy @crad - Surge 2011
Gavin M. Roy @crad - Surge 2011
BASIC.QOS
                        • Prefetch Size and Count

                        • How many messages the
                              client wants the RabbitMQ
                              to pre-deliver to the client

                        • Doesn’t work in no_ack




     Gavin M. Roy @crad - Surge 2011
TOO MANY
SHORT LIVED
CONNECTIONS
   Gavin M. Roy @crad - Surge 2011
APACHE +
MOD_PHP

    Gavin M. Roy @crad - Surge 2011
CONNECTION
PERSISTENCE

    Gavin M. Roy @crad - Surge 2011
100 SERVERS
125 BACKENDS
1 REQUEST/S
12,500 CONN/S
    Gavin M. Roy @crad - Surge 2011
AMQP CONNECTION SEQUENCE
Client                                                             Server
Protocol Header Frame
                                                          Connection.Start
Connection.StartOk
                                                          Connection.Tune
Connection.TuneOk
                                                          Connection.Open
Connection.OpenOk

Channel.Open
                                                          Channel.OpenOk




                        Gavin M. Roy @crad - Surge 2011
685 BYTES PER
8.16 MB/S IN
STARTUP @
12,500 CONN/S
    Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY




     Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY
• Light-weight PHP client for RabbitMQ

• Used the experimental JSON-RPC Channel plugin
• Handles the overhead of the HTTP, JSON-RPC to AMQP
   analog

• Fire and forget message publishing

• Can not be transparently load balanced



                                                       48

                    Gavin M. Roy @crad - Surge 2011
TOO MANY
CONNECTIONS
   Gavin M. Roy @crad - Surge 2011
VELOCITY TOO
HIGH FOR ONE
NODE?
    Gavin M. Roy @crad - Surge 2011
MEMORY
UTILIZATION
DISPROPORTIONATE
TO QUEUE DEPTHS

     Gavin M. Roy @crad - Surge 2011
CLUSTER
NODE
FAILURES
    Gavin M. Roy @crad - Surge 2011
UP TO 60
SECONDS TO
DETECT
   Gavin M. Roy @crad - Surge 2011
MESSAGES
DROPPED ON
THE FLOOR*
   Gavin M. Roy @crad - Surge 2011
* 2.6.0 ADDED
ACTIVE/ACTIVE
QUEUES IN
CLUSTERS
    Gavin M. Roy @crad - Surge 2011
ACTIVE/
PASSIVE
FAILOVER?
Recommendation is to use Pacemaker and DRBD or shared
storage for Mnesia databases.



                     Gavin M. Roy @crad - Surge 2011
RECOVERY?



Gavin M. Roy @crad - Surge 2011
CLUSTERING ALTERNATIVES
        Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                                    to 11

FEDERATION

                     • Exchange Plugin

                     • Provides ability to route all
                          messages to a x-federation
                          exchange to another broker




      Gavin M. Roy @crad - Surge 2011
SHOVEL PLUGIN
• From Queue on one broker to an
   exchange on another broker.

• Effective for cross-cluster data
   broadcasting




                    Gavin M. Roy @crad - Surge 2011
TRENDING & MONITORING
• RabbitMQ Message Velocity Rates Gotcha

• Management API Plugin exposes RESTful API including
   monitoring data points

• 2.6.1 allows monitoring tag for users to allow data access
   without full admin access

• Queue.Declare(passive=True) gets queue depths as well

• Plug in to Graphite, Nagios,or your favorite tool

                                                               61

                     Gavin M. Roy @crad - Surge 2011
PHOTO CREDITS
•   “skateboard kids 7”
    © 2007 Volker Neumann
    http://flickr.com/prawnpie

•   “methodcn contest”
    ©2009 Erin O’Hara
    http://flickr.com/erinohara73/

•   USPS Truck Accident
    Valley City Fire Department
    http://www.valleycityfire.com




                                                                62

                              Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ
                                     to 11




QUESTIONS?

   Gavin M. Roy @crad - Surge 2011

Weitere ähnliche Inhalte

Andere mochten auch

The Future of Messaging: RabbitMQ and AMQP
The Future of Messaging: RabbitMQ and AMQP The Future of Messaging: RabbitMQ and AMQP
The Future of Messaging: RabbitMQ and AMQP Eberhard Wolff
 
High powered messaging with RabbitMQ
High powered messaging with RabbitMQHigh powered messaging with RabbitMQ
High powered messaging with RabbitMQJames Carr
 
Kinesis vs-kafka-and-kafka-deep-dive
Kinesis vs-kafka-and-kafka-deep-diveKinesis vs-kafka-and-kafka-deep-dive
Kinesis vs-kafka-and-kafka-deep-diveYifeng Jiang
 
Introduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQIntroduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQDmitriy Samovskiy
 
Ruled based expert system
Ruled based expert systemRuled based expert system
Ruled based expert systemAmit Dahal
 
Introduction to Rule-based Applications
Introduction to  Rule-based ApplicationsIntroduction to  Rule-based Applications
Introduction to Rule-based Applicationsgiurca
 
Rule Based Architecture System
Rule Based Architecture SystemRule Based Architecture System
Rule Based Architecture SystemFirdaus Adib
 
Monitoring Kafka w/ Prometheus
Monitoring Kafka w/ PrometheusMonitoring Kafka w/ Prometheus
Monitoring Kafka w/ Prometheuskawamuray
 
Expert system 21 sldes
Expert system 21 sldesExpert system 21 sldes
Expert system 21 sldesYasir Khan
 
Voice based repair shop floor picking
Voice based repair shop floor pickingVoice based repair shop floor picking
Voice based repair shop floor pickingAshish Saxena
 
Dissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal ArchitectureDissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal ArchitectureAlvaro Videla
 

Andere mochten auch (12)

The Future of Messaging: RabbitMQ and AMQP
The Future of Messaging: RabbitMQ and AMQP The Future of Messaging: RabbitMQ and AMQP
The Future of Messaging: RabbitMQ and AMQP
 
High powered messaging with RabbitMQ
High powered messaging with RabbitMQHigh powered messaging with RabbitMQ
High powered messaging with RabbitMQ
 
Kinesis vs-kafka-and-kafka-deep-dive
Kinesis vs-kafka-and-kafka-deep-diveKinesis vs-kafka-and-kafka-deep-dive
Kinesis vs-kafka-and-kafka-deep-dive
 
Introduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQIntroduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQ
 
Ruled based expert system
Ruled based expert systemRuled based expert system
Ruled based expert system
 
Introduction to Rule-based Applications
Introduction to  Rule-based ApplicationsIntroduction to  Rule-based Applications
Introduction to Rule-based Applications
 
Rule Based Architecture System
Rule Based Architecture SystemRule Based Architecture System
Rule Based Architecture System
 
Monitoring Kafka w/ Prometheus
Monitoring Kafka w/ PrometheusMonitoring Kafka w/ Prometheus
Monitoring Kafka w/ Prometheus
 
Rule Based System
Rule Based SystemRule Based System
Rule Based System
 
Expert system 21 sldes
Expert system 21 sldesExpert system 21 sldes
Expert system 21 sldes
 
Voice based repair shop floor picking
Voice based repair shop floor pickingVoice based repair shop floor picking
Voice based repair shop floor picking
 
Dissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal ArchitectureDissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal Architecture
 

Ähnlich wie Scaling RabbitMQ to 11

Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...
Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...
Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...Continuent
 
On Rabbits and Elephants
On Rabbits and ElephantsOn Rabbits and Elephants
On Rabbits and ElephantsGavin Roy
 
Openstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMsOpenstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMsSanjeev Rampal
 
Stability patterns devoxx_pl_2017
Stability patterns devoxx_pl_2017Stability patterns devoxx_pl_2017
Stability patterns devoxx_pl_2017Daniel Lebrero
 
Chicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveChicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveCloudCamp Chicago
 
Chicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveChicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveAWS Chicago
 
Go - A Key Language in Enterprise Application Development?
Go - A Key Language in Enterprise Application Development?Go - A Key Language in Enterprise Application Development?
Go - A Key Language in Enterprise Application Development?C4Media
 
Ready, Set, SD-WAN: Best Practices for Assuring Branch Readiness
Ready, Set, SD-WAN: Best Practices for Assuring Branch ReadinessReady, Set, SD-WAN: Best Practices for Assuring Branch Readiness
Ready, Set, SD-WAN: Best Practices for Assuring Branch ReadinessThousandEyes
 
London JBUG - Connecting Applications Everywhere with JBoss A-MQ
London JBUG - Connecting Applications Everywhere with JBoss A-MQLondon JBUG - Connecting Applications Everywhere with JBoss A-MQ
London JBUG - Connecting Applications Everywhere with JBoss A-MQJBUG London
 
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
How did we move the mountain? - Migrating 1 trillion+ messages per day across...How did we move the mountain? - Migrating 1 trillion+ messages per day across...
How did we move the mountain? - Migrating 1 trillion+ messages per day across...HostedbyConfluent
 
Powering Fast Data and the Hadoop Ecosystem with VoltDB and Hortonworks
Powering Fast Data and the Hadoop Ecosystem with VoltDB and HortonworksPowering Fast Data and the Hadoop Ecosystem with VoltDB and Hortonworks
Powering Fast Data and the Hadoop Ecosystem with VoltDB and HortonworksHortonworks
 
Apache Kafka® at Dropbox
Apache Kafka® at DropboxApache Kafka® at Dropbox
Apache Kafka® at Dropboxconfluent
 
F5 Networks - - OpenStack Summit 2016/Red Hat NFV Mini Summit
F5 Networks -  - OpenStack Summit 2016/Red Hat NFV Mini SummitF5 Networks -  - OpenStack Summit 2016/Red Hat NFV Mini Summit
F5 Networks - - OpenStack Summit 2016/Red Hat NFV Mini Summitkimw001
 
21st Docker Switzerland Meetup - ISTIO
21st Docker Switzerland Meetup - ISTIO21st Docker Switzerland Meetup - ISTIO
21st Docker Switzerland Meetup - ISTIONiklaus Hirt
 
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsPortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsTimothy Spann
 
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Andreas Grabner
 

Ähnlich wie Scaling RabbitMQ to 11 (20)

Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...
Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...
Harnessing the Power of Master/Slave Clusters to Operate Data-Driven Business...
 
On Rabbits and Elephants
On Rabbits and ElephantsOn Rabbits and Elephants
On Rabbits and Elephants
 
Openstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMsOpenstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMs
 
Stability patterns devoxx_pl_2017
Stability patterns devoxx_pl_2017Stability patterns devoxx_pl_2017
Stability patterns devoxx_pl_2017
 
Chicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveChicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at Cohesive
 
Chicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at CohesiveChicago AWS user group meetup - May 2014 at Cohesive
Chicago AWS user group meetup - May 2014 at Cohesive
 
Go - A Key Language in Enterprise Application Development?
Go - A Key Language in Enterprise Application Development?Go - A Key Language in Enterprise Application Development?
Go - A Key Language in Enterprise Application Development?
 
RabbitMQ & Hutch
RabbitMQ & HutchRabbitMQ & Hutch
RabbitMQ & Hutch
 
Ready, Set, SD-WAN: Best Practices for Assuring Branch Readiness
Ready, Set, SD-WAN: Best Practices for Assuring Branch ReadinessReady, Set, SD-WAN: Best Practices for Assuring Branch Readiness
Ready, Set, SD-WAN: Best Practices for Assuring Branch Readiness
 
London JBUG - Connecting Applications Everywhere with JBoss A-MQ
London JBUG - Connecting Applications Everywhere with JBoss A-MQLondon JBUG - Connecting Applications Everywhere with JBoss A-MQ
London JBUG - Connecting Applications Everywhere with JBoss A-MQ
 
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
How did we move the mountain? - Migrating 1 trillion+ messages per day across...How did we move the mountain? - Migrating 1 trillion+ messages per day across...
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
 
Powering Fast Data and the Hadoop Ecosystem with VoltDB and Hortonworks
Powering Fast Data and the Hadoop Ecosystem with VoltDB and HortonworksPowering Fast Data and the Hadoop Ecosystem with VoltDB and Hortonworks
Powering Fast Data and the Hadoop Ecosystem with VoltDB and Hortonworks
 
Routing in the cloud
Routing in the cloudRouting in the cloud
Routing in the cloud
 
Apache Kafka® at Dropbox
Apache Kafka® at DropboxApache Kafka® at Dropbox
Apache Kafka® at Dropbox
 
F5 Networks - - OpenStack Summit 2016/Red Hat NFV Mini Summit
F5 Networks -  - OpenStack Summit 2016/Red Hat NFV Mini SummitF5 Networks -  - OpenStack Summit 2016/Red Hat NFV Mini Summit
F5 Networks - - OpenStack Summit 2016/Red Hat NFV Mini Summit
 
Stonehenge
StonehengeStonehenge
Stonehenge
 
21st Docker Switzerland Meetup - ISTIO
21st Docker Switzerland Meetup - ISTIO21st Docker Switzerland Meetup - ISTIO
21st Docker Switzerland Meetup - ISTIO
 
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsPortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
 
Ha of load balancer
Ha of load balancerHa of load balancer
Ha of load balancer
 
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
 

Kürzlich hochgeladen

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Kürzlich hochgeladen (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 

Scaling RabbitMQ to 11

  • 1. Scaling RabbitMQ to 11 Gavin M. Roy @crad - Surge 2011
  • 2. Scaling RabbitMQ to 11 INTERACTIVE POLLS • http://pollev.com/gmr • SMS: 22333 2 Gavin M. Roy @crad - Surge 2011
  • 3. myYearbook.com • #1 Comscore Teen Destination, Top Trafficked Sites in US • Use of RabbitMQ dates back to 2009 • Multiple clusters for different purposes • Browser <-> Backend Message Bus • Messaging, Event Processing 3 Gavin M. Roy @crad - Surge 2011
  • 4. DISASTER PORN Gavin M. Roy @crad - Surge 2011
  • 5. ABOUT AMPQ • Platform, Vendor Neutral Messaging format • 0.9.1 finalized in November 2008 • JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix, IONA Technologies, Cisco Systems, Envoy Technologies, Twist Process Innovations • Members now include Bank of America, Barclays Banks, Microsoft Corporation, Novell, VMWare • 1.0 Final: October 2011 5 Gavin M. Roy @crad - Surge 2011
  • 6. ABOUT RABBITMQ • Born in 2006, Released in 2007, Acquired in 2010 • Written in Erlang/OTP • Talks AMQP 0.8 and 0.9.1 • Features include: • Clustering, Active/Active HA Queues • Management UI and API • Plugin Architecture 6 Gavin M. Roy @crad - Surge 2011
  • 7. ABOUT YOU & RABBITMQ Gavin M. Roy @crad - Surge 2011
  • 8. RABBITMQ & AMQP CONCEPTS Gavin M. Roy @crad - Surge 2011
  • 9. PUBLISHERS Gavin M. Roy @crad - Surge 2011
  • 10. CONSUMERS Gavin M. Roy @crad - Surge 2011
  • 11. QUEUES • Durability: Queue survives reboot • Auto-Delete: Delete the queue when consumer goes away • Exclusive: Only one consumer may consume • x-expires: Auto-Delete after given duration • x-message-ttl: Auto-discard a message after ttl 11 Gavin M. Roy @crad - Surge 2011
  • 12. MESSAGES • Have client specified properties: • Content type, Encoding, Timestamp, App-Id, User-Id, Headers • Delivery Mode: • 1: Non-Persistent • 2: Persistent 12 Gavin M. Roy @crad - Surge 2011
  • 13. MESSAGE ROUTING It’s not just about Queues. Gavin M. Roy @crad - Surge 2011
  • 14. ROUTING KEY In addition to the exchange, determines what is done with a message. Gavin M. Roy @crad - Surge 2011
  • 15. CC & BCC More routing options per message, but violates AMQP. Gavin M. Roy @crad - Surge 2011
  • 16. HEADERS EXCHANGE More on this later. Gavin M. Roy @crad - Surge 2011
  • 17. EXCHANGES Gavin M. Roy @crad - Surge 2011
  • 18. FANOUT • 1:N Message Delivery Pattern • No Routing Keys Involved • Queues bound to a fanout exchange get all messages sent to the exchange 18 Gavin M. Roy @crad - Surge 2011
  • 19. DIRECT • 1:1 Message Delivery Pattern • Routing Keys are direct, no wildcarding • All queues bound to a routing key get the message 19 Gavin M. Roy @crad - Surge 2011
  • 20. TOPIC EXCHANGES • Pattern Matching in Routing Keys • Publish one message type to Surge.Sessions, another to Surge.Mentions • Period delimiter, * stays within the period scope, # does not • Example patterns: *.Mentions, Surge.*, # • Slower than Fanout and Direct 20 Gavin M. Roy @crad - Surge 2011
  • 21. HEADERS EXCHANGES • Matches on Basic.Properties headers table values to the arguments table specified when binding a queue • Use x-match when declaring exchange to specify any or all headers must match • Slow implementation in Erlang 21 Gavin M. Roy @crad - Surge 2011
  • 22. PLUGIN EXCHANGES • Consistent Hash Exchange: Round-robin distribution to queues • External: RPC Exchange Plugin • Script: Similar in intent to External • Riak • Last-Value Cache • Recent History Exchange: Keeps last 20 messages routed • Global Fanout: Sends to all queues, regardless of routing key 22 Gavin M. Roy @crad - Surge 2011
  • 23. ONE MORE THING... Exchange-to-Exchange Bindings Gavin M. Roy @crad - Surge 2011
  • 24. CLUSTERS Gavin M. Roy @crad - Surge 2011
  • 25. STATS, DISK AND RAM NODES Gavin M. Roy @crad - Surge 2011
  • 26. WHERE DOES THE QUEUE LIVE? Gavin M. Roy @crad - Surge 2011
  • 27. CROSS-NODE PUBLISHING & CONSUMING Gavin M. Roy @crad - Surge 2011
  • 28. MAX CLUSTER SIZE ~32 notes is a realistic maximum node count for a cluster Gavin M. Roy @crad - Surge 2011
  • 29. SCALING ISSUES? Gavin M. Roy @crad - Surge 2011
  • 30. Scaling RabbitMQ to 11 Gavin M. Roy @crad - Surge 2011
  • 31. WHAT TO WATCH Gavin M. Roy @crad - Surge 2011
  • 32. Scaling RabbitMQ to 11 RABBITMQ Y U NO TAKE MESSAGE? Gavin M. Roy @crad - Surge 2011
  • 33. TCP BACKPRESSURE MUCH? Gavin M. Roy @crad - Surge 2011
  • 34. MEMORY MANAGEMENT Gavin M. Roy @crad - Surge 2011
  • 35. [{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}] }]. 40% Gavin M. Roy @crad - Surge 2011
  • 36. MESSAGE PERSISTENCE 50-70% performance penalty compared to non-persisted messages. * http://www.rabbitmq.com/faq.html#performance-persistent Gavin M. Roy @crad - Surge 2011
  • 37. CONSUMING Ack, Nack and Reject vs Basic.Consume(no_ack=True) Gavin M. Roy @crad - Surge 2011
  • 38. Scaling RabbitMQ to 11 Server Client Basic.Deliver Basic.[Ack/Nack/Reject] Basic.[Ack/Nack/Reject]Ok Gavin M. Roy @crad - Surge 2011
  • 39. Gavin M. Roy @crad - Surge 2011
  • 40. BASIC.QOS • Prefetch Size and Count • How many messages the client wants the RabbitMQ to pre-deliver to the client • Doesn’t work in no_ack Gavin M. Roy @crad - Surge 2011
  • 41. TOO MANY SHORT LIVED CONNECTIONS Gavin M. Roy @crad - Surge 2011
  • 42. APACHE + MOD_PHP Gavin M. Roy @crad - Surge 2011
  • 43. CONNECTION PERSISTENCE Gavin M. Roy @crad - Surge 2011
  • 44. 100 SERVERS 125 BACKENDS 1 REQUEST/S 12,500 CONN/S Gavin M. Roy @crad - Surge 2011
  • 45. AMQP CONNECTION SEQUENCE Client Server Protocol Header Frame Connection.Start Connection.StartOk Connection.Tune Connection.TuneOk Connection.Open Connection.OpenOk Channel.Open Channel.OpenOk Gavin M. Roy @crad - Surge 2011
  • 46. 685 BYTES PER 8.16 MB/S IN STARTUP @ 12,500 CONN/S Gavin M. Roy @crad - Surge 2011
  • 47. VORPAL BUNNY Gavin M. Roy @crad - Surge 2011
  • 48. VORPAL BUNNY • Light-weight PHP client for RabbitMQ • Used the experimental JSON-RPC Channel plugin • Handles the overhead of the HTTP, JSON-RPC to AMQP analog • Fire and forget message publishing • Can not be transparently load balanced 48 Gavin M. Roy @crad - Surge 2011
  • 49. TOO MANY CONNECTIONS Gavin M. Roy @crad - Surge 2011
  • 50. VELOCITY TOO HIGH FOR ONE NODE? Gavin M. Roy @crad - Surge 2011
  • 51. MEMORY UTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS Gavin M. Roy @crad - Surge 2011
  • 52. CLUSTER NODE FAILURES Gavin M. Roy @crad - Surge 2011
  • 53. UP TO 60 SECONDS TO DETECT Gavin M. Roy @crad - Surge 2011
  • 54. MESSAGES DROPPED ON THE FLOOR* Gavin M. Roy @crad - Surge 2011
  • 55. * 2.6.0 ADDED ACTIVE/ACTIVE QUEUES IN CLUSTERS Gavin M. Roy @crad - Surge 2011
  • 56. ACTIVE/ PASSIVE FAILOVER? Recommendation is to use Pacemaker and DRBD or shared storage for Mnesia databases. Gavin M. Roy @crad - Surge 2011
  • 57. RECOVERY? Gavin M. Roy @crad - Surge 2011
  • 58. CLUSTERING ALTERNATIVES Gavin M. Roy @crad - Surge 2011
  • 59. Scaling RabbitMQ to 11 FEDERATION • Exchange Plugin • Provides ability to route all messages to a x-federation exchange to another broker Gavin M. Roy @crad - Surge 2011
  • 60. SHOVEL PLUGIN • From Queue on one broker to an exchange on another broker. • Effective for cross-cluster data broadcasting Gavin M. Roy @crad - Surge 2011
  • 61. TRENDING & MONITORING • RabbitMQ Message Velocity Rates Gotcha • Management API Plugin exposes RESTful API including monitoring data points • 2.6.1 allows monitoring tag for users to allow data access without full admin access • Queue.Declare(passive=True) gets queue depths as well • Plug in to Graphite, Nagios,or your favorite tool 61 Gavin M. Roy @crad - Surge 2011
  • 62. PHOTO CREDITS • “skateboard kids 7” © 2007 Volker Neumann http://flickr.com/prawnpie • “methodcn contest” ©2009 Erin O’Hara http://flickr.com/erinohara73/ • USPS Truck Accident Valley City Fire Department http://www.valleycityfire.com 62 Gavin M. Roy @crad - Surge 2011
  • 63. Scaling RabbitMQ to 11 QUESTIONS? Gavin M. Roy @crad - Surge 2011