SlideShare ist ein Scribd-Unternehmen logo
1 von 37
Downloaden Sie, um offline zu lesen
Queue Everything
       &
Please Everyone
       ***

   Vaidik Kapoor
Who am I?
➔   Undergraduate student.
➔   Involved with:
    ✔   Mozilla as a Contributor and Rep
    ✔   Drupal as a Contributor
➔   Twitter: @vaidikkapoor
➔   Github: vaidikkp
➔   Web: vaidikkapoor.info
Web Applications
Responsiveness
Request                      Response
                      Client




    Task 1   Task 2   Task 3   Task N




Why do everything at once?
Break the Request-Response Cycle
       Request                   Response
                      Client




           Task 1              Task N




                    Task 2




                    Task 3
Meet Queues
What can Queues help with?
➔   Background Processing
    ✔   Data Processing
    ✔   Media Processing
    ✔   Updating Caches
➔   Anything that you want to offload off your
    server.
➔   Improve the overall User Experience
Redis
       +
HotQueue / PyRes
Redis + HotQueue
PRODUCER:

from hotqueue import HotQueue
queue = HotQueue("myqueue", host="localhost", port=6379, db=0)

queue.put(message)




CONSUMER / WORKER:

from hotqueue import HotQueue
queue = HotQueue("myqueue", host="localhost", port=6379, db=0)

while True:
     message = queue.get()
     # do something awesome
Redis + PyRes
➔   Python clone of Github's Resque
➔   Offers a lot more features than HotQueue
    ✔   Job failure handling
    ✔   Queue status information
➔   Comes with Monitoring System written in Ruby
Redis + PyRes
PRODUCER:

from pyres import ResQ
r = Resq()

class WebPage:
     queue = “screenies”

    @staticmethod
    def perform(urll):
         # save the screenshot

r.enqueue(WebPage, 'http://python.org')


CONSUMER:

./pyres_worker screenies
Message Queue
A system for enabling asynchronous processing
               of discrete tasks.
What can MQs help with?
➔   Everything that generic Queues can help with.
➔   Increase Reliability
➔   Cron Jobs (Celery)
Available MQ Solutions
➔   RabbitMQ
➔   Amazon Simple Queue
➔   Apache MQ
➔   Gearman
➔   Starling
➔   OpenAMQ
➔   Sun Java Message Queue System
Message Queue Protocols
➔   AMQP: Advanced Message Queue Protocol
➔   JMS: Java Messaging Service
➔   STOMP: Streaming Text Oriented Messaging
    Protocol
Criteria for Broker Selection
➔   Difficulty in Recovery
➔   Relatively low level of required maintenance
➔   Ease of Deployment
➔   Durability
➔   Persistence
➔   Community Support
➔   Cluster Support
➔   What language is it written in?
RabbitMQ
➔   A Message Broker
➔   Implements AMQP
Pika & Other AMQP Libraries
Celery
➔   An amazing Task Manager
➔   Batteries Included
➔   Uses RabbitMQ as broker (or almost anything)
➔   Libraries for most of the common web frameworks
    like Django, Flask, Pyramid
➔   Supports multiprocessing.
➔   Distribute tasks easily.
➔   Makes life easy
Celery Example
from celery.task import task

@task():
def take_screenshot(url):
  # magical code to take screenshot comes here
How to put it all together?
The General Setup

                   Task
Web App                        Broker
                  Manager




                  Worker
Database         Server (s)
Then what?
Notify Your Users
Django's Messaging System
Flask's Flashes
Poll using AJAX
WebSockets
Email
Things to Remember
Isolate
Isolate code, make reusable components.
Recycle
Remove Request Dependency
Unit Testing?
Review
➔   Choose a system according to needs
➔   Build a robust system
➔   Integrate seamlessly with your UI
➔   Isolate & Recycle: make it a habit
➔   Improve the overall UX!
➔   Please everyone, even yourself!
References
➔   PyRes - http://pyres.readthedocs.org/
➔   RabbitMQ - http://www.rabbitmq.com/
➔   Celery - http://celeryproject.org/
➔   Amazing Article -
    http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
Thank You!

Más contenido relacionado

Was ist angesagt?

Infrastructure development on windows ldn cd meetup
Infrastructure development on windows   ldn cd meetup Infrastructure development on windows   ldn cd meetup
Infrastructure development on windows ldn cd meetup Owain Perry
 
Serverless framework와 CircleCI를 통한 NoOps 맛보기
Serverless framework와 CircleCI를 통한 NoOps 맛보기Serverless framework와 CircleCI를 통한 NoOps 맛보기
Serverless framework와 CircleCI를 통한 NoOps 맛보기Kyuhyun Byun
 
Node4J: Running Node.js in a JavaWorld
Node4J: Running Node.js in a JavaWorldNode4J: Running Node.js in a JavaWorld
Node4J: Running Node.js in a JavaWorldIan Bull
 
VMUG St Louis - SDN in the Real World
VMUG St Louis - SDN in the Real WorldVMUG St Louis - SDN in the Real World
VMUG St Louis - SDN in the Real WorldChris Wahl
 
Webconf nodejs-production-architecture
Webconf nodejs-production-architectureWebconf nodejs-production-architecture
Webconf nodejs-production-architectureBen Lin
 
Level Up: 5 Expert Tips for Optimizing WordPress Performance
Level Up: 5 Expert Tips for Optimizing WordPress PerformanceLevel Up: 5 Expert Tips for Optimizing WordPress Performance
Level Up: 5 Expert Tips for Optimizing WordPress PerformancePantheon
 
Silverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applicationsSilverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applicationsBrettTasker
 
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native Codemotion
 
NGINX for Application Delivery & Acceleration
NGINX for Application Delivery & AccelerationNGINX for Application Delivery & Acceleration
NGINX for Application Delivery & AccelerationNGINX, Inc.
 
Grafana and MySQL - Benefits and Challenges
Grafana and MySQL - Benefits and ChallengesGrafana and MySQL - Benefits and Challenges
Grafana and MySQL - Benefits and ChallengesPhilip Wernersbach
 
CFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful CodeCFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful Codeindiver
 
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...Max Andersen
 
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...Codemotion
 
Altitude San Francisco 2018: HTTP Invalidation Workshop
Altitude San Francisco 2018: HTTP Invalidation WorkshopAltitude San Francisco 2018: HTTP Invalidation Workshop
Altitude San Francisco 2018: HTTP Invalidation WorkshopFastly
 
Hacking on WildFly 9
Hacking on WildFly 9Hacking on WildFly 9
Hacking on WildFly 9JBUG London
 
Get acquainted with the new ASP.Net 5
Get acquainted with the new ASP.Net 5Get acquainted with the new ASP.Net 5
Get acquainted with the new ASP.Net 5Suyati Technologies
 
Glass fish performance tuning tips from the field
Glass fish performance tuning tips from the fieldGlass fish performance tuning tips from the field
Glass fish performance tuning tips from the fieldPayara
 

Was ist angesagt? (20)

Infrastructure development on windows ldn cd meetup
Infrastructure development on windows   ldn cd meetup Infrastructure development on windows   ldn cd meetup
Infrastructure development on windows ldn cd meetup
 
Serverless framework와 CircleCI를 통한 NoOps 맛보기
Serverless framework와 CircleCI를 통한 NoOps 맛보기Serverless framework와 CircleCI를 통한 NoOps 맛보기
Serverless framework와 CircleCI를 통한 NoOps 맛보기
 
Short-Training asp.net vNext
Short-Training asp.net vNextShort-Training asp.net vNext
Short-Training asp.net vNext
 
Node4J: Running Node.js in a JavaWorld
Node4J: Running Node.js in a JavaWorldNode4J: Running Node.js in a JavaWorld
Node4J: Running Node.js in a JavaWorld
 
VMUG St Louis - SDN in the Real World
VMUG St Louis - SDN in the Real WorldVMUG St Louis - SDN in the Real World
VMUG St Louis - SDN in the Real World
 
Webconf nodejs-production-architecture
Webconf nodejs-production-architectureWebconf nodejs-production-architecture
Webconf nodejs-production-architecture
 
Level Up: 5 Expert Tips for Optimizing WordPress Performance
Level Up: 5 Expert Tips for Optimizing WordPress PerformanceLevel Up: 5 Expert Tips for Optimizing WordPress Performance
Level Up: 5 Expert Tips for Optimizing WordPress Performance
 
Silverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applicationsSilverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applications
 
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native
Webinar - Matteo Manchi: Dal web al nativo: Introduzione a React Native
 
NGINX for Application Delivery & Acceleration
NGINX for Application Delivery & AccelerationNGINX for Application Delivery & Acceleration
NGINX for Application Delivery & Acceleration
 
Grafana and MySQL - Benefits and Challenges
Grafana and MySQL - Benefits and ChallengesGrafana and MySQL - Benefits and Challenges
Grafana and MySQL - Benefits and Challenges
 
CFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful CodeCFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful Code
 
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...
Case study: JBoss Developer Studio, an IDE for Web, Mobile and Cloud applicat...
 
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...
Mobile Library Development - stuck between a pod and a jar file - Zan Markan ...
 
Altitude San Francisco 2018: HTTP Invalidation Workshop
Altitude San Francisco 2018: HTTP Invalidation WorkshopAltitude San Francisco 2018: HTTP Invalidation Workshop
Altitude San Francisco 2018: HTTP Invalidation Workshop
 
Docker, From zero to hero
Docker, From zero to heroDocker, From zero to hero
Docker, From zero to hero
 
Hacking on WildFly 9
Hacking on WildFly 9Hacking on WildFly 9
Hacking on WildFly 9
 
Mini-Training: Redis
Mini-Training: RedisMini-Training: Redis
Mini-Training: Redis
 
Get acquainted with the new ASP.Net 5
Get acquainted with the new ASP.Net 5Get acquainted with the new ASP.Net 5
Get acquainted with the new ASP.Net 5
 
Glass fish performance tuning tips from the field
Glass fish performance tuning tips from the fieldGlass fish performance tuning tips from the field
Glass fish performance tuning tips from the field
 

Andere mochten auch

12 tips on Django Best Practices
12 tips on Django Best Practices12 tips on Django Best Practices
12 tips on Django Best PracticesDavid Arcos
 
Open Networking Foundation, Year 3: Don’t bother trying to catch your breath
Open Networking Foundation, Year 3: Don’t bother trying to catch your breathOpen Networking Foundation, Year 3: Don’t bother trying to catch your breath
Open Networking Foundation, Year 3: Don’t bother trying to catch your breathOpen Networking Summits
 
Celery in the Django
Celery in the DjangoCelery in the Django
Celery in the DjangoWalter Liu
 
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Jimmy DeadcOde
 
Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python CeleryMahendra M
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmWei Lin
 
Advanced task management with Celery
Advanced task management with CeleryAdvanced task management with Celery
Advanced task management with CeleryMahendra M
 
Scaling up task processing with Celery
Scaling up task processing with CeleryScaling up task processing with Celery
Scaling up task processing with CeleryNicolas Grasset
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoTareque Hossain
 
Resftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryResftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryRidwan Fadjar
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Wei Lin
 
Distributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHDistributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHCesar Cardenas Desales
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to CeleryIdan Gazit
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryMauro Rocco
 

Andere mochten auch (18)

12 tips on Django Best Practices
12 tips on Django Best Practices12 tips on Django Best Practices
12 tips on Django Best Practices
 
Open Networking Foundation, Year 3: Don’t bother trying to catch your breath
Open Networking Foundation, Year 3: Don’t bother trying to catch your breathOpen Networking Foundation, Year 3: Don’t bother trying to catch your breath
Open Networking Foundation, Year 3: Don’t bother trying to catch your breath
 
Celery workshop
Celery workshopCelery workshop
Celery workshop
 
Django Celery
Django Celery Django Celery
Django Celery
 
Celery in the Django
Celery in the DjangoCelery in the Django
Celery in the Django
 
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
 
Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python Celery
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker Swarm
 
Advanced task management with Celery
Advanced task management with CeleryAdvanced task management with Celery
Advanced task management with Celery
 
Celery by dummy
Celery by dummyCelery by dummy
Celery by dummy
 
Scaling up task processing with Celery
Scaling up task processing with CeleryScaling up task processing with Celery
Scaling up task processing with Celery
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with django
 
Practical Celery
Practical CeleryPractical Celery
Practical Celery
 
Resftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryResftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & Celery
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
 
Distributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHDistributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZH
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & Celery
 

Ähnlich wie Queue Everything and Please Everyone

Comment améliorer le quotidien des Développeurs PHP ?
Comment améliorer le quotidien des Développeurs PHP ?Comment améliorer le quotidien des Développeurs PHP ?
Comment améliorer le quotidien des Développeurs PHP ?AFUP_Limoges
 
CNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfCNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfLibbySchulze
 
Rock Solid Deployment of Web Applications
Rock Solid Deployment of Web ApplicationsRock Solid Deployment of Web Applications
Rock Solid Deployment of Web ApplicationsPablo Godel
 
Edge Side APIs: Fast and Reliable Hypermedia APIs
Edge Side APIs: Fast and Reliable Hypermedia APIsEdge Side APIs: Fast and Reliable Hypermedia APIs
Edge Side APIs: Fast and Reliable Hypermedia APIsNordic APIs
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesWeaveworks
 
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...PROIDEA
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017Patrick Chanezon
 
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...SecureSoftwareDevOn SecureSoftwareDevOn
 
Planning to Fail #phpne13
Planning to Fail #phpne13Planning to Fail #phpne13
Planning to Fail #phpne13Dave Gardner
 
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016Derek Downey
 
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleuls
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleulsapidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleuls
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleulsapidays
 
Using Puppet - Real World Configuration Management
Using Puppet - Real World Configuration ManagementUsing Puppet - Real World Configuration Management
Using Puppet - Real World Configuration ManagementJames Turnbull
 
Docker and the Container Revolution
Docker and the Container RevolutionDocker and the Container Revolution
Docker and the Container RevolutionRomain Dorgueil
 
Writing your First Ansible Playbook
Writing your First Ansible PlaybookWriting your First Ansible Playbook
Writing your First Ansible PlaybookSana Khan
 
Ranger admin dev overview
Ranger admin dev overviewRanger admin dev overview
Ranger admin dev overviewTushar Dudhatra
 
Porting Rails Apps to High Availability Systems
Porting Rails Apps to High Availability SystemsPorting Rails Apps to High Availability Systems
Porting Rails Apps to High Availability SystemsMarcelo Pinheiro
 

Ähnlich wie Queue Everything and Please Everyone (20)

Comment améliorer le quotidien des Développeurs PHP ?
Comment améliorer le quotidien des Développeurs PHP ?Comment améliorer le quotidien des Développeurs PHP ?
Comment améliorer le quotidien des Développeurs PHP ?
 
CNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfCNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdf
 
Rock Solid Deployment of Web Applications
Rock Solid Deployment of Web ApplicationsRock Solid Deployment of Web Applications
Rock Solid Deployment of Web Applications
 
Edge Side APIs: Fast and Reliable Hypermedia APIs
Edge Side APIs: Fast and Reliable Hypermedia APIsEdge Side APIs: Fast and Reliable Hypermedia APIs
Edge Side APIs: Fast and Reliable Hypermedia APIs
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slides
 
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...
DevOpsDays Warsaw 2015: Automating microservices in Syncano – Michał Kobus & ...
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017
 
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...
Introducing DevSecOps by Madhu Akula - Software Security Bangalore - May 27 2...
 
Docker & Daily DevOps
Docker & Daily DevOpsDocker & Daily DevOps
Docker & Daily DevOps
 
Docker and-daily-devops
Docker and-daily-devopsDocker and-daily-devops
Docker and-daily-devops
 
Planning to Fail #phpne13
Planning to Fail #phpne13Planning to Fail #phpne13
Planning to Fail #phpne13
 
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016
 
Docker in prod
Docker in prodDocker in prod
Docker in prod
 
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleuls
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleulsapidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleuls
apidays LIVE Paris 2021 - Edge Side APIs by Kevin Dunglas, Les Tilleuls
 
Using Puppet - Real World Configuration Management
Using Puppet - Real World Configuration ManagementUsing Puppet - Real World Configuration Management
Using Puppet - Real World Configuration Management
 
Docker and the Container Revolution
Docker and the Container RevolutionDocker and the Container Revolution
Docker and the Container Revolution
 
Writing your First Ansible Playbook
Writing your First Ansible PlaybookWriting your First Ansible Playbook
Writing your First Ansible Playbook
 
Ranger admin dev overview
Ranger admin dev overviewRanger admin dev overview
Ranger admin dev overview
 
Porting Rails Apps to High Availability Systems
Porting Rails Apps to High Availability SystemsPorting Rails Apps to High Availability Systems
Porting Rails Apps to High Availability Systems
 
DevOps and Chef improve your life
DevOps and Chef improve your life DevOps and Chef improve your life
DevOps and Chef improve your life
 

Queue Everything and Please Everyone

  • 1. Queue Everything & Please Everyone *** Vaidik Kapoor
  • 2. Who am I? ➔ Undergraduate student. ➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor ➔ Twitter: @vaidikkapoor ➔ Github: vaidikkp ➔ Web: vaidikkapoor.info
  • 5. Request Response Client Task 1 Task 2 Task 3 Task N Why do everything at once?
  • 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
  • 8. What can Queues help with? ➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches ➔ Anything that you want to offload off your server. ➔ Improve the overall User Experience
  • 9. Redis + HotQueue / PyRes
  • 10. Redis + HotQueue PRODUCER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) queue.put(message) CONSUMER / WORKER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) while True: message = queue.get() # do something awesome
  • 11. Redis + PyRes ➔ Python clone of Github's Resque ➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information ➔ Comes with Monitoring System written in Ruby
  • 12. Redis + PyRes PRODUCER: from pyres import ResQ r = Resq() class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshot r.enqueue(WebPage, 'http://python.org') CONSUMER: ./pyres_worker screenies
  • 13. Message Queue A system for enabling asynchronous processing of discrete tasks.
  • 14. What can MQs help with? ➔ Everything that generic Queues can help with. ➔ Increase Reliability ➔ Cron Jobs (Celery)
  • 15. Available MQ Solutions ➔ RabbitMQ ➔ Amazon Simple Queue ➔ Apache MQ ➔ Gearman ➔ Starling ➔ OpenAMQ ➔ Sun Java Message Queue System
  • 16. Message Queue Protocols ➔ AMQP: Advanced Message Queue Protocol ➔ JMS: Java Messaging Service ➔ STOMP: Streaming Text Oriented Messaging Protocol
  • 17. Criteria for Broker Selection ➔ Difficulty in Recovery ➔ Relatively low level of required maintenance ➔ Ease of Deployment ➔ Durability ➔ Persistence ➔ Community Support ➔ Cluster Support ➔ What language is it written in?
  • 18. RabbitMQ ➔ A Message Broker ➔ Implements AMQP
  • 19. Pika & Other AMQP Libraries
  • 20. Celery ➔ An amazing Task Manager ➔ Batteries Included ➔ Uses RabbitMQ as broker (or almost anything) ➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid ➔ Supports multiprocessing. ➔ Distribute tasks easily. ➔ Makes life easy
  • 21. Celery Example from celery.task import task @task(): def take_screenshot(url): # magical code to take screenshot comes here
  • 22. How to put it all together?
  • 23. The General Setup Task Web App Broker Manager Worker Database Server (s)
  • 30. Email
  • 32. Isolate Isolate code, make reusable components.
  • 35. Review ➔ Choose a system according to needs ➔ Build a robust system ➔ Integrate seamlessly with your UI ➔ Isolate & Recycle: make it a habit ➔ Improve the overall UX! ➔ Please everyone, even yourself!
  • 36. References ➔ PyRes - http://pyres.readthedocs.org/ ➔ RabbitMQ - http://www.rabbitmq.com/ ➔ Celery - http://celeryproject.org/ ➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes