Message Queues - A Primer is an introduction to message queues and how they work with PHP.

1. Mike Willbanks | Barnes & Noble
Message Queues : A Primer

2. Housekeeping…
• Talk
– Slides will be posted after the talk.
• Me
– Sr. Web Architect Manager at NOOK
Developer
– Prior MNPHP Organizer
– Open Source Contributor
– Where you can find me:
• Twitter: mwillbanks G+: Mike Willbanks
• IRC (freenode): mwillbanks Blog:
http://blog.digitalstruct.com
• GitHub: https://github.com/mwillbanks

3. Agenda
• Message Queues?
• Protocols and Software
• Picking a Message Queue
• Best Practices

4. A Definition
“Message queues and mailboxes are software-engineering
components used for interprocess communication, or for inter-
thread communication within the same process. They use a queue
for messaging – the passing of control or of content.”

5. What is messaging?
“Messaging describes the sending and receiving of data (in the
form of messages) between systems. Messages are exchanged
between programs or applications, similar to the way people
communicate by email but with guarantees on delivery, speed,
security and the absence of spam.”

6. Producer Messages Consumer
Messages
Task
Messages
General Anatomy
Producer creates a message and pushes it to the queue; the
consumer reads from the queue and processes the message.

7. Describing Message Queues
l Pub/Sub
l FIFO buffer
l Push / Pull
l A way to communicate between
applications / systems.
l A way to decouple components.
l A way to offload work.

8. Why to use a Message Queue
l Offload Heavy Work
l Integration with Legacy Systems
l Asynchronous Processing
l Parallel Processing
l Process consistency
l Scalability

9. Unix Foundations
l “Write programs that work together.”
l “Do it in the background.”

10. Why it matters
l Web systems need to be geared to run things
asynchronously.
l Distribution of load
l System integrity

11. You’ve seen them before; they are used in most applications to
help them scale.
MESSAGE QUEUE EXAMPLES

15. When to make use of message queues in case you’re wondering.
GENERIC USE CASES

16. Notifications
Email, SMS, Push Messaging….

17. Photo Processing
Thumbnails, Resizing, Watermarking, Converting…

18. Video Processing
Resampling, Audio Overlay, Type Conversion…

19. Analytics
Web Server Logs, Log Aggregation, PHP Errors, etc.

20. Integrations
Save local first; push second.

21. AMQP
STOMP
XMPP
Vendor Specific
PROTOCOLS

22. AMQP
Advanced Message Queuing Protocol

23. Overview of AMQP
l AMQP Working Group (Community and
Vendor)
l Platform agnostic protocol.
l Completely open, interoperable and broadly
applicable.
l Many severs available and many client
libraries.

24. How it Works
Producer, Exchange, Queue, Consumer

25. How it Works
l AMQP utilizes exchanges, queues and
bindings.
l An exchange are routers with routing tables.
l A binding defines the routing rules.
l A queue is where the messages wait for a
consumer.

26. Exchanges

27. Understanding Exchanges
l Fanout Exchange
l No routing keys involved. Any message that is
sent to the exchange is sent to all queues bound
to that exchange.
l Direct Exchange
l Routing keys involved. A queue binds to the
exchange to request messages that match a
routing key exactly.
l Topic Exchange
l Routing keys involved. A queue binds to the
exchange to request messages that match a
routing key pattern.

28. Implementations
www.rabbitmq.com
Very popular and common message queue owned by
VMware.
qpid.apache.org
Long standing project; apache foundation.
www.openamq.org
Long standing project; ZeroMQ partner, no news since
2009.

29. Building a Queue
l An exchange, queue and bindings must be defined
first. Publishing can then commence after.
l Create the queue
l Create the exchange
l Bind to the queue.

30. Persistence?
l Default behavior is no persistence.
l How important are the messages?
l Just about all items have a level of persistence if you
would like them to survive on reboot.
l Mark exchanges, queues and messages as DURABLE.

31. PECL AMQP
l Extension compatible with AMQP specification
0-9-1.
l pecl install amqp

32. AMQP Client

33. AMQP Worker

34. STOMP
Simple (or Streaming) Text Orientated Messaging Protocol

35. Overview
l Simple protocol
l Behaviors follow very simple commands.
l Most message queues can communicate over
STOMP.

36. /queue/
msg
Connect Send Disconnect
/queue/ S
msg T
O
Connect Subscribe Disconnect
M
P
P
H
S
P
E
R
V
Read E
R
Ack
How It Works
l When you send in a message, you tell it which
queue to go to.
l When you subscribe you request a queue.

37. Sever Implementations
activemq.apache.org
One of the oldest message queues existing;
a apache foundation project
activemq.apache.org/apollo
Next generation ActiveMQ
www.rabbitmq.com
Very popular and common message queue owned by
VMware.
www.jboss.org/hornetq
Supported by Red Hat Middle Ware division, picking
up steam.

38. PECL Stomp
l pecl install stomp
l That was easy J

39. STOMP Client

40. STOMP Worker

41. XMPP
Extensible Messaging and Presence Protocol
(Although not really a “Message Queue”)

42. Overview
l Best for real-time data.
l Leveraging pub/sub can turn it into more of a generic message
system.
l Multiple libraries
l JAXL - https://github.com/abhinavsingh/JAXL
l Xmpp - https://github.com/alexmace/Xmpp

43. Publish
to, id,
message
Subscribe
id,
from, to,
message
Sub1 Sub2 Sub3
XEP-0060: Publish-Subscribe
l Specification for implementing Publish Subscribe
models.
l Extension to the original XMPP specification.

44. Publish

45. Message

46. Various others; including Gearman, ZeroMQ, etc.
SPECIAL MESSAGE QUEUES

47. Overview
l There are job servers available that are more
flexible or more specific.
l Extreme Flexibility
l Job Severs
l Cloud Messaging

48. ZeroMQ
The ultimate in message queue flexibility.
Socket library that acts as a concurrency framework.
Contains a PHP extension.

49. Several Types of Queues
Request / Reply
Publish / Subscribe
Parallel Pipeline
Fair Queuing
And more…

50. ZeroMQ Client Example

51. ZeroMQ Worker Example

52. Gearman
l Application framework for farming out work.
l Job sever for asynchronous or synchronous
messages.

53. Gearman
l Pass a job to the job server
l Worker receives the job and processes
l Ability to persist if enabled; default is in-memory.

54. Gearman Client Example

55. Gearman Worker Example

56. Beanstalkd
l Asynchronous Job Queue
l Good scheduling system (aka delays).
l User land PHP clients.

57. Application Code
Beanstalkd Client API
(Pheanstalk)
Beanstalkd Server
Beanstalkd Worker API
(Pheanstalk)
Worker Application Code
Beanstalkd
l Pass a job to the job server
l Worker receives the job and processes
l Ability to persist to binlog; default is in-memory

58. Pheanstalk Client

59. Pheanstalk Worker

60. What you want to look for.
PICKING A MESSAGE QUEUE

61. The Trifecta
l?
Horizonta
Vertical?
Scale
Messages
Messages Messages
Messages Messages
Messages
Messages Messages
Messages
ry?
ed? In-Memo
R a w Sp e ce?
es?
Performance Durability
Persisten
# Messag n?
Replicatio

62. Standards
• A recognized standard?
– AMQP? STOMP? XMPP?
• How many developers?
– Will an unfortunate event kill off the
product?
• Undocumented protocol?
– Forget about it!

63. Delivery Policies
Store Message Saved?
Receive Message TTL?
Process Message ACK?

64. Message Policies
• Handling in-active messages?
• Messages that failed processing?
• Time to live?
• Retry?
• Ability to check message status?

65. Routing Policies
• Fanout?
• Direct?
• Topic?
• Broadcast?
• Etc?

66. Security Policies
• Require authentication?
– LDAP/AD integration?
• Connection Restrictions?
• Require logging?
– Compliance
– Intrusion Detection

67. Rules of the road my friends…
BEST PRACTICES

68. Messages
• Formatting
– JSON or XML are great options.
• Please no serialized PHP objects.
• Message Size
– Only as large as necessary.
• Don’t send a binary object through the queue.

69. Workers
• Dumb as possible!
– Single message type
• Easier to scale.
– Single operation
• Easy to debug and far more flexible.

70. PHP Daemons
• Prevent Memory Leaks
– Detection
– Cycle Workers
• Handle Signals
– Properly shutdown!
– Watch out for OS service kills
• Sleeping is good J

72. Database
Worker Server Web Server
Message Queue
Environment Integration
Web Server -> Message Queue
Worker Server -> Message Queue

73. Database
Web Server
Worker Server
Worker Server
Message Queue
Send to 1
Message Queue
High Availability
Insert multiple message queue servers + multiple worker nodes.
Each worker node can connect to as many message queue servers
as necessary.

74. Supervisord
Process monitoring, logging and more!

75. Installing Supervisord
• Requires: python-setuptools
sudo easy_install supervisor!
• Generate Configuration
sudo echo_supervisord_conf > /etc/
supervisord.conf!
!

77. These slides will be posted to SlideShare & SpeakerDeck.
SpeakerDeck: http://speakerdeck.com/u/mwillbanks
Slideshare: http://www.slideshare.net/mwillbanks
Twitter: mwillbanks
G+: Mike Willbanks
IRC (freenode): mwillbanks
Blog: http://blog.digitalstruct.com
GitHub: https://github.com/mwillbanks
QUESTIONS?