Talk given at Zendcon showing how to implement several messaging patterns with RabbitMQ and PHP

1. Integrating PHP
With RabbitMQ
ÁlvaroVidela | The NetCircle
Zendcon 2010
Tuesday, November 2, 2010

2. Who?
Tuesday, November 2, 2010

3. About Me
• Development Manager at TheNetCircle.com
• Writing “RabbitMQ in Action” for Manning
• Blog: http://videlalvaro.github.com/
• Twitter: @old_sound
Tuesday, November 2, 2010

4. Why Do I need
RabbitMQ?
Tuesday, November 2, 2010

5. The User
Tuesday, November 2, 2010

6. I don’t want to wait
till your app resizes
my image!
Tuesday, November 2, 2010

7. The Product Owner
Tuesday, November 2, 2010

8. Can we also notify the
user friends when she
uploads a new image?
Tuesday, November 2, 2010

9. Can we also notify the
user friends when she
uploads a new image?
I forgot to mention we need it for tomorrow…
Tuesday, November 2, 2010

10. The Sysadmin
Tuesday, November 2, 2010

11. Dumb!You’re delivering
full size images!
The bandwidth bill has
tripled!
Tuesday, November 2, 2010

12. Dumb!You’re delivering
full size images!
The bandwidth bill has
tripled!
We need this fixed for yesterday!
Tuesday, November 2, 2010

13. The Developer in the
other team
Tuesday, November 2, 2010

14. I need to call your PHP
stuff but from Python
Tuesday, November 2, 2010

15. I need to call your PHP
stuff but from Python
And also Java starting next week
Tuesday, November 2, 2010

16. You
Tuesday, November 2, 2010

17. FML!
Tuesday, November 2, 2010

18. Is there a solution?
Tuesday, November 2, 2010

19. RabbitMQ & AMQP
Tuesday, November 2, 2010

20. AMQP
Tuesday, November 2, 2010

21. AMQP
• Advanced Message Queuing Protocol
• Suits Interoperability
• Completely Open Protocol
• Binary Protocol
• AMQP Model
• AMQP Wire Format
Tuesday, November 2, 2010

22. AMQP Model
• Exchanges
• Message Queues
• Bindings
• Rules for binding them
Tuesday, November 2, 2010

23. AMQP Wire Protocol
• Functional Layer
• Transport Layer
Tuesday, November 2, 2010

24. Message Flow
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html
Tuesday, November 2, 2010

25. Exchange Types
• Fanout
• Direct
• Topic
Tuesday, November 2, 2010

26. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
Fanout_Exchange.html
Tuesday, November 2, 2010

27. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
Direct_Exchange.html
Tuesday, November 2, 2010

28. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
Topic_Exchange.html
Tuesday, November 2, 2010

29. Usage Scenarios
Tuesday, November 2, 2010

30. Usage Scenarios
• Batch Processing
Tuesday, November 2, 2010

31. Usage Scenarios
• Batch Processing
• Image Uploading
Tuesday, November 2, 2010

32. Usage Scenarios
• Batch Processing
• Image Uploading
• Distributed Logging
Tuesday, November 2, 2010

33. Scenario
Batch Processing
Tuesday, November 2, 2010

34. Requirements
Tuesday, November 2, 2010

35. Requirements
• Generate XML
Tuesday, November 2, 2010

36. Requirements
• Generate XML
• Distribution Over a Cluster
Tuesday, November 2, 2010

37. Requirements
• Generate XML
• Distribution Over a Cluster
• Elasticity - Add/Remove new workers
Tuesday, November 2, 2010

38. Requirements
• Generate XML
• Distribution Over a Cluster
• Elasticity - Add/Remove new workers
• No Code Changes
Tuesday, November 2, 2010

39. Design
Tuesday, November 2, 2010

40. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

41. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

42. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

43. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

44. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

45. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();
$conn->close();
Tuesday, November 2, 2010

46. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

47. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

48. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

49. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

50. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

51. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

52. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

53. Scenario
Upload Pictures
Tuesday, November 2, 2010

54. Requirements
Tuesday, November 2, 2010

55. Requirements
• Upload Picture
Tuesday, November 2, 2010

56. Requirements
• Upload Picture
• Reward User
Tuesday, November 2, 2010

57. Requirements
• Upload Picture
• Reward User
• Notify User Friends
Tuesday, November 2, 2010

58. Requirements
• Upload Picture
• Reward User
• Notify User Friends
• Resize Picture
Tuesday, November 2, 2010

59. Requirements
• Upload Picture
• Reward User
• Notify User Friends
• Resize Picture
• No Code Changes
Tuesday, November 2, 2010

60. Design
Tuesday, November 2, 2010

61. Design
Tuesday, November 2, 2010

62. Design
Tuesday, November 2, 2010

63. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
Tuesday, November 2, 2010

64. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
Tuesday, November 2, 2010

65. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
Tuesday, November 2, 2010

66. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
Tuesday, November 2, 2010

67. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
Tuesday, November 2, 2010

68. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

69. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

70. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

71. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

72. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

73. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true,
false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
while(count($channel->callbacks)) {
$channel->wait();
}
Tuesday, November 2, 2010

74. Consumer Code
$consumer = function($msg){
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};
Tuesday, November 2, 2010

75. Consumer Code
$consumer = function($msg){
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};
Tuesday, November 2, 2010

76. Consumer Code
$consumer = function($msg){
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};
Tuesday, November 2, 2010

77. Consumer Code
$consumer = function($msg){
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};
Tuesday, November 2, 2010

78. Consumer Code
$consumer = function($msg){
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};
Tuesday, November 2, 2010

79. Scenario
Distributed Logging
Tuesday, November 2, 2010

80. Requirements
Tuesday, November 2, 2010

81. Requirements
• Several Web Servers
Tuesday, November 2, 2010

82. Requirements
• Several Web Servers
• Logic Separated by Module/Action
Tuesday, November 2, 2010

83. Requirements
• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels:
Tuesday, November 2, 2010

84. Requirements
• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels:
• Info,Warning, Error
Tuesday, November 2, 2010

85. Requirements
• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels:
• Info,Warning, Error
• Add/Remove log listeners at will
Tuesday, November 2, 2010

86. Design
Tuesday, November 2, 2010

87. Design
Tuesday, November 2, 2010

88. Design
Tuesday, November 2, 2010

89. Design
Tuesday, November 2, 2010

90. Design
Tuesday, November 2, 2010

91. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
$channel->basic_publish($msg, 'logs',
server1.user.profile.info');
Tuesday, November 2, 2010

92. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
$channel->basic_publish($msg, 'logs',
server1.user.profile.info');
Tuesday, November 2, 2010

93. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
$channel->basic_publish($msg, 'logs',
server1.user.profile.info');
Tuesday, November 2, 2010

94. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
$channel->basic_publish($msg, 'logs',
server1.user.profile.info');
Tuesday, November 2, 2010

95. Consumer Code
Get messages sent by host:
server1
Tuesday, November 2, 2010

96. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('server1-logs', false, true,
false, false);
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Tuesday, November 2, 2010

97. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('server1-logs', false, true,
false, false);
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Tuesday, November 2, 2010

98. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('server1-logs', false, true,
false, false);
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Tuesday, November 2, 2010

99. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('server1-logs', false, true,
false, false);
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Tuesday, November 2, 2010

100. Consumer Code
Get all error messages
Tuesday, November 2, 2010

101. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('error-logs', false, true,
false, false);
$channel->queue_bind('error-logs', 'logs', '#.error');
Tuesday, November 2, 2010

102. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('error-logs', false, true,
false, false);
$channel->queue_bind('error-logs', 'logs', '#.error');
Tuesday, November 2, 2010

103. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('error-logs', false, true,
false, false);
$channel->queue_bind('error-logs', 'logs', '#.error');
Tuesday, November 2, 2010

104. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
$channel->queue_declare('error-logs', false, true,
false, false);
$channel->queue_bind('error-logs', 'logs', '#.error');
Tuesday, November 2, 2010

105. PHP Integration
Tuesday, November 2, 2010

106. php-amqplib
• http://github.com/tnc/php-amqplib
Tuesday, November 2, 2010

107. php-amqplib
• http://github.com/tnc/php-amqplib
• Fork of http://code.google.com/p/php-
amqplib/
Tuesday, November 2, 2010

108. php-amqplib
• http://github.com/tnc/php-amqplib
• Fork of http://code.google.com/p/php-
amqplib/
• PHP 5.3 Compatible
Tuesday, November 2, 2010

109. php-amqplib
• http://github.com/tnc/php-amqplib
• Fork of http://code.google.com/p/php-
amqplib/
• PHP 5.3 Compatible
• Improved performance
Tuesday, November 2, 2010

110. php-amqplib
• http://github.com/tnc/php-amqplib
• Fork of http://code.google.com/p/php-
amqplib/
• PHP 5.3 Compatible
• Improved performance
• Used in production one+ year
Tuesday, November 2, 2010

111. php-amqplib
$ git clone http://github.com/tnc/php-amqplib.git
<?php
require_once('./php-amqplib/amqp.inc');
$conn = new AMQPConnection(‘localhost’, 5672, guest, guest);
$channel = $conn->channel();
?>
Tuesday, November 2, 2010

112. Why RabbitMQ?
Tuesday, November 2, 2010

113. RabbitMQ
• Enterprise Messaging System
• Open Source MPL
• Written in Erlang/OTP
• Commercial Support
Tuesday, November 2, 2010

114. Features
• Reliable and High Scalable
• Easy To install
• Easy To Cluster
• Runs on:Windows, Solaris, Linux, OSX
• AMQP 0.8 - 0.9.1
Tuesday, November 2, 2010

115. Client Libraries
• Java
• .NET/C#
• Erlang
• Ruby, Python, PHP, Perl,AS3, Lisp, Scala,
Clojure, Haskell
Tuesday, November 2, 2010

116. Docs/Support
• http://www.rabbitmq.com/documentation.html
• http://dev.rabbitmq.com/wiki/
• #rabbitmq at irc.freenode.net
• http://www.rabbitmq.com/email-archive.html
Tuesday, November 2, 2010

117. One Setup for HA
Tuesday, November 2, 2010

118. Conclusion
Tuesday, November 2, 2010

119. Conclusion
• Flexibility
Tuesday, November 2, 2010

120. Conclusion
• Flexibility
• Scalability
Tuesday, November 2, 2010

121. Conclusion
• Flexibility
• Scalability
• Interoperability
Tuesday, November 2, 2010

122. Conclusion
• Flexibility
• Scalability
• Interoperability
• Reduce Ops
Tuesday, November 2, 2010

123. Questions?
Tuesday, November 2, 2010

124. Thanks!
Álvaro Videla
http://twitter.com/old_sound
http://github.com/videlalvaro
http://github.com/tnc
http://www.slideshare.net/old_sound
Tuesday, November 2, 2010