Presented at Oklahoma Drupal Users Group on November 21, 2013.

1. Message Queues and Drupal

2. About Me
●
●
●
●
Name: Brian Altenhofel
IRC: VeggieMeat
Twitter: @BrianAltenhofel
Drupal: #550616
What I Do: Commerce, Third-party APIs, Migrations, Devops,
anything with data...
Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro

4. What are message queues?

5. Consumer
Producer
Producer
Queue
Consumer
Consumer
Producer
What are message queues?
Publisher
Publisher
Publisher
Subscriber
Queue
Subscriber
Subscriber

6. You already use queues.

7. In Drupal...

8. Queues are pluggable
Backends available on Drupal.org:
AWS SQS – http://drupal.org/project/aws_sqs
Beanstalkd – http://drupal.org/project/beanstalkd
Redis – http://drupal.org/project/redis
STOMP – http://drupal.org/project/stomp
(And.... still in sandbox but ready to go...)
Openstack Marconi http://drupal.org/sandbox/BrianAltenhofel/2107851

9. Using a Different Queue
As easy as:
●
Global:
$conf['queue_default_class'] = 'MarconiQueue';
●
Specific queue
$conf['queue_class_{queue_name}'] = 'MarconiQueue';

10. Queues and Drush
●
drush queue-list
●
drush queue-run [queue_name]

11. Code.

12. Code.
function my_module_create_queue() {
$queue = DrupalQueue::get('my_queue');
$queue­>createQueue();
}

13. Code.
function my_module_create_queue_item() {
$queue = DrupalQueue::get('my_queue');
$queue_data = array(
'my_key' => 'my_value',
);
return $queue­>createItem($queue_data);
}

14. Code.
function my_module_count_queue_items() {
$queue = DrupalQueue::get('my_queue');
return $queue­>numberOfItems();
}

15. Code.
function my_module_process_queue_item() {
$queue = DrupalQueue::get('my_queue');
$item = $queue­>claimItem(30);
if ($item && $item['my_key'] == 'my_value') {
$queue­>deleteItem($item);
}
else {
$queue­>releaseItem($item);
}
}

16. Code.
function my_module_cron_queue_info() {
$queues['my_queue'] = array(
'worker callback' =>
'my_module_process_queue_item',
'time' => 60,
// Time for cron to spend on this queue, not
// the lease time.
);
return $queues;
}

17. It's really that simple.

18. Debugging
●
watchdog() is a Drupaler's best friend
●
Log everything (for both your sanity and insanity)
●
(probably want to ship these logs if you're processing a lot of items)
(if you want to get started on that):
http://www.youtube.com/watch?v=p0Av29yaBEI

19. Monitoring
●
Monitor queue depths with your favorite tool (Nagios, Zabbix,
Zenoss, etc.)
●
In a pinch, you can get an idea with $queue->numberOfItems().
●
Take action if queues get too deep

20. Any questions?
@BrianAltenhofel
IRC: VeggieMeat (MANY channels)
brian.altenhofel@vmdoh.com