You may know that queues can help with long-running tasks, but did you know they can help you make your application easier to debug, more performant, and scale in the cloud? Taking the real-world example of a contest app deployed on Azure, we'll see how easy queues can be to implement. You'll see how the smart use of queues can enable your application to handle many more users with the same code, break components across servers, and help you keep your app responsive.
3. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Social Contest
â˘Show oďŹ Azure + PHP
â˘People submit tweets to enter contest
â˘Pull speciďŹed keywords from Twitter queue (preďŹltered by
Node.js app)
â˘Human admins ďŹlter out inappropriate content
â˘Humans or computer pulls out winner from approved entries,
on timer or arbitrarily
â˘Display latest entries and latest winners to public
3
4. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Stretch goals
â˘Allow any size contest
â˘Assume global contest with distributed moderators
4
8. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Quick refresher
Performance vs. Scaling
Vertical vs. Horizontal Scaling
8
9. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Traditional database design
â˘Group by kind
â˘Keep metadata with object or in metadata tables
â˘Objects (Document, Person, Account) are most important
â˘Reinforced by TableGateway, ActiveRecord patterns, and ORM
and framework module generator defaults
â˘Works for 80% of cases
9
10. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Tradeoffs of traditional design
â˘Everythingâs in one table, even when you routinely only need a
subset
â˘Typically one master writes, replicants read
â˘Design is focused around what something is, not its state or
other attribute
â˘Requires creative solutions for horizontal scaling
â˘Encourages (but does not require) centralizing logic for a given
type of data
10
11. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Worst of allâŚ
â˘This design really didnât show oďŹ all the stuďŹ in Azure
11
12. (We had a week left)
Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Redesign time
12
13. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Redesign goals
â˘Include as much Azure stuďŹ as is reasonable
â˘Implement the stretch goals of scalability
13
14. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Queues to the rescue!
(Plus some other cool stuďŹ)
14
15. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
New approach
â˘Keep long term but fast storage
â˘Central concern is not the Thing (Entry) but Status
â Unapproved
â Approved
â Denied
â Winner
â˘Separate updates to long term storage from status changes
â˘Minimal impact to working code
15
16. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Azure queuing options
Azure Queues (duh)
â˘Simple
â˘âShortâ-lived (<7 days)
â˘Used within Azure
â˘Uses REST
â˘Can track message
processing
16
Azure Service Bus
â˘Enterprisey
â˘Long-lived
â˘In Azure or private cloud
â˘Can use AMQP, REST, or
API
â˘Can publish/subscribe
â˘Can batch requests
â˘Can guarantee FIFO
â˘etc.
See https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx
17. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
More options
â˘Anything you can install on Linux or Windows (RabbitMQ,
ZeroMQ, Kafka, Kestrel, ActiveMQ, etc.)
â˘Any relational or NoSQL database
â˘Azure Tables - Simple REST NoSQL store with a twist
17
18. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Solutions
â˘Long term storage and display retrieval: Azure Table
â˘Since entries could begin long before a conference, use
Service Bus to store changes in status
â˘Have daemons pull incoming status changes out and write
them to the Table
18
20. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Azure Table basics
20
require_once 'vendorautoload.php';â¨
use WindowsAzureCommonServicesBuilder;â¨
use WindowsAzureCommonServiceException;â¨
use WindowsAzureTableModelsEntity;â¨
use WindowsAzureTableModelsEdmType;â¨
â¨
// Create table REST proxy.â¨
$tableRestProxy = ServicesBuilder::getInstance()->createTableService($connectionString);â¨
â¨
try {â¨
// Create table.â¨
$tableRestProxy->createTable("mytable");â¨
} catch(ServiceException $e){â¨
// Handle exception based on error codes and messages.â¨
}â¨
â¨
$entity = new Entity();â¨
$entity->setPartitionKey("pk");â¨
$entity->setRowKey("1");â¨
$entity->addProperty("PropertyName", EdmType::STRING, "Sample");â¨
â¨
try {â¨
$tableRestProxy->insertEntity("mytable", $entity);â¨
} catch(ServiceException $e){â¨
// Handle exception based on error codes and messages.â¨
}
21. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Create and send with Service Bus
21
require_once 'vendorautoload.php';â¨
use WindowsAzureServiceBusModelsQueueInfo;â¨
use WindowsAzureCommonServiceException;â¨
use WindowsAzureCommonServicesBuilder;â¨
â¨
$serviceBusRestProxy = ServicesBuilder::getInstance()-
>createServiceBusService($connectionString);â¨
â¨
try {â¨
$queueInfo = new QueueInfo("myqueue");â¨
$serviceBusRestProxy->createQueue($queueInfo);â¨
} catch(ServiceException $e) {â¨
// handle errorâ¨
}â¨
â¨
try {â¨
// Create message.â¨
$message = new BrokeredMessage();â¨
$message->setBody("my message");â¨
â¨
// Send message.â¨
$serviceBusRestProxy->sendQueueMessage("myqueue", $message);â¨
} catch(ServiceException $e) {â¨
// handle errorâ¨
}
22. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Receive with Service Bus
22
require_once 'vendorautoload.php';â¨
use WindowsAzureServiceBusModelsQueueInfo;â¨
use WindowsAzureCommonServiceException;â¨
use WindowsAzureCommonServicesBuilder;â¨
â¨
$serviceBusRestProxy = ServicesBuilder::getInstance()-
>createServiceBusService($connectionString);â¨
â¨
try {â¨
// Set the receive mode to PeekLock (default is ReceiveAndDelete).â¨
$options = new ReceiveMessageOptions();â¨
$options->setPeekLock(true);â¨
â¨
// Receive message.â¨
$message = $serviceBusRestProxy->receiveQueueMessage("myqueue", $options);â¨
echo "Body: ".$message->getBody()."<br />";â¨
echo "MessageID: ".$message->getMessageId()."<br />";â¨
â¨
// *** Process message here ***â¨
â¨
// Delete message.â¨
$serviceBusRestProxy->deleteMessage($message);â¨
} catch(ServiceException $e){â¨
// handle errorâ¨
}
23. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Benefits
â˘Queues add safety: if processing fails, main store is unchanged
â˘App doesnât wait for process-update-delete cycle
â Concerns more separated
â˘Can move queue processing to separate machines
â˘Trivial to move to diďŹerent stores for each status
â˘Very performant with up-to-second data
23
24. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Challenges
â˘No full ACID
â˘Safety is largely in the application layer
â˘Potential for race conditions
â Humans suck
24
25. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Takeaways
â˘Look for more than just long processes
â˘Use to decouple functions
â˘Look for status changes
â˘Is the type of data the most important aspect of your data?
â It usually is!
â˘Design for replacement of components
â Makes changes easier (not easy)
25
26. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Links
â˘Azure: http://azure.microsoft.com/en-us/
â˘Social Contest:â¨
https://github.com/MusketeersMe/SocialContest
â˘Me
â @SandyS1
â http://phparch.com/
â˘Feedback! https://joind.in/event/dc-php-march-2015
â˘Lone Star PHP (April 16â18): http://lonestarphp.com
â˘php[tek] (May 18â22) http://tek.phparch.com
â˘Slides will be at: http://www.slideshare.net/SandySmith
26
27. Architecting with Queues - Sandy Smith - DCPHP - 3/11/15
Image credits
â˘Questions? by Valerie Everettâ¨
https://ďŹic.kr/p/5zEjFG
â˘Expanded coke zero can! by Audin Malminâ¨
https://ďŹic.kr/p/5Ldjx8
27