In this session, we learn about best practices to help you integrate serverless applications using Amazon Simple Queue Service (Amazon SQS). We explore how to tune AWS Lambda and Amazon SQS to scale your existing applications without having to worry about provisioning capacity. We also discuss how to bridge high-scale serverless applications with limited server-full applications such as relational databases.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Best practices for queue processing in
serverless applications
Luay Kawasme
Sr. Manager, Software Development
Amazon SQS
Amazon Web Services
M A D 3 1 3

2. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build microservices with serverless technologies
No infrastructure to
provision or manage
Scales automatically
by unit of consumption
Pay for value Built-in security
highly available compute

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What is Amazon SQS?

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What is Amazon Lambda?
Synchronous push model
Pull model
Asynchronous push model
Alexa Skill

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Lambda invocation integrations
Amazon S3
Amazon DynamoDB
Amazon Kinesis Data Streams
Amazon SNS
Amazon Simple Email Service
Amazon SQS
Amazon Cognito
AWS CloudFormation
Amazon CloudWatch Logs
Amazon CloudWatch Events
AWS CodeCommit
AWS Config
Amazon Alexa
Amazon Lex
Amazon API Gateway
AWS IoT Button
and more…

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Pipes and filters
A single event triggers a sequence of processing steps. Typically, an event traverses
a pipeline and goes through several transformations
(e.g., filtering, enrichment).
Message
Queue
Lambda function
Queue

8. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Serverless message processing
1. Message inserted into to a queue
2. Lambda polls queue and
invokes function
Queue Lambda Function

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Serverless message processing
1. Message inserted into to a queue
2. Lambda polls queue and
invokes function
3. Lambda removes message from
queue
Queue Lambda Function

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SQS to Lambda
1. Message inserted into to a queue
2. Lambda polls queue and invokes
function
3. Lambda removes message
from queue
Serverless processing
Lambda polls queue for messages
Processed in batches
Message processing
Messages are stored durably
Queue accepts messages at any rate
No need to provision capacity
Visibility timeout for failure processing
Queue Lambda function

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Lambda async push vs. Amazon SQS
AWS Lambda async invoke
• Direct integration
• Automatic scaling
• No batching
• Simple DLQ policy
• No backlog visibility
• No extra queue features
• No extra cost
Amazon SQS + AWS Lambda
• Two resources
• Automatic scaling
• Supports batching
• Advanced DLQ policy
• Amazon CloudWatch metrics
• All SQS features
• Cost of requests to SQS

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Advanced Amazon SQS features
Advanced DLQ policy
Batching
Delayed messages
Server-side encryption with customer managed key (CMK)
Retention period
Message attributes
CloudWatch metrics
Purge operation

14. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consuming from Amazon SQS
C
C
C

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Inside Amazon SQS: Visibility timeouts

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SQS to Lambda: Details
Lambda
function
Message
batch
Queue
Message
batch
Message
batch
Message
batch
Message
batch
Lambda
function
Lambda
function
Lambda
function
Lambda
function
Lambda
function
Message
batch

28. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

30. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Highly scalable processing
1. Message inserted into to a queue
2. Lambda polls queue and
invokes function
3. Lambda removes message from
queue
Queue Lambda Function

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Downstream dependency with a limit
1. Message inserted into to a queue
2. Lambda polls queue and
invokes function
3. Lambda removes message from
queue
Queue Lambda Function

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Limiting concurrency

34. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SNS or Amazon SQS as an event source?
Comparison SNS as an event source for Lambda SQS as an event source for Lambda
Scale/
concurrency
Automatic scaling, per function concurrency Same
Durability Durable storage across multiple AZs Same
Consumption
model
Messages pushed to Lambda; Amazon SNS can
“fanout” the same message
Lambda polls for messages and invokes function; each
message is consumed and deleted by Lambda
Resilience Once delivered, message is deleted, even if
Lambda function does not complete processing
Lambda function is responsible for deleting message
after processing; ability to monitor and control queue
Persistence/
retries
Delivery retry logic that extends up through
potentially 13 hours
Message retention for up to 14 days

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SQS to Lambda: Tips
• For low-concurrency, high-duration functions, set Visibility Timeout to 5x
function duration. This will avoid receiveCount retries and DLQ.
• Check that visibilityTimeout is always greater than function duration to avoid
duplicate invokes.
• Even a fraction of errors will induce backpressure and interfere with ramping up
to full concurrency.
• The number of invocations does not equal throughput. Maximum batching is
not guaranteed.

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Luay Kawasme
@luay_ca
https://www.linkedin.com/in/luayk/