RedisConf17 breakout session

1. Michael Labib, Global Specialist Data Architect
Turbo-charge your apps with
Amazon ElastiCache for Redis
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
ElastiCache
@michaelslabib

2. “Generally, the craft of programming is the factoring of a set of
requirements into a set of functions and data structures.”
GET quoteREDIS:6379>
-- Douglas Crockford --

3. What to Expect from the Session
LRANGE agenda 0 2
1) “Amazon ElastiCache Overview”
2) “Engine Enhancements”
3) “Usage Patterns”
REDIS:6379>

4. SMEMBERS featuresREDIS:6379>
Amazon
ElastiCache
1) “Easy to deploy & monitor”
2) “Enhanced Redis Engine“
3) “Highly Available with fast auto-failover”
4) “Fully managed; Zero admin”
5) “Cost Effective”
6) “Redis and Memcached”
7) “Microsecond performance”
8) “Secured in VPC”

5. SMEMBERS features
REDIS:6379>
Amazon
ElastiCache
1) “Easy to deploy & monitor”
AWS
Config
Amazon
CloudWatch
AWS
CloudTrail
AWS
CloudFormation
AWS
Management
Console
AWS CLI
and SDKs
alarm
REDIS:6379>
hget feature:details “deploy-monitor”
Amazon
SNS
Email
Notification
AWS
Lambda

6. SMEMBERS features
REDIS:6379>
Amazon
ElastiCache
REDIS:6379>
hget feature:details “enhancements”
2) “Enhanced Redis Engine”
Optimized Swap Memory
•Mitigate the risk of increased swap usage
during syncs and snapshots.
Dynamic write throttling
•Improved output buffer management when
the node’s memory is close to being
exhausted.
Smoother failovers
•Clusters recover faster as replicas avoid
flushing their data to do a full re-sync with
the primary.

7. SMEMBERS ec-team:open-source:contributors
Amazon
ElastiCache
REDIS:6379>
https://raw.githubusercontent.com/antirez/redis/2.8/00-RELEASENOTES
(from https://github.com/antirez/redis/issues/1814)
1) “Kevin McGehee”
2) “Qu Chen”
hgetall ec-team:open-source:contributionsREDIS:6379>
1) PSYNC2 (Redis 4.0)
https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES
3) BGSAVE (Redis 3.2)
https://raw.githubusercontent.com/antirez/redis/3.2/00-RELEASENOTES
5) MIGRATE (Redis 3.0)
7) MASTER TIMEOUT (Redis 2.8)
https://raw.githubusercontent.com/antirez/redis/3.0/00-RELEASENOTES
https://raw.githubusercontent.com/antirez/redis/2.8/00-RELEASENOTES
9) INCREASE 2Billion+ KEYS in a DATASET (Redis 2.8)
2)
4)
6)
8)
10)

8. SMEMBERS features
REDIS:6379>
Amazon
ElastiCache
5) “Cost Effective”
REDIS:6379>
hget feature:details “cost”
• Get Started for Free! - Get 750 hours free per month for an entire year
with a micro node
• On Demand – Pay as low of 1.7 cents / hour
• Reserved Instances – Optimize further by reserving 1 or 3 year capacity
and save up to 65% using ElastiCache Ris
• No Cross-AZ Data Transfer Costs – No charge associated with
multi-AZ cluster communication

9. ElastiCache Redis Reference Architecture
Availability Zone B Availability Zone CAvailability Zone A
REDIS:6379> hget feature:details “ref-arch”

10. ElastiCache Redis Reference Architecture
Availability Zone B Availability Zone CAvailability Zone A
Private SubnetPrivate Subnet Private Subnet
REDIS:6379> hget feature:details “ref-arch”

11. ElastiCache Redis Reference Architecture
Availability Zone B Availability Zone CAvailability Zone A
Private SubnetPrivate Subnet Private Subnet
REDIS:6379> hget feature:details “ref-arch”
security group security group security group

12. ElastiCache Redis Reference Architecture
Availability Zone B Availability Zone CAvailability Zone A
Private SubnetPrivate Subnet Private Subnet
REDIS:6379> hget feature:details “ref-arch”
security group security group security group
Primary
Elasticache Redis Cluster
Amazon S3
bucket
REDIS RDB
snapshot

13. ElastiCache Redis Reference Architecture
Availability Zone B Availability Zone CAvailability Zone A
Private SubnetPrivate Subnet Private Subnet
REDIS:6379> hget feature:details “ref-arch”
security group security group security group
Primary
Elasticache Redis Cluster
security group
Public Subnet
security group
Public Subnet Public Subnet Low Latency
Secure
Direct Access
Amazon S3
bucket
REDIS RDB
snapshot
security group

14. ElastiCache for Redis Failure Scenarios
“Everything fails, all the time.”
GET quote:failureREDIS:6379>
-- Werner Vogels, CTO Amazon.com --

15. Availability Zone A
slots 0 - 5454 slots 5455 – 10909
Redis Cluster
slots 10910 – 16363
Availability Zone B Availability Zone C
slots 5455 – 10909 slots 5455 – 10909slots 0 - 5454 slots 0 - 5454
slots 10910 – 16363 slots 10910 – 16363
Scenario 1: Single Primary Failure

16. Availability Zone A
slots 0 - 5454 slots 5455 – 10909
Redis Cluster
Scenario 1: Single Primary Failure
slots 10910 – 16363
Availability Zone B Availability Zone C
slots 5455 – 10909 slots 5455 – 10909slots 0 - 5454 slots 0 - 5454
slots 10910 – 16363
Mitigation:
1. Automatic Failure Detection & Replica Promotion (~15-30s)
2. Repair Failed Node
slots 10910 – 16363

17. Availability Zone A
slots 0 - 5454 slots 5455 – 10909
Redis Cluster
Scenario 2: Majority of Primaries Fail
slots 10910 – 16363
Availability Zone B Availability Zone C
slots 5455 – 10909 slots 5455 – 10909slots 0 - 5454 slots 0 - 5454
slots 10910 – 16363slots 10910 – 16363

18. Availability Zone A
slots 0 - 5454 slots 5455 – 10909
Redis Cluster
slots 10910 – 16363
Availability Zone B Availability Zone C
slots 5455 – 10909 slots 5455 – 10909slots 0 - 5454 slots 0 - 5454
Mitigation: Redis enhancements on ElastiCache
• Automatic Failure Detection and Replica Promotion
• Repair Failed Nodes
slots 10910 – 16363slots 10910 – 16363
Scenario 2: Majority of Primaries Fail

20. Usage Patterns
Session
Management
Database
Caching
APIs
(HTTP responses)
IOT
Streaming Data
Analytics
(Filtering/Aggregation)
Pub/Sub
Social Media
(Sentiment Analysis)
Standalone
Database
(Metadata Store)
Leaderboards

21. Database Caching
Clients
Amazon
ElastiCache
Redis
Amazon
DynamoDB
Elastic Load
Balancing
Amazon
EC2
Amazon
RDS
AWS
Lambda
write-through
reads/
writes
DDB streams
mysql.lambda_async
Simplified write-through
with RDS and DynamoDBreads/
writes
https://aws.amazon.com/caching/database-caching/

22. Amazon
Kinesis
Analytics
Amazon
Kinesis
Streams
Amazon
Kinesis
Streams
Amazon
ElastiCache
(Redis)
cleansed
stream
Streaming Data Enrichment / Processing
DataSources
raw
stream
Subscribers
Lambda Function 1
ContinualData
Filtering/Enrichment
Real-time
Pub/Sub
Lambda Function 2

23. Big Data Architectures using Redis
Amazon Kinesis
DataSources
AWS Lambda
Apache Storm
on EMR
Spark Streaming
on EMR
Amazon
Kinesis App
Amazon
EC2
AWS IoT
Amazon
ElastiCache
Collect
Store
Process
Amazon
S3
Apache Kafka
AWS
Lambda
Custom
App
Spark on
EMR
Analyze

24. Event Data
AWS
Lambda
Amazon
ElastiCach
e
Amazon
DynamoDB
Streams
Amazon
SES
Amazon
S3
Amazon
Cognito
Amazon
SNS
Amazon
CloudWatch
(Events/Logs)
Amazon
Kinesis
Streams
Amazon
API
Gateway
AWS
CloudFormation
AWS
CodeCommit
Amazon
Lex
Amazon
Echo
Amazon
ElastiCache
Redis
AWS
Lambda
ElastiCache as an
event store
event sources
Node.js,Java,
Python
Dashboards,
Apps, etc.
Sub
Pub

25. Rules Engine
Amazon
ElastiCache
Redis
AWS
Lambda
Direct Integration
LambdaSNS SQS
S3 KinesisDDB
AWS
IoT Devices
AWS
IoT
Sensor Store
IoT powered by ElastiCache
https://aws.amazon.com/blogs/database/managing-iot-and-time-series-data-with-amazon-elasticache-for-redis/

26. Mobile Apps Powered by ElastiCache
Amazon API
Gateway
AWS
Lambda
Amazon
ElastiCache
Redis
GEOADD
GEORADIUS
Search points of interest
Update points of interest
https://aws.amazon.com/blogs/database/amazon-elasticache-utilizing-redis-geospatial-capabilities/
Amazon
DynamoDB
DDB streams
Amazon
EC2

27. Chat Apps powered by ElastiCache
https://aws.amazon.com/blogs/database/amazon-elasticache-utilizing-redis-geospatial-capabilities/
Clients
Chat Apps
Application
Load Balancer
WebSockets
Amazon
ElastiCache
Redis
PubSub
Serverpersistent
connections
Elastic
Beanstalk

28. Amazon ElastiCache Summary
 Enhanced Redis engine for increased robustness to
support your critical workloads
 Cost-effective solution with various pricing options to
optimize your TCO
 Native support with the AWS ecosystem of services
 Fully managed with zero administration allowing you to
focus on your business

29. Thank You!