Learn about how to build resilient systems in the cloud by understanding the underlying infrastructure and build you app to best use it. The talk covers running multiple copies of your app, timeouts, retries&backoffs, database scaling options.

1. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
O N L I N E - D E V C O N F Z A
Resiliency and availability design
patterns for the cloud
Cobus Bernard
Senior Developer Advocate
AmazonWeb Services
02.04.20
@cobusbernard
cobusbernard
cobusbernard

2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Can you guess whatwillhappen?

4. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributed Systems are hard

5. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failures areagivenand
everythingwilleventuallyfail
over time.
Werner Vogels
CTO – Amazon.com
“ “

6. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resiliency:Abilityfor asystemtohandle and
eventuallyrecover from unexpected conditions
conditions

7. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partialfailure mode

8. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do we build resilient software systems?

9. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thecostof downtime
Annual Fortune
1000 application
downtime costs
(IDC)
$1.25 to
$2.5B
Average cost of a
data breach
(Ponemon
Institute)
$3.6M
Cost/hr of a
critical
application
failure (IDC)
$500K
to $1M
Average cost/hr
of downtime
(Ponemon
Institute)
$474K
Average cost per
lost or stolen
record (Ponemon
Institute)
$141

10. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do we build resilient software systems?

11. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
People
Application
Network & Data
Infrastructure

12. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutAvailability

13. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Systemavailability
Availability =
Normal Operation Time
Total Time
MTBF**
MTBF** + MTTR*
=
* Mean Time To Repair (MTTR)
**Mean Time Between Failure (MTBF)

14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reading homework

15. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
A = 1 – (1 – Ax)2
Part X
Part X

16. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

17. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Componentredundancyincreases availability
significantly!

18. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Global Infrastructure
• 22 Regions with 70 Availability Zones
• 5 Regions coming soon: Cape Town
Jakarta, Milan, Spain, Osaka
• 100Gbps redundant network
• 99.99% SLA

19. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully-scaledAvailabilityZone

20. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highlyredundant regional network

21. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region and availabilityzones
Region
Availability zone a Availability zone b Availability zone c
data center
data center
data center
1 or more data centers per AZ
2 or more AZs per region (new regions min 3)
data center
data center
data center
data center
data center
data center

22. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How about a global architecture?

23. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Once upon a time …
Origin

24. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And Now …
Origin
~300ms

25. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin
Improve latency for end-users
Origin

26. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improve availabilityand disasterrecovery
Applications in
US West
Applications in
US East
Users from
San Francisco
Users from
New York
Service 1
Service 2
Service 3
Service 4
Service 1
Service 2
Service 3
Service 4

27. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
So should we go for a global architecture?

28. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Perfect your regional architecture first!

29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about Multi-AZ

30. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

31. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

32. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

33. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
new master
Elastic Load
Balancing (ELB)

34. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged by AWS regional
services such as Amazon S3,
Amazon DynamoDB, Amazon
Aurora, Amazon ELBs, etc.
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

35. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about auto scaling

36. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-Scaling
FixedVariable

37. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
Elastic Load
Balancing (ELB)
X

38. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about the responsibility models

39. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWSoperationalresponsibilitymodels
On-Premises Cloud
Less More
Compute Virtual Machine
EC2 Elastic Beanstalk AWS LambdaFargate
Databases MySQL MySQL on EC2
RDS MySQL RDS Aurora Aurora Serverless DynamoDB
Storage Storage
S3
Messaging ESBs
Amazon MQ Kinesis SQS / SNS
Analytics
Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena
Firehose

40. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about databases

41. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common resiliencyissueswithDatabases??
REPLICATION BACKUPSSCALING

42. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
Network
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
PutItem

44. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
GetItem
Network

45. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon.com, Nike, Netflix, Duolingo, Lyft, Airbnb, Samsung,
Toyota, and Capital One depend on the scale and performance of
DynamoDB to support their workloads.
10 trillion requests
per day
20 million
requests per second

46. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases
Relational Key-value Document In-memory Graph Time-series Ledger
DynamoDB NeptuneAmazon RDS
Aurora CommercialCommunity
Timestream QLDBElastiCacheDocumentDB

47. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read-Writeseparation
Master Read Replica Read Replica Read Replica
Instance InstanceInstance

48. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseFederation
Users
DB
Products
DB
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

49. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseSharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
Master
(Read) Replica
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

50. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about backups

51. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

52. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Backup service

53. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion
https://aws.amazon.com/blogs/database/preventing-accidental-table-deletion-in-dynamodb/

54. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion (sql)

55. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Practice and testrecoveryfrom your backups!!

56. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about timeouts, backoff &
retries!

57. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

58. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = default = Infinite
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

59. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout

60. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

61. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

62. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@timeout_decorator.timeout(5, timeout_exception=StopIteration)
def timed_get(url):
return requests.get(url)
https://pypi.org/project/timeout-decorator/

63. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How elsecould wehaveprevented theerror?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

64. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Wait 4s before Retry
Wait 8s before Retry
Wait 16s before Retry
Backing off betweenretries
Releasing connectionsBackoff

65. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
SimpleExponentialBackoffisnotenough:AddJitter

66. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: add jitter0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

67. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Idempotent operation
No additional effect if it is called more than
once with the same input parameters.

68. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about health checking!

69. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
database Email
Probing for health
Cluster

70. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Email
database
Shallowhealthcheck
Instance
Cache node
Cluster
Are you healthy?
yes

71. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
database
Shallowhealthcheck
Instance
Cache node
Cluster
Are you healthy?
yes
Email

72. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Email
database
Deep healthcheck
Instance
Cache node
Cluster
Are you healthy?
yes
Are you healthy?
yes
yes
yes
yes

73. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Email
database
Deep healthcheck
Instance
Cache node
Cluster
Are you healthy?
no
Are you healthy?
no
yes
yes
yes

74. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prioritize shallow health checks during
hard times.
Cache.

75. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about load shedding.

76. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

77. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

78. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cheaply reject excess work

79. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

80. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Be careful when selecting the right metric

81. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Don’tbeoverlyoptimisticandtakeonmorethanyoucan.
Findanoperationalmetrictorejectwhatyoucannottakein.
Favorcachedandstaticcontent
PrioritizeELBhealthcheck(shallow)pings
Inanoverloadsituationyouhavepreciousresources,donotletany
ofitgotowaste.
Load Shedding

82. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutresiliency (chaos) engineering

83. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fire Drills

84. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaosengineering
https://github.com/Netflix/SimianArmy

85. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Chaos Engineeringis the discipline of
experimentingon a distributedsystem
in orderto buildconfidence in the system’s
capabilitytowithstand turbulentconditionsin
production.”
http://principlesofchaos.org

86. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failureinjection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack
https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org

87. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
STEADY
STATE
HYPOTHESIS
RUN
EXPERIMENT
VERIFY
FIX!
PhasesofChaosEngineering

88. Thank you!
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cobus Bernard
Senior DeveloperAdvocate
Amazon Web Services
O N L I N E - D E V C O N F Z A
@cobusbernard
cobusbernard
cobusbernard