At AWS, we obsess over operational excellence. We have a deep understanding of system availability, informed by over a decade of experience operating the cloud and our roots of operating Amazon.com for nearly a quarter-century. One thing we've learned is that failures come in many forms, some expected, and some unexpected. It's vital to build from the ground up and embrace failure. A core consideration is how to minimize the "blast radius" of any failures. In this talk, we discuss a range of blast radius reduction design techniques that we employ, including cell-based architecture, shuffle-sharding, availability zone independence, and region isolation. We also discuss how blast radius reduction infuses our operational practices.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How AWS Minimizes the Blast
Radius of Failures
Peter Vosshall
VP & Distinguished Engineer
Amazon Web Services
A R C 3 3 8

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. “If anything can go wrong, it will.”
Murphy’s Law

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blast radius
• Who is impacted?
• How many workloads?
• What functionality?
• How many locations?

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. “As a thought exercise, how could you
cut the blast radius for a similar event
in half?”
Correction of Errors Template

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Region isolation
Availability Zone independence
Cell-based architecture
Shuffle sharding
Operational practices

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS regions
Region New region (coming soon)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Regional service endpoints
ec2.us-west-1.amazonaws.com
ec2.us-east-2.amazonaws.com

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared nothing architecture
X
X
X
us-west-1 us-east-2

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region Virtual Private Cloud (VPC) Peering
Region A
VPC A VPC B
Region B

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region Virtual Private Cloud (VPC) Peering
Region A
VPC A
VPC Peering
Connection
VPC B
Region B

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region Virtual Private Cloud (VPC) Peering
Region A
VPC A
VPC Peering
Connection
VPC B
Region B
X

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region Virtual Private Cloud (VPC) Peering
Region A
VPC A
VPC Peering
Connection
VPC B
Cross
Region
Orchestrator
Region B
X

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS regions
Region isolation Single region blast radius

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
Region
Availability zone Availability zone
Availability zone

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
Region
Availability zone Availability zone
Availability zone

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
Region
Availability zone Availability zone
Availability zone
• Geographically spread to avoid
correlated failure
• Geographically close for low latency

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
Region
Availability zone Availability zone
Availability zone
• n data centers per AZ (1 or more)
• n AZs per region (typically 3+)
• 57 AZs across 19 regions globally
• Coming soon: 15 additional AZs
across 5 new regions

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Region
Availability zone a
Instances
DB Instance DB instance
standby
Availability zone b
Instances
Multi-AZ architecture

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Region
Availability zone a
Instances
DB instance DB instance
standby
Availability zone b
Instances
Multi-AZ architecture
DB instanceDB instance
standby

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone Availability zone
Availability zone
• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged in S3’s design for
99.999999999% durability
Multi-AZ Zero blast radius!

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Zonal services
• Amazon EC2 instances
• Amazon EBS volumes
• Amazon EMR clusters
• AWS CloudHSM instances
• NAT gateways
• etc.
• Zone-specific resources
• Zone-local control planes
• Availability Zone independence

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Control planes and data planes
• Control plane: administration of resources
• Data plane: usage of resources
• Separating control plane from data plane reduces blast radius
Service Control plane Data plane
Amazon DynamoDB DescribeTable API Query API
Amazon EC2 RunInstances API A running EC2 instance
AWS Lambda CreateFunction API Invoke API

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Zone a control plane
Zone a
Zone a data plane
Zone b control plane
Zone b
Zone b data plane
Zone c control plane
Zone c
Zone c data plane
Availability Zone independence

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Zone a control plane
Zone a
Zone a data plane
Regional control plane
Zone b control plane
Zone b
Zone b data plane
Zone c control plane
Zone c
Zone c data plane
Availability Zone independence

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
✔ ✔
✔
✔ ✔
Zone a control plane
Zone a
Zone b data plane
Regional control plane
Zone b control plane
Zone b
Zone b data plane
Zone c control plane
Zone c
Zone c data plane
Availability Zone independence

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blast Radius with Availability Zones
Zone a

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability zone failure
Zone a Zone a

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Theoretical blast radius
Zone a Zone a

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Abstracted Architecture

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Typical service application
Compute
Service
Storage
Compute
Service
Storage
Compute
Service
Storage

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cell-based architecture
Compute
Cell 0
Compute
Cell n
Service
Storage
Compute
Cell 1
Storage Storage

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cells and Availability zones
Zone a Zone a

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zone failure
Zone a Zone a

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Theoretical blast radius
Zone a Zone a

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cells and Availability Zones – zonal services
Zone a Zone a

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability zone failure
Zone a Zone a

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partial availability zone failure
Zone a Zone a

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Theoretical blast radius
Zone a Zone a

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
System properties
Cell 0
Service
Cell 1 Cell n
• Workload isolation
• Failure containment
• Scale-out vs. scale-up
• Testability
• Manageability
✔X ✔

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
System properties
Cell 0
Service
Cell 1 Cell n
• Workload isolation
• Failure containment
• Scale-out vs. scale-up
• Testability
• Manageability
Cell n+1

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Core considerations
Cell 0
Service
Cell 1 Cell n
• Cell size
• Router robustness
• Partitioning dimension
• Cross-cell use cases
• Cell migration

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cell size tradeoffs
Smaller cells
• Reduced blast radius
• Easier to test
• Cells easier to operate
Larger cells
• Cost efficiency
• Reduced splits
• System easier to operate

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Router robustness
Cell 0
Service
Cell 1 Cell n
• Remaining shared component
• Stress testing
• Battle hardening
• Keep it simple!
• “Thinnest Possible Layer”

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partitioning dimension
Cell 0
Service
Cell 1 Cell n
• System-specific
• Needs careful analysis
• Cut with grain of domain

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cut with the grain

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-cell use cases
Cell 0
Service
Cell 1 Cell n
• May be unavoidable…keep to
small subset of workload
• Scatter/gather queries
• Batch operations
• Coordinated writes
• Cell migration

70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cell migration
Cell 0
Service
Cell 1 Cell n
• Relocation of workloads
• Driven by
• Heat management
• Size balancing
• Scale-out
• Similar to VM live migration
• Careful coordination between
router and cells

71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Theoretical blast radius
Zone a Zone a

72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
X X X X X X XX
♤♡♢ ⚀ ⚁ ⚂ ⚃♧♢

75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cell-based architecture
XX
♤♡♢ ⚀ ⚁ ⚂ ⚃♧♢

76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shuffle sharding
XX
♤♡♢ ⚀ ⚁⚂ ⚃♡ ♤ ♧♢ ⚀⚂♧ ⚁⚃♢ ♢
✔ ✔
♡ ♧♢

77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shuffle sharding
Nodes = 8
Shard size = 2
Combinations = 28
Overlap % customers
0 53.6%
1 42.8%
2 3.6%

78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shuffle sharding
Nodes = 100
Shard size = 5
Combinations = 75 million!
Overlap % customers
0 77%
1 21%
2 1.8%
3 0.06%
4 0.0006%
5 0.0000013%

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shuffle sharding

80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

81. “I'd wager that every new AWS
engineer knows within their first week,
if not their first day, that we never
want to touch more than one zone at
a time.”
Werner Vogels

82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Software deployments
• Staggered deployments
• Across regions
• Across availability zones
• Across cells
• Fractional deployments
• Within deployment units
• Automated tests and canaries
• Automated rollback

83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automation is key
• Reviewable
• Testable
• Predictable
• Repeatable

84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
End-to-end ownership
• Service teams own everything!
• No “throw it over the wall”
• Direct feedback loop to reduce
blast radius

85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In conclusion…

86. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In conclusion…

87. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In conclusion…

88. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Peter Vosshall
vosshall@amazon.com

89. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.