1. Scalable &
Available
Patterns for Success

2. Derek Collison
@derekcollison
dcollison@vmware.com
derek.collison@gmail.com

3. Special Thanks
Jonas Bonér
twitter: @jboner
http://jonasboner.com/
http://www.slideshare.net/jboner/scalability-
availability-stability-patterns

4. Background
• Scalable Apps maintain performance under load
• More requests, More users, More data
• Available Apps maintain the experience during failures
• Hardware failures, Network splits/partitioning
• Simple Designs tend to scale better

5. Background
Good
Performance
is good

6. Background
Predictable
Performance
is king!

7. Background
Understand
your data!

8. Background
Understand
the user
experience!

9. Background
Measure
everything
(can’t fix what you don’t know)

10. Background
Don’t be a failure of
your own success

11. Background
Master the
• Good Performance is good
• Predictably Good Performance is king!
Tradeoffs
• Measure everything (can’t fix what you don’t know)
• Understand app and your data!)
(For your your data
• Understand your user experience
• Don’t be a failure of your own success

12. Master the Tradeoffs
Performance
vs Scalability

13. Master the Tradeoffs
Latency vs
Throughput

14. Master the Tradeoffs
Availability vs
Consistency

15. Lots of ways to skin a cat!

16. Scalability
Patterns

17. Performance
vs
Scalability

18. How do I know if I have a
performance problem?
If your system is slow for a single
request/user

19. How do I know if I have a
scalability problem?
If your system is fast for a
single request/user but slow for
many users

20. Latency
vs
Throughput

21. You should strive for
maximal throughput
with
acceptable latency

22. Performance vs Scalability
Response Time
Concurrent Requests

23. Know what to scale!
• CPU or IO Bound?
• Scale up or Scale out?
• Waiting on IO? What? Disk/Net/Other System?
• How many components are used per request?
• Know who and what the slowest will be!

24. Scalability Patterns
Behavior

25. Scalability Patterns:
Behavior
✓Event-Driven Architectures
✓Load-Balancing
✓Parallel Computing

26. Event-Driven Architecture
✓Events
✓Messaging
✓Asynchronous
✓Non-blocking

27. Messaging
✓Publish-Subscribe
✓Queuing
✓Request-Reply
✓Store and Forward

28. Messaging - Publish Subscribe
1:N
Subscriber
Publisher Subject Subscriber
Subscriber

29. Messaging - Queuing
1:1
Subscriber
Publisher Queue Subscriber
Message #1
Subscriber

30. Messaging - Queuing
1:1
Subscriber
Publisher Queue Subscriber
Message #2
Subscriber

31. Messaging - Queuing
1:1
Subscriber
Publisher Queue Subscriber
Message #3
Subscriber

32. Messaging - Request Reply
1:1
Reply Subscriber
Publisher Subject Subscriber
Subscriber

33. Messaging Patterns
✓Addressing, discovery
✓Command and control
✓Load-balancing
✓N-way scalability

34. Messaging
✓Standards
✓ AMQP (wire)
✓ JMS (api)
✓Products
✓ RabbitMQ
✓ ZeroMQ
✓ ActiveMQ
✓ TIBCO
✓ MQSeries

35. Asynchronous and
Non-Blocking
✓Don’t wait, go doing something else
✓Never block
✓All callbacks all the time can get messy!
✓Good language/framework support
✓functional closures
✓co-routines

36. Load Balancing
✓Multiple endpoints to perform work
✓Can be semantically aware
✓Chainable: DNS, hardware, software
✓Endpoints can be Hardware, VM,
process, thread, co-routine, fiber, etc.

37. Load Balancing
Selection
✓Random
✓Round Robin
✓Weighted
✓Dynamically “aware”
✓Least connections
✓Least loaded

38. Load Balancing
Technologies
✓DNS Round Robin
✓Anycast
✓Reverse Proxies
✓Clustering
✓Hardware Load Balancers

39. Load Balancing
Reverse Proxies
✓Nginx
✓HAProxy
✓Apache (mod_proxy)
✓Squid

40. Parallel Computing
✓Divide and Conquer
✓Worker queues
✓Map Reduce
✓UE = Unit of Execution
✓VM, process, thread, co-routine, fiber, callback

41. Parallel Computing
Worker Queues
✓Good for offloading tasks
✓Need bounded time check in master
✓Async result processing
✓Fork/Join pattern

42. Parallel Computing
MapReduce
✓Used internally at Google
✓Variation of Fork and Join
✓Distributed
✓Originally used for logs processing

43. Parallel Computing
MapReduce
✓Google’s MapReduce
✓Hadoop
✓Amazon’s Elastic MapReduce
✓RIAK uses it internally for queries

44. Scalability Patterns
State

45. Scalability Patterns: State
Harder than scaling
behavior

46. Scalability Patterns: State
✓Master Record
✓Replication
✓Sharding
✓Caching
✓NoSQL
✓Concurrency

47. Master Record
✓Normally Relational Databases (RDBMS)
✓NoSQL Databases emerging
✓Can’t lose this data
✓Scaling can be a challenge

48. Master Record: Scaling
✓Traditonally Scale Up
✓Technology will help here
✓SSD (50k-100k IOPs)
✓More memory/cores per box
✓Faster network connectivity
✓Clustering Appliances

49. Clustering Appliances
64 bit
Infiniband SSD

50. Master Record: Scaling
✓Scaling Reads vs Writes?
✓Scaling Reads with Slaves
✓Synchronous (Speed of Light)
✓Asynchronous

51. Master Record: Scaling
How do we scale
OUT?

52. Master Record: Replication
✓Synchronous vs Asynchronous
✓Master / Slave Replication
✓Master / Master Replication
✓Tree Replication
✓Buddy Replication

53. Replication: Master / Slave

54. Replication: Master / Master

55. Replication: Tree

56. Replication: Buddy

57. Sharding
✓Partitioning state
✓Requests need to know where to go
✓Distributed Hash
✓Load Balancer
✓Messaging

58. Sharding: Paritioning

59. Sharding: Replication

60. Sharding: Over-provision
✓Use N partitions
✓Use Y replicas
✓Use message based requests
✓First back wins
✓Therefore user wins (Google Search)

61. Master Record: RDBMS
Do we really need an
RDBMS?

62. Master Record: RDBMS
Don’t underestimate RDBMS
or
the ability of a single machine

63. Master Record: RDBMS
What about alternatives?

64. NoSQL
✓Key-Value
✓Column Databases
✓Document Databases
✓Graph Databases
✓Datastructure Databases

65. NoSQL
✓Key-Value: (Memcache, Redis, Riak)
✓Column Databases: (Cassandra, Vertica)
✓Document Databases: (MongoDB, CouchDB)
✓Graph Databases: (Neo4J, AllegroGraph)
✓Datastructure Databases: (Redis, Hazelcast)

66. NoSQL in the wild
✓Google: Bigtable, Colossus
✓Twitter: Redis
✓Amazon: Dynamo, SimpleDB
✓Yahoo: HBase (Hadoop)
✓Facebook: Cassandra, HBase

67. Caching
✓Cache early and often
✓Usually biggest bang for the buck
✓Referential Transparency
✓Polyglot APIs coming
✓NoSQL stores
✓Cache invalidation is still hard!

68. Caching
✓HTTP (HTML, JS, CSS, Images, Media)
✓Key/Value Data
✓Semantic Data structures

69. HTTP Caching
✓Varnish
✓Squid
✓Pound
✓Nginx
✓Rack-cache

70. HTTP Caching
CDN
✓Akamai
✓Limelight
✓Level3
✓Digital Fountain (Qualcomm)
✓aiCache

71. HTTP Caching
CDN

72. HTTP Caching
✓Lives in browsers, proxies, CDNs, apps
✓Hard to control, so do it right!
✓Master page controls other resources
✓master page not cached (at least too far)
✓read-only resources
✓change link in master page

73. Key/Value Caching
✓Memcache
✓Redis
✓Riak
✓Voldemort

74. Data Structure Caching

75. Data Structure Caching
✓Standalone
✓Augment RDBMS
✓In Memory or on Disk

76. Data Structure Caching
✓Data Types
✓Strings, Hashes, Lists, Sets, Sorted Sets
✓Atomic Operations
✓Push, pop, ranges, set operations (intersect, union)

77. Caching Patterns
✓Write Through
✓Write Behind
✓Replicated
✓P2P

78. Cache Invalidation
✓TTL (Time to Live)
✓Bounded FIFO or LIFO
✓Explicit cache invalidation
✓Explicit non-use of read-only resource
✓Harder problem the more master items used

79. Scalability Key Points
✓The problem is not where you think ;)
✓Autoscaling is a myth
✓Can’t fix what you can’t measure
✓Scaling master record writes is hard
✓Scaling reads is more tractable
✓What is the opex cost of your choices?

80. Availability
Patterns

81. What do you do
when things go
bad?

82. Availability Patterns
Available vs Consistent

83. Availability Patterns
Available

84. We have been here
before, right?

85. Yes, we have been
here before!?

86. Scalability Patterns
Behavior

87. Scalability Patterns:
Behavior
✓Event-Driven Architectures
✓Load-Balancing
✓Parallel Computing

88. Scalability Patterns
State

89. Scalability Patterns: State
✓Master Record
✓Replication
✓Sharding
✓Caching
✓NoSQL
✓Concurrency

90. But let’s talk more
about your data

91. Availability Patterns
Available vs Consistent

92. Brewer’s CAP Theorem

93. Brewer’s CAP Theorem

94. You can only pick 2
Consistency
Availability
Partition Tolerance

95. Centralized Systems
✓If the system is centralized
✓no P (network partitions)
✓So you get both:
✓Availability
✓Consistency

96. Distributed Systems
✓If the system is distributed
✓you will have P! (network partitions)
✓So you get pick one:
✓Availability
✓Consistency

97. CAP in reality
✓There is only once choice to make:
✓When there is a network partition,
which do you sacrifice?
✓Availability
✓Consistency

98. BASE

99. What is BASE?

100. BASE
Basically
Available
Soft State
Eventually Consistent

101. Eventually Consistent
✓Great tradeoff for the right kind of data
✓Can’t be used everywhere
✓Works in more places than you think
✓Solved speed of light problem

102. Availability Patterns
Failover

103. Availability Patterns:
Failover
✓Failover is complex
✓Switch time is critical
✓Failback is equally as complex

104. Availability Patterns:
Failover
Copyright Michael Nygaard

105. Availability Patterns:
Failback
Copyright Michael Nygaard

106. Availability Patterns:
Replication
✓Synchronous vs Asynchronous
✓Master / Slave Replication
✓Master / Master Replication

107. Availability Patterns:
Redirection
✓DNS
✓Load Balancers
✓Secondary Sites

108. Availability Key Points
✓Always have a dial tone
✓Syntactically correct is good
✓Semantically correct is better
✓Be transparent

109. Background
Beating the
• Good Performance is good
• Predictably Good Performance is king!
dead horse
• Measure everything (can’t fix what you don’t know)
• Understand your data
• Understand your user experience
• Don’t be a failure of your own success

110. Background
Understand
• Good Performance is good
• Predictably Good Performance is king!
your data!
• Measure everything (can’t fix what you don’t know)
• Understand your data
• Understand your user experience
• Don’t be a failure of your own success

111. Background
Understand
• Good Performance is good
• Predictably Good Performance is king!
your user!
• Measure everything (can’t fix what you don’t know)
• Understand your data
• Understand your user experience
• Don’t be a failure of your own success

112. Background
Understand
• Good Performance is good
the
• Predictably Good Performance is king!
• Measure everything (can’t fix what you don’t know)
experience!
• Understand your data
• Understand your user experience
• Don’t be a failure of your own success

113. Background
Master the
• Good Performance is good
• Predictably Good Performance is king!
Tradeoffs
• Measure everything (can’t fix what you don’t know)
• Understand app and your data!)
(For your your data
• Understand your user experience
• Don’t be a failure of your own success

114. Thank You

115. Thank You

116. Questions?

117. Derek Collison
@derekcollison
dcollison@vmware.com
derek.collison@gmail.com