What does it take to scale a system? We'll learn how going distributed can pay dividends in areas like availability and fault tolerance by examining a real-world case study. However, we will also look at the inherent pitfalls. When it comes to distributed systems, for every promise there is a peril.

1. The Economics of Scale Tyler Treat
WorkivaPromises and Perils of Going Distributed
September 19, 2015

2. About The Speaker
• Backend engineer at Workiva
• Messaging platform tech lead
• Distributed systems
• bravenewgeek.com @tyler_treat
tyler.treat@workiva.com

3. About The Talk
• Why distributed systems?
• Case study
• Advantages/Disadvantages
• Strategies for scaling and resilience patterns
• Scaling Workiva

4. What does it mean to “scale” a system?

5. Scale Up vs. Scale Out
❖ Add resources to a node
❖ Increases node capacity, load
is unaffected
❖ System complexity unaffected
Vertical Scaling
❖ Add nodes to a cluster
❖ Decreases load, capacity is
unaffected
❖ Availability and throughput w/
increased complexity
Horizontal Scaling

6. Okay, cool—but what does this actually mean?

7. Let’s look at a real-world example…

8. How does Twitter work?

9. Just write tweets to a big database table.

11. Getting a timeline is a simple join query.

13. But…this is crazy expensive.

14. Joins are sloooooooooow.

15. Prior to 5.5, MySQL used table-level locking. Now
it uses row-level locking. Either way,
lock contention everywhere.

16. As the table grows larger, lock contention on
indexes becomes worse too.

17. And UPDATES get higher priority than SELECTS.

19. So now what?

20. Distribute!

21. Specifically, shard.

22. Partition tweets into different databases using
some consistent hash scheme
(put a hash ring on it).

24. This alleviates lock contention and improves
throughput…
but fetching timelines is still extremely costly
(now scatter-gather query across multiple DBs).

25. Observation: Twitter is a consumption mechanism
more than an ingestion one…
i.e. cheap reads > cheap writes

26. Move tweet processing to the write path
rather than read path.

27. Ingestion/Fan-Out Process
1. Tweet comes in
2. Query the social graph service for followers
3. Iterate through each follower and insert tweet ID into
their timeline (stored in Redis)
4. Store tweet on disk (MySQL)

29. Ingestion/Fan-Out Process
• Lots of processing on ingest, no computation on reads
• Redis stores timelines in memory—very fast
• Fetching timeline involves no queries—get timeline from
Redis cache and rehydrate with multi-get on IDs
• If timeline falls out of cache, reconstitute from disk
• O(n) on writes, O(1) on reads
• http://www.infoq.com/presentations/Twitter-Timeline-Scalability

30. Key Takeaway: think about your access patterns
and design accordingly.
Optimize for the critical path.

31. Let’s Recap…
• Advantages of single database system:
• Simple!
• Data and invariants are consistent (ACID transactions)
• Disadvantages of single database system:
• Slow
• Doesn’t scale
• Single point of failure

32. Going distributed solved the problem,
but at what cost?
(hint: your sanity)

33. Distributed systems are all about trade-offs.

34. By choosing availability, we give up consistency.

35. This problem happens all the time on Twitter.
For example, you tweet, someone else replies, and
I see the reply before the original tweet.

36. Can we solve this problem?

37. Sure, just coordinate things before proceeding…
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, good.”

38. Sooo what do you do when Justin Bieber tweets to
his 67 million followers?

40. Coordinating for consistency is expensive
when data is distributed
because processes
can’t make progress independently.

43. Source: Peter Bailis, 2015 https://speakerdeck.com/pbailis/silence-is-golden-coordination-avoiding-systems-design

44. Key Takeaway: strong consistency is slow and
distributed coordination is expensive (in terms of
latency and throughput).

45. Sharing mutable data at large scale is difficult.

46. If we don’t distribute, we risk scale problems.

47. Let’s say we want to count the number of times a
tweet is retweeted.

48. “Get, add 1, and put”
transaction will not
scale.

49. If we do distribute, we risk consistency problems.

51. What do we do when our system is partitioned?

53. If we allow writes on both sides of the partition,
how do we resolve conflicts when the partition
heals?

54. Distributed systems are hard!

55. But lots of good research going on to solve these
problems…
CRDTs
Lasp
SyncFree
RAMP transactions
etc.

56. Twitter has 316 million monthly active users.
Facebook has 1.49 billion monthly active users.
Netflix has 62.3 million streaming subscribers.

57. How do you build resilient systems at this scale?

58. Embrace failure.

59. Provide partial availability.

60. If an overloaded service is not essential to
core business, fail fast to prevent availability or
latency problems upstream.

63. It’s better to fail predictably than fail in
unexpected ways.

64. Use backpressure to reduce load.

67. Flow-Control Mechanisms
• Rate limit
• Bound queues/buffers
• Backpressure - drop messages on the floor
• Increment stat counters for monitoring/alerting
• Exponential back-off
• Use application-level acks for critical transactions

68. Bounding resource utilization and failing fast
helps maintain predictable performance and
impedes cascading failures.

69. Going distributed means more wire time.
How do you improve performance?

71. Cache everything.

73. “There are only two hard things in computer
science: cache invalidation and naming
things.”

75. Embrace asynchrony.

78. Distributed systems are not just about workload
scale, they’re about organizational scale.

79. In 2010, Workiva released a product to streamline
financial reporting.

80. A specific solution to solve a very specific
problem, originally built by a few dozen
engineers.

81. Fast forward to today:
a couple hundred engineers,
more users, more markets, more solutions.

82. How do you ramp up new products quickly?

83. You stop thinking in terms of products and start
thinking in terms of platform.

84. From Product to Platform

85. At this point, going distributed is all but necessary.

86. Service-Oriented Architecture allows us to
independently build, deploy, and scale discrete
parts of the platform.

87. Loosely coupled services let us tolerate failure.
And things fail constantly.

88. Shit happens — network partitions, hardware
failure, GC pauses, latency, dropped packets…

89. Build resilient systems.

90. –Ken Arnold
“You have to design distributed systems with
the expectation of failure.”

91. Design for failure.

92. Consider the trade-off between
consistency and availability.

93. Most important?

94. Don’t distribute until you have
a reason to!

95. Scale up until you have to
scale out.

96. –Paul Barham
“You can have a second computer once you’ve
shown you know how to use the first one.”

97. And when you do distribute,
don’t go overboard.
Walk before you run.

98. Remember, when it comes to
distributed systems…
for every promise there’s a peril.

99. Thanks!
@tyler_treat
github.com/tylertreat
bravenewgeek.com