Cassandra gives operations a lot of control over the system by forcing them to make a lot of decisions they'd rather not around cluster topology changes. Hecuba2 is a tool that helps to automate that. Hecuba2 has a library component and an agent component. The library provides an API for manipulating Cassandra topologies and the agent runs on all Cassandra hosts and converges the existing topology to the generated topology. Hecuba2 is running in production at Spotify and has been remarkably bug free since being rolled out. It supports creating a cluster, expanding a cluster, and replacing nodes. This talk will cover the design of Hecuba2 and how to deploy it. About the Speaker Radovan Zvoncek Backend Engineer, Spotify After graduating a master degree in distributed systems I've joined Spotify as a backend engineer. For the past three years I've been involved in Cassandra operations, as well as the cultivation of the Cassandra ecosystem at Spotify.

1. Hecuba2
Cassandra Operations Made Easy
Radovan Zvoncek
zvo@spotify.com

2. Agenda
1 Two peace stories
2 Cassandra infrastructure at Spotify
3 What exactly is Hecuba2?
4 Wrap up
2© DataStax, All Rights Reserved.

3. About Radovan
Likes pancakes
© DataStax, All Rights Reserved. 3

4. About Radovan
Likes pancakes
Now knows where to get them
© DataStax, All Rights Reserved. 4

5. About Radovan
Likes pancakes
Now knows where to get them
Works at Spotify
© DataStax, All Rights Reserved. 5

6. Spotify
Music streaming service
● ~ 100 million active users
● ~ 2 billion playlists
© DataStax, All Rights Reserved. 6

7. Spotify
Music streaming service
● ~ 100 million active users
● ~ 2 billion playlists
Happy Apache Cassandra user
● ~ 100 Cassandra clusters
● ~ 1000 nodes altogether
© DataStax, All Rights Reserved. 7

8. The Playlist Cluster
The largest cluster we have
● 2 x 45 nodes
● ~ 1TB of data on each node
© DataStax, All Rights Reserved. 8

9. The Playlist Cluster
The largest cluster we have
● 2 x 45 nodes
● ~ 1TB of data on each node
© DataStax, All Rights Reserved. 9

10. The Playlist Cluster
The largest cluster we have
● 2 x 45 nodes
● ~ 1TB of data on each node
Had to expand by 50%
© DataStax, All Rights Reserved. 10

11. The previous large cluster expansion went so wrong…
Peace Story #1: The Playlist Expansion
© DataStax, All Rights Reserved. 11

12. The previous large cluster expansion went so wrong…
How exactly to do the expansion now?
Peace Story #1: The Playlist Expansion
© DataStax, All Rights Reserved. 12

13. The previous large cluster expansion went so wrong…
How exactly to do the expansion now?
What tokens should the new nodes have?
Peace Story #1: The Playlist Expansion
© DataStax, All Rights Reserved. 13

14. Peace Story #1: The Playlist Expansion
The previous large cluster expansion went so wrong…
How exactly to do the expansion now?
What tokens should the new nodes have?
How to bootstrap the nodes?
© DataStax, All Rights Reserved. 14

15. Peace Story #1: The Playlist Expansion
Turns out, we needed just one command:
© DataStax, All Rights Reserved. 15
hecuba2-cli expand-cluster
dc1-playlistcassandra-a{31..45}.foo.net

16. Peace Story #1: The Playlist Expansion
Turns out, we needed just one command:
And one peer review
© DataStax, All Rights Reserved. 16
hecuba2-cli expand-cluster
dc1-playlistcassandra-a{31..45}.foo.net

17. Peace Story #1: The Playlist Expansion
Turns out, we needed just one command:
And one peer review
And then a week of waiting
© DataStax, All Rights Reserved. 17
hecuba2-cli expand-cluster
dc1-playlistcassandra-a{31..45}.foo.net

18. Peace Story #1: The Playlist Expansion
18

19. Peace Story #2: The Slush Incident
© DataStax, All Rights Reserved. 19

20. Peace Story #2: The Slush Incident
© DataStax, All Rights Reserved. 20

21. Peace Story #2: The Slush Incident
Got paged with 2 out of 3 nodes being down
© DataStax, All Rights Reserved. 21
Datacenter: dc1
==========
Address Rack Status Load Owns Token
dc1-slush-1.foo.net rac1 Up Normal 797.27 GB 33.33% 0
dc1-slush-2.foo.net rac1 Down Normal 798.58 GB 33.33% 56...
dc1-slush-3.foo.net rac1 Down Normal 797.58 GB 33.33% 11...

22. Peace Story #2: The Slush Incident
Turns out, we needed just two commands
© DataStax, All Rights Reserved. 22
hecuba2-cli replace-nodes
--old-host dc1-slush-2.foo.net
--new-host dc1-slush-4.foo.net
hecuba2-cli replace-nodes
--old-host dc1-slush-3.foo.net
--new-host dc1-slush-5.foo.net

23. Peace Story #2: The Slush Incident
What gave us two peer reviews, such as:
© DataStax, All Rights Reserved. 23

24. Peace Story #2: The Slush Incident
After a while, we ended up with
© DataStax, All Rights Reserved. 24
Datacenter: dc1
==========
Address Rack Status Load Owns Token
dc1-slush-1.foo.net rac1 Up Normal 797.27 GB 33.33% 0
dc1-slush-4.foo.net rac1 Up Normal 798.58 GB 33.33% 56...
dc1-slush-5.foo.net rac1 Up Normal 797.58 GB 33.33% 11...

25. The Peace Stories
Very pleasant experience operating Cassandra @ Spotify
All because our infrastructure
© DataStax, All Rights Reserved. 25

26. Agenda
1 Two peace stories
2 Cassandra infrastructure at Spotify
3 What exactly is Hecuba2?
4 Wrap up
26© DataStax, All Rights Reserved.

27. Cassandra Infrastructure @ Spotify
Let’s just create a Cassandra cluster like a Spotifier
© DataStax, All Rights Reserved. 27

28. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
© DataStax, All Rights Reserved. 28

29. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
© DataStax, All Rights Reserved. 29

30. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
More info about System-Z
Modelling Microservices at Spotify
by Petter Måhlén
https://youtu.be/7XDA044tl8k
© DataStax, All Rights Reserved. 30

31. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
Step 2: Install the operating system
© DataStax, All Rights Reserved. 31

32. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
Step 2: Install the operating system
© DataStax, All Rights Reserved. 32

33. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
Step 2: Setup the cluster
© DataStax, All Rights Reserved. 33

34. Creating C* Cluster Like A Spotifier
Step 1: Get some machines
Step 2: Setup the cluster
© DataStax, All Rights Reserved. 34
hecuba2-cli create-cluster
--cluster-name "My new cluster"
--owner mySquad dc1-userdatacass-
{1..3}.foo.net

35. Creating C* Cluster Like A Spotifier
© DataStax, All Rights Reserved.
Step 1: Get some machines
Step 2: Setup the cluster
Step 3: Check PR
35

36. Creating C* Cluster Like A Spotifier
© DataStax, All Rights Reserved.
Step 1: Get some machines
Step 2: Setup the cluster
Step 3: Check PR
Step 4: Wait
36

37. Creating C* Cluster Like A Spotifier
© DataStax, All Rights Reserved.
Step 1: Get some machines
Step 2: Setup the cluster
Step 3: Check PR
Step 4: Wait
This will get picked up by our conf. management system and
● Install software
● Configure Cassandra
37

38. Configuring Cassandra
© DataStax, All Rights Reserved. 38

39. Configuring Cassandra
Is (almost) all about putting things into config files
© DataStax, All Rights Reserved. 39

40. Configuring Cassandra
Is (almost) all about putting things into config files
Mostly tokens and seeds
● Tokens are clunky large strings
● V-nodes would help, but...
© DataStax, All Rights Reserved. 40

41. Configuring Cassandra
Is (almost) all about putting things into config files
Mostly tokens and seeds
● Tokens are clunky large strings
● V-nodes would help, but...
Also node bootstrap
● All nodes joining at once is not desired
© DataStax, All Rights Reserved. 41

42. Configuring Cassandra
Is (almost) all about putting things into config files
Mostly tokens and seeds
● Tokens are clunky large strings
● V-nodes would help, but...
Also node bootstrap
● All nodes joining at once is not desired
Both are handled by Hecuba2
© DataStax, All Rights Reserved. 42

43. Agenda
1 Two peace stories
2 Cassandra infrastructure at Spotify
3 What exactly is Hecuba2?
4 Wrap up
43© DataStax, All Rights Reserved.

44. What Exactly is Hecuba2
© DataStax, All Rights Reserved. 44
Hecuba
YAML
File
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
SeedPro
vider
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cassandra NodeHecuba2
Client
Library
C*
Process

45. Hecuba YAML File
Represents the truth about the cluster
© DataStax, All Rights Reserved. 45

46. Hecuba YAML File
dc1-mytestcass-1.foo.net:
cluster_name: mytestcass
dc: dc1
seed: true
token: 0
dc1-mytestcass-2.foo.net:
cluster_name: mytestcass
dc: dc1
seed: false
token: 56713727820156410577229101238628035242
© DataStax, All Rights Reserved. 46

47. Hecuba2 Client Library
Manipulates the Hecuba YAML file
© DataStax, All Rights Reserved. 47

48. Hecuba2 Client Library
Manipulates the Hecuba YAML file
Does this in a smart way
© DataStax, All Rights Reserved. 48

49. Hecuba2 Client Library
Manipulates the Hecuba YAML file
Does this in a smart way
Doubling the cluster
© DataStax, All Rights Reserved. 49
Double the size
New node
Existing node stayed in place

50. Manipulates the Hecuba YAML file
Does this in a smart way
Doubling the cluster
Expand by 50%
Hecuba2 Client Library
© DataStax, All Rights Reserved. 50
Add 2 nodes
New node
Existing node stayed in place
Existing node moved

51. Hecuba2 Server-Side Components
© DataStax, All Rights Reserved. 51
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cassandra Node
C*
Process

52. Hecuba2 Server-Side Components
Again, three things:
● hecuba2-agent
○ State machine managing the C* process
© DataStax, All Rights Reserved. 52

53. hecuba2-agent
© DataStax, All Rights Reserved. 53

54. hecuba2-agent
© DataStax, All Rights Reserved. 54

55. Hecuba2 Server-Side Components
Again, three things:
● hecuba2-agent
○ State machine managing the C* process
● hecuba2-jmxproxy
○ nodetool with JSON output
© DataStax, All Rights Reserved. 55

56. Hecuba2 Server-Side Components
Again, three things:
● hecuba2-agent
○ State machine managing the C* process
● hecuba2-jmxproxy
○ nodetool with JSON output
● hecuba2-seedprovider
○ Picks seeds from Hecuba YAML
© DataStax, All Rights Reserved. 56

57. How Does It All Work Together
© DataStax, All Rights Reserved. 57

58. How Does It All Work Together
© DataStax, All Rights Reserved. 58
Hecuba2
Client
Library

59. How Does It All Work Together
© DataStax, All Rights Reserved. 59
Hecuba
YAML
File
Hecuba2
Client
Library
Creates
Hecuba2
Not in scope

60. How Does It All Work Together
© DataStax, All Rights Reserved. 60
Hecuba
YAML
File
Hecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates

61. How Does It All Work Together
© DataStax, All Rights Reserved. 61
Hecuba
YAML
File
Hecuba
YAML
File
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
C*
Process
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider

62. How Does It All Work Together
© DataStax, All Rights Reserved. 62
Hecuba
YAML
File
Hecuba
YAML
File
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
C*
Process
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider

63. How Does It All Work Together
© DataStax, All Rights Reserved. 63
Hecuba
YAML
File
Hecuba
YAML
File
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cron
executes
C*
Process

64. How Does It All Work Together
© DataStax, All Rights Reserved. 64
Hecuba
YAML
File
Hecuba
YAML
File
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cron
executes
calls
C*
Process

65. How Does It All Work Together
© DataStax, All Rights Reserved. 65
Hecuba
YAML
File
Hecuba
YAML
File
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cron
executes
calls
manages
C*
Process

66. How Does It All Work Together
© DataStax, All Rights Reserved. 66
Hecuba
YAML
File
Hecuba2
Agent
manages
Cron
executes
Hecuba2
jmx-
proxy
Hecuba2
SeedPro
vider
Hecuba
YAML
File
calls
calls Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
C*
Process

67. How Does It All Work Together
© DataStax, All Rights Reserved. 67
Hecuba
YAML
File
Hecuba2
Agent
manages
Cron
executes
Hecuba2
jmx-
proxy
Hecuba2
SeedPro
vider
Hecuba
YAML
File
calls
calls Hecuba2
Agent
Hecuba2
jmx-
proxy
Hecuba2
Seed
Provider
Cassandra NodeHecuba2
Client
Library
Text
Editor
Hecuba2
Not in scope
Alternative
Creates
Distribution
Manual
Puppet
C*
Process

68. To Recap
Hecuba2 manages cluster topologies
● create-cluster
● expand-cluster
● replace-node
Many things are out of scope
Missing features:
● parallelism @ hecuba2-agent
● cluster shrinking
© DataStax, All Rights Reserved. 68

69. Agenda
1 Two peace stories
2 Cassandra infrastructure at Spotify
3 What exactly is Hecuba2?
4 Wrap up
69© DataStax, All Rights Reserved.

70. Our Experience So Far
It’s been in use for a year
It hasn’t let us down yet
But it has surprised us by being more robust than we thought
State machine testable, visualisable, and easily extensible
Peer review for changes
© DataStax, All Rights Reserved. 70

71. FAQ
Why is it called Hecuba2?
Does it support v-nodes?
Does it support Cassandra version X?
Can I use it on Y?
Is it FOSS?
© DataStax, All Rights Reserved. 71

72. Actual Q
© DataStax, All Rights Reserved. 72

73. Thank You!
bases-ext@spotify.com
zvo@spotify.com
Eventually https://github.com/spotify/hecuba2
© DataStax, All Rights Reserved. 73