How and why Scalarium is using CouchDB to control and manage thousands of machines on Amazon EC2

1. Scalarium and CouchDB
Jonathan Weiss
@jweiss
@scalarium
Peritor GmbH

2. Scalarium.com
EC2 Cluster Management
 Automatic & dynamic configuration over lifetime
of a cluster via Chef runs
 Self healing clusters
 Auto scaling clusters
 One click deployment
 One click cluster cloning
 Monitoring & alerting
 Logs & accounting
 Firewalls & backups
 Access & rights management
2

3. Use case wooga.com
One of wooga’s Facebook games
 1.5 million DAU (Daily Active Users)
 ~ 130 servers
 ~ 150.000 requests per minute
on average
 Complex stack with different
databases and replication
 Multiple deployments of the game
on multiple clusters
3

4. Scalarium.com
http://www.flickr.com/photos/av8pix/3391322123/ 4

5. Data
Rather static but important Dynamic and non-critical
 Cluster configuration  Accounting & events
 Server descriptions  Monitoring and metering
and current state
 Lifecycle events:
setup, deploy, configure, shutdown
 Applications and deployment
definitions
 EC2 assets like
EBS, elastic IPs, SSH keys
5

6. Data
Rather static but important Dynamic and non-critical
6

7. Why CouchDB?
 High Availability
 Easy Replication
 Clustering
 Robustness
 Short Recovery Time
 Jan Lehnardt
7

8. Architecture
8

9. Architecture
9

10. Interaction
10

11. SimplyStored
Ruby library with ActiveRecord inspired capabilities
 Models
 Associations  Generates map & reduce
 Validations
 Callbacks
 Dynamic finders
 S3 attachments
 Paranoid delete
http://github.com/peritor/simply_stored
11

12. SimplyStored
12

13. SimplyStored
13

14. Conflict & Resolution
14

15. Conflict & Resolution
Re-load and retry: Automatically done by SimplyStored
Discard and report: Internal report
15

16. RockingChair
In-memory CouchDB as a Ruby library
 Just a big Hash
 Understands views generated by SimplyStored
 Speeds up your tests
 Tests can run in parallel
 Nice for debugging
http://github.com/jweiss/rocking_chair
16

17. Operations
17

18. Backup
Slaves in different EC2 availability zones and regions
Offsite copy of data and index files
18

19. Hot Standby
19

20. Cold Standby
http://www.flickr.com/photos/dnkemontoh/2680822579 20

21. Keeping views fresh
 Cronjobs for crawling all design docs and calling all
views
 On master & standby!
 Rollout new views without using them first
 And while we are there, compact views&dbs
21

22. Performance
22

23. I/O
CouchDB heavily depends on file system I/O
 EBS RAIDs for better performance
 Different volumes for data and indices
 Be generous with RAM & CPU
http://www.flickr.com/photos/walkn/5471945439
23

24. include_docs
 Nice feature to load related data in one go…but slow
 random I/O on read
 Emit the data you need to the view index
24

25. Design Documents
 Maintained in a framework/model-layer
 Using md5 of content to decide if update needed
 Whenever one view definition changed,
all views got recomputed
25

26. Design Documents
Separate design document per view
Fine granular control
 Update
 Delete
 Compact
26

27. One DB to rule them all
27

28. Separate DBs
28

29. Wishlist
 Build-in caching
 include_docs to load from index
 Improved performance
of building views
 Build-in solution for
automatic sharding
 Automatic compaction
 Metrics & monitoring
29

30. Couchbase 2.0
30

31. Q&A
Talk to us if you want to use
Scalarium 3 month for free
@scalarium
31