4. Our platform - late 2012
tbc tbc
MetaBroadcast platform
Video and audio metadata Profiles and activity from video and
from 20+ sources Analytic requests and groupings
audio products, social networks
19. What is Atlas?
BBC
Data ingest
server DB
PA
C4
Update bus HTTP server
etc...
ES
20. Data model
• columns to model annotations
• secondary indexes
• index.direct(keyspace, SEGMENT_URI_INDEX_CF, ConsistencyLevel.CL_QUORUM).
from(segment.getCanonicalUri()).
to(segment.getIdentifier()).
index().execute(requestTimeout, TimeUnit.MILLISECONDS);
21. ID generation
• give external data our own ID on ingest
• needs to be user-friendly:
http://www.radiotimes.com/programme/cf2/eastenders
• mongo: findAndModify()
• solution: uses Astyanax client with its distributed locking
• more details: http://metabroadcast.com/blog/let-
cassandra-identify-your-data
22. Where we’re at
• already live with some data
• alpha release of schedule endpoint coming soon
• later: roll out across other endpoints
24. Ops in Cassandra
• we love Puppet
• it’s great for automation and deployment
• MongoDB: 1 file
• Cassandra: 2 files!
• oh... tokens
25. Cassandra Tokens
• define where data is written to
in a cluster
• therefore balanced tokens =
balanced cluster
• tokens should be rack aware
• tools available to provide appropriate tokens
for you
26. Cassandra plays nicely with AWS
• datacentre / rack aware
• AWS Region = Datacentre
• AWS Availability Zone = Rack
• only recently introduced in MongoDB but simple to
implement in Cassandra
• horizontally (and vertically) scalable
27. Monitoring
• Nagios is a little threadbare for Cassandra
• basic TCP service check
• stats from API not very helpful
• nodetool and CLI tools useful
• manual effort to integrate them
• if only there was some useful service...
28. OpsCenter
• wonderful for an overview
• not so much for alerting ;)
• ohai API
• can integrate metrics into Nagios
29. Disaster Recovery
• we operate a 4 node cluster presently
• replication factor of 3 with quorum read/writes
• DR complicated by tokens
• cluster should be balanced
• snapshot + S3 Backups
30. Cluster Happiness and Headaches
• little maintenance overhead
• cluster rebalancing
• uncommon maintenance procedure
• schema changes are cumbersome
• little scope for rollback, can put cluster in unrecoverable state
31. Summary
• Mongo is good, Atlas has outgrown it
• Cassandra isn’t a drop-in replacement
• Ops more complex but so far so good