Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
•Als PPTX, PDF herunterladen•
15 gefällt mir•4,796 views
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Ähnlich wie Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB (20)
Mehr von Cody Ray
Mehr von Cody Ray (9)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
- 1. 1 Stat, 2 Stat, 3 Stat A Trillion Cody A. Ray Dev-Ops @ BrightTag
- 2. Outline 1. Initial Attempt: MongoDB 2. Ideal Stats System: KairosDB? 3. Making KairosDB Work for Us
- 3. What Kind of Stats? Counting! sum, min, max, etc Any recurrence relation: yn = f(x, y0, …, yn-1)
- 4. The First Pass: MongoDB ● JSON Documents, Schema-less, Flexible ● Aggregation Pipeline, MapReduce ● Master-Slave Replication ● Atomic Operators!
- 5. http://fearlessdeveloper.com/race-condition-java-concurrency/ read counter counter = 0 read counter counter = 0 increment value by 1 increment value by 1 write value to counter = 1 write value to counter = 1 incorrect value of counter = 1
- 9. Simple, Right? What’s the Problem? Only 3500 writes/second! (m1.large) up to 7000 wps (with m1.xlarge)
- 11. Redundancy → Mongo Explosion!!!
- 13. Feel the Pain ● Scale 3x. 3x != x. Big-O be damned. ● Managing 50+ Mongo replica sets globally ● 10s of $1000s of dollars “wasted” each year
- 14. Ideal Stats System? ● Linearly scalable time-series database ● Store arbitrary metrics and metadata ● Support aggregations, other complex queries ● Bonus points for o good for storing both application and system metrics o Graphite web integration
- 15. Enter KairosDB ● “fast distributed scalable time series” db ● General metric storage and retrieval ● Based upon Cassandra o linearly scalable o tuned for fast writes o eventually consistent, tunable replication
- 16. Adding Data [ { "name": "archive_file_tracked", "datapoints": [[1359788400000, 123]], "tags": { "host": "server1", "data_center": "DC1" } } ]
- 17. Querying Data { "start_absolute": 1357023600000, "end_absolute": 1357455600000 "metrics": [{ "name": "abc.123", "tags": { "host": ["foo", "foo2"], "type": ["bar"] }, "aggregators": [{ "name": "sum", "sampling": { "value": 10, "unit": "minutes" }}]}]}
- 18. The Catch(es) ● Lack of atomic operations o + millisecond time granularity ● Bad support for high cardinality “tags” ● Headache managing Cassandra in AWS
- 19. The Catch(es) ● Lack of atomic operations o + millisecond time granularity ● Bad support for high cardinality “tags” ● Headache managing Cassandra in AWS
- 20. Cassandra on AWS
- 21. Agathon
- 22. The Catch(es) ● Lack of atomic operations o + millisecond time granularity ● Bad support for high cardinality “tags” ● Headache managing Cassandra in AWS
- 25. Custom Data [ { "name": "archive_file_tracked", "datapoints": [[1359788400000, "value,metadata,...", "string"]], "tags": { "host": "server1", "data_center": "DC1" } } ] https://github.com/proofpoint/kairosdb/tree/feature/custom_data
- 26. Custom Data [ { "name": "archive_file_tracked", "datapoints": [[1359788400000, "value,metadata,...", "string"]], "tags": { "host": "server1", "data_center": "DC1" } } ] https://github.com/proofpoint/kairosdb/tree/feature/custom_data
- 27. The Catch(es) ● Lack of atomic operations o + millisecond time granularity ● Bad support for high cardinality “tags” ● Headache managing Cassandra in AWS
- 28. Pieces of the Solution ● Shard the data o avoids concurrency race conditions ● Pre-aggregation o solves time-granularity issue ● Stream processing, exactly-once semantics
- 31. StormStormStorms StormStormKafkas StormStormZoos StormStormStormStorm App Server 1 suro StormStormMongos StormStormKairoses StormStormStormStorm App Server 2 suro Stats 1.5 Stats 2.0 Stats 1.5 Stats 2.0 1.5 2.0 1.5 2.0
- 32. groupBy(timeRange, metric, tags) Kafka Broker Partition Kafka Broker Partition Kafka Spout Kafka Spout Transforms 30s Writer Bolt30s Writer Bolt30m Writer Bolt 30s Writer Bolt30s Writer Bolt30s Writer Bolt Kafka Layer Spout Layer Transform Layer Persistence Layer shuffle() round-robin(?) round-robin (haproxy) KairosDB Layer KairosDB Cluster
- 38. Pieces of the Solution ● Shard the data o avoids concurrency race conditions ● Pre-aggregation o solves time-granularity issue ● Stream processing, exactly-once semantics
- 39. Pieces of the Solution ● Shard the data o avoids concurrency race conditions ● Pre-aggregation o solves time-granularity issue ● Stream processing, exactly-once semantics
- 41. Pieces of the Solution ● Shard the data o avoids concurrency race conditions ● Pre-aggregation o solves time-granularity issue ● Stream processing, exactly-once semantics
- 43. Pieces of the Solution ● Shard the data o avoids concurrency race conditions ● Pre-aggregation o solves time-granularity issue ● Stream processing, exactly-once semantics
- 44. Non-Transactional 123 Transactional “[9, 123]” Opaque Transactional “[9, 123, 120]”
- 45. Transactional Tags [ { "name": "archive_file_tracked", "datapoints": [[1359788400000, 123]], "tags": { "txid": 9, "prev": 120 } } ]
- 49. The Catch(es) ● Lack of atomic operations o + millisecond time granularity ● Bad support for high cardinality “tags” ● Headache managing Cassandra in AWS
- 50. Does It Work? … the counts still match! (whew)
- 51. Average latency remains < 10 seconds
- 52. Stats 1.0 vs Stats 1.5 Performance Replacing 9 mongo sets with 2
- 53. Cody A. Ray, BrightTag cray@brighttag.com Open Source: github.com/brighttag Slides: bit.ly/gluecon-stats
- 54. These following slides weren’t presented at Gluecon. You may find them interesting anyway. :)
- 55. Bolt Bolt Spout Bolt spout each group by persistent aggregate eachshuffle Trident State each each each each each each group by persistent aggregate Trident State Trident → Storm Topology Compilation
- 57. 2 3 2 4
- 58. Tuning Rules 1. Number of workers should be a multiple of number of machines 1. Number of partitions should be a multiple of spout parallelism 1. Parallelism should be a multiple of number of workers 1. Persistence parallelism should be equal to the number of workers
- 59. multi get: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) multi get: values of ((ts2, metric1)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) multi get: values of ((ts4, metric2), (ts3, metric4)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) group by (ts, metric) http://svendvanderveken.wordpress.com/2013/07/30/scalable-real-time-state-update-with- storm/ Batch from Kafka Persistent Aggregate value = ... (ts1, metric1) value = ... (ts2, metric2) value = ... (ts2, metric3) value = ... (ts2, metric1) value = ... (ts4, metric2) value = ... (ts3, metric4)
- 60. multi get: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) multi get: values of ((ts2, metric1)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) multi get: values of ((ts4, metric2), (ts3, metric4)) reducer / combiner multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) group by (ts, metric) value = ... (ts1, metric1) value = ... (ts2, metric2) value = ... (ts2, metric3) value = ... (ts2, metric1) value = ... (ts4, metric2) value = ... (ts3, metric4) http://svendvanderveken.wordpress.com/2013/07/30/scalable-real-time-state-update-with- storm/ Persistent AggregateBatch from Kafka
- 61. multi get: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) reducer / combiner reducer / combiner reducer / combiner value = ... value = ... value = ... (ts1, metric1) (ts1, metric1) (ts1, metric1) value = ... value = ... (ts2, metric3) value = ... (ts2, metric2) (ts2, metric3) value = ... (ts2, metric3) value = ... (ts2, metric2) multi put: values of ((ts1, metric1), (ts2, metric2), (ts2, metric3)) http://svendvanderveken.wordpress.com/2013/07/30/scalable-real-time-state-update-with- storm/ From the batch From the underlying persistent state
Hinweis der Redaktion
- Great for developers! As we’ll see, not great for operations.
- Remember the read-update-write concurrency problem. Without atomic operations, multiple processes will clobber each other. In this example, you can see that the counter starts at 0 and two processes try to increment it at the same time… and an update is lost.
- Even with bloated Java, atomic increments in mongo are pretty damn simple
- We have three types of stats (using dynamic mongo keys) per (timestamp, clientId). Each of these needs incremented...
- … so we just use mongo’s $inc atomic increment operator
- m1.xlarge has more disks so it can parallelize writes better
- Sharding writes across multiple mongos works… we’ve done this for the past few years. The writer writes to a random mongo and the reader aggregates across all mongos.
- … but if you want redundancy for your data, you have a mongo explosion. For every 1 node of write capacity, you have to add 3 nodes. Not cost effective and a lot of administration overhead
- This is even more pronounced when you take multiple regions into account. We have two layers of reader/aggregation apps and a lot of mongos in each region.
- Spend a lot of time and money managing a stats system that we’ve clearly outgrown
- After a few years of operational experience at scale… we decided to step back and rethink what an ideal system would look like
- KairosDB sounds like it fits the bill…. and its based upon Cassandra which has a big community and we had experience running in production. Because of this, we knew how KairosDB would scale linearly and how to tune it and such.
- KairosDB is a RESTful web service wrapper around Cassandra with a Cassandra schema tuned for timeseries data. Data is added to KairosDB by POSTing a JSON document. Supports batch adding multiple metrics (and multiple datapoints per metric) with a single call. The metric has a name, a set of datapoints (timestamp and float or int values), and a set of tags. Tags are arbitrary metadata that you can associate with metrics.
- Similarly, just POST a JSON document to query KairosDB. Can batch read multiple metrics within a time range but can’t batch reads for multiple time ranges (yet?) Query for metrics by name and filter by tags. Can also aggregate and down-sample data (e.g., sum all data in this time range into 10 minute buckets)
- Of course, all was not rosy. If we want to parallelize writes, we have to handle the read-update-write concurrency problem discussed earlier. Plus, if you have higher volume than one metric per millisecond, you HAVE to pre-aggregate since all KairosDB operations are idempotent. KairosDB also doesn’t handle high-cardinality tags very well… but this is well-known in the community and I expect it to be addressed soon-ish. Lastly, as anyone who has managed Cassandra in AWS knows… it can be tedious.
- Let’s start with the low hanging fruit. I think a lot of companies probably write their own tools for this. Netflix has Priam which we used for a while, contributed some stuff back, but eventually took a different direction.
- An example of the problem managing Cassandra on AWS is managing the security groups. You have 30 to 100 Cassandra instances in 3-6 regions and they all talk to each other. So you have to manage a lot of security group rules manually or use a tool. Priam was backed by SimpleDB. So it just changed the problem from managing IPs in security groups to managing them in SimpleDB.
- We already had all these IPs and such (from a libcloud-based internal tool), we didn’t want to have to manually manage the IPs in SimpleDB. We also wanted something that would run outside AWS since we have different environments in different clouds, and we didn’t want to run a coprocess on all Cassandra instances. So we built Agathon which is easier to extend to different backends, can run in any cloud or raw hardware, and runs as a normal centralized web service. It also handles providing seeds for bootstrapping Cassandra nodes and such. Anyway, its on Github and its low hanging fruit. Check it out.
- The next problem was high cardinality “tags”... so you can’t really store user ids or IP addresses or transaction ids in them.
- This issue stems from KairosDB underlying Cassandra schema. There are 3 column families. The main one is the data_points family which has a row key consisting of the metric name, base timestamp, and serialized key-value pairs from the tags. Each row stores three weeks of data, with each column name being a millisecond offset from the base timestamp in the row key and the value is the actual value itself.
- In NoSQL you generally have to precompute your queries and build indexes manually, so there’s a row_key_index column family. The row key is the metric name and each column name points to a row key in the data_points column family. As you can see, distinct tag combinations each result in a new column. With high-cardinality tags, this can quickly reach Cassandra’s 2 billion column limit.
- So instead of storing high-cardinality data in a tag, we might be able to store it in the value itself. There’s a feature branch (not in master yet) that adds support for custom data. Getting this merged is Brian Hawkins, the main guy behind KairosDB, top priority for the next release. Even so, we’ve been running with this in production for a few months now and its looking pretty solid. No problems with it so far.
- As you can see, you specify a string data for example and tell Kairos that this datapoint is of the “string” type, which is built-in to KairosDB custom_data by default. You can also provide your own custom data types as KairosDB plugins as we’ll see in a bit.
- Okay, now we can get to the grandaddy of problems here. Parallelizing writes without running into race conditions.
- So this solution has several pieces. The standard way to avoid the concurrency race condition is to only have one process responsible for reading/updating/writing a particular piece of data. I already mentioned that pre-aggregation is required to solve the time-granularity issue. And lastly is ensuring that each message/stat is processed exactly once. This is a hard but important problem since many stats systems are considered systems of record, used for billing purposes or identifying discrepancies.
- When you put these pieces together, you typically end up with something like this. Multiple queues and a lot of worker processes. This leads to complex dependency chains and brittle configurations. Not fun to manage.
- The folks at Twitter built (or, rather, bought) Storm/Trident to make this easier. Storm provides a higher-level abstraction than raw message passing, queues, and workers. Storm provides two main primitives: spouts and bolts. Spouts are the sources of data. The emit streams of tuples to downstream bolts. Bolts can perform arbitrary calculations: filtering, transformations, reading data from other sources and merging, writing to persistent data stores. Anything. This chain of Spouts and Bolts forms a “topology” You can even join or merge streams and perform other higher level operations easily using Trident, an API on top of Storm. Queuing between workers happens seamlessly and only when required by a repartition operation, like shuffling or sharding. Storm provides two built-in queues to choose from: 0MQ or Netty.
- So how does this fit into the big picture? For us, it looks something like this. We launched a new product which needed support for more general metrics so we built this out and called it the Stats 2.0 pipeline. But for our existing high-volume apps which write directly to Mongo, we needed a nice transition from our existing infrastructure to this new stats pipeline (aka Stats 2.0). So we built a transitional Stats 1.5 pipeline which uses Storm for pre-aggregation and then writes to Mongo. These two pipelines use a lot of the same infrastructure but different Storm topologies and persistent layers.
- Logically, this pipeline can be broken down to these layers. At the top, we have our Kafka brokers which use partitions as the unit of parallelism. The first layer within Storm is the Kafka spouts which read from the Kafka partitions. The tuples are emitted from the spout to the transform layer. In our stats topology, this is parsing the message JSON, splitting messages to multiple metrics, picking the pre-aggregation bucket, and deciding the final metric names. The last layer in Storm provides the actual read-update-write operations for persistence. And finally we have the KairosDB layer itself. In red, you see the repartitioning operations which divide each layer. Tuples are randomly distributed between the 2 spout worker threads and all the transform worker threads. This could actually move data between nodes too. The groupBy between the transform and persistence layers is essentially sharding based on the given fields. So only a single worker thread does the aggregation for a set of data. If your set of fields form a decent partitioner, you can spread the load across your nodes pretty evenly. Finally, the persistence layer talk to a local haproxy load balancer which round-robins between different KairosDB nodes. And of course Cassandra has its own partitioning built-in.
- This is one of our Trident topologies. What’s nice is that those logical layers are pretty evident in this topology. It also makes it much easier to persist to state, do partitions and aggregations, and even batches tuples which trades better throughput for increased latency.
- Its easy to see that this is the spout layer...
- … and the transform layer...
- … and finally the persistence layer
- But where is KairosDB in this topology? Both the Kafka spout and KairosDB state are declared in advance and you can easily substitute other spouts or states in the topology (in theory at least :).
- So let’s review our checklist.
- We first wanted to shard the data to avoid any concurrency race conditions.
- We did that by grouping by specific fields here.
- Next we wanted to pre-aggregate the data.
- This is handled in the awesome presistentAggregate function that Trident provides. This does the actual read-update-write process. It takes the group of keys from the incoming batch of tuples, queries the persistent state for the current value, and reduces/combines them all using whatever aggregator you choose, and finally writes them back out to the state. Here we’re just doing a Sum across all the values. (There are some extra slides at the end which illustrate this process)
- Finally lets see how we can ensure exactly-once semantics in this system.
- Thankfully, Trident makes this easy too. It supports three levels of transactionality by default. Non-Transactional provide at-least once semantics. If a tuple fails processing, it may be replayed and thus double counted (or more). Using transactional spouts/states can ensure that doesn’t happen. These store the transaction ID alongside the data in the database. And finally Opaque Transactional provides a stronger guarantee. It can provide exactly-once semantics even in the face of failure of your spout, such as if Kafka goes down. In addition to the transaction ID, it also stores the previous value so it can reason out when a batch has completed processing. Trident provides a few standard serializers for storing all the needed information for the desired level of transactionality. Its either a raw value or a stringified JSON array with a transaction ID and possibly the previous value.
- Before I knew about the high-cardinality issue, my first approach was to store the transaction ID and previous value as tags in KairosDB. We’ve already talked about how that turned out. (Badly)
- So instead we have to store these in the KairosDB value itself using the Custom Data support. We just map the serializers we saw earlier to KairosDB custom data types.
- And then when we create the metric, we also tell KairosDB what type of data the metric holds.
- These Trident custom data types for KairosDB and the full KairosDB state implementation for Trident are both open source and available on GitHub. Check ‘em out.
- So we’ve workarounds for all these catches and arrived at a flexible and very scalable distributed infrastructure for stats. It wasn’t that bad, was it? :)
- As a final note, if you’ve ever tried to track down stats discrepancies, you’ll know how bad it hurts. So we actually deployed the Stats 1.5 pipeline in parallel to the existing system to make sure that it was producing the same values. Great Success! (The Stats 2.0 pipeline was green-field so there wasn’t a previous system for comparison)
- So its correct, but is it operating quickly enough? If you look up how to monitor a pipeline like this, you’re probably going to see references to this tool: stormkafkamon. Its pretty great, but its a bit dated. It doesn’t work with the latest Kafka, etc. There’s a fork on the BrightTag GitHub page that’s updated for the latest versions of everything. The main question the business always asks is “how backed up is the stats system”, so I also modified the output so it tells it how much lag there is. This parses the message from Kafka so it has some assumptions. I’m working on generalizing this a bit more for others to use. Here you can see that we’re still near-real-time, enough for our needs anyway. You can decrease this latency by scaling out your persistence layer farther. This is where the engineering tradeoff of latency vs. resources/$$$ comes in.
- Again, looking at writing to mongo directly vs. pre-aggregating with Storm. The bottom line is that sharding and pre-aggregation can drastically reduce the number of writes to mongo. We’ve reduced the number of mongo instances by 5x here! That’s a huge win. We’ve gone from being bound by disk I/O to being bound by Mongo locking. KairosDB/Cassandra doesn’t have the same locking penalty, so this should be an even bigger win with it.
- Trident provides a nice fluent interface and a lot of powerful operations for building a topology, but how does that map to the underlying Storm primitives? A Trident topology compiles to an efficient storm topology (spouts and bolts) by dividing between repartitioning operations like shuffle and groupBy. The Topology we showed (with the Kafka + Transform + Persistence layers where the Transform does a Stream split into two aggregation buckets) would compile like this.
- If you call .name() on the various bits of the stream, you can actually see how Trident compiled into the Storm spouts and bolts in the Storm UI. Here you can see the 30s and 30m persistence bolts, the Transform bolt, and the spout. Note that the spout actually appears as a bolt because the Kafka-Spout provides a controller Spout which coordinates reads from the spout bolt threads to the kafka broker/partitions.
- You can also see how the parallelismHints in the topology translate to executors (aka threads) and tasks in the Storm UI. We codified the tuning rules into a StormParallelism helper which is parametized for your machine configuration and topology layout. Note that we have 2 spout0 threads since we have 2 kafka partitions, each 30s and 30m aggregator has 3 threads since there are 3 storm hosts (one of each aggregator per host) and the transform layer has 24 executors since its the only CPU-bound part of the topology; that’s 3 hosts * 2 cores each * 4 threads per core = 24 threads.
- There’s not a lot of Storm tuning info out there, but there’s enough if you’ll read a few Gist-based guides. I found they really boil down to these big-4 rules of storm tuning. https://gist.github.com/codyaray/ac2eceb3ff92fa0eaf6b
- This is how persistentAggregate works internally. It takes a State like KairosState which implements IBackingMap with the two methods: multiGet and multiPut. All the tuples in the current batch are grouped by the fields (in this case, the timestamp and metric name). Then these keys are parallelized to the persistence parallelism into several multiGet calls. Then all of these values are reduced/combined together and written back to the state in a bulk mutiPut request. (Usually you want persistence parallelism = number of workers, or one thread per worker node, to improve cache efficiency and reduce the bulk request overhead).
- But this still isn’t the full picture of the reducer/combiner; we can dive into how this read-update-write pipeline works even farther.
- Each multiGet call performs the underlying bulk request and then passes the key-value pairs from the batch + the key-value pairs from the multiGet into the given reducer/combiner. For example, the Sum() aggregator we’ve used here just adds up all the previous stored values and the values from the batch. The final value for each key is then written back to the underlying state in a bulk multiPut request.