Weitere ähnliche Inhalte Ähnlich wie Building an Event-oriented Data Platform with Kafka, Eric Sammer (20) Kürzlich hochgeladen (20) Building an Event-oriented Data Platform with Kafka, Eric Sammer 1. building a system for machine and
event-oriented data
e. sammer | @esammer
kafka summit 2016
3. © 2015 Rocana, Inc. All Rights Reserved.
what we do
3
• we build a system for the operation of modern data centers
• triage and diagnostics, exploration, trends, advanced analytics of complex
systems
• our data: logs, metrics, human activity, anything that occurs in the data center
• “enterprise software” (i.e. we build for others.)
• today: how we built what we built
4. © 2015 Rocana, Inc. All Rights Reserved.
our typical customer use cases
4
• >100K events / sec (8.6B events / day), sub-second end to end latency, full
fidelity retention, critical use cases
• quality of service - “are credit card transactions happening fast enough?”
• fraud detection - “detect, investigate, prosecute, and learn from fraud.”
• forensic diagnostics - “what really caused the outage last friday?”
• security - “who’s doing what, where, when, why, and how, and is that ok?”
• user behavior - ”capture and correlate user behavior with system performance,
then feed it to downstream systems in realtime.”
6. © 2015 Rocana, Inc. All Rights Reserved.
high level architecture – sources
6
7. © 2015 Rocana, Inc. All Rights Reserved.
high level architecture – sinks
7
8. © 2015 Rocana, Inc. All Rights Reserved.
guarantees
8
• no single point of failure exists
• all components scale horizontally[1]
• data retention and latency is a function of cost, not tech[1]
• every event is delivered provided no more than N - 1 failures occur (where N is
the kafka replication level)
• all operations, including upgrade, are online[2]
• every event is (or appears to be) delivered exactly once[3]
[1] we’re positive there’s a limit, but thus far it has been cost.
[2] from the user’s perspective, at a system level.
[3] when queried via our UI. lots of details here.
10. © 2015 Rocana, Inc. All Rights Reserved.
modeling our world
10
• everything is an event
• each event contains a timestamp, type, location, host, service, body, and type-
specific attributes (k/v pairs)
• build specialized aggregates as necessary - just optimized views of the data
11. © 2015 Rocana, Inc. All Rights Reserved.
event schema
11
{
id: string,
ts: long,
event_type_id: int,
location: string,
host: string,
service: string,
body: [ null, string ],
attributes: map<string>
}
12. © 2015 Rocana, Inc. All Rights Reserved.
event types
12
• some event types are standard
– syslog, http, log4j, generic text record, …
• users define custom event types
• producers populate event type
• transformations can turn an event of type A into B
• event type metadata tells downstream systems how to interpret body and
attributes
13. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic syslog event
13
event_type_id: 100, // rfc3164, rfc5424 (syslog)
body: … // raw syslog message bytes
attributes: { // extracted fields from body
syslog_message: “DHCPACK from 10.10.0.1 (xid=0x45b63bdc)”,
syslog_severity: “6”, // info severity
syslog_facility: “3”, // daemon facility
syslog_process: “dhclient”,
syslog_pid: “668”,
…
}
14. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic http event
14
event_type_id: 102, // generic http event
body: … // raw http log message bytes
attributes: {
http_req_method: “GET”,
http_req_vhost: “w2a-demo-02”,
http_req_path: “/api/v1/search?q=service%3Asshd&p=1&s=200”,
http_req_query: “q=service%3Asshd&p=1&s=200”,
http_resp_code: “200”,
…
}
17. © 2015 Rocana, Inc. All Rights Reserved.
data processing
17
• each processing job gets a full stream of the fire hose, decides what it wants to
consider or operate on
• output of “non-terminal” jobs always just events
• result: all processing jobs are composable
• many jobs take user rules or configuration from our ui
18. © 2015 Rocana, Inc. All Rights Reserved.
the jobs
18
• transformation engine: configuration-based data transformation
• metric aggregation: olap cube construction of time series data (e.g. host 17
user cpu time)
• model build/eval: train/evaluate various kinds of models (e.g. anomaly
detection)
• trigger engine: detect complex patterns in the stream, emit events on match
(e.g. complex event processing, automated workflow, alerting)
• action engine: perform some action upon receiving a specific event type (e.g.
email notification, 3rd party api invocation)
• storage: write all the things to hdfs
19. © 2015 Rocana, Inc. All Rights Reserved.
transformation use cases
19
22. © 2015 Rocana, Inc. All Rights Reserved.
aggregation
22
• mostly for time series metrics
• two halves: on write and on query
• data model: (dimensions) => (aggregates)
• on write
– reduce(a: A, b: A) => A over window
– store “base” aggregates, all associative and commutative
• on query
– perform same aggregate or derivative aggregates
– group by the same dimensions
– we use SQL (Impala)
23. © 2015 Rocana, Inc. All Rights Reserved.
aside: late arriving data (it’s a thing)
23
• never trust a (wall) clock
• producer determines event time, rest of the system uses this always
• data that shows up late always processed according to event time
• apache beam describes these issues perfectly
• this is real and you must deal with it
24. © 2015 Rocana, Inc. All Rights Reserved.
ex: service event volume by host and minute
24
• dimensions: ts, window, location, host, service, metric
• on write, aggregates: count, sum, min, max, last
• epoch, 60000, us-west-2a, w2a-demo-1, sshd, event_volume =>
17, 42, 1, 10, 8
• on query:
– SELECT floor(ts / 60000) as bin, host, service, metric, sum(value_sum) FROM
events WHERE ts BETWEEN x AND y AND metric = ”event_volume” GROUP BY
bin, host, service, metric
• if late arriving data existed in events, the same dimensions would repeat with a
another set of aggregates and would be rolled up as a result of the group by
• tl;dr: normal window aggregation operations
25. © 2015 Rocana, Inc. All Rights Reserved.
extension, pain, and advice
26. © 2015 Rocana, Inc. All Rights Reserved.
extending the system
26
• custom producers
• custom consumers
• event types
• parser / transformation plugins
• custom metric definition and aggregate functions
• custom processing jobs on landed data
27. © 2015 Rocana, Inc. All Rights Reserved.
pain (aka: the struggle is real)
27
• lots of tradeoffs when picking a stream processing solution
– samza: right features, but low level programming model, not supported by vendors.
missing security features.
– storm: too rigid, too slow. not supported by all Hadoop vendors.
– flink: relatively new, fledgling community. growing.
– spark streaming: tons of issues initially, but lots of community energy. improving.
• stack complexity, (relative im)maturity
• beam-style retractions required for correct, timely, efficient aggregates of
complex metrics (non-assoc/commutative)
28. © 2015 Rocana, Inc. All Rights Reserved.
if you’re going to try this…
28
• read all the literature on stream processing[1]
• treat it like the distributed systems problem it is
• understand, make, and make good on guarantees
• find the right abstractions
• never trust the hand waving or “hello worlds”
• fully evaluate the projects/products in this space
• understand it’s not just about search
[1] wait, like all of it? yea, like all of it.
29. © 2015 Rocana, Inc. All Rights Reserved.
things I didn’t talk about
29
• reprocessing data when bad code / transformations are detected
• dealing with data quality issues (“the struggle is real” part 2)
• the user interface and all the fancy analytics
– data visualization and exploration
– event search
– anomalous trend and event detection
– metric, source, and event correlation
– motif finding
– noise reduction and dithering
• event delivery semantics (e.g. at least once, exactly once, etc.)
• alerting, notification, and other subsystems
30. © 2015 Rocana, Inc. All Rights Reserved.
questions?
thank you.
@esammer | esammer@rocana.com