1
Kostas Kloudas
@KLOUBEN_K
BERLIN BUZZWORDS
JUNE 13, 2017
Complex Event Processing with
Flink The state of FlinkCEP
2
Original creators of Apache
Flink®
Providers of the
dA Platform, a supported
Flink distribution
What is CEP?
3
CEP: Complex Event Processing
 Detecting event patterns
 Over continuous streams of events
 Often arriving out-of-order...
CEP: Complex Event Processing
5
Input
CEP: Complex Event Processing
6
Pattern
Input
CEP: Complex Event Processing
7
Pattern
Output
Input
CEP: use-cases
 IoT
 Intrusion detection
 Inventory Management
 Click Stream Analysis
 Trend detection in financial s...
What is Stream Processing?
9
Stream Processing
10
Computation
Computations on
never-ending
“streams” of events
Distributed Stream Processing
11
Computation
Computation
spread across
many machines
Computation Computation
Stateful Stream Processing
12
Computation
State
Result depends
on history of
stream
13
Stream Processors are a natural fit
for CEP
FlinkCEP
14
Pattern
Output
FlinkCEP
Input
What does FlinkCEP offer?
15
Pattern Definition
16
Pattern
Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
17
Pattern
P2
P1
Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
 Combined by Contig...
FlinkCEP Individual Patterns
 Unique Name
 Condition : which elements to accept
• Simple e.g shape == rectangle
• Iterat...
FlinkCEP Complex Patterns
 Combine Individual Patterns
 Contiguity Conditions
• how to select relevant events given an i...
FlinkCEP Contiguity Conditions
21
Pattern
Input
FlinkCEP Contiguity Conditions
22
Pattern
OutputInput
Strict Contiguity
• matching events strictly follow each other
FlinkCEP Contiguity Conditions
23
Pattern
OutputInput
FlinkCEP Contiguity Conditions
24
Pattern
Relaxed Contiguity
• non-matching events to simply be ignored
Input Output
FlinkCEP Contiguity Conditions
25
Pattern
Input Output
FlinkCEP Contiguity Conditions
26
Pattern
Input Output
FlinkCEP Contiguity Conditions
27
Pattern
Input Output
Non-Deterministic Relaxed Contiguity
• allows non-deterministic act...
FlinkCEP Contiguity Conditions
28
Pattern
NOT patterns:
• for strict and relaxed contiguity
• for cases where an event sho...
FlinkCEP Summary
29
 Quantifiers
• oneOrMore(), times(), optional()
 Conditions
• Simple & Iterative
 Time Constraints
...
 Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
30
Ru...
 Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
31
Ob...
32
Observation B Quantifiers
 Start/End: single event
 Middle: multiple events
• .oneOrMore()
Start
End
Mid
ev.from == A...
33
Observation C Conditions
 Start -> Simple
• properties of the event
 Middle/End -> Iterative
• Depend on previous eve...
34
 Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
Ob...
35
 We opt for relaxed continuity
Observation E Contiguity
Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myItera...
Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myItera...
Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myItera...
Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myItera...
Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input...
Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input...
Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input...
Documentation
 FlinkCEP documentation:
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
43
4
Thank you!
@KLOUBEN_K
@ApacheFlink
@dataArtisans
45
Stream Processing
and Apache Flink®'s
approach to it
@StephanEwen
Apache Flink PMC
CTO @ data ArtisansFLINKFORWARD IS C...
We are hiring!
data-artisans.com/careers
Nächste SlideShare
Wird geladen in …5
×

Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP

1.051 Aufrufe

Veröffentlicht am

Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput.

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP

  1. 1. 1 Kostas Kloudas @KLOUBEN_K BERLIN BUZZWORDS JUNE 13, 2017 Complex Event Processing with Flink The state of FlinkCEP
  2. 2. 2 Original creators of Apache Flink® Providers of the dA Platform, a supported Flink distribution
  3. 3. What is CEP? 3
  4. 4. CEP: Complex Event Processing  Detecting event patterns  Over continuous streams of events  Often arriving out-of-order 4
  5. 5. CEP: Complex Event Processing 5 Input
  6. 6. CEP: Complex Event Processing 6 Pattern Input
  7. 7. CEP: Complex Event Processing 7 Pattern Output Input
  8. 8. CEP: use-cases  IoT  Intrusion detection  Inventory Management  Click Stream Analysis  Trend detection in financial sector  ...yours? 8
  9. 9. What is Stream Processing? 9
  10. 10. Stream Processing 10 Computation Computations on never-ending “streams” of events
  11. 11. Distributed Stream Processing 11 Computation Computation spread across many machines Computation Computation
  12. 12. Stateful Stream Processing 12 Computation State Result depends on history of stream
  13. 13. 13 Stream Processors are a natural fit for CEP
  14. 14. FlinkCEP 14 Pattern Output FlinkCEP Input
  15. 15. What does FlinkCEP offer? 15
  16. 16. Pattern Definition 16 Pattern
  17. 17. Pattern Definition  Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle) 17 Pattern P2 P1
  18. 18. Pattern Definition  Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle)  Combined by Contiguity Conditions • ...later 18 Pattern P2 P1
  19. 19. FlinkCEP Individual Patterns  Unique Name  Condition : which elements to accept • Simple e.g shape == rectangle • Iterative e.g rectangle.surface < triangle.surface  Quantifiers (or not) • Looping/Optional oneOrMore(),times(#),optional() 19 Pattern P2 P1
  20. 20. FlinkCEP Complex Patterns  Combine Individual Patterns  Contiguity Conditions • how to select relevant events given an input mixing relevant and irrelevant events  Time Constraints • within(time) e.g. all events have to come within 24h 20 Pattern P2 P1
  21. 21. FlinkCEP Contiguity Conditions 21 Pattern Input
  22. 22. FlinkCEP Contiguity Conditions 22 Pattern OutputInput Strict Contiguity • matching events strictly follow each other
  23. 23. FlinkCEP Contiguity Conditions 23 Pattern OutputInput
  24. 24. FlinkCEP Contiguity Conditions 24 Pattern Relaxed Contiguity • non-matching events to simply be ignored Input Output
  25. 25. FlinkCEP Contiguity Conditions 25 Pattern Input Output
  26. 26. FlinkCEP Contiguity Conditions 26 Pattern Input Output
  27. 27. FlinkCEP Contiguity Conditions 27 Pattern Input Output Non-Deterministic Relaxed Contiguity • allows non-deterministic actions on relevant events
  28. 28. FlinkCEP Contiguity Conditions 28 Pattern NOT patterns: • for strict and relaxed contiguity • for cases where an event should invalidate a match Input
  29. 29. FlinkCEP Summary 29  Quantifiers • oneOrMore(), times(), optional()  Conditions • Simple & Iterative  Time Constraints • Event and Processing time  Different Contiguity Constraints • Strict, relaxed, non-deterministic relaxed, NOT
  30. 30.  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 30 Running Example: retailer A B M1 M2 M3 M4 M5
  31. 31.  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 31 Observation A Individual Patterns Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  32. 32. 32 Observation B Quantifiers  Start/End: single event  Middle: multiple events • .oneOrMore() Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  33. 33. 33 Observation C Conditions  Start -> Simple • properties of the event  Middle/End -> Iterative • Depend on previous events Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  34. 34. 34  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h Observation D Time Constraints Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  35. 35. 35  We opt for relaxed continuity Observation E Contiguity
  36. 36. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Individual Patterns
  37. 37. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Quantifiers
  38. 38. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Conditions
  39. 39. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Time Constraint
  40. 40. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  41. 41. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  42. 42. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  43. 43. Documentation  FlinkCEP documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html 43
  44. 44. 4 Thank you! @KLOUBEN_K @ApacheFlink @dataArtisans
  45. 45. 45 Stream Processing and Apache Flink®'s approach to it @StephanEwen Apache Flink PMC CTO @ data ArtisansFLINKFORWARD IS COMING BACKTO BERLIN SEPTEMBER11-13, 2017 BERLIN.FLINK-FORWARD.ORG -
  46. 46. We are hiring! data-artisans.com/careers

×