This document provides an overview of Apache Apex, an open source unified streaming and fast batching platform. It discusses key aspects of Apex including its application programming model using operators and directed acyclic graphs, native Hadoop integration using YARN and HDFS, partitioning and scaling operators for high throughput, windowing support, fault tolerance, and data locality features. Examples of building a data processing pipeline and its logical and physical plans are also presented.

1. Thomas Weise <thomas@datatorrent.com>
Dec 2nd, 2015
Introduction to Open Source Unified Streaming and Fast Batching Platform
Apache Apex

2. © 2015 DataTorrent
Apex Platform Overview
2

3. © 2015 DataTorrent
Apache Malhar Library
3

4. © 2015 DataTorrent
Native Hadoop Integration
4
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state

5. © 2015 DataTorrent
Application Programming Model
5
 A Stream is a sequence of data tuples
 An Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance in single-threaded
 Directed Acyclic Graph (DAG) is made up of operations and streams
Directed Acyclic Graph (DAG)
Output StreamTuple Tuple
er
Operator
er
Operator
er
Operator
er
Operator

6. © 2015 DataTorrent
Application Specification
6

7. © 2015 DataTorrent
Partitioning and Scaling Out
7
• Operators can be dynamically
scaled
• Flexible Streams split
• Parallel partitioning
• MxN partitioning
• Unifiers

8. © 2015 DataTorrent
Advanced Windowing Support
8
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency

9. © 2015 DataTorrent
Platform Features
9
Stateful Fault Tolerance Processing Semantics Data Locality
 Supported out of the box
– Application state
– Application master state
– No data loss
 Automatic recovery
 Lunch test
 Buffer server
 At least once
 At most once
 Exactly once
 Stream locality for placement of
operators
 Rack local – Distributed
deployment
 Node local – Data does
not traverse NIC
 Container local – Data
doesn’t need to be
serialized
 Thread local – Operators
run in same thread
 Data locality

10. © 2015 DataTorrent
Dynamic Updates
10
 Dynamic topology updates
– Properties of operators can be changed
– New operators can be added

11. © 2015 DataTorrent
Data Processing Pipeline Example
App Builder
11

12. © 2015 DataTorrent
Data Processing Pipeline Example
Logical Plan
12

13. © 2015 DataTorrent
Data Processing Pipeline Example
Physical Plan
13

14. © 2015 DataTorrent
Data Processing Pipeline Example
Real Time Visualization
14

15. © 2015 DataTorrent
Resources
15
Apache Apex Community Page
Apache Apex LinkedIn Group

16. © 2015 DataTorrent
We Need Your Vote
16
Vote for our Apex Session to see us in Hadoop Summit Europe!

17. End
17