Apache Apex (incubating) is a next generation native Hadoop big data platform. This talk will cover details about how it can be used as a powerful and versatile platform for big data. Presented by Pramod Immaneni at Data Riders Meetup hosted by Nexient on Apr 5th, 2016

1. Pramod Immaneni <pramod@datatorrent.com>
PPMC Member, Architect @DataTorrent Inc
Apr 5th, 2016
The next generation native Hadoop platform
Introduction to Apache Apex (incubating)

2. © 2015 DataTorrent
What is Apex
2
• Platform and framework to build scalable and fault-tolerant
distributed applications
• Hadoop native
• Build any custom logic in your application
• Unobtrusive API to facilitate distributed application development
• Runtime engine to ensure fault tolerance, scalability and
data flow
• Process streaming or batch big data
• High throughput and low latency
• Realtime applications

3. © 2015 DataTorrent
Applications on Apex
3
• Distributed processing
• Application logic broken into components called operators that run in a distributed fashion
across your cluster
• Natural programming model
• Code as if you were writing normal Java logic
• Maintain state in your application variables
• Scalable
• Operators can be scaled up or down at runtime according to the load and SLA
• Fault tolerant
• Automatically recover from node outages without having to reprocess from beginning
• State is preserved
• Long running applications
• Operational insight – DataTorrent RTS
• See how each operator is performing and even record data

4. © 2015 DataTorrent
Apex Platform Overview
4

5. © 2015 DataTorrent
Apache Malhar Library
5

6. © 2015 DataTorrent
Native Hadoop Integration
6
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state

7. © 2015 DataTorrent
Application Development Model
7
 A Stream is a sequence of data tuples
 A typical Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output
Stream
Tupl
e
Tupl
e
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator

8. © 2015 DataTorrent
Advanced Windowing Support
8
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency

9. © 2015 DataTorrent
Application in Java
9

10. © 2015 DataTorrent
Operators
10

11. © 2015 DataTorrent
Operators (contd)
11

12. © 2015 DataTorrent
Partitioning and unification
12
NxM PartitionsUnifier
0 1 2 3
Logical DAG
0 1 2
1
1 Unifier
1
20
Logical Diagram
Physical Diagram with operator 1 with 3 partitions
0
Unifier
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneck
Unifier
Unifier0
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier

13. © 2015 DataTorrent
Advanced Partitioning
13
0
1a
1b
2 3 4Unifier
Physical DAG
0 4
3a2a1a
1b 2b 3b
Unifier
Physical DAG with Parallel Partition
Parallel Partition
Container
uopr
uopr1
uopr2
uopr3
uopr4
uopr1
uopr2
uopr3
uopr4
dopr
dopr
doprunifier
unifier
unifier
unifier
Container
Container
NICNIC
NICNIC
NIC
Container
NIC
Logical Plan
Execution Plan, for N = 4; M = 1
Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiers
Cascading Unifiers
0 1 2 3 4
Logical DAG

14. © 2015 DataTorrent
Dynamic Partitioning
14
• Partitioning change while application is running
ᵒ Change number of partitions at runtime based on stats
ᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitions
ᵒ Supports re-distribution of state when number of partitions change
ᵒ API for custom scaler or partitioner
2b
2c
3
2a
2d
1b
1a1a 2a
1b 2b
3
1a 2b
1b 2c 3b
2a
2d
3a
Unifiers not shown

15. © 2015 DataTorrent
How tuples are partitioned
15
• Tuple hashcode and mask used to determine destination partition
ᵒ Mask picks the last n bits of the hashcode of the tuple
ᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuples
ᵒ Can also be used for specifying custom serialization
tuple: {
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x11)
Partition
00 1
01 2
10 3
11 4

16. © 2015 DataTorrent
Custom partitioning
16
• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x00)
Partition
00 1
00 2
00 3
00 4

17. © 2015 DataTorrent
Fault Tolerance
17
• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint state
ᵒ Frequency configurable per operator
ᵒ Asynchronous and distributed by default
ᵒ Default store is HDFS
• Automatic detection and recovery of failed operators
ᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so
that there is no loss of data
• Application master state checkpointed

18. © 2015 DataTorrent
Processing Guarantees
18
Atleast once
• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lost
ᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following mechanisms to achieve
exactly-once behavior in fault recovery scenarios
ᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operations
Atmost once
• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is
sufficient
Exactly once
• At least once + state recovery + operator logic to achieve end-to-end
exactly once

19. © 2015 DataTorrent
Stream Locality
19
• By default operators are deployed in containers (processes) randomly
on different nodes across the Hadoop cluster
• Custom locality for streams
ᵒ Rack local: Data does not traverse network switches
ᵒ Node local: Data is passed via loopback interface and frees up network
bandwidth
ᵒ Container local: Messages are passed via in memory queues between
operators and does not require serialization
ᵒ Thread local: Messages are passed between operators in a same thread
equivalent to calling a subsequent function on the message

20. © 2015 DataTorrent
Data Processing Pipeline Example
App Builder
20

21. © 2015 DataTorrent
Monitoring Console
Logical View
21

22. © 2015 DataTorrent
Monitoring Console
Physical View
22

23. © 2015 DataTorrent
Real-Time Dashboards
Real Time Visualization
23

24. © 2015 DataTorrent
Resources
24
Apache Apex Community Page - http://apex.incubator.apache.org/

25. End
25

26. © 2015 DataTorrent
Extra Slides

27. © 2015 DataTorrent
Application Programming Model
27
 A Stream is a sequence of data tuples
 An Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output StreamTuple Tuple
er
Operator
er
Operator
er
Operator
er
Operator

28. © 2015 DataTorrent
Partitioning and Scaling Out
28
• Operators can be dynamically
scaled
• Flexible Streams split
• Parallel partitioning
• MxN partitioning
• Unifiers

29. © 2015 DataTorrent
Fault Tolerance Overview
29
Stateful Fault Tolerance Processing Semantics Data Locality
 Supported out of the box
– Application state
– Application master state
– No data loss
 Automatic recovery
 Lunch test
 Buffer server
 At least once
 At most once
 Exactly once
 Stream locality for placement of
operators
 Rack local – Distributed
deployment
 Node local – Data does
not traverse NIC
 Container local – Data
doesn’t need to be
serialized
 Thread local – Operators
run in same thread
 Data locality

30. © 2015 DataTorrent
Machine Data Application
Logical View
30

31. © 2015 DataTorrent
Machine Data Application
Physical View
31