David Yan, PMC Member, Apache Apex & Architect, DataTorrent To make Apache Apex operable, DataTorrent RTS includes a component called dtGateway, which provides a comprehensive REST API to let users query for cluster and application metrics, manage application packages, monitor running applications, etc. This talk will introduce features and cover the internal dynamics of dtGateway.

1. © 2016 DataTorrent
David Yan
PMC Member, Apache Apex
Architect, DataTorrent
June 1, 2016
dtGateway
Making Apache Apex operable

2. © 2016 DataTorrent
What is dtGateway
2
• Web server communicating in JSON via REST API
• You can use the REST API to:
Get performance metrics of Apex application instances, operators,
containers
Retrieve container logs, dynamically change operator properties,
and change the DAG of a running Apex application
Manage Apex application packages
Launch and kill applications
Query and visualize application data in real time
• Powers the DataTorrent RTS Web UI

3. © 2016 DataTorrent
Links
3
• dtGateway Overview:
http://docs.datatorrent.com/dtgateway/
• dtGateway REST API Specification:
http://docs.datatorrent.com/dtgateway_api/

4. © 2016 DataTorrent
Internal Dynamics
4
HDFS
Apex STRAMs
Node
Managers
Resource Manager
dtGateway
REST over
HTTP
REST over
HTTP
Hadoop Java
API
Hadoop Java
API
HTTP
Apex Worker
Containers
RP
C
This is awesome!

5. © 2016 DataTorrent
Security and Access Control
5
• dtGateway by default runs under a dedicated UNIX user “dtadmin”
• Supports front-end authentication: password, Kerberos, LDAP, PAM,
ActiveDirectory and generic JAAS
• Supports back-end Kerberos handshake with Hadoop components
when Hadoop secure mode is turned on.
• When authentication is enabled, dtGateway will impersonate the
logged-in user when launching applications
• RBAC: Users are assigned to roles, and each role has a set of
permissions.
• For more info on security in dtGateway:
http://docs.datatorrent.com/dtgateway_security/

6. © 2016 DataTorrent
Use Case for Monitoring: Ad Tech
6

7. © 2016 DataTorrent
Use Case for Monitoring: Ad Tech
7
Metrics captured using REST API
● Number of Planned Containers and Allocated Containers
● Total memory allocated
● Latency, both end-to-end and at individual operator level
● Number of failed containers
● Application Status (Fire alert if FAILED, KILLED)
● Operator status - the # operators in PENDING_DEPLOY state
Open Source Example:
https://github.com/DataTorrent/examples/tree/master/tools

8. © 2016 DataTorrent
Resources
8
• Apache Apex website - http://apex.apache.org/
• Subscribe - http://apex.apache.org/community.html
• Download - http://apex.apache.org/downloads.html
• Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex
• Facebook - https://www.facebook.com/ApacheApex/
• Meetup - http://www.meetup.com/topics/apache-apex
• Startup Program – Free Enterprise License for Startups, Educational Institutions,
Non-Profits - https://www.datatorrent.com/startups/
• Cloud Trial - https://www.datatorrent.com/download/cloud-trial/

9. © 2016 DataTorrent
We Are Hiring
9
• jobs@datatorrent.com
• Back-End Engineers
• Front-End Engineers
• QA Automation Engineers
• Solutions Engineers
• Build and Release Engineers