http://flink-forward.org/kb_sessions/flink-security-enhancements/ Recent security enhancements to Flink make it easy to access secure data and to protect the associated credentials. In this talk we’ll describe and demonstrate the new features, including Kerberos-based access to HDFS and Kafka, transport security (TLS), and service-level authorization which protects your Flink cluster from unauthorized access.

1. Flink Security
Enhancements
Eron Wright – eron.wright@emc.com
DELL EMC
@eronwright

2. 2 of 11
New Security Features
1. Kerberos Authentication Support
2. Service-Level Authorization
3. Transport Security (SSL/TLS)

3. 3 of 11
Existing Capability
• Hadoop Delegation Token (DT)
• CLI usesKerberosto authenticateto HDFS
• HDFSprovidesa DT, which CLI passesto the Flinkcluster
• Clusteris ableto accessHDFSfilesonbehalfof theuser
• Limitations
• YARN mode only
• Not usefulto non-Hadoopservices,e.g. Kafka.
• Note: Still supported
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
delegation token

4. 4 of 11
Kerberos Authentication Support
• “Cluster-Level Kerberos Identity”
• Keytab-based
• Sharedby alljobs, notjob-specific
• Enables Kerberos authentication
• DataSourcesandSinks(HDFS,Kafka…)
• StateBackends(ZooKeeper…)
• Protects state data
• ACL onznodes,files
• Supported in standalone and YARN
deployment modes
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
keytab

5. 5 of 11
Service-Level Authorization
• “Restrict access to your Flink cluster”
• Protects all endpoints:
• Akka System(control path)
• Intra-ClusterDataTransfer
• WebUI
• BlobTransfer(JARs…)
• Simple shared secret
• Configuredor generated
• Storedonclient (~/.flink/…)
• Storedincluster
• Supported in standalone and YARN
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
keytab secret

6. 6 of 11
Transport-Level Security (SSL/TLS)
• “SSL for all connections”
• May be enabled on a per-endpoint basis
• WebUIis problematic
• Supported in standalone and YARN TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTPS
Flink
Cluster
keytab secret TLS cert(s)

7. Demo

8. 8 of 11
Configuration
• Configure Kerberos Identity:
– security.enabled: true
– security.keytab: /path/to/keytab
– security.principal: name@realm
• Configure Service-Level Authorization:
– security.cookie: (secret cookie)
• Configure Transport-Level Security:
– security.ssl.enabled: true
– security.ssl.keystore: /path/to/keystore
– security.ssl.keystore-password: (password)
– security.ssl.key-password: (password)
– security.ssl.truststore: /path/to/truststore
– security.ssl.truststore-password: (password)
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTPS
Flink
Cluster
keytab secret TLS cert(s)

9. Summary

10. 10 of 11
Project Status
• Targeted for: Flink 1.2
• Contributors:
– Vijay Srinivasaraghavan (Dell EMC)
– Suresh Krishnappa (Dell EMC)
• Design Doc: Secure Data Access on Google Docs
• JIRAs:
– FLINK-3929 - Support for Kerberos Authentication with Keytab Credential
– FLINK-3930 - Implement Service-Level Authorization
– FLINK-3931 - Implement Transport Encryption (SSL/TLS)
– FLINK-3932 - Implement State Backend Security
• Code:
– Github: https://github.com/EronWright/flink/tree/feature-flink-security