Weitere Àhnliche Inhalte
Ăhnlich wie Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apache Knox (20)
Mehr von Hortonworks (20)
KĂŒrzlich hochgeladen (20)
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apache Knox
- 1. Page 1 © Hortonworks Inc. 2014
Discover HDP 2.1
New Features for Security & Apache Knox
Hortonworks. We do Hadoop.
- 2. Page 2 © Hortonworks Inc. 2014
Speakers
Justin Sears
Hortonworks Product Marketing Manager
Vinay Shukla
Hortonworks Director of Product Management
& owner of Hortonworks security roadmap
Kevin Minder
Hortonworks Engineer & Committer for
Apache Knox Gateway project
- 3. Page 3 © Hortonworks Inc. 2014
Agenda
âąâŻ Security for Hadoop REST/HTTP API â Knox Gateway
âąâŻ HDFS Security â ACLs
âąâŻ SQL Security â Next Generation Hive Authorization
- 4. Page 4 © Hortonworks Inc. 2014
OPERATIONS*TOOLS*
Provision,
Manage &
Monitor
DEV*&*DATA*TOOLS*
Build &
Test
A Modern Data ArchitectureAPPLICATIONS*DATA**SYSTEM*
REPOSITORIES*
RDBMS* EDW* MPP*
Business**
Analy<cs*
Custom*
Applica<ons*
Packaged*
Applica<ons*
Governance
&Integration
ENTERPRISE HADOOP
Security
Operations
Data Access
Data Management
SOURCES*
OLTP,&ERP,&
CRM&Systems&
Documents,&&
Emails&
Web&Logs,&
Click&Streams&
Social&
Networks&
Machine&
Generated&
Sensor&
Data&
GeolocaCon&
Data&
- 5. Page 5 © Hortonworks Inc. 2014
HDP 2.1: Enterprise Hadoop
HDP 2.1
Hortonworks Data Platform
**
Provision,*
Manage*&*
Monitor*
&
Ambari&
Zookeeper&
Scheduling*
&
Oozie&
Data*WorkïŹow,*
Lifecycle*&*
Governance*
*
Falcon&
Sqoop&
Flume&
NFS&
WebHDFS&
YARN*:*Data*Opera<ng*System&
DATA**MANAGEMENT*
SECURITY*DATA**ACCESS*
GOVERNANCE*&*
INTEGRATION*
Authen<ca<on*
Authoriza<on*
Accoun<ng*
Data*Protec<on*
&
Storage:&HDFS&
Resources:&YARN&
Access:&Hive,&âŠ&&
Pipeline:&Falcon&
Cluster:&Knox&
OPERATIONS*
Script*
&
Pig&
*
*
Search*
*
Solr&
*
*
SQL*
*
Hive/Tez,&
HCatalog&
*
*
NoSQL*
*
HBase&
Accumulo&
*
*
Stream*
**
Storm&
&
*
*
Others*
*
InTMemory&
AnalyCcs,&&
ISV&engines&
1& °& °& °& °& °& °& °& °& °&
°& °& °& °& °& °& °& °& °& °&
°& °& °& °& °& °& °& °& °& °&
°&
°&
N*
HDFS**
(Hadoop&Distributed&File&System)&
Batch*
*
Map&
Reduce&
*
*
- 6. Page 6 © Hortonworks Inc. 2014
HDP 2.1: Enterprise Hadoop
HDP 2.1
Hortonworks Data Platform
**
Provision,*
Manage*&*
Monitor*
&
Ambari&
Zookeeper&
Scheduling*
&
Oozie&
Data*WorkïŹow,*
Lifecycle*&*
Governance*
*
Falcon&
Sqoop&
Flume&
NFS&
WebHDFS&
YARN*:*Data*Opera<ng*System&
DATA**MANAGEMENT*
DATA**ACCESS*
GOVERNANCE*&*
INTEGRATION*
OPERATIONS*
Script*
&
Pig&
*
*
Search*
*
Solr&
*
*
SQL*
*
Hive/Tez,&
HCatalog&
*
*
NoSQL*
*
HBase&
Accumulo&
*
*
Stream*
**
Storm&
&
*
*
Others*
*
InTMemory&
AnalyCcs,&&
ISV&engines&
1& °& °& °& °& °& °& °& °& °&
°& °& °& °& °& °& °& °& °& °&
°& °& °& °& °& °& °& °& °& °&
°&
°&
N*
HDFS**
(Hadoop&Distributed&File&System)&
Batch*
*
Map&
Reduce&
*
*
SECURITY*
Authen<ca<on*
Authoriza<on*
Accoun<ng*
Data*Protec<on*
&
Storage:&HDFS&
Resources:&YARN&
Access:&Hive,&âŠ&&
Pipeline:&Falcon&
Cluster:&Knox&
- 7. Page 7 © Hortonworks Inc. 2014
Security: Rings of Defense
Perimeter Level Security
âąâŻ Network Security (i.e. Firewalls)
âąâŻ Apache Knox (i.e. Gateways)
Authentication
âąâŻ Kerberos
OS Security
Authorization
âąâŻ MR ACLs
âąâŻ HDFS Permissions
âąâŻ HDFS ACLs
âąâŻ HiveATZ-NG
âąâŻ HBase ACLs
âąâŻ Accumulo Label Security
Data Protection
âąâŻ Core Hadoop
âąâŻ Partners
- 8. Page 8 © Hortonworks Inc. 2014
Security for Hadoop REST API â
Apache Knox Gateway
- 9. Page 9 © Hortonworks Inc. 2014
Current Hadoop Client Model
âąâŻ FileSystem and MapReduce Java APIs
âąâŻ HDFS, Pig, Hive and Oozie clients (that wrap the Java APIs)
âąâŻ Typical use of APIs is via âEdge Nodeâ that is âinsideâ cluster
âąâŻ Users SSH to Edge Node and execute API commands from shell
HadoopUser
Edge
Node
SSH!
- 10. Page 10 © Hortonworks Inc. 2014
Why Knox?
Simplified
Access
Single Hadoop access point
Rationalized REST API
hierarchy
Consolidated API calls
Multi-cluster support
Client DSL
Centralized
Security
Eliminate SSH âedge nodeâ
Central API management &
audit
Service-level authorization
Identity
Management
SSO Integration
LDAP & AD integration
Knox eliminates the clientâs requirements for intimate knowledge of cluster topology
- 11. Page 11 © Hortonworks Inc. 2014
Hadoop REST API Security: Drill-Down
REST
Client
Enterprise
Identity
Provider
LDAP/AD
Knox
Gateway
GW
GW
Firewall
Firewall
DMZ
L
B
Edge
Node/
Hadoop
CLIs
Edge
Node/
Hadoop
CLIs
RPC
HTTP
HTTP HTTP
LDAP
RPC
Hadoop Cluster 2
Masters
Slaves
NN
RM Oozie
Web
HCat
HS2
HBase
DN NM
Hadoop Cluster 2
Masters
Slaves
NN
RM Oozie
Web
HCat
HS2
HBase
DN NM
- 12. Page 12 © Hortonworks Inc. 2014
Knox Summary
âąâŻ Simplifies Client Interaction with REST Web Services
âąâŻ Abstracts away complexities of Kerberos
âąâŻ Integrates with LDAP, Site Minder & other protocols in future
âąâŻ Provides Authorization to each Web Service with IP, User, Group
policies
âąâŻ Able to secure multiple clusters through a single-endpoint
- 13. Page 13 © Hortonworks Inc. 2014
HDFS Access Control List (ACL)
- 14. Page 14 © Hortonworks Inc. 2014
HDFS Permissions Model Before HDP 2.1
âąâŻHDFS permissions at a File & Directory level
âąâŻManaged by a set of 3 distinct user classes
ââŻâownerâ, âgroupâ and âothersâ
âąâŻ3 permissions for each user class
ââŻRead (ârâ), Write (âwâ), Execute (âeâ)
ââŻFor Files, ârâ for read, âwâ for write
ââŻFor Directories, ârâ to list content, âwâ to create/delete files +
directories, âxâ for access child of directory
Owner
Group
Others
HDFS
Directory
⊠rwx
⊠rwx
⊠rwx
- 15. Page 15 © Hortonworks Inc. 2014
HDFS File Permissions Example
âąâŻ Authorization requirements:
â⯠In a sales department, they would like a single user Maya (Department
Manager) to control all modifications to sales data
â⯠Other members of sales department need to view the data, but canât modify it.
â⯠Everyone else in the company must not be allowed to view the data.
âąâŻ Can be implemented via the following:
Read/Write perm for
user maya
User
Group
Read perm for
group sales
File with sales data
- 16. Page 16 © Hortonworks Inc. 2014
HDFS Extended ACLs in HDP 2.1
âąâŻ Problem
ââŻNo longer feasible for Maya to control all modifications to the file
â⯠New Requirement: Maya, Diane and Clark are allowed to make modifications
â⯠New Requirement: New group called executives should be able to read the sales data
ââŻCurrent permissions model only allows permissions at 1 group and 1 user
âąâŻ Solution: HDFS Extended ACLs
ââŻNow assign different permissions to different users and groups
Owner
Group
Others
HDFS
Directory
⊠rwx
⊠rwx
⊠rwx
Group D ⊠rwx
Group F ⊠rwx
User Y ⊠rwx
- 17. Page 17 © Hortonworks Inc. 2014
HDFS Extended ACLs in HDP 2.1
New Tools for ACL Management (setfacl, getfacl)
ââŻhdfs dfs -setfacl -m group:execs:r-- /sales-data!
ââŻhdfs dfs -getfacl /sales-dataâš
# file: /sales-dataâš
# owner: mayaâš
# group: salesâš
user::rw-âš
group::r--âš
group:execs:r--âš
mask::r--âš
other::--!
How do you know if a directory has ACLs set?
ââŻhdfs dfs -ls /sales-dataâš
Found 1 itemsâš
-rw-r-----+  3 maya sales          0 2014-03-04
16:31 /sales-data!
- 18. Page 18 © Hortonworks Inc. 2014
HDFS Extended ACLs in HDP 2.1
Default ACLs
ââŻhdfs dfs -setfacl -m default:group:execs:r-x /
monthly-sales-data!
ââŻhdfs dfs -mkdir /monthly-sales-data/JAN!
ââŻhdfs dfs âgetfacl /monthly-sales-data/JAN!
â⯠# file: /monthly-sales-data/JANâš
# owner: mayaâš
# group: salesâš
user::rwxâš
group::r-xâš
group:execs:r-xâš
mask::r-xâš
other::---âš
default:user::rwxâš
default:group::r-xâš
default:group:execs:r-xâš
default:mask::r-xâš
default:other::---"
- 19. Page 19 © Hortonworks Inc. 2014
SQL-Style Security for Hive âATZ-NG
- 20. Page 20 © Hortonworks Inc. 2014
Hive Authorization Before HDP 2.1
HiveAuthorizationProvider(HAP) as the base interface
1.⯠StorageBasedAuthorizationProvider
ââŻUses HDFS permissions to make authorization decision
ââŻHDFS dir permission = Table Permission
ââŻCoarse grained, no column level security
ââŻSecure://hive.apache.org/docs/hcat_r0.5.0/authorization.pdf
2.⯠DefaultHiveAuthorizationProvider â BROKEN
HORTONWORKS RECOMMENDATION: DO NOT USE
ââŻRDBMS style authorization provider
ââŻDoes not check all operations
ââŻDoes not check policy grants
- 21. Page 21 © Hortonworks Inc. 2014
Hive Authorization in HDP 2.1
âąâŻMany paths into Hive
ââŻHive CLI, Beeline, Oozie, Hue, Pig, HCatalog, etc.
ââŻAdmin type users use CLI, Pig, HCatalog
ââŻBusiness users use O/JDBC, Beeline
âąâŻOther security concerns
ââŻAuthentication is enforced. It is a pre-requisite to meaningful
authorization
ââŻNo direct access to HDFS â cluster is Kerberized and restricts
access
ââŻHive Metastore is protected and allows only authorized access
ââŻViews are used to provide row/column level access with ATZ-NG
- 22. Page 22 © Hortonworks Inc. 2014
Hive ATZ-NG â Architecture
HDFS
Metastore
HiveServer2
O/JDBC Beeline CLI
âąâŻ ATZ-NG is called for O/JDBC & Beeline CLI
âąâŻ Standard SQL GRANT / REVOKE for management
âąâŻ Privilege to register UDF restricted to Admin user
âąâŻ Policy integrated with Table/View life cycle
Storage Based Authorization
Hive
CLI
OozieHue
PIG HCat
Ambari
0. Enable HiveATZ-NG
1. Authentication
UDFs
Protected â fine grained
Protected -- coarse grained
Restrict direct access to Metastore
Protect HDFS with Kerberos & HDFS ACL
ATZ-NG
2. Authorization
- 23. Page 23 © Hortonworks Inc. 2014
Hive ATZ-NG Details
Hive ATZ NG
SQL standard-based authorization
Manually config Hive to enable, Hive restart required
Grants on tables or views to roles or users
GRANT/REVOKE action ON [table | view] to role | user!
Policy stored in Hive Metastore
Table/View lifecycle auto-synced with policy stored in Hive Metastore
Grant/Revoke does integrity check, prevents invalid policies
Show grants on user | table | view | role & shows policy
Supports delegated administration
All data need to be readable/writable by Hive user, combined with HDFS ACL,
need not be owned by Hive user
Back up of Policy same as Hive Metastore backup
Check on the ability to register UDF
- 24. Page 24 © Hortonworks Inc. 2014
What about MR/Pig/Hive CLI?
âąâŻAll these are ETL run by privileged users
âąâŻProtect them at coarse grained level with
StorageBasedAuthorization
- 25. Page 25 © Hortonworks Inc. 2014
Summary
ATZ-NG is a superior approach for Hive Authorization because it
delivers:
1.⯠Familiar & DBA-friendly approach for defining security policies
for Hive Tables. No additional education required to understand
how to take advantage of this.
2.⯠Integrated and error-free policy definition approach which
works in lock-step with the lifecycle of tables and views.
3.⯠Minimal additional operational overhead to take advantage of
ATZ-NG; from no required MR/YARN restart through leveraging
pre-existing Hive Metastore (and associated handling - back-up,
recovery, etc.)
- 26. Page 26 © Hortonworks Inc. 2014
Hive ATZ-NG Example
Page 26
- 27. Page 27 © Hortonworks Inc. 2014
Scenario
âąâŻObjective: Share Product Management Roadmap
securely
âąâŻActors:
ââŻAdmin Role â Specified in hive-site
ââŻAdmin role controls role memberships
ââŻProduct Management Role
ââŻShould be able to create, read all road map details.
ââŻMembers: Vinay Shukla, Tim Hall
ââŻEngineering Role
ââŻShould be able to read (see) all roadmap details
ââŻMembers: Kevin Minder, Larry McCay
- 28. Page 28 © Hortonworks Inc. 2014
Step 1: Admin role Creates Roles, Adds Users
1.⯠CREATE ROLE PM;
2.⯠CREATE ROLE ENG;
3.⯠GRANT ROLE PM to user timhall with admin option;
4.⯠GRANT ROLE PM to user vinayshukla;
5.⯠GRANT ROLE ENG to user kevinminder with admin option;
6.⯠GRANT ROLE ENG to user larrymccay;
- 29. Page 29 © Hortonworks Inc. 2014
Step 2: Super-user Creates Tables/Views
create table hdp_hadoop_plans (
id int,
hadoop_roadmap string,
hdp_roadmap string
);
- 30. Page 30 © Hortonworks Inc. 2014
Step 3: Users or Roles Assigned To Tables
1.⯠GRANT ALL ON hdp_hadoop_plans TO ROLE PM;
2.⯠GRANT SELECT ON hdp_hadoop_plans TO ROLE
ENG;
- 31. Page 31 © Hortonworks Inc. 2014
Learn More
Hortonworks.com/labs/
security/
Register for the other six
Discover HDP 2.1 Webinars
Hortonworks.com/webinars
Next on the Security Roadmap