Security is a tradeoff between usability and safety and should be driven by the perceived threats. https://www.bigdataspain.org/2017/talk/keeping-enterprises-big-data-secure Big Data Spain 2017 November 16th - 17th Kinépolis Madrid

2. © Hortonworks Inc. 2017
Keeping Your Enterprise’s
Big Data Secure
Nov 2017
Page 1
Owen O’Malley
owen@hortonworks.com
@owen_omalley

3. © Hortonworks Inc. 2017
Security
Page 2
• What are the important threats?
• How to minimize attack surfaces?
• How do we control who has access?
• What data do we have?

4. © Hortonworks Inc. 2017
Threat: Accidental File Deletion
Page 3
• Need HDFS permissions
• Trash needs to be enabled too!
• Accountability
• Audit log is very important
• Avoid group accounts

5. © Hortonworks Inc. 2017
Threat: Accidental Killing Tasks
Page 4
• Need Linux Container Executor
• Small setuid executable
• Enables isolation between users
• Also provides local file protection
• In the works, Docker containers
• Stronger task killing
• Does not require user accounts

6. © Hortonworks Inc. 2017
Threat: Pretending to be a User
Page 5
• Need Kerberos
• Provides mutual authentication
• User logs in to get ticket
• Delegation tokens authenticate jobs
• Good for 1 day, renewable for 7
• Renewed and cancelled by YARN

7. © Hortonworks Inc. 2017
Threat: User accesses private data
Page 6
• Need Kerberos
• HDFS permissions are critical
• ACLs add additional flexibility
• Define user-to-group mapping
• Need group directories
• Minimize user or global spaces

8. © Hortonworks Inc. 2017
Threat: Pretending to be a Service
Page 7
• Need Kerberos integration
• Each service needs a keytab
• Hadoop’s UserGroupInformation
• loginFromKeytab
• When you are testing, test for longer
than 24 hours

9. © Hortonworks Inc. 2017
Threat: Remote Access
Page 8
• Need Service Level Authorization
• Which users can use each service
• Apache Ranger simplifies this
• Need firewall around cluster
• Never let users log in to masters

10. © Hortonworks Inc. 2017
Security Architecture
Page 9

11. © Hortonworks Inc. 2017
Threat: Eavesdropping Inside Data Center
Page 10
• Super user can watch network traffic
• Need wire encryption
• RPC encryption
• Data transfer
• MapReduce Shuffle
• Very expensive
• Generally not recommended

12. © Hortonworks Inc. 2017
Threat: Eavesdropping Outside Data Center
Page 11
• Intranet hard to control
• Need HTTPS encryption for outside
• Set up SSL certificates
• Create a master certificate to sign
all of the others.
• Have users add master certificate
to their browsers

13. © Hortonworks Inc. 2017
Threat: Physical access
Page 12
• Very Rare
• Attacker can get to physical box
• Hopeless
• Can remove hard drives
• Includes access to retired drives
• Need raw file system encryption

14. © Hortonworks Inc. 2017
Threat: Attackers gets Hadoop Admin
Page 13
• Attacker is an Hadoop Admin
• But not root
• Need HDFS Encryption Zones
• Directory sub-tree encryption
• Each file gets unique key
• Client decrypts data

15. © Hortonworks Inc. 2017
HDFS Encryption
Page 14
• Each zone has master key
• Each file gets unique sub-key
• HDFS stores sub-key encrypted with
master key
• Client uses sub-key to decrypt file
• Noticeable performance impact

16. © Hortonworks Inc. 2017
KeyProvider API
Page 15
• Key management
• Allows 3rd
party plugins
• Named keys
• Key versions and key rolling
• Ranger provides a Key Management
Server

17. © Hortonworks Inc. 2017
Ranger KMSRanger KMS
Transparent Data Encryption in HDFS
NNNN
A B
C D
HDFS
Client
HDFS
Client
A B
C D
A B
C D
DN DN DN
Benefits
 Selective encryption of relevant
files/folders
 Fine grained access controls
 Transparent to end application w/o
changes
 Ranger KMS integrated to external HSM
(Safenet Luna) adding to reliability/security
of KMS
SafeNet-
Luna HSM

18. © Hortonworks Inc. 2017
Threat: Sensitive Tabular Data
Page 17
• Need to protect sensitive columns
• Example - Personal information
• Restrict rows based on filters
• Example – Different Regions
• HDFS permissions don’t work
• Need Hive Server 2

19. © Hortonworks Inc. 2017
Hive Architecture with Storage-Based Auth
Page 18

20. © Hortonworks Inc. 2017
Hive Architecture with Hive Server 2
Page 19

21. © Hortonworks Inc. 2017
Configuring Hive Server 2
Page 20
• Configure Ranger security policies
• Or use Hive SQL Standard Auth
• Restrict permissions in HDFS
• Only Hive has access
• All normal users use Hive Server 2

22. © Hortonworks Inc. 2017
Attribute-Based Access Control (ABAC)
Page 21
• Ranger policies allow you to
control access with ABAC.
• Combination of the subject,
action, resource, and
environment
• Uses descriptive attributes: AD
group, Atlas-based tags or
classifications, geo-location, etc.
• Consistent with NIST 800-162
• Avoid role proliferation and
manageability issues

23. © Hortonworks Inc. 2017
Data Masking and Anonymization
Page 22
• Some users need limited access
• Very difficult problem
• Ranger’s dynamic masking
• Redact – Replace character sets
• Hash – Replace with hash
• Tokenization – Replace with ID
• User defined

24. © Hortonworks Inc. 2017
ORC Column Encryption
Page 23
• Encrypts just the sensitive columns
• Columns may use different keys
• Uses Ranger KMS
• Local key for each column & file
• Unencrypted data can be statically
masked
• Follow progress in ORC-14.

25. © Hortonworks Inc. 2017
Tabular Access from Spark
Page 24
• Require fine-grain access control
• Hive has created LLAP
• Live, Long & Process
• Caches hot columns and partitions
• Runs sub-query on daemons
• Created an LLAP Context for Spark

26. © Hortonworks Inc. 2017
Apache Atlas – Data Governance
Page 25
• As data and clusters grow need to
track the data.
• Atlas provides Data Governance
• What data do you have?
• Where did it come from?
• Extensible data modelling
• Integrates with other system

27. © Hortonworks Inc. 2017
Apache Atlas: Lineage and Impact

28. © Hortonworks Inc. 2017
Apache Atlas: Classification
• Categorize and curate data assets for easier discovery
• Associate context with data– Governance, Security, Business
GOVERNANCEGOVERNANCE
SECURITYSECURITY
BUSINESSBUSINESS

29. Metadata Catalog Search : Basic
Search for a hive_table classified as ‘PII’ and name starting with ‘prov’
Filter by
Data Asset type
Filter by
Data Asset type
Filter by
Classification
Filter by
Classification
Search text
Wildcards: prov*, *sum*
Logical expressions: prov* AND *sum*
Search text
Wildcards: prov*, *sum*
Logical expressions: prov* AND *sum*

30. Metadata Catalog Search : Advanced
Filter by
Data asset type
Filter by
Data asset type
Search for a hive_table named ‘employees’ and owner ‘hive’
DSL search with SQL like syntax
Select columns from impressions table in raw database
hive_column where table.name=‘impressions’ and table.db.name
= ‘raw’
DSL query
string
DSL query
string

31. © Hortonworks Inc. 2017
Key Take Aways
Page 30
• Think about security in terms of
threats.
• Think holistically about security.
• Consider encryption and masking.
• Create Data Catalog with Atlas.
• Identify and classify data.
• Understand data propagation.

32. © Hortonworks Inc. 2017
Thank You!
Page 31
Owen O’Malley
@owen_omalley
owen@hortonworks.com