Apache HDFS Extended Attributes and Transparent Encryption

Extended Attributes and Transparent
Encryption in Apache Hadoop
Uma Maheswara Rao G
Yi Liu (刘轶)
Copyright © 2014 Intel Corporation.

Who we are?
• Uma Maheswara Rao G
- Software Engineer at Intel
- PMC member at Apache Hadoop
- Active committer of Apache Hadoop
@UmaMaheswaraG | umamahesh@apache.org
• Yi Liu (刘轶)
- Software Engineer at Intel
- Active committer of Apache Hadoop
- PMC/committer at Apache Tajo
- Senior security expert of Big data

Intel BigData Team
• Global team, local focus
• Worldwide (China, US and India) teams, >80% in China
• Local collaborations (industry & academic) a high priority
• Greater impact thru open source
• Active open source development (Spark, Hadoop, HBase, Storm, etc.)
• Widely used in the industry (from Facebook to Alibaba to Cloudera to China Mobile …)
• Strong influence in the open source community
• ~10 project committers in the team
• Technology and innovation oriented
• Next generations of Big Data Technologies
• Real-time, in-memory, complex analytics (statistic modeling, machine learning, graph analysis,
…)
• Bridging advanced research and real-world applications

Agenda
• Extended Attributes
• Transparent Encryption

HADOOP Ecosystem
Batch
Processing
MAPREDUCE,
HIVE, PIG
HBase
SQL
HDFS
(Hadoop Distributed File System)
YARN
(Resource Management)
Search
Stream
DATA INTEGRATION (Sqoop, Flume …)
…Machine
Learning
ZooKeeper
SPARK

HDFS Extended Attributes
HDFS-2006

Introduction
• Allows user to associate addition metadata with files/directories
• XAttrs will be set as Key-Value pair on any INode
• File System will not interpret the XAttrs
• Derived from Linux XAttrs feature, so it is functionally similar.
• Allows users to set own encoding format to XAttrs

Namespaces of XAttrs
• XAttrs should be prefixed with namespace
• HDFS support 5 XAttrs namespaces
•Access permission defined by file/directory permission bits
•For Sticky directories, only owner and privileged users can write
USER
•Only visible and accessed by privileged usersTRUSTED
•Not visible to users
•Only available for System kernel
SYSTEM
•Not visible to users
•Only available for System kernel for storing security information
SECURITY
•They are like SYSTEM attributes, but they can be accessed the files/directories under
./reserved/raw by the super users only.RAW

Implementation details
• XAttrs stored as separate INode feature in Namenode
• XAttrs will be persisted as part of INode information
• XAttrs will be validated against the Namespaces at the Namenode

Configuration
• dfs.namenode.xattrs.enabled
Whether the support of XAttrs is enabled in HDFS.
• dfs.namenode.fs-limits.max-xattrs-per-inode
Max number of XAttrs per Inode. Default 32 bytes.
• dfs.namenode.fs-limits.max-xattr-size
Max combined size of name and value of XAttrs. Default 16384 bytes

Use Cases
• Storing the Encrypted Data Encryption Keys as XAttrs in HDFS
Encrypted cluster environment
• Storing policy for Heterogeneous Storage

Transparent Encryption in Hadoop
(HADOOP-10150 & HDFS-6134)

Outlines
• Transparent to upper layer applications and transparent access to
encrypted files by all HDFS clients.
• High performance, it’s not bottleneck.
• Encryption is independent of the file type, data format.
• Scalable key management.
• End-to-end encryption: data can only be encrypted and decrypted
by the client. This satisfies two typical requirements for encryption:
at-rest encryption and in-transit encryption.
• Security: HDFS never handles unencrypted data or data encryption
keys.

Write file
NNNN
DNDNDN
DFS
Client
KMS
5. Encrypt data
using DEK
4. Decrypt EDEK and get DEK
Fill EDEK cache
in background
2. EDEK from
cache and persist
to File metadata.
Backing
keystore
HDFS

Read file
15
NNNN
DNDNDN
DFS
Client
KMS
4. Decrypt EDEK and get DEK
2. Read EDEK from
File metadata.
Backing
keystore
6. Decrypt data
using DEK
HDFS

Implementation details
• Pread support.
• Original file and Cipher file have the same length and 1:1
corresponding by using AES-CTR
• Use AES-NI support on Intel platform to improve encryption
performance, 20x speedup.
• We define encryption zone and files are transparently
encrypted/decrypted in the zone.
• We use two layer keys: encryption zone key (EZK), and data
encryption key (DEK) which is encrypted by EZK. Each file has a
different DEK.
16 Copyright © 2014 Intel Corporation.

Encryption/Decryption for HDFS Blocks

User Ops
• Create Key
hadoop key create <keyname> [-cipher <cipher>] [-size <size>]
[-description <description>]
[-attr <attribute=value>]
[-provider <provider>]
• Roll Key
hadoop key roll <keyname> [-provider <provider>]
• Delete Key
hadoop key delete <keyname> [-provider <provider>]
• List Keys
hadoop key list [-provider <provider>] [-metadata]

Admin Ops
• Create Encryption Zone
hdfs crypto -createZone -keyName <keyName> -path <path>
• List Encryption Zones
hdfs crypto -listZones

Usage Example
• As a normal user, create a new encryption key:
$ hadoop key create myKey
• As the super user, create a new empty directory and make it an
encryption zone:
$ sudo -u hdfs hadoop fs -mkdir /zone
$ sudo -u hdfs hdfs crypto -createZone -keyName myKey -path /zone
• Change its ownership to the normal user:
$ sudo -u hdfs hadoop fs -chown myuser:myuser /zone
• As the normal user, put a file in, read it out:
$ hadoop fs -put helloWorld /zone
$ hadoop fs -cat /zone/helloWorld

Performance
TestDFSIO Benchmark
AES-NI enabled

Call for Collaborations
• Close collaborations with local ecosystems
• Intel Big Data engineering teams, industry partners and academic research
• Building next generations of Big Data Technologies
• Real-time, in-memory, complex analytics, etc.
• Bridging advanced research and real-world applications
• Highly impactful through open source, university research (e.g., UC Berkeley) and
industry adoptions (e.g., Alibaba, Cloudera, etc.)

Q & A
Thanks!

24
Notices and Disclaimers:
• Copyright © 2014 Intel Corporation.
• Intel, the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as
the property of others.
• All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product
specifications and roadmaps.
• Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.
Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations
and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance
tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other
products.
For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.
• Optimization Notice
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel
microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the
availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel.
Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not
specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference
Guides for more information regarding the specific instruction sets covered by this notice.
• Intel technologies may require enabled hardware, specific software, or services activation. Check with your system manufacturer or
retailer.
• No computer system can be absolutely secure. Intel does not assume any liability for lost or stolen data or systems or any damages
resulting from such losses.
• You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel
products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which
includes subject matter disclosed herein.
• No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
• The products described may contain design defects or errors known as errata which may cause the product to deviate from publish.

Apache HDFS Extended Attributes and Transparent Encryption

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Apache HDFS Extended Attributes and Transparent Encryption

Ähnlich wie Apache HDFS Extended Attributes and Transparent Encryption (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache HDFS Extended Attributes and Transparent Encryption