The document discusses building a data lake using Amazon S3 and Amazon Glacier for storage. It covers topics like what is big data, what is a data lake, achievable business outcomes from a data lake, securing the data lake, and examples of what can be done with analytics services on AWS. The presentation provides examples of using services like Amazon Comprehend, Amazon Transcribe, Kinesis, Athena and QuickSight for natural language processing, audio analysis, real-time streaming and visualization.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a Data Lake in Amazon S3 &
Amazon Glacier
Joyjeet Banerjee
Enterprise Solutions Architect
AWS
S T G 4 0 1 - R

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• What is big data?
• What is a data lake?
• Achievable business outcomes
• Securing a data lake
• Art of the possible
• Lab Materials & code: http://bit.ly/2I9qHO7

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is big data?

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is big data?
Volume
Velocity
Variety

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is big data? (cont.)
Volume—unconstrained growth!
GB
TB
PB
ZB
EB
Source: IDC, The Internet of Things: Getting Ready to Embrace Its Impact on the Digital Economy, March 2016.
RDB
DWH
Data
lake
Sources of data:
Relational
NoSQL
Web servers
Mobile
Third-party feeds
IoT
Clickstream

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is big data? (cont.)
Velocity
• Real-time streaming
• Near real-time
• Periodic
• Batch jobs

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is big data? (cont.)
Variety—structured, unstructured, and multi-structured
Transactions ERP Connected devices Social mediaWeb logs/cookies
Records StreamsFiles

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The AWS Cloud was built for big data
Agility: Try more, fail fast, go
big, or start small, and process
data at any scale
Scalability: Run jobs anytime,
without guessing capacity or
limiting functionality
Get to insights faster:
Focus on data science, not the
heavy undifferentiated lift of
managing raw data
Broadest and deepest
capabilities: Access 70+
managed big data services to
address any workload
Low cost: Pay only for the IT
you use, when you use it
Data migrations made easy:
Move exabyte-scale data to the
cloud quickly and cost-
effectively

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is a data lake?

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is a data lake?
A data lake is a new and increasingly
popular architecture to store and
analyze massive volumes of
heterogeneous data in a centralized
repository

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Characteristics of a data lake
Future-proofFlexible
access
Dive in
anywhere
Collect
anything

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of a data lake
Quickly ingest and store
any type of data, at any
scale, and at low cost
Have a single source of
truth and quickly search
and find the relevant data
Easily query the data
through a unified set of
tools

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Defining the AWS data lake
Data lake is an architecture with a virtually limitless
centralized storage platform capable of
categorization, processing, analysis, and
consumption of heterogeneous data sets
Key data lake attributes
• Decoupled storage and compute
• Rapid ingest and transformation
• Secure multi-tenancy
• Query in place
• Schema on read
• Low cost of storage

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
High performance
▪ Multiple upload
▪ Range GET
▪ Amazon S3 Select
Why Amazon Simple Storage Service (Amazon S3) for
the data lake?
Secure
▪ Granular object-level controls
▪ Full compliance, and audit
capability
▪ Encryption @ rest & transit
Durable
Desired for 11 nines
of durability
Available
Designed for 99.99%
availability
Easy to use
▪ Simple REST API
▪ AWS SDKs
▪ Read-after-create consistency
▪ Event notification
▪ Lifecycle policies
Scalable & affordable
▪ Store as much as you need
▪ Scale storage and compute
independently
▪ No minimum usage commitments
▪ Pay for what you use
Integrated
▪ Amazon Redshift/Amazon
Redshift Spectrum
▪ Amazon EMR
▪ Amazon Athena
▪ Amazon DynamoDB

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Central storage
Secure, cost effective
storage in Amazon S3
Amazon S3
Catalog & search
Access & search metadata
DynamoDB Amazon ES
Access & user interface
Give your users easy & secure access
Amazon
API Gateway
AWS
IAM
Amazon
Cognito
Protect & secure
Use entitlements to ensure data is secure and users’ identities are verified
AWS STS Amazon
CloudWatch
AWS
CloudTrail
AWS KMS
Amazon
Athena
Amazon
QuickSight
Amazon
EMR
Amazon
Redshift
Processing & analytics
Use predictive and prescriptive
analytics to gain better understanding
Amazon
Kinesis Data
Firehose
AWS Direct
Connect
AWS
Snowball
AWS DMS
Data ingestion
Get your data into Amazon S3
quickly and securely

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Driving business outcomes
Modern data architectures on AWS

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Outcome 1: modernize and consolidate
• Insights to enhance business applications and create new digital services
Outcome 2: innovate for new revenues
• Personalization, demand forecasting, risk analysis
Outcome 3: real-time engagement
• Interactive customer experience, event-driven automation, fraud detection
Outcome 4: automate for expansive reach
• Automation of business processes and physical infrastructure
Business outcomes on a modern data architecture

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A platform to build business outcomes from data
Purchases
Movement
Influence
Ingest/
Collect
Consume/
Visualize
Store
Process/
Analyze
Revenue lift
Market acquisition
Customer delight
Brand advocacy
Inventory
optimization
Supply chain
efficiency

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (real-time)
Ingest ServingData
sources
Scale (batch)
Modernize and consolidate
Insights to enhance business applications, new digital servicesStart with the business case and the personas
Data analysts
Data scientists
Business users
Engagement platforms
Automation/events
Transactions
Web logs/
cookies
ERP

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (real-time)
Ingest ServingData
sources
Scale (batch)
Modernize and consolidate
Insights to enhance business applications, new digital servicesStart with the business case, and the personas
Data warehouse
Amazon Redshift
Legacy apps
Amazon RDS
Schemaless
Amazon ES
direct query
Athena
Near-zero latency
DynamoDB
Semi/Unstructured
Amazon EMR
Amazon S3
Staged data
(data lake)Transactions
Web logs/
cookies
ERP
Data analysts
Data scientists
Business users
Engagement platforms
Automation/events

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (real-time)
Ingest ServingData
sources
Scale (batch)
Modernize and consolidate
Insights to enhance business applications, new digital servicesProcess data for ETL, cleansing, tagging, and place into staged data (data lake)
Data warehouse
Amazon Redshift
Legacy apps
Amazon RDS
Schemaless
Amazon ES
direct query
Athena
Near-zero latency
DynamoDB
Semi/Unstructured
Amazon EMR
Amazon S3
Staged data
(data lake)
AWS DMS
DX
Internet
interfaces Amazon S3
Raw data
AWS Glue
ETL
Lab 2
Lab 4
Transactions
Web logs/
cookies
ERP
Data analysts
Data scientists
Business users
Engagement platforms
Automation/events
Amazon
QuickSight
Amazon
API Gateway

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (real-time)
Ingest ServingData
sources
Scale (batch)
Innovate for new revenues
Insights to enhance business applications, new digital services
Data warehouse
Amazon Redshift
Legacy apps
Amazon RDS
Schemaless
Amazon ES
direct query
Athena
Near-zero latency
DynamoDB
Semi/Unstructured
Amazon EMR
Amazon S3
Staged data
(data lake)
AWS DMS
DX
Internet
interfaces Amazon S3
Raw data
Advanced
analytics
Lab 3
AWS Glue
ETL
Transactions
Web logs/
cookies
ERP
Data analysts
Data scientists
Business users
Engagement platforms
Automation/events

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (real-time)
Ingest ServingData
sources
Scale (batch)
Real-time engagement
Events are captured in the speed layer
Amazon S3
Staged data
(data lake)
AWS DMS
DX
Internet
interfaces Amazon S3
Raw data
Amazon
Kinesis
Connected
devices
Social media
Event capture
Kinesis
Stream analysis
Kinesis Data Analytics
Lab 1
AWS Glue
ETL
Advanced
analytics
Transactions
Web logs/
cookies
ERP
Data warehouse
Amazon Redshift
Legacy apps
Amazon RDS
Schemaless
Amazon ES
direct query
Athena
Near-zero latency
DynamoDB
Semi/Unstructured
Amazon EMR
Data analysts
Data scientists
Business users
Engagement platforms
Automation/events

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Securing a data lake

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security
▪ AWS Identity and Access
Management (IAM) policies
▪ Bucket policies
▪ Access control lists (ACLs)
▪ Private VPC endpoints to
Amazon S3
▪ Pre-signed Amazon S3 URLs
Encryption
▪ SSL endpoints
▪ Server-side encryption
(SSE-S3)
▪ Amazon S3 server-side
encryption with provided
keys (SSE-C, SSE-KMS)
▪ Client-side encryption
Audit & compliance
▪ Buckets access logs
▪ Lifecycle management
policies
▪ Versioning & MFA deletes
▪ Certifications—HIPAA, PCI,
SOC 1/2/3, more
Implement the right cloud security controls

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Art of the possible

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Broad range of possibilities
Analytics services to
support streaming &
batch datasets
Direct integration with
artificial intelligence
services
Import data into
machine learning/
deep learning
modeling services

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics services on AWS
Broadest and deepest portfolio purpose-built for builders
Amazon EMR Amazon
EC2
Amazon
Glacier
Amazon S3
Kinesis
Amazon
Redshift
DynamoDB
Collect Orchestrate Store Analyze
AWS Lambda
AWS IoT
Core
AWS Data Pipeline
KinesisData
Analytics
Amazon
SNS
AWSSnowball
Amazon
SWF
Athena
AWS Glue
Amazon Aurora
Amazon
QuickSight
Amazon
SageMaker
AWSDirectConnect

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS AI/ML stack
Frameworks &
infrastructure
AWS Deep Learning AMI
GPU
(P3 Instances)
MobileCPU
IoT (AWS
Greengrass)
Vision:
Amazon Rekognition Image
Amazon Rekognition Video
Speech:
Amazon Polly
Amazon Transcribe
Language:
Amazon Lex, Amazon Translate,
Amazon Comprehend
Apache
MXNet
PyTorch
Cognitive
Toolkit
Keras
Caffe2
& Caffe
TensorFlow Gluon
Application
services
Platform
Services
Amazon Machine
Learning
Mechanical
Turk
Spark &
Amazon EMR
Amazon
SageMaker
AWS
DeepLens

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis
Data Firehose
S3
bucket
AthenaAmazon ML
Twitter stream Amazon QuickSight
Example use case:
Social media analysis & visualization

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap—What we covered
• Big data
• Data lake
• Achievable business outcomes
• Securing a data lake
• Art of the possible

33. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Joyjeet Banerjee
Enterprise Solutions Architect
AWS

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Comprehend
Natural language processing
Data lake (Amazon
S3)
Amazon
Comprehend
Data lake (Amazon
S3)
Athena
Natural language processing
Amazon EMR
Amazon
QuickSight
Unstructured
text

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Transcribe
Audio to text analysis
Lambda
Data lake
Athena
Audio Input
Amazon
QuickSight
Amazon
Comprehend
Amazon
Transcribe

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine
translation
NLP
Data lake
Amazon
QuickSight
Lambda
Athena
Social Media Analysis & Visualization
Streams
AWS
Glue
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Kinesis
Data Firehose
Twitter

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Outcome 1: Modernize and consolidate
Common initiatives
• Insights: 360° view of the business
• Digitization: web service that gives on-demand insights
• Data monetization: enrich, aggregate, and sell business data

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Outcome 2: Innovate for new revenues
Common initiatives
• Personalization: refine market approaches on optimal segments
• Predict demand: guide business owners to select best scenarios
• Risk measurement: create freedom to act by quantifying exposures

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Outcome 3: Real-time engagement
Common initiatives
• Interactive CX: natural customer journeys with adaptive interfaces
• Event-driven automation: triggered execution of business process
• Fraud detection: protect customer and business interests