This document outlines a presentation about building flexible data lake architectures on AWS. It discusses challenges with traditional data warehousing and siloed data sources. A data lake architecture using AWS services like S3, Athena, and Glue is presented as a way to ingest and analyze all data in one centralized location. The presentation also covers real-time analytics platforms using Kinesis and batch vs streaming data processing. It concludes with a demo of visualizing data in S3 using Athena and Amazon QuickSight.

1. Cloudwick© 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Data Ingestion, Transformation, and Visualization
with Amazon Web Services
Sponsored by Amazon Web Services and Intel (Venue )
Presented by Arun Kumar Palathumpattu (Cloudwick)

2. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Cloudwick

3. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Agenda Cloudwick
• Introduction
• Common Challenges in Data Analytics Environments
• Creating Data Lake – Single source of truth
• Build Effective Data work flow
• Demo
• Q&A

4. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Top 5 Challenges in Building Data Analysis
Environments
Cloudwick

5. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
# 1 Market Landscape
Cloudwick

6. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Cloudwick
Image : http://mwicorp.com/lake-okeechobee-water-transfer/
Structured
# 2 Issues of Storing All the Data

7. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
# 3 Time it Takes to Find the Useful Information
Cloudwick
Image : https://img.clipartfest.com/0ffb2c38607970437e20a6fdd2872eb9_4-benefits-from-taking-guitar-time-value-of-money-clipart_500-500.png

8. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
# 4 Security
Cloudwick
Image : https://insights.ubuntu.com/2017/03/20/three-flaws-at-the-heart-of-iot-security/

9. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
# 5 Skill Gap
Cloudwick
https://www.linkedin.com/pulse/wonder-three-years-analytics-big-data-skills-most-demand-hosseini

10. Evolution of Data Architectures
1985: Data Warehouse Appliances Benefits
• Consolidated multiple decision support
environments (i.e. databases) into a single
architecture
• Best performance available at time of conception,
hence the expensive licenses
• Worked well with structured, columnar data
• Could build customized data marts on top
Shared Storage Tier
(NAS Appliance)
Compute
Node
Compute
Node
Compute
Node
Compute
Node
• Proprietary software license paid per node per
year
• Gold-plated hardware available only from the
vendor with per node per year cost
Constraints
• Proprietary software license paid per node per year
• Gold-plated hardware available only from the
vendor with per node per year cost
• Could not handle unstructured data sets
• Heavy ETL & data cleansing

11. Evolution of Data Architectures
2006: Hadoop Clusters
CPU
Memory
HDFS Storage
Hadoop Master Node
CPU
Memory
HDFS Storage
CPU
Memory
HDFS Storage
Improvements
• Open source based software license!!!
• Commodity white box servers!!!!
• Could handle structured & unstructured data sets
• Many different applications within the framework
(MapReduce, Spark, Hive, Pig, HBase, Presto, etc.)
Constraints
• HDFS 3X replication to protect against node failure
gets expensive at scale
• 500 TB data set = 1.5 PB cluster
• Local storage means you must scale and pay for
CPU & memory resources when adding data
capacity
• General purpose, monolithic cluster with many
different apps on same hardware
• Still a data silo

12. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Legacy Data Architectures Exist as Isolated Data Silos
Hadoop
Cluster
SQL
Database
Data
Warehouse
Appliance

13. Evolution of Data Architectures
2009: Decoupled EMR Architecture
CPU
Memory
Hadoop Master Node
CPU
Memory
CPU
Memory
Improvements
• Decoupled storage & compute
• Scale CPU and memory resources independently
and up & down
• Only pay for the 500 TB data set (not 3X)
• Multi-physical facility replication via S3
• Multiple clusters can run in parallel against shared
data in S3
• Each job gets its own optimized cluster. i.e. Spark
on memory intensive, Hive on CPU intensive, HBase
on I/O intensive, etc.
Constraints
• Still have a cluster to provision and manage
• Must expose EMR cluster to SQL users via Hive,
Presto, etc.
S3 as HDFS

14. Evolution of Data Architectures
2012: Amazon Redshift – Cloud DW
Improvements
Constraints
• Still have to load data into a schema
Leader node
Compute node
10 GigE
(HPC)
Ingestion
Backup
Restore
Customer VPC
Internal VPC
BI tools SQL clientsAnalytics tools
Compute node Compute node
JDBC/ODBC
• Automated installation, patching, backups
• No servers to manage and maintain
• MPP Columnar relational database
• $1,000 / TB / Year
• Accessible to any ODBC or JDBC BI Tool

15. Evolution of Data Architectures
Today: Clusterless Improvements
• No cluster/infrastructure to manage
• Business users and analysts can write SQL without
having to provision a cluster or touch infrastructure
• Pay by the query
• Zero Administration
• Process data where it lives
Constraints
• Limited to SQL, Hive and Spark jobs today. More
frameworks to come!
SQL Interface in web
browser
Athena for SQL
S3 Data Lake
Glue for ETL
S3 Data Lake
Spark & Hive Interface in
web browser

16. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a Flexible Data Lake
Architecture on AWS

17. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Enter Data Lake Architectures
Data Lake is a new and increasingly
popular architecture to store and analyze
massive volumes and heterogeneous
types of data.
Separating your storage and compute
allows you to scale each component as
required
“How can I scale up with the
volume of data being generated?”

18. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Benefits of a Data Lake – All Data in One Place
Store and analyze all of your data,
from all of your sources, in one
centralized location.
“Why is the data distributed in
many locations? Where is the
single source of truth ?”
Quickly ingest data
without needing to force it into a
pre-defined schema.
“How can I collect data quickly
from various sources and store it
efficiently?”

19. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Benefits of a Data Lake – Schema on Read
“Is there a way I can apply multiple
analytics and processing frameworks
to the same data?”
A Data Lake enables ad-hoc
analysis by applying schemas
on read, not write.

20. Benefits of an AWS S3 Data Lake
Fixed Cluster Data Lake AWS S3 Data Lake
• Limited to only the single tool contained on
the cluster (i.e. Hadoop or data warehouse
or Cassandra, etc.). Use cases & ecosystem
tools change rapidly
• Expensive to add nodes to add storage
capacity
• Expensive to replicate data against node
loss
• Complexity in scaling local storage capacity
• Long refresh cycles to add additional
storage equipment
• Decouple storage and compute by making
S3 object based storage, not a fixed tool
cluster the data lake
• Flexibility to use any and all tools in the
ecosystem. The right tool for the job
• Future proof your architecture. As new use
cases and new tools emerge you can plug
and play current best of breed.

21. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Designed for 11 9s
of durability
Designed for
99.99% availability
Durable Available High performance
Multiple upload
Range GET
Store as much as you need
Scale storage and compute
independently
No minimum usage commitments
Scalable
Amazon EMR
Amazon Redshift
Amazon DynamoDB
Integrated
Simple REST API
AWS SDKs
Read-after-create consistency
Event notification
Lifecycle policies
Easy to use
S3 for data lake

22. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Building a Data Lake on AWS
Kinesis Firehose
Athena
Query Service

23. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Processing & Analytics
Real-time Batch
AI & Predictive
BI & Data Visualization
Transactional &
RDBMS
AWS Lambda
Apache Storm
on EMR
Apache Flink
on EMR
Spark Streaming
on EMR
Elasticsearch
Service
Kinesis Analytics,
Kinesis Streams
DynamoDB
NoSQL DB Relational Database
Aurora
EMR
Hadoop, Spark,
Presto
Redshift
Data Warehouse
Athena
Query Service
Amazon Lex
Speech
recognition
Amazon
Rekognition
Amazon Polly
Text to speech
Machine Learning
Predictive analytics
Kinesis Streams
& Firehose

24. Summary of AWS Analytics, Database & AI Tools
Amazon Redshift
Enterprise Data Warehouse
Amazon EMR
Hadoop/Spark
Amazon Athena
Clusterless SQL
Amazon Glue
Clusterless ETL
Amazon Aurora
Managed Relational Database
Amazon Machine Learning
Predictive Analytics
Amazon Quicksight
Business Intelligence/Visualization
Amazon ElasticSearch Service
ElasticSearch
Amazon ElastiCache
Redis In-memory Datastore
Amazon DynamoDB
Managed NoSQL Database
Amazon Rekognition
Deep Learning-based Image Recognition
Amazon Lex
Voice or Text Chatbots

25. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Encryption ComplianceSecurity
Identity and Access
Management (IAM) policies
Bucket policies
Access Control Lists (ACLs)
Private VPC endpoints to
Amazon S3
SSL endpoints
Server Side Encryption
(SSE-S3)
S3 Server Side
Encryption with
provided keys (SSE-C,
SSE-KMS)
Client-side Encryption
Buckets access logs
Lifecycle Management
Policies
Access Control Lists
(ACLs)
Versioning & MFA
deletes
Certifications – HIPAA,
PCI, SOC 1/2/3 etc.
Implement the right cloud security controls

26. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
More Efficient Data Lake Architectures
Cloudwick

27. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
What is a Modern Enterprise Data
Warehouse?
A Modern Enterprise Data warehouse
(EDW), is designed to support rapid data
growth, quick analytics over relational,
non-relational as well as streaming data,
with an easy and single interface to
consume all these types of data.

28. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Data
Consumption
Building a Modern EDW on AWS
Amazon
Kinesis
Firehose
Amazon S3
Amazon S3
Amazon
EMR
Amaz
on
Redsh
ift
Streamin
g
Un-
structured
Relational
Amaz
on
RDS
Amaz
on
Athen
a
Amazon
Quicksig
ht
Amazo
n
Machin
e
Learnin
g
Analyze
Predict
Data
Mart
EDW
Ad-hoc
query
Data
Ingestion

29. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
What is Real-time analytics platform?
Real-time Analytics platform provides an
ability to load and analyze streaming
data, and build custom streaming data
applications for specialized needs. This
platform can handle staggering amounts
of streaming data – sometimes TBs per
hour – that need to be collected, stored,
and processed continuously.

30. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Building a Real-time analytics platform (Streaming
vs batching)
Stream Delivery
Stream Analytics
Stream processing
Event Driven
Kinesis
Streami
ng

31. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Simplify Big Data Processing
ingest /
collect
store process /
analyze
consume /
visualize
data answers
Time to Answer (Latency)
Throughput
Cost

32. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Amazon QuickSight is a Business Analytics Service that lets business users
quickly and easily visualize, explore, and share insights from their data.

33. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Amazon S3 Data Lake
Visualizing the Data Lake
Amazon Athena

34. Cloudwick © 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Cloudwick
DEMO

35. Cloudwick© 2015 Cloudwick. All rights reserved. Confidential and Proprietary.
Credits
Event Sponsored by Amazon Web Services
Venue by Intel
Presented by Arun Kumar Palathumpattu (Cloudwick)
Slides credit : AWS + Cloudwick