Serverless Big Data Architectures: Serverless Data Analytics

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Radhika Ravirala, Solutions Architect, AWS
August 17, 2017
Serverless Big Data Architectures
Serverless Data Analytics

2. Agenda
Cloud Architecture Evolution – Why Serverless
Data and Analytics Flow
Key Services Overview
Design Patterns
Call to Action

3. Cloud Architecture Evolution
Virtualized Managed Serverless
Virtualized
Servers
Managed
Platforms
Serverless
Analytics

4. No servers to provision
or manage
Scales with usage
Never pay for idle Availability and fault
tolerance built in
Serverless characteristics

5. Data and Analytics Flow
Ingest/
Collect
Store
Analyze/
Process
Visualization/
Consume
Orchestrate/Transform

6. What Is the Temperature of Your Data / Access ?

7. Orchestration/Transform
AWS Big Data Services
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Batch
ETL/ELT
Realtime
ETL/ELT
Transactional
/ CDC
B.I. Tools
Data Science
Notebooks
Bulk Transport
File/Object Upload
Streaming Ingest
Commits Transactional
NoSQL
Data Lake
Streaming Storage
Dashboards
Batch Analytics
Interactive
Querying
Machine Learning/
Deep Learning
Realtime Analytics
…

8. Orchestration/Transform
AWS Big Data Services
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
= Serverless
Serverless
Managed
Virtualized
Batch
ETL/ELT
Realtime
ETL/ELT
Transactional
/ CDC
B.I. Tools
Data Science
Notebooks
Bulk Transport
File/Object Upload
Streaming Ingest
Commits Transactional
NoSQL
Data Lake
Streaming Storage
Dashboards
Batch Analytics
Interactive
Querying
Machine Learning/
Deep Learning
Realtime Analytics

9. Orchestration/Transform
AWS Big Data Services
EMR EC2
S3
RedshiftDynamoDB
AWS DMS (CDC)
AWS Lambda
Kinesis Analytics Amazon Athena
Amazon
QuickSight
Aurora
AWS Glue AWS Step
Functions
Kinesis
Streams
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
AWS
Snowball
ISV
Connectors
Kinesis
Firehose
S3 Transfer
Acceleration
= Serverless
Amazon
ElasticSearc
h

10. Key Services Overview

11. Big Data Storage for Virtually All AWS Services
Amazon S3
• Store anything
• Object storage
• Scalable
• 99.999999999% durability
• Extremely low cost

12. Amazon
DynamoDB
Fast & Flexible NoSQL Database Service
• NoSQL Database
• Seamless scalability
• Zero admin
• Single digit millisecond latency

13. Amazon
Kinesis
Real-time Streaming Platform
• Streams, Firehose, Analytics
• Real-time processing
• High throughput; elastic
• Easy to use
• Integration with S3, EMR,
Redshift, DynamoDB

14. Amazon Kinesis
Streams
• For Technical Developers
• Build your own custom
applications that process
or analyze streaming
data
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into S3, Amazon Redshift
and Amazon Elasticsearch
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver and process streams on AWS

15. AWS Lambda
• Run your code in the cloud - fully
managed and highly-available
• Triggered through API or state
changes in your setup
• Scales automatically to match the
incoming event rate
• Node.js (JavaScript), Python, Java,
and C#
• Charged per 100ms execution time
Serverless Compute

16. Amazon
Athena
Interactive Query Service
• Query directly from
Amazon S3
• Use ANSI SQL
• Serverless
• Multiple Data Formats
• Pay per query

17. AWS Glue
Fully Managed ETL Service
• Catalog data sources
• Identify data formats & data types
• Error Handling
• Manage and scale resources
• Generate ETL code
• Schedules, executes ETL jobs
New !

18. AWS Glue: services
Data Catalog
 Hive metastore compatible metadata repository of data sources.
 Crawls data source to infer table, data type, partition format.
Job Execution
 Runs jobs in Spark containers – automatic scaling based on
SLA.
 Glue is serverless - only pay for the resources you consume.
Job Authoring
 Generates Python code to move data from source to destination.
 Edit with your favorite IDE; share code snippets using Git.

19. • Fast and cloud-powered
• Easy to use, no infrastructure to
manage
• Scales to 100s of thousands of
users
• Quick calculations with SPICE
• 1/10th the cost of legacy BI
software
Business Intelligence
Amazon
QuickSight

20. Serverless Design Patterns

21. Real-time Analytics
Producer
Apache
Kafka
KCL
AWS
Lambda
Spark
Streaming
Apache
Storm
Amazon
SNS
Notifications
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
Alert
Analytics
Output KPI
Serverless
Managed
DynamoDB
Streams
Kinesis
Streams
Virtualized
Kinesis
Analytics
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Apache
FlinkSQS

22. Interactive Queries
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Producer Amazon S3
Amazon
Redshift
Amazon EMR
Presto
Impala
Spark
Interactive
Amazon
Athena
Serverless
Managed
Virtualized
QuickSight

23. Catalog & Search
Access and search metadata
Access & User Interface
Give your users easy and secure access
DynamoDB Elasticsearch API Gateway Identity & Access
Management
Cognito
QuickSight Amazon AI EMR Redshift
Athena Kinesis RDS
Central Storage
Secure, cost-effective
Storage in Amazon S3
S3
Snowball Database Migration
Service
Kinesis Firehose Direct Connect
Data Ingestion
Get your data into S3
Quickly and securely
Protect and Secure
Use entitlements to ensure data is secure and users’ identities are verified
Processing & Analytics
Use of predictive and prescriptive
analytics to gain better understanding
Security Token
Service
CloudWatch CloudTrail Key Management
Service
Data Lake Reference Architecture
= Serverless

24. Amazon S3
Data Lake
Amazon Kinesis
Streams & Firehose
Hadoop / Spark
Streaming Analytics Tools
Amazon Redshift
Data Warehouse
Amazon DynamoDB
NoSQL Database
AWS Lambda
Spark Streaming
on EMR
Amazon
Elasticsearch Service
Relational Database
Amazon EMR
Amazon Aurora
Amazon Machine Learning
Predictive Analytics
Any Open Source Tool
of Choice on EC2
Data Science Sandbox
Visualization /
Reporting
Apache Storm
on EMR
Apache Flink
on EMR
Amazon Kinesis
Analytics
Serving Tier
Clusterless SQL Query
Amazon Athena
DataSourcesTransactionalData
Amazon Glue
Clusterless ETL
Amazon ElastiCache
Redis
Data Lake and
Real-time
Analytics

25. Serverless ETL
Store Transform Store Analyze/ Process
Visualize/
Consume
Amazon S3
Apache
Kafka
Kinesis
Streams Amazon EMR
Spark
Flink
AWS Glue
AWS Lambda
ISV
Amazon S3
Apache
Kafka
Redshift
Kinesis
Streams
Data CatalogAWS Glue
DynamoDB
Streams
DynamoDB Hive M/D

26. Serverless nicely fits into big data platforms
• AWS Serverless Big Data Services
• Complements existing big data flows
• Focus on the analytics and not on infrastructure or servers
• Don’t focus on the scaling, availability, and undifferentiated
heavy lifting
• Pay only for what you use
• Easily try out different tools, analytics, and solutions

27. DEMO