AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SSID: Guest
Password: Unfabric@@2020
WildRydes: Serverless Data Processing Workshop
Adir Sharabi, AWS Solutions Architect
David Feldstein , AWS Solutions Architect
Alon Gendler, AWS Solutions Architect

Scenario: Wild Rydes (www.wildrydes.com)

Build processing infrastructure to enable operations personnel
in WildRydes headquarters to monitor the health of the unicorn fleet
Your task
Welcome to Wild Rydes
Inc., Employee #4!

AWS Services You Will Use Today

What is serverless?
No infrastructure provisioning,
no management
Automatic scaling
Pay for value Highly available and secure

AWS Lambda
Services (anything)
Changes in
data state
Requests to
endpoints
Changes in
resource state
Event source Function
Node.js
Python
Java
C#
Go
Ruby
Runtime API

Anatomy of a Lambda function
Handler() function
Function to be executed
upon invocation
Event object
Data sent during
Lambda function
Invocation
Context object
Methods available to
interact with runtime
information (request ID,
log group, more)

Serverless architectures with AWS Lambda
Lambda invoked
File put
into bucket
Amazon S3
Object
AWS Lambda
Message inserted
into to a queue
2. Function
removes
message from
queue
1. Lambda polls
queue and
invokes function
Amazon Simple Queue
Service
Message
Amazon
SQS
AWS Lambda
Lambda invoked
Request sent to
API Gateway
API Call
AWS Lambda
Amazon API Gateway

Serverless architectures
1. Lambda polls
stream
Data published to a
stream
2. Amazon
Kinesis returns
stream data
Data
Amazon Kinesis
Data Streams
AWS Lambda
Request sent to ALB
Application Load
Balancer
Request
AWS Lambda
Scheduled time
occurs
Event (time-base)
AWS Lambda

Add user sign-up, sign-in, and data synchronization to your apps
Amazon Cognito
Add user sign-up and
sign-in to your
mobile and web apps
Federate identities
and provide secure
access to AWS
resources

Fast and flexible NoSQL database service for any scale
Amazon DynamoDB
Dead simple
• GetItem(primaryKey)
• PutItem(item)
Robust depth
• Fine-grained access control
• Streams/Triggers
• Cross-region replication
• Global Tables
• TTL
• Auto Scaling
• Atomic counters
• Backup and restore
• On Demand (NEW)
• Transaction (NEW)

• Durable, massively scalable object storage
• Designed for 99.999999999% durability and
99.99% availability
• Stores trillions of objects and regularly handles
millions of requests per second
• Effectively infinite storage without provisioning
capacity
Amazon Simple Storage Service (Amazon S3)

Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process, and
store video streams
Load data streams
into data stores
Analyze data streams with
SQL or Java Flink (New)
Capture, process, and
store data streams
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Analytics
Amazon Kinesis
Video Streams

Kinesis Data Streams
• Serverless
• Elastically scalable
• Low cost, pay for what you use
• Managed encryption
• AWS and open source integrations
• Near real-time applications
• Multiple consumers

Streams: Processing a data stream with Lambda
Data
producer
Kinesis Data
Streams
Amazon
SNS
Continuously
stream data
Lambda
service
Lambda
function A
Lambda
function B
Continuously polls for
new data,
1 poll per second
Automatically invokes
your function(s) when
data found

Streams: How it works
Data
producer
Kinesis Data
Streams
AWSLambda
service
function A
(instance 1)
shard1
shard2
shard3
shard4
function A
(instance 2)
function A
(instance 3)
function A
(instance 4)
function B
(instance 1)
function B
(instance 2)
function B
(instance 3)
function B
(instance 4)
function C
(instance 1)
function C
(instance 2)
function C
(instance 3)
function C
(instance 4)

Kinesis Data Firehose
• Serverless
• Zero administration
• Seamless elasticity
• Pay for usage
• Direct-to-data store integration
• Serverless ETL using AWS Lambda

Kinesis Data Firehose: Filter, enrich, and convert
data while it is streaming
Data
producer Kinesis Data
Firehose
Amazon
Elasticsearch
Service
[Wed Oct 11 14:32:52 2017] [error] [client 127.0.0.1]
[Wed Oct 11 14:32:53 2017] [info] [client 127.0.0.1]
geo-IPservice
{
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
{
"recordId": "1",
"result": "Ok",
"data": {
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
},
{
"recordId": "2",
"result": "Dropped"
}

Kinesis Data Analytics
• Serverless
• Zero administration
• Pay for usage
• Sub 1-second processing latencies
• Filter, aggregate, and enrich data using SQL
• Programming skills not required

Kinesis Data Analytics: Pre-process data before
analysis
Data
producer
Kinesis Data
Analytics
Kinesis Data
Streams |Firehose
Pre-
processor
SQL
engine
…
• Asynchronously
invokes pre-
processing
Lambda function
Kinesis Data Streams
AWS Lambda
Kinesis Firehose
• Tumbling windows:
Lambda invoked at the
end of the window
• Sliding windows or
continuous queries:
invoked ~1 per second

• Serverless
• Query service for querying data directly from S3
• Use standard ANSI SQL queries
• Support for multiple data formats include text, CSV,
TSV, JSON, Avro, ORC, Parquet
• Pay per query - on data scanned; $5 per TB
• Query using AWS console, ODBC, JDBC or QuickSight
• Improve performance and reduces cost:
• Compression, Partitioning & Columnar formats
Amazon Athena

WildRydes Workshop
Overview

• OBJECTIVE: Collect real-time data coming from the sensors attached to your
unicorn fleet. This sensors emits data every second including the unicorn's current
location, distance traveled in the previous second, and magic points and hit points
so that our operations team can monitor the health of our unicorns from Wild
Rydes headquarters
Module 1: Real-time Data Streaming

• OBJECTIVE: Each minute, the application will emit data including the total distance
traveled in the last minute as well as the minimum and maximum readings from
health and magic points for each unicorn in our fleet
Module 2: Streaming Aggregation

• OBJECTIVE: Process the aggregated stream and write those records to an Amazon
DynamoDB table.
Module 3: Stream Processing

• OBJECTIVE: Every few minutes, Firehose will deliver the batched data to an
Amazon S3 bucket where it can be stored indefinitely for future reprocessing.
Using Amazon Athena, we'll run ad-hoc queries against our raw data.
Module 4: Data Lake

Final Architecture

• Go to: https://dataprocessing.wildrydes.com/
• User accounts
• Your own accounts
• Be careful with production/your company/shared/etc. accounts
• Make sure all services are available in the region you are running in
• You can use N. Virginia, Oregon or Ireland
• We are here to help!
• Make sure to Cleanup your resources at the end of the Workshop.
Let the Fun Begin!
SSID: Guest
Password: Unfabric@@2020

AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)

Ähnlich wie AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2) (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)