SlideShare ist ein Scribd-Unternehmen logo
1 von 33
Downloaden Sie, um offline zu lesen
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Building	a	serverless	data	pipeline
Julien	Simon,	Principal	Technical	Evangelist,	AWS	
julsimon@amazon.fr		
@julsimon
Werner	Vogels,	CTO,	Amazon.com	
AWS	re:Invent	2015
AWS	Lambda	
•  Deploy	pure	funcJons	in	Java,	Python	and	Node.js	
•  Works	nicely	with	AWS	managed	services:		
Amazon	S3,	Amazon	DynamoDB,	etc.	
•  Build	event-driven	applicaJons	
•  Build	RESTful	APIs	in	conjuncJon	with	Amazon	API	Gateway	
	
•  Pay	as	you	go:	number	of	requests	+	execuJon	Jme	(100ms	slots)
Managed	services	
+		
AWS	Lambda	
=		
Serverless	architecture
Another	way	to	put	it…	
	
Tim	Wagner,		
General	Manager,		
AWS	Lambda	
	
	
Serverless	conference,	NYC,	May	2016
MOBILE
CHAT APP
AD DATA ANALYTICS
AND ROUTING
MOBILE APP
ANALYTICS
IMAGE CONTENT
FILTERING
REAL-TIME VIDEO
AD BIDDING
NEWS CONTENT
PROCESSING
GENE SEQUENCE
SEARCH
CLOUD
TELEPHONY
DATA
PROCESSING
WEB
APPLICATIONS WEB APPLICATIONS
THREAT INTELLIGENCE
AND ANALYTICS
NEWS CONTENT
PROCESSING
GAME METRICS ANALYTICS
Selected	serverless	customers	
PRODUCT
RECOMMANDATION
https://blog.instant.cm/a-serverless-architecture-with-zero-maintenance-and-infinite-scalability-b00c2ceb4c2b
http://highscalability.com/blog/2015/12/7/the-serverless-start-up-down-with-servers.html
Instant.cm:	100%	Serverless
https://read.acloud.guru/serverless-the-future-of-software-architecture-d4473ffed864
A	Cloud	Guru:	100%	Serverless
AWS Lambda ‘Hello World’ (Python)
1.  Write a simple Lambda function in Python
2.  Create a REST API with API Gateway (resource + POST method)
3.  Deploy the API
4.  Invoke the API with ‘curl’
A simple Lambda function in Python
def lambda_handler(event,context):
   result = event['value1'] + event['value2']
   return result
aws lambda create-function --function-name myFunc 
--handler myFunc.lambda_handler --runtime python2.7 
--zip-file fileb://myFunc.zip --memory-size 128 
--role arn:aws:iam::ACCOUNT_NUMBER:role/lambda_basic_execution
curl -H "Content-Type: application/json" 
-X POST -d "{"value1":5, "value2":7}" 
https://API_ENDPOINT/STAGE/RESOURCE
12
AWS Lambda in Java with Eclipse
https://java.awsblog.com/post/TxWZES6J1RSQ2Z/Testing-Lambda-functions-using-the-AWS-Toolkit-for-Eclipse
AWS Lambda ‘Hello World’ (Java)
1.  In Eclipse, write a simple Lambda function triggered by an S3 event
2.  Unit-test the function with Junit
3.  Using the AWS Eclipse plug-in, upload and run the function in AWS
4.  Run the function again in the AWS Console
AWS Lambda with the Serverless framework
http://github.com/serverless/serverless
•  Run/test AWS Lambda functions locally, or remotely
•  Auto-deploys & versions your Lambda functions
•  Auto-deploys your REST API to AWS API Gateway
•  Auto-deploys your Lambda events
•  Support for multiple stages
•  Support for multiple regions within stages
•  Manage & deploy AWS CloudFormation resources
Building	a	serverless	data	pipeline	
Lambda
DynamoDB
Kinesis
Firehose
API Gateway
HTTP POST 

/prod/logger writeTo
Kinesis
DynamoDB
ToFirehose
S3
eventTable
DynamoDB
streams
bucket
EMR,
Redshift,
…
firehoseToS3
Kinesis StreamsLambda Lambda
KinesisTo
DynamoDB
Web apps
Step 1: create DynamoDB table
aws dynamodb create-table 
--table-name eventTable 
--attribute-definitions 
AttributeName=userId,AttributeType=N 
AttributeName=timestamp,AttributeType=N 
--key-schema 
AttributeName=userId,KeyType=HASH 
AttributeName=timestamp,KeyType=RANGE 
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 
--stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE
eventTable
DynamoDB
Step 2: IAM role for Lambda function
aws iam create-role 
--role-name writeToDynamoDB_role 
--assume-role-policy-document file://lambda_trust_policy.json
aws iam create-policy 
--policy-name writeToDynamoDB_policy 
--policy-document file://writeToDynamoDB_policy.json
aws iam attach-role-policy 
--role-name writeToDynamoDB_role 
--policy-arn WRITETODYNAMODB_POLICY_ARN
Step 3: create Lambda function
S3
DynamoDB
eventTable
Web apps
dynamodb.
put_item()
Lambda
aws lambda create-function 
--function-name writeToDynamoDB 
--role WRITETODYNAMO_DB_ROLE 
--zip-file fileb://writeToDynamoDB.zip 
--handler writeToDynamoDB.lambda_handler 
--runtime python2.7 
--memory-size 128 
--description "Write events to DynamoDB”
Step 4: create Kinesis Stream
DynamoDB
S3
eventTable
Web apps
dynamodb.
put_item()
Lambda
aws kinesis create-stream --stream-name APItoDynamoDB --shard-count 1
Kinesis Streams
Step 5: IAM role for Lambda function
aws iam create-role 
--role-name writeToKinesis_role 
--assume-role-policy-document file://lambda_trust_policy.json
aws iam create-policy 
--policy-name writeToKinesis_policy 
--policy-document file://writeToKinesis_policy.json
aws iam attach-role-policy 
--role-name writeToKinesis_role 
--policy-arn WRITETOKINESIS_POLICY_ARN
Step 6: create Lambda function
DynamoDB
S3
eventTable
Web apps
DynamoDB
streams
dynamodb.
put_item()
Lambda
aws lambda create-function 
--function-name writeToKinesis
--role WRITETOKINESIS_ROLE 
--zip-file fileb://writeToKinesis.zip 
--handler writeToKinesis.lambda_handler 
--runtime python2.7 
--memory-size 128 
--description "Write events to Kinesis”
Kinesis StreamsLambda
KinesisTo
DynamoDB
writeTo
Kinesis
Step 7: create API
DynamoDB
S3
eventTable
Web apps
DynamoDB
streams
dynamodb.
put_item()
Lambda
Painful to do with the CLI: 9 aws apigateway calls :-/
à  Use the console
à  Use a Swagger File
http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html
à  Use the Serverless framework
Kinesis StreamsLambda
KinesisTo
DynamoDB
writeTo
Kinesis
API Gateway
Step 8: create IAM role
aws iam create-role 
--role-name DynamoDBToFirehose_role 
--assume-role-policy-document file://lambda_trust_policy.json
aws iam create-policy 
--policy-name DynamoDBToFirehose_policy 
--policy-document file://DynamoDBToFirehose_policy.json
aws iam attach-role-policy 
--role-name DynamoDBToFirehose_role 
--policy-arn DYNAMODBTOFIREHOSE_POLICY_ARN
Step	9:	create	Lambda	funcIon	and	DynamoDB	trigger	
aws lambda create-function 
--function-name DynamoDBToFirehose 
--role DYNAMODBTOFIREHOSE_ROLE_ARN 
--zip-file fileb://DynamoDBToFirehose.zip 
--handler DynamoDBToFirehose.lambda_handler 
--runtime python2.7 
--memory-size 128 
--description "Write DynamoDB stream to Kinesis Firehose"
aws lambda create-event-source-mapping 
--function-name DynamoDBToFirehose 
--event-source DYNAMODB_STREAM_ARN 
--batch-size 10 
--starting-position TRIM_HORIZON
Lambda
DynamoDB
ToFirehose
eventTable
DynamoDB
streams
DynamoDB
Step	10:	create	IAM	role	
aws iam create-role 
--role-name firehoseToS3_role 
--assume-role-policy-document file://firehose_trust_policy.json
aws iam create-policy 
--policy-name firehoseToS3_policy 
--policy-document file://firehoseToS3_policy.json
aws iam attach-role-policy 
--role-name firehoseToS3_role 
--policy-arn FIREHOSETOS3_POLICY_ARN
Step	11:	create	S3	bucket	
aws s3 mb s3://jsimon-public
Lambda
DynamoDB
ToFirehose
eventTable
DynamoDB
streams
DynamoDB
Step	12:	create	Kinesis	Firehose	stream	
aws firehose create-delivery-stream 
--delivery-stream-name firehoseToS3 
--s3-destination-configuration 
RoleARN=FIREHOSETOS3_ROLE_ARN, 
BucketARN="arn:aws:s3:::jsimon-public", 
Prefix="firehose", 
BufferingHints={SizeInMBs=1,IntervalInSeconds=60}, 
CompressionFormat="GZIP", 
EncryptionConfiguration={NoEncryptionConfig="NoEncryption"}
Kinesis
Firehose
firehoseToS3
Lambda
DynamoDB
ToFirehose
eventTable
DynamoDB
streams
DynamoDB
Building	a	serverless	data	pipeline	
Lambda
DynamoDB
Kinesis
Firehose
API Gateway
HTTP POST 

/prod/logger writeTo
Kinesis
DynamoDB
ToFirehose
S3
eventTable
DynamoDB
streams
bucket
EMR,
Redshift,
…
firehoseToS3
Kinesis StreamsLambda Lambda
KinesisTo
DynamoDB
Web apps
Lines of code: 16
Number of servers: zero
Performance & scalability: maximum
https://github.com/juliensimon/aws/tree/master/serverlessPipeline
Ready	for	some	tesIng?	
hRp://api.julien.org
Upcoming	book	on	AWS	Lambda	
Wri^en	by	AWS	Technical	
Evangelist	Danilo	Poccia	
	
Early	release	available	at:	
	
h^ps://www.manning.com/
books/aws-lambda-in-acJon
Going	further	
AWS re:Invent 2014 | (MBL202) NEW LAUNCH: Getting Started with AWS Lambda
https://www.youtube.com/watch?v=UFj27laTWQA
AWS re:Invent 2015 | (DEV203) Amazon API Gateway & AWS Lambda to Build Secure and Scalable APIs
https://www.youtube.com/watch?v=ZBxWZ9bgd44
AWS re:Invent 2015 | (DVO209) JAWS: The Monstrously Scalable Serverless Framework
https://www.youtube.com/watch?v=D_U6luQ6I90
https://github.com/serverless/serverless
AWS re:Invent 2015 | (ARC308) The Serverless Company Using AWS Lambda
https://www.youtube.com/watch?v=U8ODkSCJpJU
AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda
https://www.youtube.com/watch?v=FhJxTIq81AU
Reference architectures
http://www.allthingsdistributed.com/2016/06/aws-lambda-serverless-reference-architectures.html
AWS User Groups
Lille
Paris
Rennes
Nantes
Bordeaux
Lyon
Montpellier
Toulouse
facebook.com/groups/AWSFrance/
@aws_actus
AWS Enterprise Summit – 27/10/2016, Paris
http://amzn.to/1X2yp0i
Merci !
	
Julien	Simon,	Principal	Technical	Evangelist,	AWS	
julsimon@amazon.fr	
@julsimon

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Amazon ECS (December 2015)
Amazon ECS (December 2015)Amazon ECS (December 2015)
Amazon ECS (December 2015)
 
A real-life account of moving 100% to a public cloud
A real-life account of moving 100% to a public cloudA real-life account of moving 100% to a public cloud
A real-life account of moving 100% to a public cloud
 
AWS July Webinar Series: Introducing AWS OpsWorks for Windows Server
AWS July Webinar Series: Introducing AWS OpsWorks for Windows ServerAWS July Webinar Series: Introducing AWS OpsWorks for Windows Server
AWS July Webinar Series: Introducing AWS OpsWorks for Windows Server
 
Running Docker clusters on AWS (June 2016)
Running Docker clusters on AWS (June 2016)Running Docker clusters on AWS (June 2016)
Running Docker clusters on AWS (June 2016)
 
Write less (code) and build more with serverless
Write less (code) and build more with serverlessWrite less (code) and build more with serverless
Write less (code) and build more with serverless
 
Using Amazon CloudWatch Events, AWS Lambda and Spark Streaming to Process EC...
 Using Amazon CloudWatch Events, AWS Lambda and Spark Streaming to Process EC... Using Amazon CloudWatch Events, AWS Lambda and Spark Streaming to Process EC...
Using Amazon CloudWatch Events, AWS Lambda and Spark Streaming to Process EC...
 
Continuous Delivery with AWS Lambda - AWS April 2016 Webinar Series
Continuous Delivery with AWS Lambda - AWS April 2016 Webinar SeriesContinuous Delivery with AWS Lambda - AWS April 2016 Webinar Series
Continuous Delivery with AWS Lambda - AWS April 2016 Webinar Series
 
Continuous Deployment with Amazon Web Services
Continuous Deployment with Amazon Web ServicesContinuous Deployment with Amazon Web Services
Continuous Deployment with Amazon Web Services
 
Amazon ECS (March 2016)
Amazon ECS (March 2016)Amazon ECS (March 2016)
Amazon ECS (March 2016)
 
Building a data warehouse with Amazon Redshift … and a quick look at Amazon ...
Building a data warehouse  with Amazon Redshift … and a quick look at Amazon ...Building a data warehouse  with Amazon Redshift … and a quick look at Amazon ...
Building a data warehouse with Amazon Redshift … and a quick look at Amazon ...
 
Docker Paris #28
Docker Paris #28Docker Paris #28
Docker Paris #28
 
What's New with AWS Lambda
What's New with AWS LambdaWhat's New with AWS Lambda
What's New with AWS Lambda
 
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
 
AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo  AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo
 
Meeyup aws-loadbalancing-28032015
Meeyup aws-loadbalancing-28032015Meeyup aws-loadbalancing-28032015
Meeyup aws-loadbalancing-28032015
 
Workshop AWS IoT @ SIDO
Workshop AWS IoT @ SIDOWorkshop AWS IoT @ SIDO
Workshop AWS IoT @ SIDO
 
Infrastructure as code with Amazon Web Services
Infrastructure as code with Amazon Web ServicesInfrastructure as code with Amazon Web Services
Infrastructure as code with Amazon Web Services
 
Building A Dynamic Website - 31st Jan 2015
Building A Dynamic Website - 31st Jan 2015Building A Dynamic Website - 31st Jan 2015
Building A Dynamic Website - 31st Jan 2015
 
Docker Paris #29
Docker Paris #29Docker Paris #29
Docker Paris #29
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
 

Andere mochten auch

Andere mochten auch (16)

TIAD - Is Automation Worth My Time?
TIAD - Is Automation Worth My Time?TIAD - Is Automation Worth My Time?
TIAD - Is Automation Worth My Time?
 
TIAD 2016 : Kaizen Ops by Jessica DeVita
TIAD 2016 : Kaizen Ops by Jessica DeVitaTIAD 2016 : Kaizen Ops by Jessica DeVita
TIAD 2016 : Kaizen Ops by Jessica DeVita
 
TIAD 2016 : Ethics in software development
TIAD 2016 : Ethics in software developmentTIAD 2016 : Ethics in software development
TIAD 2016 : Ethics in software development
 
TIAD 2016 : Building a Serverless Pipeline
TIAD 2016 : Building a Serverless PipelineTIAD 2016 : Building a Serverless Pipeline
TIAD 2016 : Building a Serverless Pipeline
 
A Serverless Data Pipeline
A Serverless Data PipelineA Serverless Data Pipeline
A Serverless Data Pipeline
 
(STG312) Amazon Glacier Deep Dive: Cold Data Storage in AWS
(STG312) Amazon Glacier Deep Dive: Cold Data Storage in AWS(STG312) Amazon Glacier Deep Dive: Cold Data Storage in AWS
(STG312) Amazon Glacier Deep Dive: Cold Data Storage in AWS
 
AWS re:Invent 2016: Strategic Planning for Long-Term Data Archiving with Amaz...
AWS re:Invent 2016: Strategic Planning for Long-Term Data Archiving with Amaz...AWS re:Invent 2016: Strategic Planning for Long-Term Data Archiving with Amaz...
AWS re:Invent 2016: Strategic Planning for Long-Term Data Archiving with Amaz...
 
AWS re:Invent 2016: Deep Dive on Amazon Glacier (STG302)
AWS re:Invent 2016: Deep Dive on Amazon Glacier (STG302)AWS re:Invent 2016: Deep Dive on Amazon Glacier (STG302)
AWS re:Invent 2016: Deep Dive on Amazon Glacier (STG302)
 
(CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda
(CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda(CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda
(CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda
 
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS LambdaReal-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
 
AWS re:Invent 2016: What’s New with AWS Lambda (SVR202)
AWS re:Invent 2016: What’s New with AWS Lambda (SVR202)AWS re:Invent 2016: What’s New with AWS Lambda (SVR202)
AWS re:Invent 2016: What’s New with AWS Lambda (SVR202)
 
Amazon CloudWatch Logs and AWS Lambda
Amazon CloudWatch Logs and AWS LambdaAmazon CloudWatch Logs and AWS Lambda
Amazon CloudWatch Logs and AWS Lambda
 
AWS for IoT
AWS for IoTAWS for IoT
AWS for IoT
 
AWS re:Invent 2016: Application Lifecycle Management in a Serverless World (S...
AWS re:Invent 2016: Application Lifecycle Management in a Serverless World (S...AWS re:Invent 2016: Application Lifecycle Management in a Serverless World (S...
AWS re:Invent 2016: Application Lifecycle Management in a Serverless World (S...
 
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 

Ähnlich wie Building a Serverless Pipeline

Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
Building CICD Pipelines for Serverless Applications - DevDay Austin 2017Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
Amazon Web Services
 

Ähnlich wie Building a Serverless Pipeline (20)

Building CICD Pipelines for Serverless Applications
Building CICD Pipelines for Serverless ApplicationsBuilding CICD Pipelines for Serverless Applications
Building CICD Pipelines for Serverless Applications
 
Serverless Development Deep Dive
Serverless Development Deep DiveServerless Development Deep Dive
Serverless Development Deep Dive
 
Serverless Developer Experience I AWS Dev Day 2018
Serverless Developer Experience I AWS Dev Day 2018Serverless Developer Experience I AWS Dev Day 2018
Serverless Developer Experience I AWS Dev Day 2018
 
Serverless Development Deep Dive
Serverless Development Deep DiveServerless Development Deep Dive
Serverless Development Deep Dive
 
Productionize Serverless Application Building and Deployments with AWS SAM - ...
Productionize Serverless Application Building and Deployments with AWS SAM - ...Productionize Serverless Application Building and Deployments with AWS SAM - ...
Productionize Serverless Application Building and Deployments with AWS SAM - ...
 
Deep Dive on Serverless App Development
Deep Dive on Serverless App DevelopmentDeep Dive on Serverless App Development
Deep Dive on Serverless App Development
 
Deep Dive On Serverless App Development
Deep Dive On Serverless App DevelopmentDeep Dive On Serverless App Development
Deep Dive On Serverless App Development
 
Voxxed Athens 2018 - Serverless by Design
Voxxed Athens 2018 - Serverless by DesignVoxxed Athens 2018 - Serverless by Design
Voxxed Athens 2018 - Serverless by Design
 
re:Invent ARC307 - Serverless architectural patterns and best practices.pdf
re:Invent ARC307 - Serverless architectural patterns and best practices.pdfre:Invent ARC307 - Serverless architectural patterns and best practices.pdf
re:Invent ARC307 - Serverless architectural patterns and best practices.pdf
 
AWS Summit Singapore - Lambda, Step Functions and Datadog: A Symphony
AWS Summit Singapore - Lambda, Step Functions and Datadog: A SymphonyAWS Summit Singapore - Lambda, Step Functions and Datadog: A Symphony
AWS Summit Singapore - Lambda, Step Functions and Datadog: A Symphony
 
Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
Building CICD Pipelines for Serverless Applications - DevDay Austin 2017Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
Building CICD Pipelines for Serverless Applications - DevDay Austin 2017
 
Best Practices for CI/CD with AWS Lambda and Amazon API Gateway (SRV355-R1) -...
Best Practices for CI/CD with AWS Lambda and Amazon API Gateway (SRV355-R1) -...Best Practices for CI/CD with AWS Lambda and Amazon API Gateway (SRV355-R1) -...
Best Practices for CI/CD with AWS Lambda and Amazon API Gateway (SRV355-R1) -...
 
SMC305 Building CI/CD Pipelines for Serverless Applications
SMC305 Building CI/CD Pipelines for Serverless ApplicationsSMC305 Building CI/CD Pipelines for Serverless Applications
SMC305 Building CI/CD Pipelines for Serverless Applications
 
A Practitioners Guide to Securing Your Cloud
A Practitioners Guide to Securing Your CloudA Practitioners Guide to Securing Your Cloud
A Practitioners Guide to Securing Your Cloud
 
Serverless use cases with AWS Lambda
Serverless use cases with AWS Lambda Serverless use cases with AWS Lambda
Serverless use cases with AWS Lambda
 
Serverless Applications with AWS SAM
Serverless Applications with AWS SAMServerless Applications with AWS SAM
Serverless Applications with AWS SAM
 
AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)
AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)
AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)
 
Getting Started with Serverless Computing Using AWS Lambda - ENT332 - re:Inve...
Getting Started with Serverless Computing Using AWS Lambda - ENT332 - re:Inve...Getting Started with Serverless Computing Using AWS Lambda - ENT332 - re:Inve...
Getting Started with Serverless Computing Using AWS Lambda - ENT332 - re:Inve...
 
AWS Lambda 與 Amazon API Gateway 新功能介紹
AWS Lambda 與 Amazon API Gateway 新功能介紹AWS Lambda 與 Amazon API Gateway 新功能介紹
AWS Lambda 與 Amazon API Gateway 新功能介紹
 
Application Lifecycle Management in a Serverless World | AWS Public Sector Su...
Application Lifecycle Management in a Serverless World | AWS Public Sector Su...Application Lifecycle Management in a Serverless World | AWS Public Sector Su...
Application Lifecycle Management in a Serverless World | AWS Public Sector Su...
 

Mehr von Julien SIMON

Mehr von Julien SIMON (20)

An introduction to computer vision with Hugging Face
An introduction to computer vision with Hugging FaceAn introduction to computer vision with Hugging Face
An introduction to computer vision with Hugging Face
 
Reinventing Deep Learning
 with Hugging Face Transformers
Reinventing Deep Learning
 with Hugging Face TransformersReinventing Deep Learning
 with Hugging Face Transformers
Reinventing Deep Learning
 with Hugging Face Transformers
 
Building NLP applications with Transformers
Building NLP applications with TransformersBuilding NLP applications with Transformers
Building NLP applications with Transformers
 
Building Machine Learning Models Automatically (June 2020)
Building Machine Learning Models Automatically (June 2020)Building Machine Learning Models Automatically (June 2020)
Building Machine Learning Models Automatically (June 2020)
 
Starting your AI/ML project right (May 2020)
Starting your AI/ML project right (May 2020)Starting your AI/ML project right (May 2020)
Starting your AI/ML project right (May 2020)
 
Scale Machine Learning from zero to millions of users (April 2020)
Scale Machine Learning from zero to millions of users (April 2020)Scale Machine Learning from zero to millions of users (April 2020)
Scale Machine Learning from zero to millions of users (April 2020)
 
An Introduction to Generative Adversarial Networks (April 2020)
An Introduction to Generative Adversarial Networks (April 2020)An Introduction to Generative Adversarial Networks (April 2020)
An Introduction to Generative Adversarial Networks (April 2020)
 
AIM410R1 Deep learning applications with TensorFlow, featuring Fannie Mae (De...
AIM410R1 Deep learning applications with TensorFlow, featuring Fannie Mae (De...AIM410R1 Deep learning applications with TensorFlow, featuring Fannie Mae (De...
AIM410R1 Deep learning applications with TensorFlow, featuring Fannie Mae (De...
 
AIM361 Optimizing machine learning models with Amazon SageMaker (December 2019)
AIM361 Optimizing machine learning models with Amazon SageMaker (December 2019)AIM361 Optimizing machine learning models with Amazon SageMaker (December 2019)
AIM361 Optimizing machine learning models with Amazon SageMaker (December 2019)
 
AIM410R Deep Learning Applications with TensorFlow, featuring Mobileye (Decem...
AIM410R Deep Learning Applications with TensorFlow, featuring Mobileye (Decem...AIM410R Deep Learning Applications with TensorFlow, featuring Mobileye (Decem...
AIM410R Deep Learning Applications with TensorFlow, featuring Mobileye (Decem...
 
A pragmatic introduction to natural language processing models (October 2019)
A pragmatic introduction to natural language processing models (October 2019)A pragmatic introduction to natural language processing models (October 2019)
A pragmatic introduction to natural language processing models (October 2019)
 
Building smart applications with AWS AI services (October 2019)
Building smart applications with AWS AI services (October 2019)Building smart applications with AWS AI services (October 2019)
Building smart applications with AWS AI services (October 2019)
 
Build, train and deploy ML models with SageMaker (October 2019)
Build, train and deploy ML models with SageMaker (October 2019)Build, train and deploy ML models with SageMaker (October 2019)
Build, train and deploy ML models with SageMaker (October 2019)
 
The Future of AI (September 2019)
The Future of AI (September 2019)The Future of AI (September 2019)
The Future of AI (September 2019)
 
Building Machine Learning Inference Pipelines at Scale (July 2019)
Building Machine Learning Inference Pipelines at Scale (July 2019)Building Machine Learning Inference Pipelines at Scale (July 2019)
Building Machine Learning Inference Pipelines at Scale (July 2019)
 
Train and Deploy Machine Learning Workloads with AWS Container Services (July...
Train and Deploy Machine Learning Workloads with AWS Container Services (July...Train and Deploy Machine Learning Workloads with AWS Container Services (July...
Train and Deploy Machine Learning Workloads with AWS Container Services (July...
 
Optimize your Machine Learning Workloads on AWS (July 2019)
Optimize your Machine Learning Workloads on AWS (July 2019)Optimize your Machine Learning Workloads on AWS (July 2019)
Optimize your Machine Learning Workloads on AWS (July 2019)
 
Deep Learning on Amazon Sagemaker (July 2019)
Deep Learning on Amazon Sagemaker (July 2019)Deep Learning on Amazon Sagemaker (July 2019)
Deep Learning on Amazon Sagemaker (July 2019)
 
Automate your Amazon SageMaker Workflows (July 2019)
Automate your Amazon SageMaker Workflows (July 2019)Automate your Amazon SageMaker Workflows (July 2019)
Automate your Amazon SageMaker Workflows (July 2019)
 
Build, train and deploy ML models with Amazon SageMaker (May 2019)
Build, train and deploy ML models with Amazon SageMaker (May 2019)Build, train and deploy ML models with Amazon SageMaker (May 2019)
Build, train and deploy ML models with Amazon SageMaker (May 2019)
 

Kürzlich hochgeladen

Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Kürzlich hochgeladen (20)

MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 

Building a Serverless Pipeline

  • 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Building a serverless data pipeline Julien Simon, Principal Technical Evangelist, AWS julsimon@amazon.fr @julsimon
  • 3. AWS Lambda •  Deploy pure funcJons in Java, Python and Node.js •  Works nicely with AWS managed services: Amazon S3, Amazon DynamoDB, etc. •  Build event-driven applicaJons •  Build RESTful APIs in conjuncJon with Amazon API Gateway •  Pay as you go: number of requests + execuJon Jme (100ms slots)
  • 6. MOBILE CHAT APP AD DATA ANALYTICS AND ROUTING MOBILE APP ANALYTICS IMAGE CONTENT FILTERING REAL-TIME VIDEO AD BIDDING NEWS CONTENT PROCESSING GENE SEQUENCE SEARCH CLOUD TELEPHONY DATA PROCESSING WEB APPLICATIONS WEB APPLICATIONS THREAT INTELLIGENCE AND ANALYTICS NEWS CONTENT PROCESSING GAME METRICS ANALYTICS Selected serverless customers PRODUCT RECOMMANDATION
  • 9. AWS Lambda ‘Hello World’ (Python) 1.  Write a simple Lambda function in Python 2.  Create a REST API with API Gateway (resource + POST method) 3.  Deploy the API 4.  Invoke the API with ‘curl’
  • 10. A simple Lambda function in Python def lambda_handler(event,context):    result = event['value1'] + event['value2']    return result aws lambda create-function --function-name myFunc --handler myFunc.lambda_handler --runtime python2.7 --zip-file fileb://myFunc.zip --memory-size 128 --role arn:aws:iam::ACCOUNT_NUMBER:role/lambda_basic_execution curl -H "Content-Type: application/json" -X POST -d "{"value1":5, "value2":7}" https://API_ENDPOINT/STAGE/RESOURCE 12
  • 11. AWS Lambda in Java with Eclipse https://java.awsblog.com/post/TxWZES6J1RSQ2Z/Testing-Lambda-functions-using-the-AWS-Toolkit-for-Eclipse
  • 12. AWS Lambda ‘Hello World’ (Java) 1.  In Eclipse, write a simple Lambda function triggered by an S3 event 2.  Unit-test the function with Junit 3.  Using the AWS Eclipse plug-in, upload and run the function in AWS 4.  Run the function again in the AWS Console
  • 13. AWS Lambda with the Serverless framework http://github.com/serverless/serverless •  Run/test AWS Lambda functions locally, or remotely •  Auto-deploys & versions your Lambda functions •  Auto-deploys your REST API to AWS API Gateway •  Auto-deploys your Lambda events •  Support for multiple stages •  Support for multiple regions within stages •  Manage & deploy AWS CloudFormation resources
  • 14. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST 
 /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps
  • 15. Step 1: create DynamoDB table aws dynamodb create-table --table-name eventTable --attribute-definitions AttributeName=userId,AttributeType=N AttributeName=timestamp,AttributeType=N --key-schema AttributeName=userId,KeyType=HASH AttributeName=timestamp,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 --stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE eventTable DynamoDB
  • 16. Step 2: IAM role for Lambda function aws iam create-role --role-name writeToDynamoDB_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name writeToDynamoDB_policy --policy-document file://writeToDynamoDB_policy.json aws iam attach-role-policy --role-name writeToDynamoDB_role --policy-arn WRITETODYNAMODB_POLICY_ARN
  • 17. Step 3: create Lambda function S3 DynamoDB eventTable Web apps dynamodb. put_item() Lambda aws lambda create-function --function-name writeToDynamoDB --role WRITETODYNAMO_DB_ROLE --zip-file fileb://writeToDynamoDB.zip --handler writeToDynamoDB.lambda_handler --runtime python2.7 --memory-size 128 --description "Write events to DynamoDB”
  • 18. Step 4: create Kinesis Stream DynamoDB S3 eventTable Web apps dynamodb. put_item() Lambda aws kinesis create-stream --stream-name APItoDynamoDB --shard-count 1 Kinesis Streams
  • 19. Step 5: IAM role for Lambda function aws iam create-role --role-name writeToKinesis_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name writeToKinesis_policy --policy-document file://writeToKinesis_policy.json aws iam attach-role-policy --role-name writeToKinesis_role --policy-arn WRITETOKINESIS_POLICY_ARN
  • 20. Step 6: create Lambda function DynamoDB S3 eventTable Web apps DynamoDB streams dynamodb. put_item() Lambda aws lambda create-function --function-name writeToKinesis --role WRITETOKINESIS_ROLE --zip-file fileb://writeToKinesis.zip --handler writeToKinesis.lambda_handler --runtime python2.7 --memory-size 128 --description "Write events to Kinesis” Kinesis StreamsLambda KinesisTo DynamoDB writeTo Kinesis
  • 21. Step 7: create API DynamoDB S3 eventTable Web apps DynamoDB streams dynamodb. put_item() Lambda Painful to do with the CLI: 9 aws apigateway calls :-/ à  Use the console à  Use a Swagger File http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html à  Use the Serverless framework Kinesis StreamsLambda KinesisTo DynamoDB writeTo Kinesis API Gateway
  • 22. Step 8: create IAM role aws iam create-role --role-name DynamoDBToFirehose_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name DynamoDBToFirehose_policy --policy-document file://DynamoDBToFirehose_policy.json aws iam attach-role-policy --role-name DynamoDBToFirehose_role --policy-arn DYNAMODBTOFIREHOSE_POLICY_ARN
  • 23. Step 9: create Lambda funcIon and DynamoDB trigger aws lambda create-function --function-name DynamoDBToFirehose --role DYNAMODBTOFIREHOSE_ROLE_ARN --zip-file fileb://DynamoDBToFirehose.zip --handler DynamoDBToFirehose.lambda_handler --runtime python2.7 --memory-size 128 --description "Write DynamoDB stream to Kinesis Firehose" aws lambda create-event-source-mapping --function-name DynamoDBToFirehose --event-source DYNAMODB_STREAM_ARN --batch-size 10 --starting-position TRIM_HORIZON Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
  • 24. Step 10: create IAM role aws iam create-role --role-name firehoseToS3_role --assume-role-policy-document file://firehose_trust_policy.json aws iam create-policy --policy-name firehoseToS3_policy --policy-document file://firehoseToS3_policy.json aws iam attach-role-policy --role-name firehoseToS3_role --policy-arn FIREHOSETOS3_POLICY_ARN
  • 25. Step 11: create S3 bucket aws s3 mb s3://jsimon-public Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
  • 26. Step 12: create Kinesis Firehose stream aws firehose create-delivery-stream --delivery-stream-name firehoseToS3 --s3-destination-configuration RoleARN=FIREHOSETOS3_ROLE_ARN, BucketARN="arn:aws:s3:::jsimon-public", Prefix="firehose", BufferingHints={SizeInMBs=1,IntervalInSeconds=60}, CompressionFormat="GZIP", EncryptionConfiguration={NoEncryptionConfig="NoEncryption"} Kinesis Firehose firehoseToS3 Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
  • 27. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST 
 /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps Lines of code: 16 Number of servers: zero Performance & scalability: maximum https://github.com/juliensimon/aws/tree/master/serverlessPipeline
  • 30. Going further AWS re:Invent 2014 | (MBL202) NEW LAUNCH: Getting Started with AWS Lambda https://www.youtube.com/watch?v=UFj27laTWQA AWS re:Invent 2015 | (DEV203) Amazon API Gateway & AWS Lambda to Build Secure and Scalable APIs https://www.youtube.com/watch?v=ZBxWZ9bgd44 AWS re:Invent 2015 | (DVO209) JAWS: The Monstrously Scalable Serverless Framework https://www.youtube.com/watch?v=D_U6luQ6I90 https://github.com/serverless/serverless AWS re:Invent 2015 | (ARC308) The Serverless Company Using AWS Lambda https://www.youtube.com/watch?v=U8ODkSCJpJU AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda https://www.youtube.com/watch?v=FhJxTIq81AU Reference architectures http://www.allthingsdistributed.com/2016/06/aws-lambda-serverless-reference-architectures.html
  • 32. AWS Enterprise Summit – 27/10/2016, Paris http://amzn.to/1X2yp0i