Deploy AWS Lambda by AWS Lambda!

1. Continuous Deployment in AWS Lambda
Suiting Tseng (@suitingtseng)

2. Who am I
class SoftwareEngineer(Human):
def __init__(self):
self.interests = [
"AWS"
]
self.languages = [
"Python",
"Golang"
]

3. –KKStream
We back you up to compete against
Netflix, Spotify, and KKBOX.

4. Outline
• AWS Lambda Introduction
• Lambda Deployer
• Adoption in KKStream

5. What is AWS
Lambda?

6. –AWS
AWS Lambda is a zero-administration compute
platform for back-end web developers that runs
your code for you in the AWS cloud and provides
you with a fine-grained pricing structure.

7. –AWS
AWS Lambda runs your back-end code on its own
AWS compute fleet of Amazon Elastic Compute
Cloud (Amazon EC2) instances across multiple
Availability Zones in a region, which provides the
high availability, security, performance, and
scalability of the AWS infrastructure.

8. Cloud Computing in AWS
• Elastic Compute Cloud (EC2)
• Infrastructure as a Service
• Abstract hardware

9. Cloud Computing in AWS
• Elastic Compute Cloud (EC2)
• EC2 Container Service (ECS)
• Container as a Service
• Abstract OS

10. Cloud Computing in AWS
• Elastic Compute Cloud (EC2)
• EC2 Container Service (ECS)
• AWS Lambda
• Function as a Service
• Serverless

11. Lambda Runtime
• Python 2.7
• Java OpenJDK 1.8
• Node.js v0.10.36, v4.3.2

12. Hello World
# lambda_function.py
def lambda_handler(event, context):
print “Hello, PyCon!”

13. Event & Context
• Event
• What happen? S3? DynamoDB?
• Context
• How much time remaining?

14. Execution model
• Synchronous
• Event driven

15. Execution model
• Synchronous
• Event driven

16. Sync. Call
• AWS SDK
• AWS API Gateway

17. Sync. Call
• AWS SDK
• AWS API Gateway

18. AWS SDK
import boto3
boto3.client("lambda").invoke(
FunctionName="hello-world"
)

19. Sync. Call
• AWS SDK
• AWS API Gateway

20. API Gateway Lambda
GET /videos/123

21. API Gateway Lambda
API Gateway calls Lambda

22. API Gateway Lambda
Lambda retrieve data from somewhere

23. API Gateway Lambda
Lambda response

24. API Gateway Lambda
200 OK
{
“name”: “ ”,
“characters”: [“ ”...],
……
}

25. API Gateway Lambda
Lambda scales automatically

26. Execution model
• Synchronous
• Event driven

27. Event driven
• Event sources
• S3
• Scheduled Event
• DynamoDB
• and many more…

28. raw thumbnailLambda
Users upload their photo

29. raw thumbnailLambda
S3 triggers Lambda

30. raw thumbnailLambda
Lambda resizes the image,
and put it to another bucket

31. S3 event
1 def lambda_handler(event, context):
2 s3 = boto3.client('s3')
3
4 bucket = event['Records'][0]['s3']['bucket']['name']
5 key = event['Records'][0]['s3']['object']['key']
6 print "S3 Bucket: %s" % bucket
7 print "S3 Key: %s" % key
8 ……

32. Lambda usage
• HTTP API backend
• Worker
• Cronjob

33. Version and Alias
• Version
• 1, 2, 3, 4 ….
• Incremental integer
• Immutable
• Alias
• feature -> 4
• testing -> 8
• Mutable

34. Outline
• AWS Lambda Introduction
• Lambda Deployer
• Adoption in KKStream

35. Problem
• So you wanna deploy…
• Upload a zip to S3
• Create or update function
• Configure version
• Create or update alias
• And you may have 100+ functions

36. Our goal
push

37. Our goal
notify

38. Deploy pipeline
git push origin master

39. Deploy pipeline
HTTP webhook

40. Deploy pipeline
1. Build dependency
2. Testing
3. Package
4. Upload to S3

41. Deploy pipeline
Triggers Lambda Deployer

42. Lambda deployer is also a Lambda function!

43. Behind the scene - 1
packages/
my-func-1/
my-func-2/
my-func-3/
……

44. Behind the scene - 2
packages/
my-func-1/
bfb53884976e70966a9e8a8d5133bcb2e1c06691/
eed8ccf9ea22099cef07b0a2012085cd45e1d3ad/
……

45. Behind the scene - 3
packages/
my-func-1/
bfb53884976e70966a9e8a8d5133bcb2e1c06691/
lambda.zip
function.json

46. function.json
{
"runtime": "python2.7",
"handler": "main.lambda_handler",
"timeout": 300,
"memory": 128
}

47. Behind the scene - 4
packages/
my-func-1/
bfb53884976e70966a9e8a8d5133bcb2e1c06691/
eed8ccf9ea22099cef07b0a2012085cd45e1d3ad/
testing
……

48. Deploy pipeline
Create / update function
Publish new version
Update alias

49. deployer.py
class Deployer(object):
def __init__(self):
self.s3_client = boto3.client("s3")
self.lambda_client = boto3.client("lambda")
self.slack = Slacker(open("config/slack-token").read())
self.slack_channel = json.load(open("config/slack.json"))

50. deployer.py
1 class Deployer(object):
2
3 def deploy(self, record):
4 bucket = record["s3"]["bucket"]["name"]
5 key = record["s3"]["object"]["key"]
6
7 lambda_function = LambdaFunction(bucket, key)
8 version = self.put_function(lambda_function)
9 self.put_alias(lambda_function, version)

51. main.py
1 import boto3, json
2 from deployer import Deployer
3
4 def run(event, context):
5 d = Deployer()
6 print "boto3 version:", boto3.__version__
7 print json.dumps(event, indent=2)
8 for record in event["Records"]:
9 d.deploy(record)

52. Pros
• Serverless (yes we are lazy…)
• One deployer manages
• many functions
• many stages (using alias)
• Perfectly fits in our workflow

53. In KKStream
• From 2016/02/01 to 2016/05/31
• 12 developers
• 99 functions
• 4288 deployments
• 1 fork

54. In KKStream
• Deploy itself 56 times ( code )
• failed several times…
• but very easy to rollback

55. Future work
• Add event source mapping
• More configurations
• https://github.com/KKStream/aws-lambda-deployer

56. Q & A

57. We’re hiring!

58. Meet us at our booth
on 4th floor!

59. Thank you!