This document discusses how Chargebee, a subscription management platform, used Amazon EKS and serverless tools to improve their software development and testing cycle times. It outlines how their previous environment led to bottlenecks of 2-3 days between development stages. Their new solution utilizes dynamic pre-development environments on EKS to allow features to progress from local development to production in as little as a few hours by enabling parallel and independent testing. It describes the tools used including EKS, ECR, ALB Ingress Controller, ExternalDNS and a self-service portal called Minions to provision and manage the pre-development Kubernetes clusters.

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHENNAI

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Powering Test
Environments with
Amazon EKS using
Serverless Tool
KARTHIKEYAN VENKATRAMAN | August 10, 2019

3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Subscription management platform that helps untangle
recurring revenue businesses from spaghetti billing.
● Hosted on AWS Cloud.
120 +
Currencies
20+
Payment
Gateways
8,500 +
Customers
50 +
Countries Employees
300+

4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BUSINESS CHALLENGE
With growing development teams, our key business challenge was to
improve the release cycle time.
Spin up Test Environments on the fly to empower Integration Testing
across various microservices

5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chargebee
Application
Stack

6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Monolithic application
● Multiple internal microservices
● Hosted on EC2 Instances
● In-house deployment tool

7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Development Time
Local
● Feature, Bug
Fixes
● Unit Testing
2-3 Days
DEV
● Manual and
Integrations Test
● Database
Migrations
● Feature Demos for
Product Teams
2-3 Hours
STAGING
● Database
Migrations
● Sanity
Testing
● Patch Fix
Go Live
PRODUCTION
● Database
Migrations
● Rolling
Upgrades
● GO Live
1-2 Hours
● Functional
Tests(covers 60% of
use cases)
● Local Database
Schema Generation
● Parallel Test
executions
● Go or No-Go for Dev
QA (Automated)
Deployment Stages and Timelines:

8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bottleneck with Dev
Majority of the features that land
in dev took a minimum of two
days to progress to Staging. LOCAL
I’m
waiting
for the
DEV :(
DEV
STG
PROD

9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database
Migrations
& Validations
Team
Collaborations
& Dependencies
Testing
& Bug Fixes
Infra Changes
& Environment
Updates
Delay in Dev

10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution:
Launch Dynamic PREDEV
Environments
LOCAL
I’m in Predev
and will
release to
Prod Shortly
DEV
STG
PROD
PRE
DEV
PRE
DEV
PRE
DEV
PRE
DEV

11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Immutable Setup
2. Parallelize Build
3. Perform Database Migrations with Production
alike Schema
4. Onboard New Services quickly as they evolve
5. Low Operational Overhead
6. CI/CD
7. Security & Compliance
Solution Requirements

12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Reclone Dev environments with EC2 instances.
● Containers.
Options

13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EC2 INSTANCES
CONTAINERS
We opted to go with containers using
AWS EKS.
Pros:
● Templated setup
● Identical to other
environments.
Cons:
● Infrastructure costs.
● Operational overheads.
Pros:
● Build for the future.
● Low operational overhead.
● Low infra costs.
Cons:
● Infrastructure is different from
other environments.

14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elastic Kubernetes Service (EKS)
● Prior experience with Kubernetes
● Managed control plane service
● Active community support
● Cloud agnostic
WORKER NODES
VPC
EKS
AZ-1 AZ-2 AZ-3

15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TOOLS FUNCTIONALITY PURPOSE
AWS Cloudformation
Infrastructure as a Code Provision infrastructure resources in
AWS Cloud Environment.
AWS Elastic Container
Service for Kubernetes (EKS)
Container Orchestration
Engine
Deploy, manage and scale Container applications.
AWS Elastic Container
Registry (ECR)
Container Registry Store and manage Docker Container images
AWS ALB Ingress Controller
Ingress Controller
(Maintained & Supported
by AWS)
To create application load balancer which routes
HTTP[s] traffic across different endpoints.
Tool Kit

16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TOOLS FUNCTIONALITY PURPOSE
KUBE2IAM
Fine Grained IAM
permissions to Pods
Provides IAM Credentials for Container to make
calls to AWS Service based on the Pod level
Roles.
ExternalDNS
DNS mapping for
Kubernetes Services and
Ingresses(Community
Project)
Kubernetes Services and Ingress is mapped to
AWS Route53 for DNS resolution.
HELM
Kubernetes Application
Manager(CNCF Project)
Manage Kubernetes Applications using charts.
Create, Version Control and Publish Kubernetes
Resources.
Jenkins
Open Source Automation
Tool
Used as Build Server with various Pipeline Jobs
which helps to provision Predev Environment by
communicating with EKS and ECR

17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● EKS control plane for cluster management
● Cloudformation for VPC and EKS Worker
Nodes Set up
● Docker Images in AWS ECR
● ALB Ingress Controller for Load Balancing
○ One per predev environment.
● External DNS and Route53 for DNS
● Kube2IAM for Policy management
● Cloudwatch for Monitoring.
Solution Overview
WORKER NODES
VPC
EKS
AZ-1 AZ-2
RDS (1..N)
DYNAMODB
ALB (1..N)
ROUTE 53
ECR
IAM
S3
CLOUDWATCH
CLOUDFORMATION

18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Leveraging ALB
with Kubernetes
Ingress Controller

19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
KUBE PROXYAWS-NODE-VPC
PD1(APP)
PD2(UI-CDNOTES)PD3(ELASTICMQ)
PD1(AUDIT)
PD1(ELASTICMQ) KUBE PROXY
KUBE2IAM
TILLER
PD1(UI-INV)
PD2(DB-MIG)
AWS-NODE-VPCEXTERNALDNSALB-INGRESS
CONTROLLER
KUBERNETES
CLUSTER
KUBERNETES
APISERVER
KUBE2IAM
NODEPORT NODEPORT NODEPORT
Rule
Host: *.pd1.in
Rule
Host: *.audits.pd1.in
Rule
Host: *.ui.pd1.in
Path: /invoices
APP-Target
Group(Instance
Mode)
AUIDIT-Target
Group(Instance
Mode)
UI-INV-Target
Group(Instance
Mode)
APPLICATION
LOAD BALANCER
Ingress Resource
Creation using
HELM CHARTS
DAEMONSET
PODS
APPLICATION
PODS
EKS-WORKER-NODE EKS-WORKER-NODE

20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minions -
Self Service Portal
for Predev Setup

21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● To ease developers to quickly provision Predev environments.
● Helps :
○ Provision/De-Provision environments
○ CI/CD
○ Track build and deployment status
○ Control access
Minions (Serverless Tool)

22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minions Serverless Architecture
1. Single page application (SPA).
2. Frontend served by Cloudfront with S3
origin.(vue.js).
3. AWS Cognito with User Pool for authentication.
4. API Gateway backed by Lambda function used to
perform server side functions.
5. Leveraging AWS WAF with Cloudfront and API
Gateway for additional security.
6. Lambda Function used SSM RUN Command to
trigger Jenkins commands.
7. Jenkins installs/updates/deletes environment in
EKS using Helm charts.
JENKINS
SSM
RUN COMMANDLAMBDAAPI
GATEWAY
S3CLOUDFRONT
DYNAMODBCOGNITO
DEVELOPERS
SQSSNS

23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CI/CD For Predev

24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEVELOPERS GITHUB
JENKINS
ECR
EKS
Developers commit
to their feature
branch
Docker images
pushed to ECR
Jenkins polls the
branch for changes
EKS pulls the images
from ECR to setup
pods
Jenkins updates the
pods using helm charts
CI/CD For Predev
DOCKER
IMAGES
Jenkins builds the
docker images

25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment Stages and Timelines (Post Predev Implementation):
Development
Time
LOCAL
● Feature ,Bug
Fixes.
● Unit Testing
2-3 Hours
DEV
● Database
Migrations
● Sanity Test
2-3 Hours
STAGING
● Database
migrations
● Sanity
Testing
● Patch Fix
Go Live
PRODUCTION
● Database
Migrations.
● Rolling
Upgrades
● GO Live
1-2 Hours
● Functional Tests
● Local Database
Schema
Generation
● Parallel Test
Executions
● Go or No-Go for
Dev
QA(Automated)
Development
Time
PREDEV
● Feature Testing
,Bug Fixes.
● Database
Migration
● Functional and
Integration Tests
● User Acceptance
Tests

26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You