This deck was presented at Lendingkart meetup in Bangalore covering our experiences with creating CI/CD Pipeline with Kubernetes. Here is the video link of the meetup. https://youtu.be/YraPL_NGmcs

1. CI/CD with Kubernetes
Tech Talk @Lendingkart
Mihir, Rishabh
May 18th, 2019

2. CI/CD Introduction
Why CI/CD ?
What is CI/CD ?
How we accomplished ?

3. Why CI/CD: Initial Lendingkart Platform !
● Started with 2 Backend, 2 UI Developers, 1 QA
● As new developers joined the team, it was difficult to understand and
maintain the monolithic
● Initial monolithic catered to customer & ops with minimal features.
● Stability was a major worry as we had to deploy features and bugs on a
daily basis
● Vertical scaling of our database server on a regular basis

4. Why CI/CD: Monolith
● Lending involves 3 major pieces: Origination, Credit analysis, Loan
management
● Our initial monolithic involved Customer, Origination and Credit analysis
modules (Spring MVC) structured on the lines of SOA
● For Loan management we used a third party service to avoid re-inventing
the wheel

5. Why CI/CD: Monolith to Microservice Architecture
● First microservice was in the form of lead service , followed by notification
service
● Subsequent break of our monolithic to microservice happened in the form
of customer, credit analysis, audit, reports, scheduler, webhooks, etc
● The greatest advantage was that since our requirements were changing
fast, we could make the changes quickly on the microservice and make it
more mature over time

6. Why CI/CD: Typical Microservice Architecture

7. Why CI/CD: How We Moved to Microservices Architecture
Monolithic to Microservice architecture
UI
MySQL DB
Business logic
Data Access Layer
MySQL DB MongoDB MySQL
Microservice
Customer
Microservice
Ops
Microservice
Notification
Microservice
Credit Analysis
Nginx
UI UI UI UI

8. How We Broke Our Monolithic !
● Lead
● Audit
● Customer
● Schedulers
● Webhooks
● Search
● Authentication & Authorization
● Crypto
● Notifications
(Many more)

9. Problems in Existing Release & Deployment Cycle
● Too many releases every week. (Thanks to microservices!)
● Too many failures and rollbacks required manually.
● Failed machines, applications, deployments go unnoticed and would need manual intervention to
bring up.
● Too many machines to be managed and procured for the release by devops.
● Every new project/service need new infrastructure & automation.
● QA needs to understand different release process for different projects like spring mvc, spring
boot, node js, UI etc.,
● No standard way of releasing all the projects.
● Growing infrastructure cost due to individual machine/servers for each service/replicas.

10. What is CI/CD ?

11. How CI/CD works ?

12. Gitlab CI
Gitlab repository .gitlabci.yml RunnerGitlab CI
Gitlab CI is a
powerful tool
for
continuously
test and
deploy the
code.
It contains all the
configuration of
what CI will do
with your
repository.
Runner can be a
VM, docker
container or
cluster of
machines which
runs the jobs
that are defined
in gitlabci.yml.
Gitlab Container
Registry
Every project can have its own space to store its
Docker images. Build and push the images to
registry.

13. Sample gitlab-ci.yml file

14. Docker
A container is a standard unit of software
that packages up code and all its
dependencies so the application runs
quickly and reliably from one computing
environment to another.
ROI & Cost Saving
Immutable
Rapid Deployment
Standardization

15. Dockerfile

16. KOPS (kubernetes operations)
Kops is basically a Kubernetes project used to manage Kubernetes cluster to AWS.
To create K8s cluster you should have:-
- S3 Bucket
- IAM Permissions
- DNS
Important Commands:-
kops create cluster --name <cluster_name> --zones <zone> --state <s3_bucket> --yes
kops get clusters --state="s3://k8s-lkart-clusters"
kops validate cluster --state="s3://k8s-lkart-clusters"
kops edit ig nodes --state="s3://k8s-lkart-clusters"
kops delete cluster --name <cluster_name> --state="s3://k8s-lkart-clusters" --yes

17. Kops Advantages
● Deploy clusters to existing virtual private clouds (VPC) or create a new VPC from scratch
● Supports public & private network topologies
● Provisions single or multiple master clusters
● Configurable bastion machines for SSH access to individual cluster nodes
● Rolling cluster updates
● Supports heterogeneous clusters by creating multiple instance groups

18. Kubernetes to Rescue!
In broad term, “Kubernetes is orchestration layer which decouples your application deployments from
underlying infrastructure” !

19. Kubernetes: Features/Advantages
● Orchestration layer
● Decouple Infrastructure from Applications.
● Cluster management
● Service discovery
● Make better use of your hardware resources
● Scale containerized applications and their resources on the fly.
● Declaratively manage services, which guarantees the deployed applications are always running
how you deployed them
● Health-check and self-heal your apps with auto placement, autorestart, auto replication, and
autoscaling.
● Horizontal scaling.
● Rolling updates (zero downtime).

20. Kubernetes: Architecture replication controller,
endpoints controller,
namespace controller,
and service accounts
controller
Individual node
monitoring
Node agent to communicate
with masters & manage pod
metrics Network proxy for the
pods to communicate
Storage stack
for secrets,
definitions, state
of system

21. Kubernetes: Building Blocks

22. Kubernetes: Cluster Setup
TCP
AWS Cloud
ALB
NGINX
VPC
PEERING
Non K8s VPC
Private
Subnet
Private
Subnet
legacy-service
Pod
container
Pod
container
K8s VPC (Private subnet)
lkart-service sherlock-service
Ingress
ELB
nucleus-service
Pod
container
Pod
container
Pod
container
Pod
container
vendor-tools

23. Kubernetes:Deploying First Application!
Service
Expose application
ports
Bind ELB
ConfigMap/Secrets/PV
Environment variables
Passwords
Volume mounts
Deployment/Statefulsets
Deployment.yml
Ports
Docker images
Resources
Ingress
Routing rules
SSL
Host details.

24. Phase 1: Dockerize & Deploy
● Deployed containers on existing machines without any changes in network topology.
● Logs mounted to local disks.
Challenges:
● Memory leaks
● Incorrect containers dockerized.
● No profiling
● No resource limits

25. Phase 2: Critical Applications Migration to K8s
● Started dockerizing all services.
● Deployed & Stabilized the applications.
● Stabilization of cluster.
Challenges:
● Logs, disk full, temp data
● Stateful applications.
● Encryption till the container.
● Monitoring & Alerts

26. Phase 3: Standardisation of logs & tmp data management
Pod1 Pod2
Pod3 Pod4
Pod5 Pod6
Pod7 Pod8
Tmp files
PVC
Bigqueue
PVC
Logs PVC
Filebeat
Filebeat
Filebeat
Filebeat
/tmp PV
/var/log/l
endingka
rt PV
/data/bq
PV
NFS PVCs
Node
Daemonsets
Kubernetes cluster

27. Phase 3: Certificates Management
Namespace
Secretes
Ingress
Service 1
Service 2
POD 1
POD 2
POD 3
POD 4
SSL
termination

28. Phase 4: Migrating rest of applications to k8s
Internal ELB Public ELB
Sherlock audit consumer LkartWS Nucleus
... ...
Lead

29. Phase 5: QA Environment setup
1 2 3 4

30. Phase 6: Serverless
Auto scaling groups
K8s cluster nodes
● Serverless kubernetes engine
● Pod driven auto-scaling
● Heterogenous instance groups
● Cost effective

31. Demo