@gokhansengun
Migrating
a Cloud Native App
Running on AWS to k8s
Gökhan Şengün
DevOpsDays İstanbul - Sep 2018

DEVOPS Zirvesi 2017
me
• Full-Stack Developer with 12 years of experience
• Blogging every Monday on Medium
• In Turkish and mostly about DevOps related concepts
@gokhansengun
• Currently working for
• Sending random tweets
@gokhansengun

@gokhansengun
1. What is this talk about?
Replatforming a cloud native
app running on Aws to k8s

@gokhansengun
2. How does application look like?

DEVOPS Zirvesi 2017
@gokhansengun
• Runs on
• 15 services all written with
• Heavily using Aws services like
• In 3 different Aws regions
• Complex integrations with 3rd parties
• Not a simple mobile application :-)
How does application look like?

@gokhansengun
3. What were customer’s requirements?

DEVOPS Zirvesi 2017
@gokhansengun
• Be cloud agnostic
• Run on bare metal
• Create a new installation by clicking of a button
• Improve developer experience
• Improve operational experience
• Prevent SPOF on vital system components
• Make it cost effective
• Improve observability and operational measurements
• Run hand-to-hand with legacy system to ease migration
• Have zero-downtime deployments
• Have auto-scale in infrastructure level
What were customer’s requirements?

@gokhansengun
4. How did we approach it?

DEVOPS Zirvesi 2017
@gokhansengun
• Abstract away creation of compute, network, storage resources
• to support becoming cloud agnostic
• Apply Immutable Infrastructure and Infrastructure as Code (IaC)
• to automate and ease new installations
• Containerize and orchestrate services
• to create parity between dev, test, prod environments
• to build health checks, observability and prevent SPOF
• Leave stateful parts out of the first phase’s scope
• to run hand-to-hand with legacy system
How did we approach it?

@gokhansengun
5. How did we do it actually?

@gokhansengun
5.1. Creation of Cloud Resources

DEVOPS Zirvesi 2017
@gokhansengun
• Used HashiCorp Packer to bake cloud images (ex: ami for Aws)
• Packer’s Ansible provisioner is used
• This is to use the same Ansible playbooks in on-prem case
• Used HashiCorp Terraform to create/provision cloud resources
Creation of Cloud Resources

DEVOPS Zirvesi 2017
@gokhansengun
Creation of Cloud Resources
Cloud Images
Ansible Playbooks
On-prem Servers
172.27.44.10
172.27.44.11
172.27.44.20
172.27.44.21

@gokhansengun
5.2. Containerization and Orchestration

DEVOPS Zirvesi 2017
@gokhansengun
• Docker was used as the container runtime
• Main stream, everybody knows it
• People familiar with its CLI
• Apps were quickly containerized
• Kubernetes is selected as the Container Orchestrator
• There are not many choices these days
• It is the de facto standard
• Preparing k8s definition files were easy too
• App already had health check endpoints
• Handling signals correctly
• Harbor is used for container registry
Containerization and Orchestration

@gokhansengun
5.3. Configuration and Secret Mgmt

DEVOPS Zirvesi 2017
@gokhansengun
• etcd is used as K/V store
• Configuration files are templated
• Confd was used to render them
• We found konfd not usable/flexible
• Configuration values kept installation-based
• Same template used for dev, test, staging and every installation
• Configuration change
• Triggers re-render of the template
• Triggers re-deploy of the app w/ no downtime
Configuration Management

DEVOPS Zirvesi 2017
@gokhansengun
• HashiCorp Vault is used as storage for secrets
• Confd renders secrets from template to config
• k8s-Vault Integration used for seamless integration
• Secrets are visible only inside the container
• Used Vault as CA (Certificate Authority) for X.509 Certificates
• k8s component certificates generated with it
• Used Vault to generate Aws keys for short period of time
• Will use Vault to manage SSH access to machines
• Except to k8s workers where containers run
Secret Management

@gokhansengun
5.4. k8s Cluster Setup

DEVOPS Zirvesi 2017
@gokhansengun
• Used kubeadm initially but replaced after a custom setup
• Kops and other solutions not possible due to cloud + on-prem support
• Control plane services run with systemd
• Supports both HA (production) or non-HA (test, staging) mode
• Terraform provisioners used to bootstrap the cluster for cloud case
• HashiCorp Vault is used for creating and renewing certificates
• Allowed easily creating/destroying worker nodes in auto-scaling scenario
• cfssl is used for developer box k8s certificates
k8s Cluster Setup

@gokhansengun
5.5. CI/CD Pipeline

DEVOPS Zirvesi 2017
@gokhansengun
• Jenkins is used to initiate all processes
• It is the single point of contact to whole infra
• Docker images for services auto-built, tagged and pushed
• Tagging scheme selected for the need, details irrelevant
• Spinnaker is used to manage the k8s deployments
• Workflows and accountability were the reasons of selection
• Infrastructure creation/modification is also done through Jenkins
• Called GitOps these days
• Change on the Terraform source code applied on infra
CI/CD Pipeline

@gokhansengun
5.6. Logging, Monitoring, Alarming

DEVOPS Zirvesi 2017
@gokhansengun
• Our Motto: “Do not search issues in log files, look into alarms”
• If there is a problem but no alarm, think of creating an alarm
• Prometheus used to infra and application collect metrics
• Still need to generate more operational metrics
• Still need to create more alarms based on OM data
• Grafana used to visualize collected metrics
• Container logs collected on each worker node with syslogd
• Logs then collected by existing ELK infrastructure
Logging, Monitoring, Alarming

@gokhansengun
5.7. Overall Architectural Look

DEVOPS Zirvesi 2017
@gokhansengun
Overall Architectural Look
Coordination
Environment eu-west
installation
ap-southeast-2
installation

@gokhansengun
5.8. Developer Box

DEVOPS Zirvesi 2017
@gokhansengun
• Every developer has her/his separate environment locally
• Including Jenkins/Grafana/Prometheus/ELK
• HashiCorp Vagrant is used to orchestrate and provision developer
box
• Developer may configure any test/staging environments on Aws for
stateful parts for his local machine
• Best of all – real magic: developer may develop/debug any of the 15
services on her/his host machine with IDE :-)
Developer Box

DEVOPS Zirvesi 2017
@gokhansengun
Developer Box – Developing Locally
Dev Host Box
Virtual Box
Stateful
Components
in the Cloud
App containers
run by k8s
Apps running on
the host directly
Routing
Point

DEVOPS Zirvesi 2017
@gokhansengun
• GameChanger move: It is possible to change mobile app’s endpoint
to route traffic to selected developer box from public internet
Developer Box – Connect Mobile App
Installation
in the Cloud
Proxy
Tunnel
Server
Tunnel Client
App running
locally
Tunnel between
dev-box and the cloud
Mobile App
Isolated Network
w Firewall/NAT
Dev Box

DEVOPS Zirvesi 2017
@gokhansengun
• Performance move: Developer box has lots of Docker images, a
registry mirror VM is used to cache and speed bootstraping up
Developer Box – Registry Mirror
Harbor – Private
RegistryDev Box Registry Mirror Box
Docker Hub
gcr.io
quay.io
etc

@gokhansengun
Immutability
and
Infrastructure as Code (IaC)
Tips and Benefits

DEVOPS Zirvesi 2017
@gokhansengun
• Not everything is container but everything is immutable
• Spinnaker and Harbor have its own VMs and but not containers
• Jenkins has its own VM but container
• All dependencies are included with version no latest tag
• Example: FROM jenkins/jenkins:2.121.1
• Every non-stateful system component is mortal
• May be recreated at any time without any data loss
• Even the whole cluster (k8s et all) is mortal
• Any time a new one could be created and traffic moved over
Immutability and IaC - Tips and Benefits

DEVOPS Zirvesi 2017
@gokhansengun
• Everything including infrastructure is coded
• It allows full-automation
• All system component (Harbor, Spinnaker, Jenkins) configurations
are also coded
• Example: Jenkins is a container
• Plugins coded
• Job definitions coded and kept in git
• Job configurations coded
• Job configurations baked into Jenkins during Docker build
• When run, Jenkins container becomes ready immediately
Immutability and IaC - Tips and Benefits

@gokhansengun
Thanks