How to train your micro-services: a few best practices in configuration management and service discovery when using Kubernetes as a container PaaS.

1. Kubernetes - training
micro-dragons without getting
burnt
Amir Moghimi
Senior Consultant in managed services
Sixtree, Australia

2. Traditional approach

3. Micro-services approach

4. Train microservices
Microservices architecture magnifies the need for:
● Fairly homogenous build artifacts
● Standard running platform
● Configuration and secret management
● Service Discovery

5. Polyglot programming
● Pick right tool for the job
● Multiple teams with different expertise/perspectives
● Keep developers busy learning new language(s)

6. Homogenous build artifacts
Build artifacts:
● Java Jar and War files
● Ruby Gems and Rails apps
● Node packages and apps
● Go binaries
Containerise everything (Docker):
● Universally deployable artifact

7. Dockerfile
FROM debian:jessie
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/jre
RUN apt-get update
&& apt-get install -y
openjdk-8-jre-headless
COPY my-app.jar /my-app.jar
ENV MY_APP_CONF_VAR super-cool-default-value
CMD [“java”, “-jar”, “/my-app.jar”]
docker build -t registry/image_name .
docker push registry/image_name

8. Configure and run
docker run -d
-e REDIS_NAMESPACE='staging'
-e POSTGRES_ENV_POSTGRES_PASSWORD='foo'
-e POSTGRES_ENV_POSTGRES_USER='bar'
-e POSTGRES_ENV_DB_NAME='mysite_staging'
-e POSTGRES_ADDR='docker-db-1.us-east-1.rds.amazonaws.com'
-e SITE_URL='staging.mysite.com'
-p 80:80
--restart=on-failure:10
--name container_name
registry/image_name
image_command cmd_arg1 cmd_arg2

9. Configuration hell
● Application config
○ Env vars, config files, cmd line args
● Runtime environment config
○ Web server, JVM
● Runtime dependencies config
○ Volumes, logging, monitoring, stats

10. Configuration management
● Train your app:
○ 12-factor app
● Configuration in a containerised world:
○ Log to stdout
○ Port mappings (from host to container)
○ SaaS blob storage (mount volumes only if providing a storage service)
○ Service discovery (Consul, Eureka, DNS)
○ Secrets (ideally only in memory but how?)
○ Environment Variables for everything else

11. Configuration management tools
● Docker compose
○ State management? Templating? Secrets? Service discovery? Cluster-
level volumes?
● Ansible Role (classic host-based approach + docker module)
● Kubernetes (container PaaS)

12. Kubernetes key resources
● Namespace
● Pod (container)
● Replica Set
● ConfigMap
● Secret
● Service
● Deployment

13. Kubernetes Master
API Server Replica Set
kubelet
Node
Pod
Container
Pod
Container
kubelet
Node
Pod
Container
Kubernetes Cluster
= Label
= Resource
= Process

14. Replica Set (Replication Controller)
apiVersion: v1
kind: ReplicationController
metadata:
name: my-nginx-replica-set
spec:
replicas: 3
selector:
app: dragon-web
template:
metadata:
name: nginx-pod
labels:
app: dragon-web
spec:
containers:
- name: nginx-container
image: nginx
env:
- name: LOG_LEVEL
value: INFO
ports:
- containerPort: 80
apiVersion: v1
kind: Pod
kubectl create -f my-nginx-replica-set.yml

15. ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: dragon-config
labels:
environment: non-prod
data:
dragon.how.much: very
dragon.type: fast
apiVersion: v1
kind: Pod
metadata:
name: dragon-pod
spec:
containers:
- name: dragon-container
image: dragon-image
env:
- name: DRAGON_LEVEL
valueFrom:
configMapKeyRef:
name: dragon-config
key: dragon.how.much
- name: DRAGON_TYPE
valueFrom:
configMapKeyRef:
name: dragon-config
key: dragon.type

16. Secret
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
password: MWYyZDFlMmU2N2RmCg==
username: my_admin
apiVersion: v1
kind: Pod
metadata:
name: secret-user-pod
Spec:
volumes:
name: secret-vol
secret:
secretName: my-secret
containers:
- name: nginx-container
image: nginx
volumeMounts:
name: secret-vol
mountPath: /etc/my-access-keys
readOnly: true

17. Service
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"name": "my-service"
},
"spec": {
"selector": {
"app": "dragon-web"
},
"ports": [{
"protocol": "TCP",
"port": 80,
"targetPort": 80
}]
}
}

18. Service discovery
● DNS
○ Take extra care when playing with fire
○ No control over client
○ Time sensitive protocol
○ Use only if you have a reliable DNS service, i.e. AWS Route53
● Provided environment variables
○ MY_DROGON_SERVICE_HOST=10.0.0.11
MY_DROGON_SERVICE_PORT=8080
○ Create services before using them in pods
○ Only works per namespace
● Kubernetes REST API
○ GET /api/v1/namespaces/{namespace}/services/{service_name}
DNS
HAZARD

19. Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
Declarative
Server-side
Revision tracking
Easy rollback

20. Project structure
my-dragon-microservice/
src/
kube-resources/
default-configmap.yml
default-secret.yml
service.yml
deployment.yml
pipeline-resources/
build.sh
test.sh
deploy.sh # kubectl apply -f ../kube-resources
pipeline.yml
Dockerfile

21. Environments config
dev-1/
namespace.yml
configmap.yml
secret.yml
dev-2/… dev-N/
qa-1/
namespace.yml
configmap.yml
secret.yml
qa-2/… qa-N/
prod-1/
namespace.yml
configmap.yml
secret.yml
prod-2/… prod-N/

22. Demo