The talk gives a state of the art update of experiences with deploying applications in Kubernetes on scale. If in clouds or on premises, Kubernetes took over the leading role as a container operating system. The central paradigm of stateless containers connected to storage and services is the core of Kubernetes. However, it can be extended to distributed databases, Machine Learning, Windows VMs in Kubernetes. All these applications have been considered as edge cases a few years ago, however, are going more and more mainstream today.

1. Three Years Of Running Containers on
Kubernetes in Production
June 13, 2018
OSDC Berlin
thomas@endocode.com

2. HI!
Thomas Fricke
thomas@endocode.com
CTO Endocode
● System Automation
● DevOps
● Cloud, Database and Software
Architect
● K8S since September 2015

3. ENDOCODE
● high-quality software solutions
● best software engineering practices: test driven
● well known open source projects: https://github.com/endocode
● diverse range of technologies
● decades of experience
○ software development,
○ team management
○ 100000s of server years in public and private clouds
● Be it web, mobile, server or desktop we use:
open source to meet any challenge

4. AGENDA
○ The Basics: Containers
○ Kubernetes
■ Pods
■ Networks
■ Services
○ Case Studies
■ Telephony: immmr
■ On Premises: ERP Customer
■ Machine Learning

5. *AAS Pyramide
Functions
App Engine
Containers
Virtual Machines
Google Infrastructure
Event driven functions
Web apps & APIs
GKE
Infrastructure

6. http://www.commitstrip.com/en/2016/06/24/how-to-host-a-coder-dinner-party/

7. CONTAINERS OR VIRTUALIZATION
Topic Container Virtualization
Isolation OS Level,
OS namespaces
CPU Level:
Ring 0/Ring 3
foreign CPU no yes, with emulation
foreign kernels, OS no yes kernel is
common
emulated devices no yes security
host devices direct virtio driver security
CPU performance 100% 95%
IO performance 100% <<100%
root isolation yes yes USER
directive
CPU cache attacks easy possible PoC ?

9. CAN WE MIX CONTAINERS AND HYPERVISORS?
● gVisor
● Intel Clear Container
● kvm in containers
● Docker on non Linux OS
● Mac OS: xhyve
● Windows: Hyper-V

10. LINUX NAMESPACES
Namespace Constant Isolates
Cgroup CLONE_NEWCGROUP Cgroup root directory
IPC CLONE_NEWIPC System V IPC, POSIX message queues
Network CLONE_NEWNET Network devices, stacks, ports, etc.
Mount CLONE_NEWNS Mount points
PID CLONE_NEWPID Process IDs
User CLONE_NEWUSER User and group IDs
UTS CLONE_NEWUTS Hostname and NIS domain name

11. CONTROL GROUPS
● cpuset
● cpu
● cpuacct
● memory
● devices
● freezer
● net_cls
● ns
● blkio
these are directories with fine grained sub folders

12. KERNEL CAPABILITIES
CAP_AUDIT_CONTROL, CAP_AUDIT_READ, CAP_AUDIT_WRITE, CAP_BLOCK_SUSPEND,
CAP_CHOWN,CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER, CAP_FSETID,
CAP_IPC_LOCK, CAP_IPC_OWNER, CAP_KILL, CAP_LEASE, CAP_LINUX_IMMUTABLE,
CAP_MAC_ADMIN,CAP_MAC_OVERRIDE, CAP_MKNOD, CAP_NET_ADMIN,
CAP_NET_BIND_SERVICE, CAP_NET_BROADCAST, CAP_NET_RAW, CAP_SETGID,
CAP_SETFCAP, CAP_SETPCAP, CAP_SETUID, CAP_SYS_ADMIN, CAP_SYS_BOOT,
CAP_SYS_CHROOT, CAP_SYS_MODULE, CAP_SYS_NICE, CAP_SYS_PACCT,
CAP_SYS_PTRACE, CAP_SYS_RAWIO, CAP_SYS_RESOURCE, CAP_SYS_TIME,
CAP_SYS_TTY_CONFIG, CAP_SYSLOG, CAP_WAKE_ALARM, CAP_INIT_EFF_SET
These are a lot! Use profiles to group them together!
man capabilities

13. Greek for “Helmsman”; also the root of the
words “governor” and “cybernetic”
● Runs and manages containers
● Inspired and informed by Google’s
experiences and internal systems
● Supports multiple cloud and bare-metal
environments
● Supports multiple container runtimes
● 100% Open source, written in Go
Manage applications, not machines
Kubernetes

14. kubelet
UI
kubeletCLI
API
users master nodes
The 10000 foot view
etcd
kubelet
scheduler
controllers
apiserver

15. UI
All you really care about
API
Container
Cluster

17. KUBERNETES PODS
● Core Concept the Kubernetes Microservice
● Bunch of Containers with the same
○ Lifecycle: live together, die together
○ Network: same ip address,
same 127.0.0.0/8
○ Volumes: can share data
○ One common task
○ Init Tasks
○ Live and Readiness Checks
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx

18. PODS SHOULD NEVER BE DEPLOYED ALONE
● deployments
○ replicasets
■ pods
● containers
● Jobs: pods that terminate
● Replicationcontrollers, Replicasets,
(don’t use directly)
● Daemonset: one per node
● StatefulSets (aka PetSets): predictable names
○ volumes
○ database clusters
apiVersion: apps/v1 #since 1.9
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

19. Deployment
Replicaset
v1.7.9 v1.7.9 v1.7.9
Replicaset
v1.14.0 v1.14.0 v1.14.0

22. POD in K8S
Pod Centric View
from Roland Huss
https://github.com/ro14nd-talks/kubernetes-patterns

24. The Network
Flannel
● Flannel
○ simplest version
○ IP over IP
● Calico
○ dam’n complicated
○ much more
sophisticated
● Network Policies
○ Security
○ Partitioning a cluster

25. TYPICAL APPLICATIONS
- Programming languages and their runtimes
- Various databases from various generations
- SQL
- NoSQL
- Local and sessions storage
- Message queueing

26. KUBERNETES SERVICES
● Connecting Pods to the
outside world
● identified by the selector
○ key value pair
○ app name
● ports
○ pod
○ node
○ loadbalancer
needs external support
kind: Service
apiVersion: v1
metadata:
name: nginx-service
spec:
selector:
app: nginx
type: Loadbalancer
ports:
- protocol: TCP
port: 80
targetPort: 80

27. ● strictly tiered
architecture
○ separation of
stateless
○ and persistent data
● inside the pods
○ developers are free
to use what they
want
○ contract is binding
to the outside
Network
stateless
Frontend
stateless
WebportalWebportal
Webportal
Frontend
Cache
stateless
Backend
Services
stateless
WebportalWebportalWeb Main
App
WebportalWebportal
Notification
WebportalWebportal
User Profile
Persistent
services
stateful
Networking Endpoints
Dataflow
Bigquery
Storage
Datastore
SQL
Pub Sub

28. MINDSET CHANGE
Ancient Times
● System Engineering
● Hardware Centric
● Virtual Machine
https://12factor.net/
Today
● DevOps
● Application Management
● Managed Service
● Pods

29. FROM VMs TO PODS
OS instances microservices in Pods

30. FROM VMs TO PODS
VM cluster Pods running on Kubernetes
- cattle: stateless containers
- pets:
- databases
- use managed services

31. FROM VMs TO PODS
configuration management separation of build time
and run time

32. BUSINESS VALUE
- faster deployments:
- faster time to market
- more and faster testing
- more teams possible
- faster deployment
- better quality
- less maintenance in operations
- less load
- simpler deployments

33. SUMMARY
Separation of build-time and run-time (12factor.net commandmend V)
- PODs should require only minimal parametrization for being deployed
- Secrets
- Config Maps
- Environment variables
- No Config Management necessary

34. CONTAINER LIFECYCLE MANAGEMENT
Part 1: Build-time related
- Audits, scanning of container content in the registry
- Management of ephemeral configuration
(as in regular scheduled updates of keys, …)
- Stop-gap: rebuild container often, deploy new versions
- Leaner containers
- immutable containers
- incredibly shrinking deployments

35. CONTAINER LIFECYCLE MANAGEMENT
Part 2: Runtime related
- Monitoring of pods, containers and apps/processes
- Lifecycle management
- Cleanup of nodes (minions) after POD end-of-live
- Issue with multi-tenant readiness
- Clean-up, … - issue of isolation beyond individual process (in container)

36. CASE STUDY: IMMMR

37. immmr - one number for every need
immmr combines the best
of Internet base
communication with the
advantages of mobile
communication
immmr makes it possible
to use a single mobile
number from any device
http://www.immmr.com/

38. CHALLENGES
● G10 Compliance
● Every connection must be encrypted
● Vault for Key Management

39. FROM THE TRENCHES
- Easy:
- Java with SpringBoot
- Python
- Hard:
- Ruby Gems
- Separation
- build
- deployment
- no compiler in production
- change to a static Ruby binary traveling ruby
- adapt to database supported by your cloud provider
- ruby version hell: rvm

40. FROM THE TRENCHES
- Lessons learned preparing for a security audit:
- this needed to be done anyway
- separation of stateless and persistent services is
a good idea anyway and with containers really important
- Dockerfiles need careful design to be fast
- private registry for images recommended (same region)
- quay.io
- container life cycle monitoring
- CVE database

41. RESULTS AND EXPERIENCES
- Scalable, kubified application
- Service architecture as it always should have been :-)
- Reduced technical debt and implicit knowledge
- Standardised processes and APIs for services management
- Previously, practises varied between projects
- Pod as deployment unit, single process per container
- Pods are containers sharint the same fate
- Service as load-balanced entry point
- external service
- no LB cluster hassle
- smaller deployments

42. RESILIENCE
High Availibility works
- CoreOS automatic cluster update at immmr
- no outage
- cluster continued to work
- monitoring showed all services available

43. BUSINESS VALUE
- faster deployments:
- faster time to market
- more and faster testing
- more teams possible
- faster deployment
- better quality
- less maintenance in operations
- less load
- simpler deployments
- higher availibility

44. RESULTS AND EXPERIENCES
Separation of build-time and run-time
- PODs: minimal parametrization
- config maps
- Secrets
- Environment variables
- Configuration management inside the container
- build-time issue
- It should not be deployed with the container

45. CASE STUDY: ON PREMISES
ERP COMPANY

46. CHALLENGES
● 1TB machines
● 80 cores
● 80 GB Containers
● Windows in KVM Containers
● Docker socket is a bottle neck

47. WAIT, WAIT, WAIT: DID HE SAY WINDOWS IN K8S?

48. CASE STUDY: MACHINE
LEARNING ON PREMISES

49. kubectl create
-f ml-job.yaml
call
all necessary libraries
config map
apiVersion: v1
kind: Pod
metadata:
name: machine-learning-job
spec:
containers:
- name: machine-learning-job
image: machine-learning
command:
["python","/opt/ml/ml.py" ]
...
tf.device('/cpu:0')
tf.device('/cpu:9')
tf.device('/cpu:1')
TENSORFLOW IN KUBERNETES

50. all necessary librariesapiVersion: v1
kind: Pod
metadata:
name: machine-learning-notebook
spec:
containers:
- name: ml-notebook
image: ml-notebook
command: [“ start-notebook.sh”]
...
Interactive Version
● Jupyter notebook
as Web Frontend
Kubernetes service
● persistent storage
for the model
datasets
JUPYTER NOTEBOOKS

51. 1. Kernel Module creation:
a. time consuming
b. extremely version sensitive
c. needed only once / version
i. distributed in an image (preferred for production)
ii. or in an highly priviledged container (dev and test only)
2. Library Images
a. need development packages
b. can be very time consuming
c. large, should be optimized for large scale rollout
d. might need GPU optimization tweaks
3. Once the images are created, starting a job is a task of minutes
4. ML handles large datasets
5. Interactive Services necessary
6. GPUs don’t have load balancing, but maybe we need a time limit
7. Container image creation itself could be delegated to priviledged containers using the wormhole pattern
8. Distributed Workloads
CHALLENGES

52. wormhole
pattern
docker in
docker
or with
builda
apiVersion: v1
kind: Pod
metadata:
name: debian-jessie-wormhole
spec:
containers:
- image: debian:jessie
command:
- docker
- build
- /library
name: debian-jessie-wormhole
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-socket
- mountPath: /bin/docker
name: docker-binary
...
bokeh
coverage
decorator
discover
docopt
jinja2
matplotlib
mysql-connector-python
nose
numpy
pyftpdlib
scipy
SQLAlchemy
sqlacodegen
xmltodict
zipline
requirements.txt
config map
build container
build
library image
BUILD WORKFLOW

53. TF Worker NodesTF Parameter ServerTF Client
tensorflow::Session
worker_service.proto
https://www.tensorflow.org/deploy/distributed
https://github.com/dask/dask
DISTRIBUTED TENSORFLOW

54. CASE STUDY: BIG NETWORK
BIG PRICE COMPARISON SITE

56. Compute Node
Kubernetes Layer
Calico
CNI
Plugin
KernelIP TablesIP Routes
RR Route
Reflector
BIRD
BGP
Felix
RR Route
ReflectorRR Route
Reflector
eth0 eth1
Physical fabric (L2, L3, MPLS)
Pods
Pods
Pods
Pods
Pods
Pods
Pods
Pods
Pods
Pods
CALICO Network

57. QUESTIONS?
Watch our Webinar ‘Dive into Kubernetes’ on our YouTube Channel
https://youtu.be/8694GGJlpZ8
Register for a free Google Cloud Platform Trial with $300 Google Cloud
Platform Credits
https://goo.gl/dUzDWi
Use another $200 partner credits
https://goo.gl/eYldnT