Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Assis, JFrog - Cloud Native Day Tel Aviv 2018

Copyright @ 2018 JFrog - All rights reserved
Kubernetes is hard!
Lessons learned taking our apps
to Kubernetes
Eldad Assis | DevOps Architect | JFrog

Copyright @ 2018 JFrog - All rights reserved. CloudNative Israel 2018
Me
Email: eldada@jfrog.com
Twitter: @eldadak
LinkedIn: Eldad Assis
What I do
- I solve problems @ JFrog
- Bringing Dev and Ops closer for over 15 years
- Doing CI/CD since the turn of the century
- Automation, automation and… automation!

● Docker
Containers
● Kubernetes
An open-source system for automating deployment, scaling, and
management of containerized applications.
No deep diving. No Demo… sorry
Come see me after the talk for more!
For this talk, I assume you know a bit
Recommended knowledge

1. Spin up a fully functional, big or small, environment with our products
a. Developer, QA, Support, Product, Solution… anyone!
b. JFrog Enterprise Plus
2. Per branch CI/CD
a. Integrate my branch with other products development branches
3. Better utilize our resources
4. Support a new distribution for JFrog products
Our journey to Kubernetes
The problems we wanted to solve

● Start small
● Get your application ready before jumping into Kubernetes
● Security - know what’s running in your cluster
● Limits
● Health probes
● Visibility - usable and accessible monitoring and logging system
● Work with the community!
The End
What you should take from here

Kubernetes - the myth
ZZZZ….

Kubernetes - the reality
Hmmm….

● Start with a small application example (nginx)
● Use existing demos
○ Understand them. What does each line actually means
● Set a minimal goal for getting your app to run in Kubernetes
● Get comfortable before you move on
○ Use managed K8s to free yourself of setups (AKS, EKS, GKE)
Where should you start?
Start small!

A lot has to be done on your application before you can
comfortably run it in Kubernetes
Here are some key points to consider when planning your
move to Kubernetes
Start with the application!
Is your application ready to run in Kubernetes?

● Logging
○ STDOUT/STDERR
○ Handling multiple log files
● Data Persistency
○ What kind of data needs persistency (if at all)?
● Proper handling of termination signals to init a proper shutdown
○ Controlled shutdown of the application
○ Easier recovery (after a controlled shutdown)
● Survive a restart
○ Managing leftovers from previous run
Is your app ready for Kubernetes?
It’s not just pushing it to Docker...

● Improve durability and availability
● Support running multiple instances of your application with load
balancing between them
○ Scaling up and down will be easy (horizontal scaling)
● Rolling upgrades for zero downtime
○ Backward compatibility
● Zero service unavailability due to pod rescheduling
○ Cluster scaling (down)
○ Pod evicted or crashed
○ Unplanned outage of a node
High availability as the new default!

I. Codebase - One codebase tracked in revision control, many deploys
II. Dependencies - Explicitly declare and isolate dependencies
III. Config - Store config in the environment
IV. Backing services - Treat backing services as attached resources
V. Build, release, run - Strictly separate build and run stages
VI. Processes - Execute the app as one or more stateless processes
The Twelve-Factor App (https://12factor.net/)
VII. Port binding - Export services via port binding
VIII. Concurrency - Scale out via the process model
IX. Disposability - Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity - Keep development, staging, and production as similar as possible
XI. Logs - Treat logs as event streams
XII. Admin processes - Run admin/management tasks as one-off processes

● Your application is rarely the only thing in its container
○ OS packages, OSS libs, 3rd party processes
● Security vulnerabilities
● FOSS licenses compliance
Security
What’s running with your application?

Once your application is ready, let’s talk about to how to properly
manage your applications run-time and configuration in Kubernetes
And now - Kubernetes
Properly running in Kubernetes

● Pod limits. Always!
● The app inside might need its own limits. For example:
○ Java apps
■ -Xms=1g -Xmx=2g
○ RabbitMQ
■ [rabbitmq.conf]
total_memory_available_override_value = 1GB
○ MongoDB
■ --wiredTigerCacheSizeGB=1
Know your limits
There might be more than you know….
…
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "250m"
...

● The total sum of limits on a node can be way over 100%
● This allows for resource sharing between pods
● Pods will crash! Nodes will crash!
● See out of resources handling by Kubernetes
● Be prepared
○ Think about the requested and limits values
○ Spread your application across nodes using multiple
replicas (HA)
○ Use pods priority and preemption to take control
Know your limits - don’t overload
There might be more than you know….
…
resources:
requests:
memory: "64Mi"
cpu: "20m"
limits:
memory: "2Gi"
cpu: "4"
...

● Your application must have a reliable metric for health
● readinessProbe
○ Is the app ready to start serving
● livenessProbe
○ Is the app good to continue serving
● Types
○ Exec - return 0 on success
○ Http - return < 400 on success
○ Tcp - succeed to open a socket on a given port
● For complex checks, write a script and use the exec probe
Know your app’s health
Application’s readiness and health
…
readinessProbe:
httpGet:
path: /api/system/health
port: 8080
…
livenessProbe:
exec:
command:
- mongo
- --eval
- "db.adminCommand('ping')"
…
livenessProbe:
tcpSocket:
port: 5672
…

● Init Containers - run before main container
○ Prepare storage
○ Setup configuration
● Sidecar containers - run alongside main container
○ Maintenance
○ Log collection
○ Monitoring
○ Proxy
Multiple containers in a Pod
When a single container is not enough
Application pod example.
Multiple artifactory logs forwarded by a
Fluentbit container to a log aggregator.
Pod
Application
container
Fluentbit
container
Logs
Log
collector

● Your application is made up of multiple components
○ Each represented as a yaml file or snippet
● Versioning of an application that’s made up of several yaml files is
challenging
● Having all configuration in the yaml files is great, but
○ How can I use the same yaml for different environments?
■ Local (minikube)
■ Dev cluster
■ Production
● Rolling back to earlier versions of the application
Managing an application’s lifecycle
The static nature of the yaml descriptors is challenging

● https://helm.sh/
● Helm is the package manager for Kubernetes. Like ‘yum’ for
CentOS/RedHat
● Your whole application described in a single package - helm
chart (template yamls)
● Default configuration values (values.yaml)
● Single version for every chart (Chart.yaml)
Here comes Helm!
Dynamic control over your application’s deployment

● Use same chart for dev,staging and production!
○ Each environment’s config managed in its own copy of values.yaml that is version
controlled (values-stg.yaml, values-prod.yaml)
○ The default values.yaml should be for dev or local, so a developer can use it locally without
hassle
○ Everything is configurable!
● External charts as requirements (dependencies)
○ Databases
○ Other 3rd parties
Helm - YES!
Use same chart in all environments

Helm - YES! YES!
Useful helm commands for understanding what the helm is going on...
# Lint your chart for errors and recommendations
$ helm lint <chart path>
# Download a chart for local viewing
$ helm fetch <chart>
# Get a release (application already deployed with helm) actual configuration
$ helm get <release>
# Get the status of all the resources included in a release
$ helm status <release>
# Get the actual resolved configuration without deploying anything
$ helm template <chart>
$ helm install --debug --dry-run <chart>
# Test your release (need to write test pods in your chart)
$ helm test <release>

More Helm charts
Find existing charts
https://hub.kubeapps.com/

● No more “ssh to the server and get me the logs”!
● Developers should not need kubectl access to debug their
applications
● Provide your Dev and Ops easy and reliable data
○ Monitoring
○ Logging
● Managed solutions provide OOB tooling
Visibility in Kubernetes?
We are not in Kansas anymore...

Dashboard for nodes
and pods
* Prometheus
* Grafana
Monitoring

Logs from all pods
EFK stack
* Fluentd
* ElasticSearch
* Kibana
Logging

Use with your CI
Plug your CI/CD to a Kubernetes cluster for easy environment deployment

● Development CI/CD our products to K8s for testing
○ Using our official Helm charts
○ On demand, fully isolated environment per branch per developer
○ 100’s of custom testing environments a week made up of ~50+ services
● Started moving our cloud based applications to K8s
○ Internal and external
● Official JFrog Helm charts for all our products
JFrog and Kubernetes
What are we doing with Kubernetes?

● You are not the first one to stumble on that particular problem
● It’s as if you have more developers in your team
● Fast turnaround
● Examples
○ RabbitMQ HA
○ MongoDB (Bitnami)
The community
Use already tested and managed helm charts

● Start small
● Get your application ready before jumping into Kubernetes
● Security - know what’s running in your cluster
● Limits
● Health probes
● Visibility - usable and accessible monitoring and logging system
● Work with the community!
The (real) End
What you should take from here

Be Good!
join.jfrog.com

Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Assis, JFrog - Cloud Native Day Tel Aviv 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Assis, JFrog - Cloud Native Day Tel Aviv 2018

Similar to Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Assis, JFrog - Cloud Native Day Tel Aviv 2018 (20)

More from Cloud Native Day Tel Aviv

More from Cloud Native Day Tel Aviv (20)

Recently uploaded

Recently uploaded (20)

Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Assis, JFrog - Cloud Native Day Tel Aviv 2018