Today there's a multitude of ways to get up and running with Kubernetes in the Cloud. In this talk we'll look at how easy it is to operationalize your K8s cluster deployments using the new gcontainer puppet module for Google Container Engine (GKE), Google’s Managed Kubernetes service. We'll walk you through an end to end deployment of a demo application using the gcontainer puppet module and the kubernetes module. We'll also take a deep dive into the unique value proposition that GKE brings to Kubernetes deployments, including security, scaling, federation, automated container builds, integrated private container registry and GPUs.

1. Kubernetes in the Cloud
with Puppet and Google
Container Engine

2. Mandy Waite
Ops&Infra Advocacy TLM @ Google
@tekgrrl

3. Coping with rapid growth
December
1998
January
1999
November
1998
February
1999
March 1999 April 1999
April 1999
500,000!
January 1999
150,000
December 1998
50,000
Google Search Queries per Day:
from 50,000 to 500,000 in 5 months!

5. ControlPlane
Apps

6. Chroot
BSD Jails
Solaris Zones

7. Encapsulate the application environment
“Images” - Statically Linked Bundles

8. Linux Application Containers
cgroups, namespaces, capabilities, chroots

9. Isolate applications from the heterogeneous
environments on which they run
Signed static bundles + Linux containers

10. BorgMaster
link shard
UI
shardBorgMaster
link shard
UI
shardBorgMaster
link shard
UI
shardBorgMaster
link shard
UI
shard
Schedulerscheduler
Borglet Borglet Borglet Borglet
BorgMaster
link shard
UI
shard
persistent store
(Paxos)
“The system internally we call Borg”

11. Global Work Queue
Babysitter
Batch Workloads
Production Workloads

12. Borg Cell
Batch Workloads
Production Workloads

13. task-eviction rates and causes

14. tasks per machine
Multiple
applications
per machine
CPI^2 paper,
EuroSys 2013

15. shared cell
(original)
shared cell
(compacted)
# machines
Sharing clusters
between
prod/batch helps
Segregating them would need
more machines

16. # machines
16
Sharing clusters
between
prod/batch helps
Segregating them would need
more machines
shared cell
(original)
shared cell
(compacted)
non-prod load
(compacted)
prod-only load
(compacted)
overhead

17. Resource reclamation
time
limit: amount of resource
requested
usage: actual resource
consumption

18. Resource reclamation
time
limit: amount of resource
requested
usage: actual resource
consumption
potentially reusable
resources reservation: estimate of
future usage

19. Resource reclamation
time
limit: amount of resource
requested
usage: actual resource
consumption
potentially reusable
resources reservation: estimate of
future usage

20. stranded resources
available resources
one
machine
Advanced
bin-packing
algorithms
Experimental placement
of production VM
workload, July 2014

21. Images by Connie
Zhou
Observations:
● Efficiency comes from
○ Scavenging unused allocations
○ Effective Prioritization
○ Sharing resources
○ Overcommit
○ Smarter Scheduling
● Application-centric, not machine-centric view
It is easier, more natural, and more productive
● Over 2B containers launched per week http://kubernetes.io
http://goo.gl/1C4nuo (Borg paper)

22. Kubernetes

23. We needed something like Containers in
order to make Borg possible

24. WeneededsomethinglikeContainersin
ordertomakeBorgpossible
You needed something like Kubernetes in
order to make Containers practical

25. ● Lightweight
● Hermetically sealed
● Isolated
● Easily deployable
● Introspectable
● Runnable
Containers: a quick recap
Linux processes
● Improves overall developer experience
● Fosters code and component reuse
● Simplifies operations for cloud native applications
Docker

26. Containers are awesome
you’ll want to run lots of them...

27. Greek for “Helmsman”; also the root of the
words “governor” and “cybernetic”
• Runs and manages containers
• Inspired and informed by Google’s experiences
and internal systems
• Supports multiple cloud and bare-metal
environments
• Supports multiple container runtimes
• 100% Open source, written in Go
Manage applications, not machines
Kubernetes

28. API

29. API

30. API

31. API

32. API
On Premise

33. Servers
ControlPlane
kubelet
UI
CLI
API
kubelet
kubelet
Cluster
Developers

34. Advanced Scheduling Capabilities
● Admission Control
● Resource based scheduling
● Affinity and Anti Affinity
● Bin Packing
Self-healing
Service discovery and load balancing
Horizontal scaling
Storage orchestration
Batch Execution
Secret and configuration management
Kubernetes Features

35. Google Container Engine

36. Google manages your control plane
Container Engine
Kubernetes Master
API Server
Controller Manager
Scheduler
etcd
● Daily Backups
● Monitoring, health checks,
auto repairs
● Restarts
● Resizing for larger clusters
● Auto upgrades
● 99.95% SLA

37. ...and system components on your nodes
Container Engine
Kubernetes Master
API Server
Controller Manager
Scheduler
etcd
Container Engine
Kubernetes Nodes
Logging
Monitoring
Ingress backend
Runtimes

38. Node management features
Node upgrade:
● Update Kubernetes
version
● Update node OS
Node repair:
● Automatically repair
broken nodes
Container Engine
Kubernetes Nodes
Logging
Monitoring
Ingress backend
Runtimes

40. Full audit trail
List operations
Describe an operation to get
more details
$ gcloud container operations list
NAME TYPE .. STATUS
operation-15-0a UPGRADE_MASTER .. DONE
operation-11-78 AUTO_UPGRADE_NODES .. DONE
$ gcloud container operations describe <op>
name: operation-15-0a
operationType: UPGRADE_MASTER
selfLink: ...
status: DONE
targetLink: ...
zone: us-west1-b

42. Lets Spin up a Cluster!

43. gcontainer module

44. Puppet Modules
google-gcompute - manages Google Compute Engine resources (VMs,
Disks, Networks)
google-gstorage - manages Google Cloud Storage resources (storage
buckets, ACLs)
google-gsql - manages Google Cloud SQL resources (MySQL instances, DBs,
Users)
google-gdns - manages Google Cloud DNS resources (Resource Records)

45. Puppet Modules (cont.)
google-gauth - provides the types to authenticate with Google Cloud
Platform
google-gcontainer - manages Google Container Engine resources (K8s
Clusters)
google-cloud - convenience to install all Google Cloud Platform modules

46. gauth_credential
path
provider
scopes
Google Container Engine - gauth module

47. gauth_credential
path
provider
scopes
Google Container Engine - gauth module
● serviceaccount The preferred method of specifying
credentials, does not rely on any pre-existing system
configuration that Puppet can't track. You'll need a
credential file to use this.
● defaultuseraccount If you have Google Cloud SDK setup
you can piggyback on the account currently set as active
for the user running Puppet.

48. gauth_credential
path
provider
scopes
Google Container Engine - gauth module
● The scopes your authentication request will be limited to.
● When executing actions against Google Cloud Platform,
choose the minimum privileges needed to perform those
actions so as to avoid accidentally affecting other
resources.
● For example if you want to manage virtual machines you
should request only "Compute R/W". That way you don't
accidentally modify your DNS records.

49. gcontainer_cluster
initial_node_count
node_config
machine_type
Disk_size_gb
zone
project
cred
...
Google Container Engine - gcontainer module
Fully Managed
K8s master
Default Node
Pool
Context
Project
Zone
Credentials (Service Acct)
Create
cluster

50. gcontainer_node_pool
initial_node_count
config
machine_type
Disk_size_gb
autoscaling
cluster
zone
project
cred
...
Google Container Engine - gcontainer module
Node Pool
Context
Cluster
Project
Zone
Credentials (Service Acct)
Create
Node pool
Fully Managed
K8s master
Default Node
Pool

51. Demo

52. Cluster Autoscaler (Beta)
Node Pool
create/
destroy
VMs
VMVM
Node Pool
Manager
actuator
Node
monitor
Pods Pending
Autoscaler
Control Plane
● Add/remove nodes based on
pod scheduling needs
● Specify max and min nodes
● Can scale node pools to 0

53. Preemptible VM Instances
● What Preemptible VMs are
○ Up to 80% cheaper than regular VMs. (~$0.01 per core hour)
○ Very easy to use -- just flip one switch in the UI, API or command line
○ Many of our biggest customers run huge clusters (10k+ cores) with great success and
savings.
● Things to keep in mind
○ Same great disk, OS images and network
○ Google Compute Engine can preempt (i.e. shutdown/take-away) the VM with 30
seconds of notice
○ Maximum 24 hours of uptime
○ No SLAs or guarantees of any kind but we historically see preemption rates of 5-15%

54. New and coming
● Per Second Billing (available now!)
● HA, multi-master Container Engine clusters (99.99% SLA)
● Run across multiple zones within a region (protects from zonal failures)
● Kubernetes Node Problem Detector
● Node Auto-Upgrade
● Maintenance Windows
● Cluster Autoscaling up to 1000 nodes
● NVIDIA Tesla P100 GPUs are available in alpha clusters

55. Resources
Puppet Forge: google - forge.puppet.com/google
Modules on GitHub:
github.com/GoogleCloudPlatform/puppet-google
github.com/GoogleCloudPlatform/puppet-google-auth
github.com/GoogleCloudPlatform/puppet-google-container
Google Container Engine - cloud.google.com/container-engine/
Kubernetes - kubernetes.io

56. Thank You!