Speakers: Katharina Probst, Shahid Masud, Karl Isenberg Venue: Google Cloud Next '19 Video: https://www.youtube.com/watch?v=fuHc8uwA4mA Google Kubernetes Engine has introduced a number of features to improve setting up a cluster to support multi-tenancy. These include policy management support for RBAC and other policies, usage metering, external pod identity, and sandbox pods. This session will talk about how to effectively use these features to create a cluster that can be shared by multiple tenants to run their workloads.

1. HYB214: Cost-Effective
Cluster Sharing With
GKE Multi-Tenancy
Shahid Masud - Strategic Cloud Engineer, Google Cloud
Katharina Probst - Engineering Manager, Google Cloud
Karl Isenberg - Tech Lead Manager, Cruise Automation

2. What companies care about
Cost Velocity

3. Why
What and How
Customer Perspective
1
2
3
Katharina Probst
Senior Engineering Manager, Google
Shahid Masud
Strategic Cloud Engineer, Google
Karl Isenberg
Tech Lead and Manager, Cruise Automation

4. 01
Why
Multi-Tenancy?

5. Kubernetes at a glance
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES

6. masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
One user, one cluster

7. Master 1User 1
kubelet
CLI/API/UI kubelet
Cluster 1
Master 2User 2
kubelet
CLI/API/UI
kubelet
kubelet
Cluster 2
Multiple users, multiple clusters
kubelet

8. kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Clusterkubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
How does this scale financially?

9. kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
How does this scale operationally?

10. master
user
ns3-pod1
CLI/API/UI
ns2-pod1
ns2-pod2
Cluster
user
user
user
ns1-pod1
ns1-pod2
ns3-pod2
ns3-pod3
Namespace 1
namespace
Namespace 2
Namespace 3
Alternative: Many users, one cluster

11. Namespace 3Namespace 2Namespace 1
Team A Team B Team C
Shared
Platform
Kubernetes Engine Clusters (shared)
App X App Y App Z
Production Readiness Tooling (shared)
Shared platform for production readiness

12. 02
The "What" of
Multi-Tenancy

13. Provide isolation and
fair resource sharing between
multiple users and their workloads
within a single cluster
Kubernetes Engine Multi-Tenancy

14. Trusted Semi-trusted Non-trusted
Teams within a company Platform providers
Hosting providers
Teams within a company
SaaS providers
How much do you trust the tenants?

15. Defense in depth to your pods
Second layer of defense between
containerized workloads in GKE
based on gVisor.
Defense-in-depth security
principles without application
changes, new architecture models,
or added complexity. Check out
HYB216: GKE Sandbox for
Multi-Tenancy and Security
April 11 | 9–9:50 AM
GKE Sandbox Beta

16. Trusted Semi-trusted Non-trusted
Teams within a company Platform providers
Hosting providers
Teams within a company
SaaS providers
How much do you trust the tenants?

17. 03
The “How” of
Multi-Tenancy

18. Multi-tenancy primitives
Access
Control
Resource
Sharing
InsightsRuntime
Isolation
.
1 2 3 4

19. Multi-tenancy primitives
Access
Control
Resource
Sharing
InsightsRuntime
Isolation
1 2 3 4

20. Practical for giving Google users/groups project-wide access:
Curated IAM “Roles”:
Kubernetes Engine Admin *Can do everything*
Kubernetes Engine Cluster Viewer *Can view everything*
Kubernetes Engine Cluster Admin Can manage clusters (create/delete/upgrade clusters)
Cannot view what's in the clusters (Kubernetes API)
Kubernetes Engine Developer Can do everything in a cluster (Kubernetes API)
Cannot manage clusters (create/delete/upgrade clusters)
You can curate new ones with Cloud IAM Custom Roles.
Projects + IAM

21. Control Plane (apiserver) Authorizer
Pluggable Auth
(GKE IAM)
RBAC
Admission
Control
allow etcd
Cloud IAM
Policies
{Cluster,}Role
{Cluster,}RoleBinding
allow
Pods
Authentication, Authorization, & Admission

22. Useful for:
● Giving access to pods calling Kubernetes API (with Kubernetes Service Accounts)
● Giving fine-grained access to people/groups calling Kubernetes API (with Google accounts)
Concepts:
ClusterRole A pre-set of capabilities, cluster-wide
Role ClusterRole, but namespace-scoped
ClusterRoleBinding Give permissions defined in a ClusterRole to:
● Google users/groups Beta
● Google Cloud IAM Service Accounts
● Kubernetes Service Accounts
RoleBinding ClusterRoleBinding, but namespace-scoped.
Role-Based Access Control (RBAC)

23. ● Intercept API request before resource is persisted
● Mutate/change resources and allow/deny requests
Useful for:
● Policy enforcement
● Adding defaults values to resources
Admission
Control
etcd
Admission
Plugins
allow
Kubernetes Admission Control

24. Let Google manage the service accounts that are used by your Kubernetes
workloads to access GCP services
Replaces existing workarounds like:
● Using node (VM) identity for the pod
● Download Service Account key and provide as Secret to the pod
Check out
HYB317: Keyless Entry:
Securely Access GCP
Services From Kubernetes
April 10 | 2:10–3:00 PM
Workload Identity Beta (soon)

25. Multi-tenancy primitives
Access
Control
Resource
Sharing
InsightsRuntime
Isolation
1 2 3 4

26. Resource Sharing
● Quotas
● Limit Range
● Pod Affinity/Anti-affinity
● Pod Priority

27. Multi-tenancy primitives
Access
Control
Resource
Sharing
InsightsRuntime
Isolation
.
1 2 3 4

28. Runtime Isolation
● Sandbox Pods
● Pod Security Policy
● Pod Security Context
● Network Policy

29. Team A
Pods
Pods
PodsPods
Pods
Pods
Team B
ns:teama
ns:teama
ns:teama
ns:teamb
ns:teamb
ns:teamb
cluster 1
cluster 2
cluster 3
How to manage and apply all the policies?

30. ● Centrally declare entire policy configuration
● Manage policy configuration like code
● Policy applied to all clusters
● Declarative and continuous reconciliation
Anthos Config Management Beta

31. ● Cluster operator/admin owns the
infrastructure and cares about...
○ PodSecurityPolicy
○ NetworkPolicy
○ ResourceQuota
○ Roles & ClusterRoles
○ DaemonSets
● Dev teams own apps that run on the
clusters
Typical Use Case

32. Branch Validate Review Deploy
Check out
HYB315: Secure Policy
Management for the Cloud
Services Platform
April 11 | 9–9:50 AM
Change Management for Policy Config

33. Multi-tenancy primitives
Access
Control
Resource
Sharing
InsightsRuntime
Isolation
.
1 2 3 4

34. ● View workloads’ resource usage broken down by namespace
and labels
○ memory, CPU, GPU, PD, network, etc.
● Join usage data with GCP Billing Export data to compute
resource costs per tenant
● Docs:
https://cloud.google.com/kubernetes-engine/docs/how-to/c
luster-usage-metering
GKE Usage Metering Beta

35. GKE Usage
Metering Beta

36. 04
Customer
Perspective:
Multi-Tenancy
at Cruise

37. We’re building the world’s most advanced
self-driving vehicles to safely connect people with
the places, things, and experiences they care about.
https://cruise-automation.github.io/webviz/worldview/https://getcruise.com/

38. @karlkfi
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Clusterkubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
kubelet
kubelet
kubeletMasterUser CLI/API/UI
Cluster
Cluster Proliferation
(circa 2017)

39. Workload Variety
Containers
● Web Applications
● Backend Services
● Data Processing
● Reporting Batch Jobs
● Productivity Tools
● Security Tools
● Continuous Deployment
● Platform Addons
@karlkfi
Virtual Machines & SaaS
● Databases
● Continuous Integration
● Simulation Pipelines
● Data Processing

40. Multi-Tenant Reasoning
@karlkfi
Cost Savings
● Fewer Clusters
● Larger Nodes
● Cheaper to Change
● Easier to Upgrade
● Concentrated Expertise
● Offload Work from
SREs/NetEng
Production Readiness
● Easier to Audit
● Easier to Enforce Policies
● Environments
(dev/stage/prod)
● Central Observability
● Built-in Backups
● Autoscaling
Developer Velocity
● Ingress/Egress
● Config Management
● Image Building
● Continuous Deployment

41. Hybrid Cloud Private Networking
@karlkfi
SubNetwork (us-west1)
Cloud Router
Interconnect
SubNetwork
Router
VPN
On-Premises
Network
Cloud Routes
SubNetwork (us-central1)
Cloud Router
SubNetwork
Router
Firewall RulesPrivate Cloud
DNS
Virtual Private
Network
Cloud NAT
Cloud RoutesFirewall RulesPrivate Cloud
DNS
Cloud NAT

42. GCP Environments
@karlkfi
Dev Staging Prod
NetEng
Team X
PaaS
Kubernetes
Engine
Shared
Private
Network
Cloud SQL
Kubernetes
Engine
Shared
Private
Network
Cloud SQL
Kubernetes
Engine
Shared
Private
Network
Cloud SQL
Projects
Folders & Networks

43. GKE Environments
@karlkfi
Pods
Dev Staging Prod
Team A
Team B
Team C Pods Pods
Pods Pods Pods
Pods Pods Pods
Namespaces
Clusters

44. Ingre
@karlkfi
RBACSync
GKE
Knative Build
(todo)
Spinnaker
Ingress Controller
External DNS
(todo)
Runscope
Stackdriver
Logging
Public Cloud
Load Balancer
Private Cloud
DNS
Internal
Load Balancer
Cloud Armor
Policies
External IPs
Container
Registry
Cruise PaaS
Ingress
Security
Deployment
Observability

45. In-Memory
Volume
DAYTONA
Init Container
App Container
Kubernetes Pod
Secrets
Login
Vault Integration
(Internal) cruise/daytona@karlkfi
Vault Login
Kubernetes service accounts used
for Vault authentication.
Secrets Injection
Init container side-loads secrets
GCP Service Accounts
Vault generates temporary
credentials on-demand

46. Vault Namespacing
@karlkfi
Group Permissions Path
Team Admin admin secret/<prefix>/<namespace>/*
Team Contractor list secret/<prefix>/<namespace>/*
App Service Account list, get secret/<prefix>/<namespace>/<env>/<app>/*

47. apiVersion: rbacsync.getcruise.com/v1alpha
kind: RBACSyncConfig
metadata:
name: namespace-bindings
namespace: backend
spec:
bindings:
- group: backend-gke-admin@example.com
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: namespace-admin
- group: backend-gke-user@example.com
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: namespace-editor
(OSS) https://github.com/cruise-automation/rbacsync
RBACSyncConfig
ClusterRoleBindingRoleBinding
Group Role Binding
ClusterRoleRole
Groups
GSuite
RBACSync
@karlkfi

48. Unsolved Challenges
Integration Tenancy
● Kubernetes Operators
● Google Container Registry
● Stackdriver Logging
● DataDog Metrics
● Spinnaker
Resource Isolation
● Local Disk Space
● Disk I/O
● Ingress
● Egress
@karlkfi

49. Cruise PaaS Future
● Team-Isolation -> Project-Isolation
● Quota per Namespace (ResourceQuota)
● Default Resource Limits (LimitRange)
● Security Validator (webhooks & policies)
● Image Builder Service (automated builds)
● PodSecurityPolicy
● Network Microsegmentation (Istio, NetworkPolicy)
● Team/Project Usage Metering
● Self-Service E2E TLS w/ Automatic Certificate Rotation
● Service Identity & RBAC (Istio, Workload Identity)
@karlkfi

50. If you’re big enough to build a
platform on top of Kubernetes,
you’re big enough to make it
multi-tenant.
@karlkfi

51. 05
Conclusion

52. Multi-Tenancy:
● Promotes operational
Efficiency
● Improves developer
velocity
● Saves infrastructure
costs
GKE features make is easier
to setup multi-tenant
clusters
master
user
ns3-pod1
CLI/API/UI
ns2-pod1
ns2-pod2
Cluster
user
user
user
ns1-pod1
ns1-pod2
ns3-pod2
ns3-pod3
Namespace 1
namespace
Namespace 2
Namespace 3

53. Questions?
Compose a question Vote to move great
questions to the top
Click the Dory Q&A link
1. Open the Cloud Next app
2. Tap a session
3. Click Dory Q&A

54. Your Feedback is Greatly Appreciated!
Complete the
session survey
in mobile app
1-5 star rating
system
Open field for
comments
Rate icon in
status bar