When we think about establishing a Kubernetes capability for our organization, our instinct, or perhaps just habit, might lead us to stand up a single cluster that will then be a shared resource across numerous tenants. Kubernetes offers namespaces that are intended to carve up the capacity across different users or groups of users. And while this may work well in some scenarios, it does impose certain constraints and limitations on its use. For example, it is well understood that the multitenancy in Kubernetes is soft, meaning it does not guard against deliberately malicious attacks from one tenant to another. If instead, we align tenant boundaries to Kubernetes clusters, effectively creating many single tenant clusters we can not only avoid certain limitations but we gain some significant advantages. Add a control plane for managing these sets of clusters and we have a powerful solution built on decades of maturity in machine virtualization. In this session we will present both models, multi-tenant clusters and multi-clusters and study the tradeoffs of each.

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved.
Cornelia Davis · @cdavisafc
Sr. Director of Technology
CF Day, Seattle
December 2018
Kubernetes:
One Cluster or Many?

2. Master
Worker Worker Worker
API Server Controller Mgr
DNSScheduler
Kube Proxy Kube Proxy Kube Proxy

3. @cdavisafc
Master
Namespace 2
API Server Controller Mgr
DNSScheduler
Kube Proxy Kube Proxy Kube Proxy
Namespace 1

4. @cdavisafc
Master
Namespace 2
API Server Controller Mgr
DNSScheduler
Kube Proxy Kube Proxy Kube Proxy
Namespace 1
Many Shared Components = soft multitenancy:
● Same configurations (i.e. API
Server, Controller Mgr)
● Ops affects all tenants
● Same version of K8s for all
tenants
Single
Cluster

5. @cdavisafc
The multi-tenancy working
group has been looking at
some of this...

6. @cdavisafc
Master
Namespace 2
API Server Controller Mgr
DNSScheduler
Kube Proxy Kube Proxy Kube Proxy
Namespace 1
Make tenant
aware
OR...
...
...

7. @cdavisafc
Give each tenant their own
components
https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/

8. @cdavisafc
Tenant 1
Master
API Server Controller Mgr
DNSScheduler
Worker
Kube Proxy
Worker
Kube Proxy
Worker
Kube Proxy
...
Single Tenant Clusters
● Virtual Machines for
hard multitenancy
● Limit Blast Radius
● Different Configs (i.e.
API Server, Controller
Mgr.)
● Different versions of
K8s for each tenant
Multiple
Clusters
Tenant 2
Master
API Server Controller Mgr
DNSScheduler
Worker
Kube Proxy
Worker
Kube Proxy
Worker
Kube Proxy

9. @cdavisafc
● Same configurations (i.e. API
Server, Controller Mgr)
● Ops affects all tenants
● Same version of K8s for all
tenants
Single,
multitenant
Cluster
● Virtual Machines for hard
multitenancy
● Limit Blast Radius
● Different Configs (i.e. API
Server, Controller Mgr.)
● Different versions of K8s
for each tenant
Multiple,
single tenant
Clusters

10. Thank you.

11. @cdavisafc
Transforming How The World Builds Software
© Copyright 2018 Pivotal Software, Inc. All rights Reserved.