https://alterbooth.connpass.com/event/109272/ での発表資料です

1. Rancher 2.X
First Step before Deep Dive
LINE Corporation, Verda2 Yuki Nishiwaki

2. Who you are?
Name:
● Yuki Nishiwaki
Working Experience:
● Private Cloud Development (OpenStack)
Jan 2015 -> Now
● Kubernetes as a Service Development (Rancher)
July 2018 -> Now
● Function as a Service Development (Knative)
Sep 2018 -> Now

3. Rancher and Me
● User and Developer
● Have contributed few patches to rancher 2.X related projects
○ rancher/types
■ https://github.com/rancher/types/pull/525
○ rancher/machine
■ https://github.com/rancher/machine/pull/12
○ rancher/norman
■ https://github.com/rancher/norman/pull/201
■ https://github.com/rancher/norman/pull/202
■ https://github.com/rancher/norman/pull/203
○ rancher/rancher
■ https://github.com/rancher/rancher/pull/15909
■ https://github.com/rancher/rancher/pull/15991
■ https://github.com/rancher/rancher/pull/16044
Not so many contributions yet

4. Today’s scope
● We have only 20-30 min
● Rancher have bunch of features. It will take half of day if I talk all parts
● Try to explain whole picture
● Check following article for more detail
○ https://www.slideshare.net/linecorp/lets-unbox-rancher-20-v200
○ https://github.com/ukinau/rancher-analyse

5. Rancher 2.X ?
All Data for Rancher are stored in Kubernetes
=> Depend on Kubernetes to run Rancher
1
Manage Kubernetes Cluster by multiple Provider
● Google Container Engine
● Amazon EKS
● Microsoft Azure
● Any VM Provider like OpenStack (RKE)
2
RBAC Configuration
Configure/Deploy
Additional Tools on Cluster
Addon Management
4
Resource
Resource
Resource
Admin member
RBAC Configuration
3
Sync RBAC configuration

6. Rancher 2.X

7. Rancher 2.X architecture
API Controller
Cluster Agent
Node Agent
Node Agent
Node Agent
Cluster Agent
Node Agent
Node Agent
Node Agent
Kubernetes Cluster
Kubernetes Cluster
Kubernetes Cluster

8. Resource Creation process is same as Kubernetes Resource
API Controller
ClusterA
Watch
Kubernetes ClusterReconcile
Get latest
information from
kube-apiserver
Check if any
difference
Between desired and
actual states
Do something to make
actual state desired
Reconcile
Loop
Cluster Agent
Node Agent
Don’t do actual provisioning

9. Resource Creation process is same as Kubernetes Resource
API Controller
ClusterA
Watch
Kubernetes ClusterReconcile
Get latest
information from
kube-apiserver
Check if any
difference
Between desired and
actual states
Do something to make
actual state desired
Reconcile
Loop
Cluster Agent
Node Agent

10. Rancher API is just kind of proxy for Kubernetes API
ClusterA NodeA UserA GroupA ・・・
Create Cluster
Store All data as a
Kubernetes by using CRD
CRD for Cluster CRD for Node CRD for User
Call Kubernetes API

11. Custom Resource Definition(CRD) in Kubernetes?
Kubernetes Native Resource Type Custom Resource Type
CustomResourceDefinition
ConfigMap
Pod
Nginx App A
Nginx Config
Cluster Node
Cluster
Node
Cluster A Cluster B
Node A Node B
Kubernetes allow user to create custom resource type in
addition to natively supported resource.

12. > kubectl get crd clusters.management.cattle.io -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
creationTimestamp: 2018-10-26T13:49:37Z
generation: 1
name: clusters.management.cattle.io
resourceVersion: "1278"
selfLink:
/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/clusters.management.cattle.io
uid: fa628204-d925-11e8-b840-fa163e305e2c
spec:
group: management.cattle.io
names:
kind: Cluster
listKind: ClusterList
plural: clusters
singular: cluster
scope: Cluster
version: v3
> kubectl get cluster
NAME AGE
local 1d
Example of CRD for Rancher Resource: Cluster
CRD for Cluster
Cluster Resource

13. Question: Do we really needed to have API server?
Create
Create
Why not Create Cluster Resource directly
?
Resource information tend to be large and some
of the attributes are only needed internally and
don’t have to be exposed to user.
Thus, Rancher mutate resource before try to
create resource in k8s to add some internal
information and also mutate resource before try
to return resource to user in order to drop some
attributes.
Cluster A
Cluster A
Cluster A
some attributes
some attributes
Add

14. Engineer for Rancher believe their approach is common
● Develop API Server Framework for Kubernetes API(CRD)
○ https://github.com/rancher/norman
● Support
○ Generate API function based on CRD schema
○ Determine path based on CRD schema
○ Override function to do some additional work like overriding attributes, drop attributes…

15. Resource Creation process is same as Kubernetes Resource
API Controller
ClusterA
Watch
Kubernetes ClusterReconcile
Get latest
information from
kube-apiserver
Check if any
difference
Between desired and
actual states
Do something to make
actual state desired
Reconcile
Loop
Cluster Agent
Node Agent

16. Rancher Controllers
Management Controllers
Kubernetes Cluster
Kubernetes Cluster
Kubernetes Cluster
User Controllers
User Controllers
User Controllers
Cluster Controller
Node Controller
Catalog Controller
・・・・
Alert Controller
RABC Controller
Endpoint Controller
・・・・

17. ● Run 1 Controller in a Rancher Server
● Do something across Kubernetes Clusters deployed
○ Catalog for Rancher User to deploy helm chart onto Kubernetes Clusters
○ Authentication for Rancher
○ Node Create/Update/Remove
○ Cluster Create/Update/Remove
○ Start User Controller for New Cluster
Management Controllers
Management Controllers
Kubernetes Cluster
Kubernetes Cluster

18. User Controllers
● Run 1 Controller for each Kubernetes Cluster deployed
● Do something for assigned Kubernetes Cluster
○ Sync data between Rancher Configuration and Kubernetes Cluster deployed
○ Propagate event happening in Kubernetes Cluster deployed to Rancher
○ Watch all workload(Pod...) in Kubernetes Cluster deployed and Do something
User Controllers
Kubernetes Cluster
Kubernetes Cluster

19. After deployed, How to check/update Node/Cluster state?
kube-apiserver
rancher-server
management controller
user controller
kubelet
Pod A Pod B
Target Resources
Any New Pod?
Anything Change in existing Pod?
Pod A and B is alive!
rancher cluster agent
Target ResourcesTarget Resources
Container
(kubelet)
File
(certificates)
Any change in
Container, Files?
Work as a TCP Proxy
Please check my internal state as you want
rancher node agent
api-server

20. After deployed, How to check/update Node/Cluster state?
kube-apiserver
rancher-server
management controller
user controller
kubelet
Pod A Pod B
Target Resources
Any New Pod?
Anything Change in existing Pod?
Pod A and B is alive!
rancher cluster agent
Target ResourcesTarget Resources
Container
(kubelet)
File
(certificates)
Any change in
Container, Files?
Work as a TCP Proxy
Please check my internal state as you want
rancher node agent
api-server

21. Agent TCP Proxy is used when you call k8s API

22. Please look at endpoint closely
https://<rancher-server-endpoint>/k8s/cluster/local
All requests against k8s cluster are
sent to Rancher first and Rancher
proxy it into correct cluster

23. How Rancher proxy
Kubernetes Cluster
rancher cluster agent
Kubernetes Cluster
rancher cluster agent
User can not call Kubernetes API
While cluster-agent is down
If rancher cluster agent got down for some reason

24. Rancher 2.X Updates

25. Rancher 2.0 -> 2.1: Support multiple Rancher Server
Kubernetes Cluster
Kubernetes Cluster
LB
Rancher Server play very very important role,
if it is down, following features are not available
1. Create/Update/Delete Node, Cluster
2. All management function for Node, Cluster
3. Proxy Kubernetes API request to Cluster

26. Rancher 2.0 -> 2.1: Support multiple Rancher Server
Kubernetes Cluster
Kubernetes Cluster
LB
Rancher 2.0 is not allowed operator to run
multiple rancher server because of websocket
session handling logic.

27. Rancher 2.0 -> 2.1: Support multiple Rancher Server
Kubernetes Cluster
Kubernetes Cluster
LB
Leader
From 2.1, Rancher’s websocket session
handling logic got improved and now they
allow us to run multiple rancher server for high
availability and scalability.

28. Rancher 2.0 -> 2.X: Move User Controllers to...
User Controllers
User Controllers
User Controllers
Kubernetes Cluster
Kubernetes Cluster
Kubernetes Cluster
CPU/Memory Usage increase
every time create cluster
・・・・
・
・
Management Controllers

29. Rancher 2.0 -> 2.X: Move User Controllers to...
Kubernetes Cluster
Kubernetes Cluster
Kubernetes Cluster
User Controllers
User Controllers
User Controllers
Management Controllers
CPU/Memory increasing ratio by new cluster got
improved than before because new user controller
for new cluster will be running on new cluster itself