In this talk, a closer look into the lifecycle of operators will be presented. With an understanding of how operators evolve, it becomes clear what challenges during operator upgrades. A brief overview of lifecycle management tools such as Helm, OLM, and Carvel is presented in this context. In particular, it will be discussed whether these tools can help, which restrictions apply and where further development would be desirable. At the end of this talk, you will know what operator lifecycle management is about, what its challenges are, and which tools may be used to reduce operational friction. This talk was given by Julian Fischer for DoK Day Europe @ KubeCon 2022.

1. Operator Lifecycle Management
1

2. Introduction
2

3. This Talk
3

4. • What is Operator Lifecycle Management
?
• Potential Tool
s
• Summary & Conclusions
4

5. Operator
Lifecycle Management
5

6. CRD
Lifecycle Management
6

7. Tools
7

8. Helm
8

9. 👍 Popular & matur
e
👍 Easy to us
e
👍 Templating + User provided value
s
👍 Chart Hooks - Lifecycle hooks: pre-install, post-install, pre-upgrade,
…
👍 No server-side componen
t
👍 Wide range of integrations available: ArgoCD, Crossplane, …
Helm
9

10. 👎👎👎 Poor CRD Lifecycle Management
👎 Limited Chart Dependency Managemen
t
Helm
10

11. OLM
11

12. ℹ Has a server side componen
t
👍 Designed solely for the purpose of managing operator
s
👍 Installation and upgrades are part of the OLM mode
l
👍 Installation = Creating a Subscription C
R
👍 Upgrades: updates „storedVersion“
fi
eld (after CRs have been migrated
)
👍 OperatorSDK helps with building OLM package
s
OLM
12

13. 👎 Not a general purpose package manage
r
👎 No templating mechanism as with Helm: e.g. custom label
s
👎 No lifecycle hooks (nothing similar to Helm Chart Hooks
)
👎 Upgrading CRs has to be done automatically
OLM
13

14. Carvel
14

15. ℹ originates from VMware Tanz
u
ℹ Includes kapp CLI and kapp controller (server-side
)
👍 kapp allows templating (kustomize, ytt
)
👍 kapp controller - App CR - Similar to OLM Subscriptions + templatin
g
👍 CRD upgrades similar to OLM
Carvel
15

16. 👎 Early stage project. Not widely adopted. Missing documentation
.
👎 No dependency managemen
t
👎 No hub such as ArtifactHub or OperatorHu
b
👎 Not as widely integrated into other tools
Carvel
16

17. Summary &
Conclusion
17

18. Helm doesn’t seem to
focus on operators.
18

19. Carvel is promising
but immature.
19

20. Hence, for now:
OLM.
20

21. There is room for
improvement.
21

22. Questions
?
@anynine
s
@
fi
scherjulian
22

23. Thank You!
23

24. Data Service
Automation
24

25. “Fully automating the entire lifecycle of a wide
range of data services to run on cloud-native
platforms across infrastructures at scale.”
25
Commit to a Mission

26. 26
«K8s Cluster»
Operator
Service Instance (Operand)
A single K8s cluster
with a single service instance
managed by a single Operator.

27. 27
«K8s Cluster»
Operator
Service Instance (Operand)
Service Instance (Operand)
Service Instance (Operand)
Service Instance (Operand)
A single K8s cluster
with multiple service-instances
managed by a single Operator.

28. 28
«K8s Cluster»
PG
Operator
PG Service Instance (Operand) #1
PG Service Instance (Operand) #2
A single K8s cluster
with multiple service-instances
managed by a multiple
Operators.
PG Service Instance (Operand) #3
PG Service Instance (Operand) #4
Other
Operator
Other Service Instance (Operand) #5
Other Service Instance (Operand)
Other Service Instance (Operand)
Other Service Instance (Operand)

29. 29
Many K8s clusters eac
h
with multiple service-instances
managed by a multiple
Operators.

30. 100s or 1000s of
data service instances!
30

31. Scale Matters!
31

32. Each data service
instance matters!
32

33. Methodology
33

34. Principles
34

35. • Know your target audience. Requirements and desired qualities
.
• Choose your data services, wisely. Be aware of open source licenses
.
• Strive for full lifecycle automation
.
• On-demand provisioning of dedicated service instances
.
• Rebuild failed instances instead of
fi
xing them
.
• Design for scalability.
Principles
35

36. • Operational model
fi
rst, automation second
.
• Be a backup/restore hero
.
• Solve issues on the framework level,
fi
ne-tune data service speci
fi
cally
.
• Test code. Test service instances. Test desired and undesired behavior
.
• Provide meaningful default con
fi
guration values. Except custom con
fi
g
parameters.
Principles
36

37. • Don’t touch upstream code, except for …
• Master release managemen
t
• Deliver releases into target environments quickl
y
• Collect feedback from users (e.g. through support
)
• Provide meaningful documentation. Better documentation, less support.
Principles
37

38. Data Service Automatio
n
with Kubernetes
38

39. Ways to Implement a
n
„Operator“
39

40. • Kubernetes CRDs + Custom Controller
s
• Operator SD
K
• KUDO
Data Service Automation with K8s
40

41. Stages of
Development
41

42. • Operational Model - Level 1: What a sysop/DBA would do
.
• Operational Model - Level 2: Containerization, YAML + kubectl
• Operational Model - Level 3: Operato
r
• Operational Model - Level 4: Operator Lifecycle Management
Data Service Automation with K8s
42

43. CRDs
43

44. 44
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: pgs.ds.a9s.io
spec:
# group name to use for REST API: /apis/<group>/<version>
group: ds.a9s.io
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
description: Yeah! Science!
properties:
spec:
type: object
required: ["replicas"]
properties:
postgresVersion:
type: string
# pattern: major.minor.patchlevel or major.minor > determine patchlevel automatically
default: "12.2"
# postgresPlugins:
# type: array
replicas:
type: integer
# pattern: 2n+1
minimum: 1
default: 1
# either Namespaced or Cluster. Namespaced as data service instances should belong to a namespace.
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: pgs
# singular name to be used as an alias on the CLI and for display
singular: pg
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: PostgreSQL
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- pg
- pgs

45. 45
apiVersion: ds.a9s.io/v1
kind: PostgreSQL
metadata:
name: pg-1
spec:
postgresVersion: "12.2"
replicas: 3

46. • CRD = Custom Resource De
fi
nitio
n
• Introduce custom data structures to Kubernete
s
• Kubernetes provides an endpoint for managing these object
s
• Kubernetes provides persistency by storing them in its etcd.
K8s CRDs
46

47. Controllers
47

48. 48
// [...]
// add adds a new Controller to mgr with r as the reconcile.Reconciler
func add(mgr manager.Manager, r reconcile.Reconciler) error {
// Create a new controller
// [...]
// Watch for changes to primary resource Memcached
err = c.Watch(&source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{})
if err != nil {
return err
}
// TODO(user): Modify this to be the types you create that are owned by the primary resource
// Watch for changes to secondary resource Pods and requeue the owner Memcached
err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{
IsController: true,
OwnerType: &cachev1alpha1.Memcached{},
})
// [...]
}

49. 49
func (r *ReconcileMemcached) Reconcile(request reconcile.Request)
(reconcile.Result, error) {
reqLogger := log.WithValues("Request.Namespace", request.Namespace,
"Request.Name", request.Name)
reqLogger.Info("Reconciling Memcached")
// Fetch the Memcached instance
instance := &cachev1alpha1.Memcached{}
err := r.client.Get(context.TODO(), request.NamespacedName, instance) //
Retrieve the object
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile
request.
// Owned objects are automatically garbage collected. For additional
cleanup logic use finalizers.
// Return and don't requeue
return reconcile.Result{}, nil
}
// Error reading the object - requeue the request.
return reconcile.Result{}, err
}
// Define a new Pod object (similar to a YAML Spec)
pod := newPodForCR(instance)
if err := controllerutil.SetControllerReference(instance, pod, r.scheme);
err != nil {
return reconcile.Result{}, err
}
// Check if this Pod already exists
found := &corev1.Pod{} // Empty Pod object
err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name,
Namespace: pod.Namespace}, found)
// If an error occurs and in particular the error is of the type NotFound then
we know the Pod doesn't exist.
if err != nil && errors.IsNotFound(err) {
reqLogger.Info("Creating a new Pod", "Pod.Namespace", pod.Namespace,
"Pod.Name", pod.Name)
// Create the secondary objects ... in this case a single pod.
err = r.client.Create(context.TODO(), pod)
if err != nil {
return reconcile.Result{}, err
}
// Pod created successfully - don't requeue
return reconcile.Result{}, nil
} else if err != nil {
return reconcile.Result{}, err
}
// Pod already exists - don't requeue
reqLogger.Info("Skip reconcile: Pod already exists", "Pod.Namespace",
found.Namespace, "Pod.Name", found.Name)
return reconcile.Result{}, nil
}

50. • Read custom resource object speci
fi
cation
s
• Translate primary resources into a set of secondary resources
.
• E.g. a PostgreSQL resource into a Service and a StatefulSet
.
• Watches the primary spec for changes
.
• Ensures secondary resources to comply to the desired state of the primary’s
spec.
K8s Controllers
50

51. Common Pitfalls
51

52. • Underestimate complexity and effor
t
• Insuf
fi
cient coverage of essential lifecycle operation
s
• Too little robustness, observability and predictabilit
y
• Applying automation that doesn’t
fi
t the context
52

53. What
Organizations Want
53

54. • Expose lifecycle operations using Kubernetes Custom Resources (CRDs
)
• On-Demand Provisioning of Dedicated Service-Instance
s
• Allow con
fi
guration updates
• Provide monitoring of health and statu
s
• Infrastructure-agnosti
c
• Runs on different Kubernetes
fl
avors
.
• Authentication with dedicated user for each application accessing the DSI
54

55. • Horizontal 2n+1 DSI scalability: 1, 3, 5 ….
• Automatic failure detection and fail-over. Self-healing to recover degraded
clustered service instances
.
• Host-anti-af
fi
nity. Support for multiple AZs
.
• Vertical DSI scalability: replace small pods with larger pods with even larger
pods,
…
• Provide backup and restore capabilities with the ability to create backup
schedules.
55

56. • Stream backups to external object stores
.
• Allow choosing data service versions
.
• Documentation
.
• Encryption at rest and encryption at transit
.
•
…
56

57. The Long Life of a
Service Instance
57

58. Data Service Automation
58
Create Service Instance
Delete Service Instance
Add log sink
Add metrics sink
Add alerting rule
Vertical scale-up
Horizontal scale-out Create backup
Restore backup
Update Operator
Install Operator
Delete Operator
Update Operator Con
fi
guration
Patch-level upgrade
Minor upgrade
Major upgrade
Kubernetes Node Failure
Availability Zone Failure
Network Partitioning
Network Delay Fluctuation
Network Bandwidth Fluctuation
Enable (Postgresql) extension
Disable (Postgresql) extension
Create service-binding
Delete service-binding
Change con
fi
guration setting
Create a backup schedule

59. Service Bindings
59

60. Service Bindings
60
«K8s Cluster»
PG Operator
PG Service Instance (Operand)
A Service Binding represents the
connection between an app and a
data service instance.
«Microservice»
Microservice #1
«Microservice»
Microservice #2
Service Binding #1 Service Binding #2

61. Service Bindings
61
«K8s Cluster»
PG Operator
PG Service Instance (Operand)
A Service Binding comprises a
Kubernetes Secret as well as
a user in the managed data
service, e.g. a PostgreSQL user
.
Both user and secret are unique
to a particular Service Binding.
«Microservice»
Microservice #1
«Custom Resource»
Service Binding #1
PG User #1 for
Service Binding #1
«Secret»

63. with multiple service-instances
managed by a multiple
Operators.
«Backup»
Daily Backup for 2021-10-06
«Backup Plan»
Plan to do a daily backup

64. Technology
64

65. Writing
Controllers
65

66. Reconcilin
g
External Resources
66

67. 67
«K8s Cluster»
PG Operator
PG Service Instance (Operand)
«Microservice»
Microservice #1
«Custom Resource»
Service Binding #1
PG User #1 for
Service Binding #1
«Secret»

68. Service Binding #1
«Secret»
PG User Controller
pg-user CR created CREATE USER user123 …

69. CREATE USER
69

70. 70
«Custom Resource»
Service Binding #1
«Secret»
PG User Controller
pg-user CR created CREATE USER user123 …
{
Careful ⚠ This is not a transaction
.
Atomicity is not guaranteed.
💥

71. 71
«Custom Resource»
Service Binding #1
«Secret»
PG User Controller
Inconsistent state
.
Secret ✅
Postgres user ❌

72. Be prepared to
re-reconcile by makin
g
actions idempotent.
72

73. CREATE USER
IF NOT EXISTS
73

74. 74
«Custom Resource»
Service Binding #1
«Secret»
PG User Controller

75. Summary
75

76. Questions
?
@anynine
s
@
fi
scherjulian
76

77. Thank You!
77

78. Attic
78