Senlin (OpenStack Clustering) project deep dive. Presentation used in Vancouver summit.

1. Senlin Deep Dive
Last Updated: 2015-05-15
IRC: #senlin
Server: https://git.openstack.org/stackforge/senlin
Client: https://git.opentack.org/stackforge/python-senlinclient

2. TOC
•Background
•Overview
•Core Concepts
•Relation with Other OpenStack Projects
•Current Status
•Future Topics
2

3. Background
•Why Senlin?
• Heat AutoScaling today is based on AWS design, with gaps
• Heat's mission
• https://github.com/openstack/governance/blob/master/reference/projects.yaml
• it's NOT about autoscaling, NOT about HA, ...
• It's NOT Heat's job to provide new service(s)
• AutoScaling: a new design
• Proposal: https://wiki.openstack.org/wiki/Heat/AutoScaling
• API Design: http://docs.heatautoscale.apiary.io/
• Targeting at many use cases not covered by Heat today
3
To orchestrate composite cloud applications using a declarative
template format through an OpenStack-native ReST API.

4. Overview (Auto-Scaling)
4
h-client h-api h-eng senlin-eng senlin-db
stack_create
asg_create
asg_id
resource_signal
hook_execute
policy_create
policy_id
webhook_create
webhook_url
• create cluster
• create nodes
• create cluster (stack)
• create policy object
• associate policy to cluster
• generate webhook for
policy
• scale cluster by triggering
policy
• policy works by updating
cluster (stack with new
group template)
• (reload LB if needed)
ASG ID is the stack ID
Auto-Scaling with Heat (30,000 ft)
template
(No
change
needed)

5. Overview (Auto-Scaling)
5
senlin-client
senlin-api
senlin-eng
senlin-lib senlin-db
senlin-db-lib
asg_create
Heat or 3rd party
service/tools
Auto-Scaling with and without Heat (10,000 ft)
h-client h-api h-eng
profile_create
stack_create
cluster_create
hook_execute cluster_update
stack_update

6. BPs on reworking autoscaling
BP Priority Description
autoscaling-api-resources high Heat resources invoking AS APIs
as-api-group-resource high ScalingGroup resource wrapping AS API's group functionality
as-api-policy-resource high ScalingPolicy resource wrapping AS API's policy functionality
as-api-webhook-resource high Webhook resource wrapping AS API's execution of webhooks
autoscaling-api-client high A python client for Heat to interact with AS API
autoscaling-api - A separate service for the implementation of autoscaling w/ Heat
as-engine - A separate engine/service for autoscaling support AS API
as-engine-db - A DB dedicated to autoscaling, using schema created in as-lib-db
as-lib - A separate module to be used by the AS service
as-lib-db - A DB for autoscaling bookkeeping
6
BluePrints

7. Dependencies among BPs
7
autoscaling-api-resources
as-api-group-resourcesas-api-policy-resources as-api-webhook-resources
autoscaling-api-client
autoscaling-api
as-engine
as-engine-db
as-lib
as-lib-db

8. A Struggle before Senlin Starts
• We chose to work outside Heat
• Our management supports this approach, thus cycles guaranteed
• We see a lot potentials in a standalone service
• We don't have to do everything from scratch
8
Within Heat Outside Heat
Pros Cons Pros Cons
• smooth transition
• strict reviews, thus
better quality
• long (forever?)
code churn
• eventually, a
dedicated service
is needed, and the
pain to switch over
• quick development
• less code churn to
Heat
• high requirements
of skills and cycles
• eventual switch
over, i.e. another
animal to feed in
the OpenStack zoo

9. What Do We Need?
9
Scalable
Load-Balanced
Healthy
Manageable
......

10. 10
Senlin
[Chinese Pinyin for Forest]

11. ER Diagram
11
policy
name
uuid
type
level
specCreation
Update
Deletion
Scaling
Health
profile
name
uuid
type
version
spec
Load Balancing
os.nova.server
...
os.heat.stack
«policy_type»
cooldown
cluster_plcy
cluster_id
policy_id
enabled
level
cooldown
«profile_type»action
context
action
inputs
outputs
API webhook
target
action
user
cluster
name
uuid
user
project
domain
parent
profile_id
status
node
name
uuid
cluster_id
profile_id
index
role
status
created_time
updated_timecreate()
delete()
update()
add_nodes()
del_nodes()
create()
delete()
update()
join()
leave()
Events
timestamp
object
action
status
...

12. Profiles
12
name: random_string_stack
template: random_string_stack.yaml
context:
region_name: RegionOne
environment:
- env.yaml
heat_template_version: 2014-10-16
parameters:
file:
type: string
default: {get_file: test_script.sh}
resources:
random:
type: OS::Heat::RandomString
properties:
length: 64
outputs:
result:
value: {get_attr: [random, value]}
$ senlin profile-create
-t os.heat.stack
-s string_stack.spec
mystack
string_stack.spec
random_string_stack.yaml

13. profile/cluster/node operations
13
$ senlin profile-create -t <type> -s <spec> <name>
$ senlin profile-list
$ senlin profile-show <profile>
$ senlin profile-delete <profile>
$ senlin profile-update -s <spec> <profile>
$ senlin cluster-create -p <profile> -n <size> <name>
$ senlin cluster-delete <cluster>
$ senlin cluster-list
$ senlin cluster-show <cluster>
$ senlin node-create -p <profile> -c <cluster> <name>
$ senlin node-delete <node>
$ senlin node-show <node>
$ senlin node-join -c <cluster> <node>
$ senlin node-leave <node>
$ senlin cluster-node-add -n <nodes> <cluster>
$ senlin cluster-node-del -n <nodes> <cluster>
$ senlin cluster-node-list <cluster>

14. policies - placement [design]
14
# Sample placement policy
# Spanning AZs with weights
availability_zones:
strategy:
# Valid values include:
# ROUND_ROBIN, WEIGHTED, SOURCE
type: ROUND_ROBIN
options:
zones: [AZ1, AZ2]
# Spanning regions with weights
regions:
strategy:
# Valid values include:
# ROUND_ROBIN, WEIGHTED, SOURCE
type: ROUND_ROBIN
options:
regions: [RegionOne, RegionTwo]

15. policies - deletion [prototype]
15
# Sample deletion policy that can be attached to a cluster.
# The valid values include:
# OLDEST_FIRST, OLDEST_PROFILE_FIRST, YOUNGEST_FIRST, RANDOM
criteria: OLDEST_FIRST
# Whether deleted node should be destroyed
destroy_after_deletion: True
# Length in number of seconds before the actual deletion happens
# This param buys an instance some time before deletion
grace_period: 60
# Whether the deletion will reduce the desired capability of
# the cluster as well.
reduce_desired_capacity: False

16. policies - scaling [implementing]
16
# Sample scaling-in policy
alarm:
type: SENLIN::ALARM::CEILOMETER
properties:
meter: cpu_util
op: lt
threshold: 15
period: 60
evaluations: 1
schedule:
start_time: "2015-05-07T07:00:00Z"
end_time: "2015-06-07T07:00:00Z"
handlers:
- type: webhook
action: SENLIN::ACTION::RESIZE
params:
type: CHANGE_IN_CAPACITY
number: -1
strict: True
credentials:
user: alarmer
password: secrete
- type: email
addresses: [joe@cloud.com]
alarm
• 'alarm' section is optional
• alarm type  properties
• 'query' or 'matching_metadata'?
• Monasca alarm?
• other alarm sources?
schedule
• 'schedule' section is optional
• time period for alarms
• 'schedule' when used by itself
implies a scheduled action
action
• a list of operations to perform
• webhooks generated when policy
is attached to a cluster

17. policies operations
17
$ senlin policy-create -t <type> -s <spec> <name>
$ senlin policy-list
$ senlin policy-show <policy>
$ senlin policy-delete <policy>
$ senlin policy-update -s <spec> ... <policy>
$ senlin cluster-policy-attach <cluster> <policy>
$ senlin cluster-policy-detach <cluster> <policy>
$ senlin cluster-policy-list ... <cluster>
$ senlin cluster-policy-show <cluster> <policy>
$ senlin cluster-policy-enable <cluster> <policy>
$ senlin cluster-policy-disable <cluster> <policy>
$ senlin cluster-policy-update ... <cluster> <policy>

18. senlin cluster-resize
18
usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>]
[-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>
Positional arguments:
<CLUSTER> Name or ID of cluster to operate on.
Optional arguments:
-c <CAPACITY>, --capacity <CAPACITY>
The desired number of nodes of the cluster.
-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>
A positive integer meaning the number of nodes to add,
or a negative integer indicating the number of nodes
to remove.
-p <PERCENTAGE>, --percentage <PERCENTAGE>
A value that is interpreted as the percentage of size
adjustment. This value can be positive or negative.
-t <MIN_STEP>, --min-step <MIN_STEP>
An integer specifying the number of nodes for
adjustment when <PERCENTAGE> is specified.
-s, --strict A boolean specifying whether the resize should be
performed on a best-effort basis when the new capacity
may go beyond size constraints.
-n MIN, --min-size MIN
New lower bound of cluster size.
-m MAX, --max-size MAX
New upper bound of cluster size. A value of -1
indicates no upper limit on cluster size.

19. How policies work together?
19
handler
count
adj_type,
number,
min_size,
max_size,
min_step,
strict
alarm
timer
start_time
end_time
ceilometer
creation
placementdeletion
scaling
policy
clusterbinding
enable
/disable
update
count
deletion
webhook
candidates[]
deletion
sched_hints[]
creation
NODE_CREATE
NODE_DELETE

20. Problems & Risks
•Problems
• test case coverage
• functional tests
• others?
•Risks
• dependency on python-openstacksdk? good or bad?
•Open Questions
• generic notification mechanism
• drivers for dumping events
• external workflow engine, i.e. Mistral
20