Mistral Unconference slides from Hong Kong OpenStack Design Summit

1. Mistral
Task Orchestration and Scheduling service
Task Orchestration and Scheduling service
•
Home: launchpad.net/mistral
•
Wiki: wiki.openstack.org/wiki/Mistral
•
IRC: #openstack-mistral
•
Me: Renat Akhmerov <
rakhmerov@mirantis.com>

2. Background
•
Realized by Murano team as a generic external
state machine with HA and scalability
•
Discussed with and supported by the authors of the
Convection proposal, Joshua Harlow and
TaskFlow team
•
Joshua, Thanks!

3. How is Mistral related to
TaskFlow & Convection
•
Mistral implements the Convection (Workflow
as a Service) and goes far beyond the initial ideas
•
TaskFlow library will be used for
implementation
•
TaskFlow library will have Mistral engine to
execute tasks over Mistral in a distributed manner

4. Mistral Mission
•
Build a Task Orchestration and Scheduling
service for OpenStack
•
Provide an easy and flexible mechanism for executing
workflows that consist of interrelated tasks in a cloud
environment
•
Help other OpenStack projects automate their
internal workflows
•
Provide HA and Scalability for task/workflow execution

5. Mistral Users
•
Infrastructure OpenStack developers : Nova,
Heat, Murano, Solum etc.
•
System administrators : to use capabilities like
Cloud Cron to be able to schedule helper cloudwide tasks (local VM processes, RESTful services,
creating/terminating VMs, etc.)
•
Enterprise application developers : automate
business processes consisting of multiple
distributed processing steps

6. Reasons to use Mistral
1. High Availability : ability to continue a workflow from the point it crashed at
2. Scalability: Mistral knows what parts of a workflow can be executed in parallel
3. Scheduling: workflows/tasks can be scheduled for periodical execution (e.g. every
Tuesday at 3 am)
4. Observable state: can always see the details of the current execution state and
history
5. Offloading dependency management : given what needs to be completed
Mistral executes whatever tasks needed for that
6. Additional integration points : a 3rd party application can hook into deployment
processes represented as Mistral workflows using events
7. Formalized task graphs are just easier to manage and understand : they
can be visualized, analyzed and optimized

7. Use Cases
•
Cloud Cron: a system administrator can schedule
cloud tasks for periodical execution
•
Cloud Environment Deployment : a deployment
tool or a system can represent deployment steps as
a Mistral workflow and use Heat for every step as a
software orchestration tool
•
Event Based Scheduling : e.g. Live Migration on
CPU 100% from Ceilometer

8. Cloud Cron
•
Single point of configuration
and control
•
Wide range of tasks : local
processes, RESTful services
etc.
•
Monitoring & Management
•
HA & Scalability

9. Cloud Environment
Deployment
•
Formalized deployment
workflow
•
Individual tasks use Heat to
install applications
•
Dependency Management
offloaded to Mistral
•
Multiple flows in parallel
•
Mistral provides HA and
scalability
•
No need to use locking

10. Event Based Scheduling
•
Live migration is an example
•
Workflow execution on
external events
•
Various event types
•
Various event sources

11. Key Mistral Features
•
Uploading custom graphs of tasks
•
Driven by simple DSL: YAML/JSON/XML
•
Task Scheduling: using UI or Cron patterns
•
Generic service: doesn’t do the actual work itself
and acts as an HA coordinator signaling to
workers about what needs to be done

12. Main Concepts
•
Task Graph - a set of related
tasks
•
Workflow - a subgraph involved
into one execution
•
Flow - a path that can be
processed independently
•
Target task - a task that needs to
get completed in a particular case
•
Action - a particular piece of work
associated with a task (signal)

13. DSL
# Tasks
# Actions
tasks:
Task1:
action: action1
actions:
action1:
transport: rest1
Task2:
action: action2
action2:
transport: amqp1
Task3:
requires: Task1, Task2
action: action3
action3:
transport: poll
# Transports
transports:
rest1:
type: REST
url: http://some.url
amqp1:
type: AMQP
host: my.host.org
port: 5672
routing-key: a_queue
…
poll:
type: POLL

14. Data Flow (Concept)
1. Select a subset of input
data
2. Process it
3. Merge it back in
4. Pass it on
5. Merge with parallel
branches

15. What about Mistral workers?
•
Core engine doesn’t start any workers
•
Workers are dynamic, configured in
DSL
•
Types of workers:
•
•
•
Client application nodes
REST services
Other processes accessible by AMQP, SSH
etc.
•
BUT!
•
Dedicated workers can be allocated
using toolsets created on top of Mistral
core engine

16. Can Mistral do anything
else?
•
Easy to build additional toolsets and frameworks
•
Potential capabilities include:
•
Domains (Namespaces)
•
Event subscription
•
Role Based Access Control
•
Task priorities
•
Task collocation
•
Language-oriented frameworks (Python, Java, etc.)
•
Plugin system to introduce new DSL keywords

17. Is Mistral similar to Amazon
Simple Workflow
•
In many ways Mistral is similar to Amazon SWF
•
Amazon SWF is oriented to language bindings
(Java, Ruby, Python)
•
Conceptual differences in underlying model
•
Mistral is targeting to be much simpler
•
Mistral may later have Amazon SWF adapter

18. Current Status
•
Documented all the core ideas and concepts
•
Prepared a draft of DSL/API specification
•
Planned the Roadmap
•
Started prototyping core ideas
•
Introduced at HK OpenStack design summit! :)

19. Roadmap
•
Nov 25th, 2013.
•
•
Dec 10th, 2013.
•
•
DSL v1.0 and API v1.0 implementation
Feb 20th, 2014. Release 0.2:
•
•
Mistral PoC: Basic task orchestration, DSL, API and Scheduling
Jan 20th, 2014. Release 0.1
•
•
DSL/API v1.0 specifications
Task engine HA & scalability, Web UI for monitoring & management
March 20th, 2014. Release 0.3:
•
Triggers (Cron, Events), Web UI for triggers (Cron, Events)

20. Thanks!
Q&A
Q&A
Q&A