Use Python, Flask, Docker, Celery and more to start the fun!

1. DIY: A simple orchestrator built step by step
TIAD PARIS 2015

2. ChallengesHi, I’m Ahmet Demir
Also known as @ahmet2mir
Opensource Enthusiast @D2SI

3. Why Orchestration

4. All IT want

5. But we’ve got

6. When we try to
assemble them
http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823

7. Choose the orchestrator

8. What we orchestrate ?
IT Process: disaster recovery, restart process
Infrastructure: Build a VM, upgrade it
Application Deployment: Deploy an application on x nodes
Developpements: Continuous integration, Continuous Delivery

9. State of the art

10. Eliminate "server management deployment"
FusionDirectory - http://www.fusiondirectory.org
m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php
Fabric - http://www.fabfile.org/
Archipel - http://archipelproject.org/
AppScale - https://github.com/AppScale/appscale
OpenBuild - http://openbuildservice.org/
CompatibleOne - http://compatibleone.org/
Opsi - http://www.opsi.org

11. Eliminate config manager
Puppet - https://puppetlabs.com/
Chef - https://chef.io
SaltStack - http://saltstack.com/
Ansible - http://www.ansible.com/home
CFengine - http://cfengine.com/

12. Eliminate "server task oriented"
Rundeck - http://rundeck.org/

13. Eliminate Only Cloud oriented
Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki
Openstack Heat - https://github.com/openstack/heat

14. Finally 1/3
Terraform - https://terraform.io/
Terraform provides a flexible abstraction of resources and providers. This
model allows for representing everything from physical hardware, virtual
machines, and containers, to email and DNS providers
#Go

15. Finally 2/3
Salt Cloud - https://salt-cloud.readthedocs.org
Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate
Salt into cloud providers in a clean way so that minions on public cloud
systems can be quickly and easily modeled and provisioned.
#Python

16. Finally 3/3
OpenScore - http://openscore.github.io/
Flow-based orchestration engine to manage deployed applications
Allows you to rapidly automate your DevOps use cases using ready-made
workflows. Based on Java and HP OO engine.
#Java

17. OpenStack Heat based:
Resource based, CRUD, Graph dependency, describe in Yaml
Terraform:
Execution Plan, Diff on infra and all heat features, describe in “json like” format
Salt Cloud:
Easy modules integration but no plan, no graph
OpenScore:
Service in Yaml, no plan but runbook featured
Features

18. API access ?
How to scale ? If I had more than 100 requests in same time ?
Where can I store all my services ?
How can I track ?
Is modules integration easy ?
What about lifecycle ?
Versioning the service ?
Multi-user ?
Organization > 100 peoples ?
High Availability ?
But
Try to solve or ...

19. Do It Yourself

21. Simple to maintain
Think tomorrow
Infrastructure agnostic
Easy development
Don't forget Windows users :)
Fundamentals

22. NetworkStorage Compute
Orchestrator
Referential Config Manager
Catalog
HP MP, OpenCart...
Monitoring
Nagios...
CI / CD
Jenkins, Bamboo ...
Service Designer
Big Picture

23. Engine
API
Resource runner
Resource 1 Resource 2 Resource 3 Resource 4
Run 1 & 2 in parallel Then run 3 Then run 4
Service Designer & Orchestrator

24. Components

25. If you think to something, you will find your answer in Python
Both of Devs and Ops can use Python and use it quickly
No compilation and not "magic"
Write less and do more
Limits the number of times your brain has to do a "context switch."
Helps productivity
The language

26. YAML is simple to read and write
Many projects use it: Heat, Fig etc...
Service Description: YAML

27. Many tasks management
Easy to scale
Dedicate queue to some tasks (DMZ, Windows...)
Asynchronous task queue/job
Real-time operation, but supports scheduling
Tasks are executed concurrently
Used in production systems to process millions of tasks a day
Tasks elery

28. Messaging broker
Common platform to send and receive messages
Safe place to live until received
Can hits one million messages per second
Message Delivery

29. rapid and agile web development
No migrations
Not a relational database
Schemaless
Database

30. Let’s Get Started

32. Step 0: Build workers and broker
Rabbit MQ
Celery 1 Celery 2
Flower GUI

33. orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo
orion ~ $ cd demo
orion demo $ cd 00_celery_rabbit
# start celery
orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d
# start rabbitmq
orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d
# create rabbitmq user
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost
myuser ".*" ".*" ".*"
# gui for flower and rabbitmq
orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d
# Visit http://localhost:5555
Step 0: Demo

34. Step 1: Execute a simple resource
Rabbit MQ
Celery 1 Celery 2
AWS Instance
Resource
add in queue
Flower GUI

35. Celery 1
Wrapper
AWS
Wrapper
Docker
Docker
CRUD
AWS CRUD
Pure Python Package deployed
with pip
Wrapper to simplify execution

36. orion demo $ cd 01_aws_resource
# virtual python environment
orion 01_aws_resource $ virtualenv .
orion 01_aws_resource $ source bin/activate
orion 01_aws_resource $ pip install -r requirements.txt
# add credentials in client_provider.py
# edit client_provider with some values (optionnal)
…
# run the resource
orion 01_aws_resource $ python client.py
Step 1: Demo

37. Step 2: Store outside
Rabbit MQ
Celery 1 Celery 2
Flower GUI
MongoDBApplication

38. orion demo $ cd 02_db_provider
# virtual python environment
orion 02_db_provider $ virtualenv .
orion 02_db_provider $ source bin/activate
orion 02_db_provider $ pip install -r requirements.txt
# start mongodb
orion 02_db_provider $ fig -f mongodb.yml -p demo up -d
# add credentials in add_*.py
orion 02_db_provider $ python add_aws.py
orion 02_db_provider $ python add_docker.py
# visit http://localhost:8080
Step 2: Demo

39. Application
Step 3: Service level
Rabbit MQ
Celery 1 Celery 2
Service
AWS Instance
Resource
Flower GUI
Docker Resource
1 2
MongoDB

40. orion demo $ cd 03_app
# virtual python environment
orion 03_app $ virtualenv .
orion 03_app $ source bin/activate
orion 03_app $ pip install -r requirements.txt
# run the service
orion 03_app $ python teecli.py create sample01.yml
Step 3: Demo

41. Commit API
Commit Admin GUI
Commit Referential Management
Commit all modules and wrappers
Build a service catalog
Role based access to the app
http://teerex.io (soon)
https://github.com/ahmet2mir/teerex (soon)
Next steps