Wrapper Formula Presentation at SaltConf 2016

1. Wrapper Formulas
Put Your Formulas in a Formula So You Can
Orchestrate Your Orchestration!

2. About me
• Principal Software Engineer, Virtustream
• Previously:
– System Administrator
– Site Reliability Engineer
– Automation Engineer
• 4+ years experience with Saltstack

3. Virtustream
– Virtustream is a member of the EMC Federation
– Providing IaaS, Object Storage, and Managed Services
– Mission Critical Enterprise workloads in the Cloud

4. Platform Engineering
• Deploy and Manage:
– Mesos Platform
– Openstack Platform

5. Salt at Virtustream
• 15,000+ managed nodes
• 25+ sites
• Multi-Exabyte Object Storage
• VMware
• OpenStack
• Apache Mesos

6. Quick Review

7. Beginnings
• Deployed SaltStack primarily for remote
execution on our object storage platform
• Built up some states and pillar data
• Kept the state and pillar trees in a GitHub
repository

8. Beginnings
• Worked reasonably well for a single project
• Difficult to test
• Difficult to deploy
• Not multi-project friendly:
– Poor code reuse
– Divergence across projects
– Lacks consistent patterns

9. Individual Formulas (what they solve)
• Promotes code reuse
• Cookiecutter projects make it easy to get everyone using the same patterns.
• Debian packaging solves the distribution problem
• State and Execution modules bundled
• TeamCity runs our tests, builds, and deployments
• Version controlled
• Peer reviewed
• Tests executed on pull request and added to comments
• Test individual states
• Pillar data stored with project
• We’re up to 140+ formulas built this way!

10. Individual Formulas (results)
• Our project’s /srv/salt
directory is far less
populated
• /srv/formulas directory is
heavily populated

11. Questions

12. Individual Formulas (what they don’t solve)
• Poor support for formula interdependency
• Reusable pillar data is not exposed
• Does not allow testing of pillar data

13. Wrapper Formulas
• Captures a single
platform or product
• Ties individual
formulas together

14. Wrapper Formulas
• Orchestration and/or states from each formula
are included in one orchestration state
• This orchestration state orders the installation
and executes blocking health checks between
steps as necessary

15. Wrapper Formulas
• Includes
orchestration
and states from
each formula

16. Wrapper Formulas
• Extends orchestration
to enforce order

17. Wrapper Formulas
• Formulas bring their own orchestration as
needed.
• Contains all non-environment pillar data for the
platform
• Allows testing of the entire platform while still
allowing individual formula testing

18. Installs to known location
• Orchestration and state data installs into /srv/salt/ext/
• Pillar data installs into /srv/pillar/ext/
• This is done so that we know that the state and pillar
data came from a wrapper formula

19. Turn up a whole stack with one command!
• Now when we need to turn up a new datacenter
or testing environment it’s as easy as:
– salt-run state.orchestrate ext.tw.orchestration.bootstrap

20. Mesos Platform
• Our flagship wrapper formula
• Runs on bare metal in production and staging
• Runs on Vagrant and Openstack VMs for
development
• Chad Heuschober’s talk after this for more
details

21. Next steps
• Full platform testing via TeamCity
• Migration of other projects to this model
• Event driven orchestration?

22. Questions