A tutorial for systems administrators and DevOps teams on writing SaltStack modules for infrastructure management and application deployment and configuration management. Presented by Joseph Hall, SaltStack senior engineer, at the OpenWest 2013 conference.

1. Writing
Salt Modules
Joseph Hall
Senior Engineer at SaltStack, Inc.
Presented at OpenWest 2013
1Friday, June 7, 13

2. The future of technology is dependent upon our ability to effectively utilize infrastructures at any scale.
2Friday, June 7, 13

3. What Salt Does
Salt is a Remote Execution framework, which enables functionality such as:
• Configuration Management
• Cloud Management
• Monitoring
• Alerting
• Responding to Alerts
• etc.
3Friday, June 7, 13

4. What do these things all have in
common?
Every component on the previous slide is a function of distributed computing.
So why not also use Salt to do the things that we think about, when we think about distributed
computing?
4Friday, June 7, 13

5. Salt Venn Diagram
Salt
Chef
Puppet
CFEngine
5Friday, June 7, 13

6. Not Playing the Same Game
Flickr photo by kawwsu29 Flickr photo by ant.photos
6Friday, June 7, 13

7. Things that Should Be Salted
Flickr photo by gdgt buzz Flickr photo by Toronto Home Theater
Flickr photo by IntelFreePress Photo by Joseph Hall
7Friday, June 7, 13

8. A Basic Use Case
Writing a simple web crawler
• Need to collect various web pages
• Use multiple nodes to collect data
• Return web content to salt-master for storage
8Friday, June 7, 13

9. Using urllib2 for Our Crawler
• Easy to use
• Ships with Python
• Negates need to install 3rd party dependencies
• If a more complex spider is needed, a 3rd party library can be used
o If the 3rd party module is not installed, the module needs to know not to make itself
available, via the __virtual__() function
9Friday, June 7, 13

10. Execution Module Basics
• The __virtual__() function
o Not technically required, but is a best practice
o Executed when salt-minion is loaded
o Identifies the module name
o Commonly used to identify whether a module should be made available
• Private functions
o Name starts with underscore
o Only used internally by the module
o Will not show up in sys.doc
• Public functions
o Called directly by salt commands
o Will show up in sys.doc
10Friday, June 7, 13

11. A Few Basics
• PEP-8 is your friend
• The Salt Style Guide supersedes PEP-8
• Always use docstrings
• Avoid late imports
• Use (proper) logging
o Do not use warn/error when info is correct
o Do not use info when debug is correct
o Do not add log messages to __virtual__()
11Friday, June 7, 13

12. Using the Salt Logging System
import logging
log = logging.getLogger(__name__)
log.info('Always useful to users')
log.warn('Something isn't right, but I can still
continue')
log.error('Something is definitely wrong, and should
be fixed')
log.debug('Only developers care about me')
12Friday, June 7, 13

13. Sample __virtual__() Function
try:
import urllib2
HAS_URLLIB2 = True
except:
HAS_URLLIB2 = False
def __virtual__():
if HAS_URLLIB2 is True:
return 'crawler'
return False
13Friday, June 7, 13

14. Sample Private Function
def _query(url):
result = urllib2.urlopen(url)
return {
'url': result.url,
'code': result.code,
'msg': result.msg,
'headers': result.headers.dict,
'content': result.read(),
}
14Friday, June 7, 13

15. Sample Public Function
def fetch(urls=None):
ret = {}
if type(urls) is str:
ret[urls] = _query(urls)
elif type(urls) is list:
for url in urls:
ret[url] = _query(url)
return ret
15Friday, June 7, 13

16. Using Docstrings
def fetch(urls=None):
'''
Fetch a URL
CLI Example::
salt myminion crawler.download http://
www.mydomain.com/
'''
...SNIP...
16Friday, June 7, 13

17. Using our New Module
(command line)
Testing/debugging locally:
• salt-call --local -l debug crawler.fetch http://
tinyurl.com/cnyypk
From the salt-master:
• salt myminion crawler.fetch http://tinyurl.com/
cnyypk
17Friday, June 7, 13

18. Ready for more?
Let's write a salt runner!
18Friday, June 7, 13

19. Bootstrapping a Runner
• manage.py is a good example runner
• cp manage.py yourrunnername.py
• Make changes to suit your needs
o status() has the actual client in it
o up() and down() use the data to form a report
19Friday, June 7, 13

20. Using the LocalClient
import salt.client
client = salt.client.LocalClient(
__opts__['conf_file'],
)
minions = client.cmd(
'*',
'test.ping',
timeout=__opts__['timeout'],
)
20Friday, June 7, 13

21. Returning Output
import salt.output
data = some_stuff()
salt.output.display_output(data, '', __opts__)
21Friday, June 7, 13

22. Which Minions to Use?
• Maybe we don't want to use all of our minions for crawling.
• Just using the minions that have our crawler installed is sloppy and indeterminate
• So let's explicitly define which minions to use, in a configurable location
22Friday, June 7, 13

23. Setting Up a Pillar
# cat /srv/pillar/top.sls
base:
'spider*':
- crawler
# cat /srv/pillar/crawler.sls
crawler.enabled: True
23Friday, June 7, 13

24. Targeting the Pillar
minions = client.cmd(
'crawler.enabled:True',
'test.ping',
timeout=__opts__['timeout'],
expr_form='pillar',
)
24Friday, June 7, 13

25. Fetching From a Minion
def fetch(urls=None):
minions = up()
client = salt.client.LocalClient(
__opts__['conf_file'])
data = client.cmd(minions[0],
'crawler.fetch', arg=[urls],
timeout=__opts__['timeout'])
for minion in data.keys():
for url in data[minion]:
data[minion][url]['content'] = ''
salt.output.display_output(
data, '', __opts__)
25Friday, June 7, 13

26. Resources
Slides Available At:
http://tinyurl.com/utos-salt-modules-2013
Sources Available At:
https://github.com/techhat/salt-crawler/
Find me on Twitter:
@techhat
26Friday, June 7, 13