This document discusses using Chef on SmartOS to automate configuration of servers provisioned on the Joyent cloud platform. It provides an overview of SmartOS and Joyent, explains how to install and configure the knife-joyent plugin to manage servers, and describes useful cookbooks and resources for configuring services and system settings on SmartOS.

1. Proprietary and
Confidential
Automating Joyent
SmartMachines with Chef
Chef on SmartOS
Eric Saxby
@sax @ecdysone @sax

2. Who am I?
Proprietary and
Confidential
■ Application developer
operational experience with many
technologies, project by project
■ BSD/AIX/Ubuntu
Solaris in 2002, but I was very much
out of my element
■ Switched to DevOps-y team 18 months ago
Multiple back end services for a large e-commerce site,
transitioning to SmartOS
■ Now I’m at Wanelo

3. From a certain point of view...
Proprietary and
Confidential

4. What is Wanelo?
Proprietary and
Confidential
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.

5. Proprietary and
Confidential
Marketing-free shopping across 100s of
thousands of unique stores

6. Proprietary and
Confidential
Personal feed of products from any
store on the internet

7. Technology overview
Proprietary and
Confidential
■ MRI Ruby 1.9.3 & Rails 3.2
■ PostgreSQL 9.2.4, Solr 3.6
■ Joyent Cloud, SmartOS
ZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace
■ Circonus, Chef + Opscode
Monitoring, graphing, alerting, automation
■ Amazon S3 + Fastly CDN
■ NewRelic, statsd, Graphite, nagios

8. What’s SmartOS?
Proprietary and
Confidential
■ Illumos branch optimized for cloud
computing
■ Developed by Joyent for their public
cloud

9. What’s Illumos?
Proprietary and
Confidential
■ It’s what OpenSolaris became after Oracle
killed the project
■ Umbrella for various distributions, each
committed to pushing their improvements
upstream
■ http://wiki.illumos.org/display/illumos/About+illumos

10. What does SmartOS look
Proprietary and
Confidential
■ Compute Node — physical server
■ Global Zone — host OS (SmartOS)
■ Non-Global Zone — like a virtual machine, with
native system calls (no fake hardware layer)
■ Very secure
■ Can run KVM for guest OS (Ubuntu, Centos)

11. How is it deployed?
Proprietary and
Confidential
■ Can manage from global zone (imgadm,
zoneadm)
■ Tools provide APIs
■ Smart Data Center (Joyent’s tools, can be licensed)
■ Project FIFO (SDC API in free package)
■ Joyent Public Cloud
■ Many compute nodes working in a cluster,
PXE booted from a head node

12. ■ Service Management Facility (SMF)
If init.d and monit and god were one thing, and
actually awesome
Why should I care?
Proprietary and
Confidential
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity
■ Application Latency
Zones are OS virtualization, so faster
Processes are scheduled in global zone kernel,
not in a hardware virtualization layer

13. ■ # cores, RAM required =~
# processes
Lower latency == less cost
Proprietary and
Confidential
■ # processes required =~
requests/second of site
■ Requests/second of single process =~
request latency
$$$

14. On to Chef!
Proprietary and
Confidential

15. Terminology
Proprietary and
Confidential
■ Image / Dataset — OS at a particular version,
snapshotted at base state
■ Flavor / Package— RAM, CPU shares
■ API URL — Each data center has its own URL
■ Server ID / Zonename — Each zone gets a
UUID

16. knife-joyent
Proprietary and
Confidential

17. Installation/Configuration
Proprietary and
Confidential
■ Update knife.rb
■ Add to Gemfile
knife[:joyent_username] = 'sax'
knife[:joyent_keyname] = 'EricSaxby'
knife[:joyent_keyfile] = "#{ENV['HOME']}/.ssh/id_rsa"
knife[:joyent_api_url] = 'https://us-sw-1.api.joyentcloud.com/'
■ Add first public key in cloud API
https://my.joyentcloud.com
gem 'knife-joyent'

18. Managing keys
Proprietary and
Confidential
■ No role based access, but at least you can
make each user upload their own key
knife joyent key add -f ~/.ssh/id_rsa -k KeyName
knife joyent key delete KeyName
■ Passphrase protected keys are annoying
Each API request includes data signed with the private
key. Ruby does not have a good way of signing private
keys with ssh-agent.

19. Creating servers!
Proprietary and
Confidential
■ See what images are available
knife joyent image list
cf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartos
f4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartos
da144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux
■ base / base64 — minimal install, you add what
you need
■ smartosplus — many more things pre-
installed, but can get in the way
13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows

20. Creating servers!
Proprietary and
Confidential
■ See what flavors are available
knife joyent flavor list
Name RAM Disk Swap
Extra Small 512 MB 0 GB 15 GB 1 GB
Small 1GB 1 GB 30 GB 2 GB
Medium 2GB 2 GB 60 GB 4 GB
Medium 4GB 4 GB 120 GB 8 GB
Large 8GB 8 GB 240 GB 16 GB
Large 16GB 16 GB 480 GB 32 GB
■ Custom networking can be done in a custom
flavor (ie public or private VLAN, routes)

21. Creating servers already!
Proprietary and
Confidential
knife joyent server create
--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b
--flavor 'Medium 2GB'
-N server.domain.com
-E environment
-d distro
-r run_list
■ No Omnibus, so you have to provide your own
distro bootstrap template
https://gist.github.com/sax/5457464

22. knife joyent server list
See what's there...
Proprietary and
Confidential
a597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachine
sdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB
5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachine
sdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GB
b3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachine
sdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB
■ ID / zonename
■ Name
■ Run state
■ Type
■ Image
■ IP addresses
■ RAM
■ Disk

23. Other management
Proprietary and
Confidential
knife joyent server delete <server_id>
knife joyent server start <server_id>
knife joyent server stop <server_id>
knife joyent server reboot <server_id>
knife joyent server resize <server_id> -f <flavor>
knife joyent snapshot create <server_id> <snapshot_name>
■ Snapshots are full ZFS snapshots
Copy-on-write snapshot of local file system.
Each snapshot is locally mounted in zone at
/checkpoints

24. So now you have a
smartmachine...
Proprietary and
Confidential

25. What's different?
Proprietary and
Confidential
■ Things you expect in /usr/local are in /opt/local
■ For historical reasons
■ If you're used to Linux, this can be annoying
■ Joyent is working on a more Linux friendly image
■ For now, add /opt/local/bin to PATH
■ Many configs are in /opt/local/etc instead of /etc
■ Some utilities are different
■ This is not the grep you're looking for....
■ Symlink your "correct" version into /opt/local/bin
■ Add /opt/local/lib to CFLAGS and LDFLAGS

26. Caveats?
Proprietary and
Confidential
■ Zones inside of zones inside of...
■ Vagrant does not currently work with SmartOS
■ VirtualBox only works in Bridged network mode
■ Local integration tests do not work

27. Where are all the things?
Proprietary and
Confidential
■ Services
■ svcs -a
■ svcadm < enable | disable | clear > service
■ Packages
■ pkgin search packagename
■ pkgin -y install packagename

28. Public vs. Private IP
Proprietary and
Confidential
■ ipaddr_extensions gem
■ Adds 'privateaddress' attribute to ohai
■ Useful to add this to bootstrap
■ Smartmachines may have a public IP and a
private IP
■ Recipes can be configured to use ipaddress or
privateaddress

29. System preparation
Proprietary and
Confidential
■ smartos cookbook
■ https://github.com/modcloth-cookbooks/smartos
■ fixes chef providers
■ smartmachine_functions
■ links nicer utils into /opt/local/bin
■ https://github.com/higanworks-cookbooks/
smartmachine_functions
■ fixes chef providers
■ provides access to Joyent metadata API
or

30. Useful LWRPs
Proprietary and
Confidential

31. SMF
Proprietary and
Confidential
■ https://github.com/modcloth-cookbooks/smf
■ Chef knows how to use SMF, not how to configure it
■ Uses nokogiri, which requires libxslt
smf 'postgres' do
user 'postgres'
group 'postgres'
project 'postgres'
start_command 'postgres-service.sh start'
stop_command 'postgres-service.sh stop'
working_directory '/var/pgsql/data'
environment 'PATH' => '/opt/postgres/bin'
end

32. SMF (cnt'd)
Proprietary and
Confidential
smf 'postgres' do
user 'postgres'
group 'postgres'
project 'postgres'
start_command 'postgres-service.sh start'
stop_command 'postgres-service.sh stop'
stop_timeout 120
restart_command 'postgres-service.sh restart'
refresh_command 'postgres-service.sh reload'
working_directory '/var/pgsql/data'
environment 'PATH' => '/opt/postgres/bin'
end
service 'postgres' do
supports :status => true,
:restart => true, :reload => true
end

33. Resource Control /
Proprietary and
Confidential
■ https://github.com/wanelo-chef/resource-control
■ configure max file descriptors, shared memory, etc
■ Bunch up master/worker processes to view in
prstat -J
resource_control_project "postgres" do
comment "PostgreSQL 9.2"
users "postgres"
project_limits "max-shm-memory" => 12000000,
"max-lwps" => 6
process_limits "max-file-descriptor" => {
"value" => 32768, "deny" => true
}
action :create
end

34. Role Based Access Control
Proprietary and
Confidential
■ https://github.com/modcloth-cookbooks/rbac
■ Allows delegation of authority without sudo
■ Implementation currently too simple, only useful for SMF
delegation
rbac 'solr' do
user 'wanelo'
action :add_management_permissions
end

35. Contributing to cookbooks
Proprietary and
Confidential
■ ~95% just require SMF, correct package names
■ ~5% of those need a special init script
■ The rest usually require custom compile
`postgres -D /path/to/data` not granular enough
`pg_ctl -D /path/to/data < start | stop | reload | refresh >`
--with-libraries=/opt/local/lib
--with-includes=/opt/local/include
LDFLAGS='-R/opt/local/lib -L/opt/local/lib'

36. Comments? Questions? Find
me.
https://github.com/wanelo
https://github.com/wanelo-chef
https://github.com/wanelo-chef/smartos-chef-
repo
Proprietary and
Confidential
@sax @ecdysone @sax