BATOU
multi-(component¦host¦environment¦.*)
deployment
Wednesday, 3.July 13
@theuni
Wednesday, 3.July 13
Wednesday, 3.July 13
Wednesday, 3.July 13
Wednesday, 3.July 13
AUTOMATING
DEPLOYMENTS IS
HARD
Wednesday, 3.July 13
HOW DOES
CONVERGENCE HELP?
Wednesday, 3.July 13
HOW DOES THIS
WORK WITH BATOU?
Wednesday, 3.July 13
SOME PERSPECTIVE
Wednesday, 3.July 13
Wednesday, 3.July 13
IT'S NOT THAT BAD.
Wednesday, 3.July 13
Wednesday, 3.July 13
service
deployment
Fabric,
Capistrano, ...
system
configuration
Puppet, Chef, ...
provisioning
kickstart, Razor,
imaging .....
FTP
bash
mkzopeinstance
zc.buildout
fabric
Wednesday, 3.July 13
CONVERGENCE
Wednesday, 3.July 13
"Everything that follows is a
result of what you see here."
(Dr. Alfred Lanning; I, Robot)
Wednesday, 3.July 13
SIMPLE
os.mkdir('foo')
with open('foo/bar', 'w') as myfile:
myfile.write('asdf')
os.chmod('foo/bar', 0755)
Wednesday, 3.Ju...
•unexpected system state
•can't resume
•unnecessary updates
os.mkdir('foo')
with open('foo/bar', 'w') as myfile:
myfile.wr...
CORRECT(?)
if not os.path.isdir('foo'): os.unlink('foo')
if not os.path.exists('foo'): os.mkdir('foo')
try:
os.lstat('foo/...
SIMPLE
File('foo/bar',
content='asdf',
mode=0755,
leading=True)
Wednesday, 3.July 13
class File(Component):
namevar = 'path'
def configure(self):
self += Presence(
self.path, leading=self.leading)
self += Mo...
class File(Component):
namevar = 'path'
def configure(self):
self += Presence(
self.path, leading=self.leading)
self += Mo...
class File(Component):
namevar = 'path'
def configure(self):
self += Presence(
self.path, leading=self.leading)
self += Mo...
class File(Component):
namevar = 'path'
def configure(self):
self += Presence(
self.path, leading=self.leading)
self += Mo...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Presence(Component):
namevar = 'path'
leading = False
def configure(self):
if self.leading:
self += Directory(
os.pa...
class Directory(Component):
namevar = 'path'
leading = False
def verify(self):
if not os.path.isdir(self.path):
raise bato...
class Directory(Component):
namevar = 'path'
leading = False
def verify(self):
if not os.path.isdir(self.path):
raise bato...
class Directory(Component):
namevar = 'path'
leading = False
def verify(self):
if not os.path.isdir(self.path):
raise bato...
class Directory(Component):
namevar = 'path'
leading = False
def verify(self):
if not os.path.isdir(self.path):
raise bato...
class Directory(Component):
namevar = 'path'
leading = False
def verify(self):
if not os.path.isdir(self.path):
raise bato...
CONVERGENCE
resume where
needed
handle many
system states
transparently
avoid
unnecessary
updates
Wednesday, 3.July 13
COMPONENTS
composition of
simple components
no magic bullet, just a
lot easier to factor
your code
configure - verify - upd...
Wednesday, 3.July 13
SINGLE-
COMMAND
Wednesday, 3.July 13
REPEATABLE
RELIABLE
Wednesday, 3.July 13
SIMPLE
Wednesday, 3.July 13
ENTROPY
Wednesday, 3.July 13
EXPRESSIVENESS
READABILITY
Wednesday, 3.July 13
REUSABLE
Wednesday, 3.July 13
PLATFORM
INDEPENDENCE
Wednesday, 3.July 13
DOMAIN
AGNOSTIC
Wednesday, 3.July 13
NO ADDITIONAL
RUNTIME
DEPENDENCIES
Wednesday, 3.July 13
CONTINUITY
Wednesday, 3.July 13
MINIMAL
DOWNTIMES
Wednesday, 3.July 13
Wednesday, 3.July 13
PRACTICAL USAGE
Wednesday, 3.July 13
REQUIREMENTS
Python 2.7
SSH
virtualenv
Mercurial
Wednesday, 3.July 13
ENVIRONMENTS
[environment]
service_user = myservice
host_domain = flyingcircus.io
branch = production
[hosts]
multikarl00 ...
LOCAL
$ bin/batou-local dev localhost
Updating Hello > File(hello) > Presence(hello)
Updating Hello > File(hello) > Conten...
REMOTE
$ bin/batou-remote prod
test02.gocept.net: connecting
test01.gocept.net: connecting
test01.gocept.net: bootstrappin...
OVERRIDES
class Hello(Component):
hostname = "foo"
[environment]
...
[component:hello]
hostname = bar
Wednesday, 3.July 13
SECRETS
class Hello(Component):
db_password = none
secrets/production.cfg
[hello]
db_password = reallysecretstuff
Wednesda...
SECRETS
class Hello(Component):
db_password = none
secrets/production.cfg
[hello]
db_password = reallysecretstuff
SciFi
bu...
PROVIDE/REQUIRE
class MyApp(Component):
def configure(self):
self.provide('appserver',
self.host.fqdn)
class HAProxy(Compo...
PLATFORMS
class HAProxy(Component):
...
@platform('flyingcircus.io', HAProxy)
class SystemWideHAProxy(Component):
def conf...
VFS MAPPING
./
...
./work
./work/_/etc/haproxy.cfg
class HAProxy(Component):
def configure(self):
self += File('/etc/hapro...
FEATURES
class MyApp(Component):
features = ['instance', 'jobrunner']
def configure(self):
if 'instance' in self.features:...
Wednesday, 3.July 13
CONVERGENCE
COMPOSITION
DETAILS
Wednesday, 3.July 13
QUESTIONS?
Wednesday, 3.July 13
batou.readthedocs.org
pypi.python.org/pypi/batou
bitbucket.org/gocept/batou
Wednesday, 3.July 13
Nächste SlideShare
Wird geladen in …5
×

batou - multi(component|host|environment|.*) deployment

1.350 Aufrufe

Veröffentlicht am

Talk given at EuroPython 2013

Veröffentlicht in: Technologie, Sport
  • Als Erste(r) kommentieren

batou - multi(component|host|environment|.*) deployment

  1. 1. BATOU multi-(component¦host¦environment¦.*) deployment Wednesday, 3.July 13
  2. 2. @theuni Wednesday, 3.July 13
  3. 3. Wednesday, 3.July 13
  4. 4. Wednesday, 3.July 13
  5. 5. Wednesday, 3.July 13
  6. 6. AUTOMATING DEPLOYMENTS IS HARD Wednesday, 3.July 13
  7. 7. HOW DOES CONVERGENCE HELP? Wednesday, 3.July 13
  8. 8. HOW DOES THIS WORK WITH BATOU? Wednesday, 3.July 13
  9. 9. SOME PERSPECTIVE Wednesday, 3.July 13
  10. 10. Wednesday, 3.July 13
  11. 11. IT'S NOT THAT BAD. Wednesday, 3.July 13
  12. 12. Wednesday, 3.July 13
  13. 13. service deployment Fabric, Capistrano, ... system configuration Puppet, Chef, ... provisioning kickstart, Razor, imaging ... Wednesday, 3.July 13
  14. 14. FTP bash mkzopeinstance zc.buildout fabric Wednesday, 3.July 13
  15. 15. CONVERGENCE Wednesday, 3.July 13
  16. 16. "Everything that follows is a result of what you see here." (Dr. Alfred Lanning; I, Robot) Wednesday, 3.July 13
  17. 17. SIMPLE os.mkdir('foo') with open('foo/bar', 'w') as myfile: myfile.write('asdf') os.chmod('foo/bar', 0755) Wednesday, 3.July 13
  18. 18. •unexpected system state •can't resume •unnecessary updates os.mkdir('foo') with open('foo/bar', 'w') as myfile: myfile.write('asdf') os.chmod('foo/bar', 0755) SIMPLISTIC Wednesday, 3.July 13
  19. 19. CORRECT(?) if not os.path.isdir('foo'): os.unlink('foo') if not os.path.exists('foo'): os.mkdir('foo') try: os.lstat('foo/bar') except OSError: pass else: if os.path.isdir('foo/bar'): shutil.rmtree('foo/bar') else: os.unlink('foo/bar') if (os.path.exists('foo/bar') and open('foo/bar', 'r').read() != 'asdf'): open('foo/bar', 'w').write('asdf'): current = os.stat('foo/bar').st_mode if stat.S_IMODE(current) != 0755: os.chmod('foo', 0755) Wednesday, 3.July 13
  20. 20. SIMPLE File('foo/bar', content='asdf', mode=0755, leading=True) Wednesday, 3.July 13
  21. 21. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) Wednesday, 3.July 13
  22. 22. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) Wednesday, 3.July 13
  23. 23. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) composition operator Wednesday, 3.July 13
  24. 24. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) composition operator order matters Wednesday, 3.July 13
  25. 25. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass Wednesday, 3.July 13
  26. 26. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run "anywhere" Wednesday, 3.July 13
  27. 27. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target run "anywhere" Wednesday, 3.July 13
  28. 28. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target run "anywhere" after all sub- components Wednesday, 3.July 13
  29. 29. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target only if needed run "anywhere" after all sub- components Wednesday, 3.July 13
  30. 30. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target only if needed run "anywhere" after all sub- components keep delegating! Wednesday, 3.July 13
  31. 31. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) Wednesday, 3.July 13
  32. 32. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) could be done with recursive composition Wednesday, 3.July 13
  33. 33. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) could be done with recursive composition refactor with sub-components if too complex Wednesday, 3.July 13
  34. 34. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) all methods optional: no configure() could be done with recursive composition refactor with sub-components if too complex Wednesday, 3.July 13
  35. 35. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) all methods optional: no configure() could be done with recursive composition pattern: just wipe out what's wrong refactor with sub-components if too complex Wednesday, 3.July 13
  36. 36. CONVERGENCE resume where needed handle many system states transparently avoid unnecessary updates Wednesday, 3.July 13
  37. 37. COMPONENTS composition of simple components no magic bullet, just a lot easier to factor your code configure - verify - update Wednesday, 3.July 13
  38. 38. Wednesday, 3.July 13
  39. 39. SINGLE- COMMAND Wednesday, 3.July 13
  40. 40. REPEATABLE RELIABLE Wednesday, 3.July 13
  41. 41. SIMPLE Wednesday, 3.July 13
  42. 42. ENTROPY Wednesday, 3.July 13
  43. 43. EXPRESSIVENESS READABILITY Wednesday, 3.July 13
  44. 44. REUSABLE Wednesday, 3.July 13
  45. 45. PLATFORM INDEPENDENCE Wednesday, 3.July 13
  46. 46. DOMAIN AGNOSTIC Wednesday, 3.July 13
  47. 47. NO ADDITIONAL RUNTIME DEPENDENCIES Wednesday, 3.July 13
  48. 48. CONTINUITY Wednesday, 3.July 13
  49. 49. MINIMAL DOWNTIMES Wednesday, 3.July 13
  50. 50. Wednesday, 3.July 13
  51. 51. PRACTICAL USAGE Wednesday, 3.July 13
  52. 52. REQUIREMENTS Python 2.7 SSH virtualenv Mercurial Wednesday, 3.July 13
  53. 53. ENVIRONMENTS [environment] service_user = myservice host_domain = flyingcircus.io branch = production [hosts] multikarl00 = nginx, haproxy multikarl01 = postgres, redis, memcached, crontab multikarl12 = supervisor, logrotate, doctotext, myapp multikarl13 = supervisor, logrotate, doctotext, myapp Wednesday, 3.July 13
  54. 54. LOCAL $ bin/batou-local dev localhost Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) $ bin/batou-local dev localhost $ Wednesday, 3.July 13
  55. 55. REMOTE $ bin/batou-remote prod test02.gocept.net: connecting test01.gocept.net: connecting test01.gocept.net: bootstrapping test02.gocept.net: bootstrapping OK OK Deploying test01.gocept.net/hello Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) OK Deploying test02.gocept.net/hello Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) OK Wednesday, 3.July 13
  56. 56. OVERRIDES class Hello(Component): hostname = "foo" [environment] ... [component:hello] hostname = bar Wednesday, 3.July 13
  57. 57. SECRETS class Hello(Component): db_password = none secrets/production.cfg [hello] db_password = reallysecretstuff Wednesday, 3.July 13
  58. 58. SECRETS class Hello(Component): db_password = none secrets/production.cfg [hello] db_password = reallysecretstuff SciFi but close Wednesday, 3.July 13
  59. 59. PROVIDE/REQUIRE class MyApp(Component): def configure(self): self.provide('appserver', self.host.fqdn) class HAProxy(Component): def configure(self): self.backends = self.require('appserver') Wednesday, 3.July 13
  60. 60. PLATFORMS class HAProxy(Component): ... @platform('flyingcircus.io', HAProxy) class SystemWideHAProxy(Component): def configure(self): self += File('/etc/haproxy', ensure='symlink', link_to=self.parent.haproxy_cfg.path) Wednesday, 3.July 13
  61. 61. VFS MAPPING ./ ... ./work ./work/_/etc/haproxy.cfg class HAProxy(Component): def configure(self): self += File('/etc/haproxy') [environment] ... [vfs] sandbox = Developer Wednesday, 3.July 13
  62. 62. FEATURES class MyApp(Component): features = ['instance', 'jobrunner'] def configure(self): if 'instance' in self.features: ... [hosts] hosta = myapp:instance hostb = myapp:jobrunner hostc = myapp:instance, myapp:jobrunner hostd = myapp Wednesday, 3.July 13
  63. 63. Wednesday, 3.July 13
  64. 64. CONVERGENCE COMPOSITION DETAILS Wednesday, 3.July 13
  65. 65. QUESTIONS? Wednesday, 3.July 13
  66. 66. batou.readthedocs.org pypi.python.org/pypi/batou bitbucket.org/gocept/batou Wednesday, 3.July 13

×