Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Groovy DevOps in the Cloud for OpenSlava 2014

926 Aufrufe

Veröffentlicht am

Talk focuses on a set of tools to automate the provisioning of (cloud) servers using Groovy libraries and Gradle plugins. We explore how to leverage those to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud - elegant and groovy. Java/Groovy developers interested in reusing their existing skills for infrastructure provisioning and learning more about problems encountered during system operations should come to this presentation.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Groovy DevOps in the Cloud for OpenSlava 2014

  1. 1. 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Twitter: @aestasit ••••• 03
  4. 4. What's this presentation about? Our take on: DevOps Intrastructure Provisioning Continuous Integration Continuous Delivery •••• 04
  5. 5. Technologies Groovy - http://groovy.codehaus.org Gradle - http://gradle.org Jenkins - http://jenkins-ci.org Puppet - http://puppetlabs.com AWS - http://aws.amazon.com ••••• 05
  6. 6. Developers + Operations = ? 06
  7. 7. Silos 07
  8. 8. Conflicts 08
  9. 9. Risk 09
  10. 10. Agile 10
  11. 11. What is DevOps? 11
  12. 12. C.A.M.S. Culture : People over processes and tools. Software is made by and for people. Automation : Automation is essential for DevOps to gain quick feedback. Measurement : DevOps finds a specific path to measurement. Quality and shared (or at least aligned) incentives are critical. Sharing : Creates a culture where people share ideas, processes, and tools. • • • • 12
  13. 13. It's not about tools! 13
  14. 14. It's about culture! 14
  15. 15. We started with the tools! 15
  16. 16. Infrastructure as code Automate the provisioning and maintenance of servers: Build from source control Utilize existing tools Ensure testability ••• 16
  17. 17. Configuration propagation 17
  18. 18. Configuration propagation 18
  19. 19. Changes Imagine manually uploading *.class files and repackaging JAR directly on production servers when you have an urgent code change. 19
  20. 20. Deployment is automatic! 20
  21. 21. And, so, should be... 21
  22. 22. infrastructure configuration changes! 22
  23. 23. No manual changes! 23
  24. 24. Building an automation toolkit Automation is key We are JVM hackers Fragmented ecosystem ••• 24
  25. 25. Initial toolset Gradle Groovy Ant Python/WLST Shell scripts ••••• 25
  26. 26. Required tooling Infrastructure connectivity Infrastructure provisioning Infrastructure virtualization Infrastructure testing •••• 26
  27. 27. First Blood 27
  28. 28. Ant + Gradle ant.taskdef( name: 'scp', classname: 'o.a.t.a.t.o.ssh.Scp', classpath: configurations.secureShell.asPath) ant.taskdef( name: 'sshexec', classname: 'o.a.t.a.t.o.ssh.SSHExec', classpath: configurations.secureShell.asPath) 01. 02. 03. 04. 0056.. 07. 08. 09. 28
  29. 29. Simple call ant.sshexec( host: host, username: user, password: password, command: command, trust: 'true', failonerror: failOnError) 01. 02. 03. 04. 05. 06. 07. 29
  30. 30. Next step: wrapper function def ssh(String command, Properties props, boolean failOnError = false, String suCommandQuoteChar = "'", String outputProperty = null) { ... } 01. 02. 03. 04. 05. 06. 07. 30
  31. 31. Next step: wrapper function def scp(String file, String remoteDir, Properties props) { ... } 01. 02. 03. 04. 05. 31
  32. 32. Task example I task installFonts << { forAllServers { props -> ssh('yes | yum install *font*', props) } } 01. 02. 03. 04. 05. 32
  33. 33. Task example II task uninstallNginx << { forAllServers { props -> 01. 02. 03. 04. 05. 06. 07. 08. 09. 33 ssh('/etc/init.d/nginx stop', props) ssh('yes | yum remove nginx', props, true) ssh('rm -rf /etc/yum.repos.d/nginx.repo', props) ssh('rm -rf /var/log/nginx', props) ssh('rm -rf /etc/nginx /var/nginx', props) } }
  34. 34. Drawbacks New connection each time Excplicit repeating parameters Complex scripts are hard to maintain Tasks are not idempotent •••• 34
  35. 35. Sshoogr 35
  36. 36. Sshoogr features Groovy-based SSH DSL for: Remote command execution File uploading/downloading Tunneling ••• 36
  37. 37. Why Groovy? Groovy is perfect choice for scripting Gradle build scripts are Groovy Very mature, concise syntax Extremely easy to produce DSL We wrote a book about it! ••••• 37
  38. 38. Shameless plug 38
  39. 39. Sshoogr usage (import) @Grab( group='com.aestasit.infrastructure.sshoogr', module='sshoogr', version='0.9.16') import static com.aestasit.ssh.DefaultSsh.* 01. 02. 03. 04. 05. 39
  40. 40. Sshoogr usage (defaults) defaultUser = 'root' defaultKeyFile = new File('secret.pem') execOptions { verbose = true showCommand = true } 01. 02. 03. 04. 05. 06. 40
  41. 41. Sshoogr usage (connection) remoteSession { url = 'user2:654321@localhost:2222' exec 'rm -rf /tmp/*' exec 'touch /var/lock/my.pid' remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 41
  42. 42. Sshoogr usage (multi-line content) remoteFile('/etc/yum.repos.d/puppet.repo').text = ''' [puppet] name=Puppet Labs Packages baseurl=http://yum.puppetlabs.com/el/ enabled=0 gpgcheck=0 ''' 01. 02. 03. 04. 05. 06. 07. 42
  43. 43. Sshoogr usage (file copying) remoteSession { scp { from { localDir "$buildDir/application" } into { remoteDir '/var/bea/domain/application' } } } 01. 02. 03. 04. 05. 06. 43
  44. 44. Sshoogr usage (command result) def result = exec(command: '/usr/bin/mycmd', failOnError: false, showOutput: false) if (result.exitStatus == 1) { result.output.eachLine { line -> 01. 02. 03. 04. 05. 06. 07. 08. 09. 44 if (line.contains('WARNING')) { throw new RuntimeException("Warning!!!") } } }
  45. 45. Sshoogr usage (shortcuts) if (ok('/usr/bin/mycmd')) { ... } if (fail('/usr/bin/othercmd')) { ... } 01. 02. 03. 04. 05. 06. 45
  46. 46. Sshoogr usage (tunnels) tunnel('1.2.3.4', 8080) { int localPort -> def url = "http://localhost:${localPort}/flushCache" def result = new URL(url).text if (result == 'OK') { 01. 02. 03. 04. 05. 06. 07. 08. 09. 46 println "Cache is flushed!" } else { throw new RuntimeException(result) } }
  47. 47. Sshoogr usage (prefix/suffix) prefix('sudo ') { exec 'rm -rf /var/log/abc.log' exec 'service abc restart' } suffix(' >> output.log') { exec 'yum -y install nginx' exec 'yum -y install mc' exec 'yum -y install links' } 01. 02. 03. 04. 05. 06. 07. 08. 09. 47
  48. 48. Still problems Complex scripts are still not easy to maintain Scripts are usually not idempotent •• 48
  49. 49. Puppet 49
  50. 50. Why Puppet? More mature than competition Large community Readable DSL Good acceptance from DEVs and OPs No need to learn Ruby ;) ••••• 50
  51. 51. Puppet example 51
  52. 52. Puppet provisioning 52
  53. 53. Puppet provisioning 53
  54. 54. Puppet provisioning 54
  55. 55. Puppet provisioning 55
  56. 56. Puppet state management 56
  57. 57. Puppet state management 57
  58. 58. Puppet state management 58
  59. 59. Puppet modules 59
  60. 60. Puppet modules 60
  61. 61. Puppet modules 61
  62. 62. Sshoogr + Gradle + Puppet 62
  63. 63. Upload modules task uploadModules << { remoteSession { exec 'rm -rf /tmp/repo.zip' scp { from { localFile "${buildDir}/repo.zip" } into { remoteDir "/root" } } ... 01. 02. 03. 04. 05. 06. 07. 08. 63
  64. 64. Upload modules ... exec 'rm -rf /etc/puppet/modules' exec 'unzip /tmp/repo.zip -d /etc/puppet/modules' } } 01. 02. 03. 04. 05. 64
  65. 65. Apply manifests task puppetApply(dependsOn: uploadModules) << { remoteSession { 01. 02. 03. 04. 05. 06. 07. 08. 09. 65 scp { from { localFile "${buildDir}/setup.pp" } into { remoteDir "/tmp" } } exec 'puppet apply /tmp/setup.pp' } }
  66. 66. What we solved? Separated infrastructure state description and operations tasks Scripts became more maintainable and idempotent •• 66
  67. 67. In the meanwhile... We started developing complex/generic Puppet modules Modules need proper testing ...on different platforms ••• 67
  68. 68. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! ••• 68
  69. 69. PUnit 69
  70. 70. PUnit Simple testing tool for verifying remote server state Uses Sshoogr and JUnit Reuse reporting features of JUnit As simple as ... •••• 70
  71. 71. PUnit example (derby) class DerbyInstallTest extends BasePuppetIntegrationTest { @Before void installDerby() { apply("include derby") } ... } 01. 02. 03. 04. 05. 06. 07. 08. 71
  72. 72. PUnit example (derby) @Test void ensureDerbyRunning() { command('service derby status > derbystatus.log') assertTrue fileText("/root/derbystatus.log") .contains('Derby') assertTrue fileText("/root/derbystatus.log") .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 72
  73. 73. PUnit example (derby) @Test void ensureCanConnect() { Thread.sleep(10000) uploadScript() command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 'testDataScript.sql > derbytest.log') ... 01. 02. 03. 04. 05. 06. 07. 73
  74. 74. Continuous integration 74
  75. 75. Jenkins De-facto standard Stable There is a plugin for that! ••• 75
  76. 76. Jenkins build 76
  77. 77. Next problem? 77
  78. 78. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? ••• 78
  79. 79. Amazon Web Services 79
  80. 80. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety •••• 80
  81. 81. Gramazon 81
  82. 82. Gramazon Groovy-based API for interacting with EC2 Integration with Gradle •• 82
  83. 83. Gramazon example I task startInstance(type: StartInstance) { keyName 'cloud-do' securityGroup 'cloud-do' instanceName 'gramazon/cloud-do' stateFileName 'cloud-do.json' ami 'ami-6f07e418' instanceType 't1.micro' waitForStart true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 83
  84. 84. Gramazon example II task terminateInstance(type: TerminateInstance) { stateFileName 'cloud-do.json' } 01. 02. 03. 84
  85. 85. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 85
  86. 86. Next issue? 86
  87. 87. Imgr 87
  88. 88. Imgr A tool for building images Inspired by Packer •• 88
  89. 89. Supports Shell Puppet •• 89
  90. 90. Configuration example 90
  91. 91. Summary 91
  92. 92. Images, manifests, tasks 92
  93. 93. The big picture 93
  94. 94. Aetomation 94
  95. 95. Conclusions Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Don't be afraid to try new technologies Infrastructure configuration is outsourcable now! Automate! •••••• 95
  96. 96. Next steps? Create more documentation and examples Add more DSL convience methods Extend integration with Gradle Add Windows connectivity/scripting support (groowin) Define richer model for EC2 and potentially other clouds Extend support for other provisioning tools •••••• 96
  97. 97. Reading material 97
  98. 98. The Phoenix Project 98
  99. 99. Continuous Delivery 99
  100. 100. Release It 100
  101. 101. Programming Amazon EC2 101
  102. 102. Gradle in Action 102
  103. 103. Groovy 2 Cookbook 103
  104. 104. Technologies to follow Vagrant - http://www.vagrantup.com/ Docker - https://www.docker.io/ Packer - http://www.packer.io/ Qemu - http://wiki.qemu.org/ jclouds - http://jclouds.apache.org/ Cloudbees - http://www.cloudbees.com/ •••••• 104
  105. 105. One more thing... 105
  106. 106. It's all Open Source! 106
  107. 107. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle PUnit: https://github.com/aestasit/puppet-unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr ••••• 107
  108. 108. Seeking contributors! 108
  109. 109. Questions? 109
  110. 110. Thank you! 110

×