Jenkins entwined with deployment and pragmatism

1. Jenkins entwined with deployment and pragma>sm Eric Ritchie DevOps Engineer Novadex +49 (0) 7142 / 788905-‐0 eric.ritchie@novadex.com Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.

2. Eric Ritchie bids you welcome! Never heard of him? Aside from looking like this, he is: §  currently working as DevOps Engineer and System Administrator at Novadex GmbH §  a Zend Framework and PHP ZCE & MySQL DBA §  a Linux System Administrator with 18 years experience §  a PHP developer since version 3.0.18 (or so) §  known to speak at a conference or two Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 2

3. LeaerMaschine: Custom-‐tailored Communica>on Personalized data stream Cross-‐Channel Output Data protec>on Security Content management E-‐commerce integra>on CRM integra>on Portal Op>on Campaign management Collabora>on Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 3

4. What I read in a book once… „DevOPs should aim for 10 deployments per day“ Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 4

5. Novadex’s key tool: Jenkins §  Great for Con?nuous Integra?on §  Can hide complex processes behind a nice buDon §  Deployment remains one of those complex processes §  §  Requires a lot of scrip>ng/manual work This risks our deployment goal Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 5

6. An example of that scrip>ng <target name="package"> <echo message="...using UPSTREAM_BUILD_NUMBER received from triggering job for packagenaming. UPSTREAM_BUILD_NUMBER is [${UPSTREAM_BUILD_NUMBER}]"/> <!-‐-‐ leaermaschine-‐complete packaging -‐-‐> <!-‐-‐ wipe the dir from possible previous copy opera>on -‐-‐> <delete includeemptydirs="true"> <fileset dir="${package.contents.basedir}" includes="**/*"/> <fileset dir="${package.basedir}/DEBIAN" includes="**/*"/> </delete> <!-‐-‐ prepare debian control file -‐-‐> <copy todir="${package.basedir}/DEBIAN" overwrite="true"> <fileset file="resource/package/test/control"/> <filterset begintoken="[[" endtoken="]]"> <filter token="version" value="${UPSTREAM_BUILD_NUMBER}-‐1"/> <filter token="name" value="leaermaschine-‐complete"/> <filter token="depends" value=""/> <filter token="descrip>on" value="Complete Leaermaschine applica>on"/> </filterset> </copy> <!-‐-‐ copy pos>nstalla>on script -‐-‐> <copy file="${basedir}/resource/package/test/pos>nst_lm" tofile="${package.basedir}/DEBIAN/pos>nst"/> <chmod file="${package.basedir}/DEBIAN/pos>nst" perm="775"/> <copy todir="${package.contents.basedir}"> <fileset dir="${export.basedir}"> <snip> </fileset> </copy> <exec executable="sudo" dir="${working.basedir}" failonerror="true"> <arg line="sudo dpkg -‐b ${package.basedir} ${dist.basedir}/leaermaschine-‐complete_${UPSTREAM_BUILD_NUMBER}-‐1_all.deb"/> </exec> </target> Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 6

7. Deployment v1.x? §  Applica?on bundled into a .deb package §  §  §  As shown on the previous slide (yeah, that’s what it did) Shell scrip>ng available for in deployment manipula>ons SCP’d to target machine, installed via SSH exec §  Need to set up SSL cer>ﬁcates to allow password free access Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 7

8. Problems? §  Who is comfortable with bash shell scrip?ng? §  What happens when something goes wrong? §  §  Not really all that cluster friendly §  §  Each version of the .deb package is cached Need to grant unlimited access to live servers, argh! §  §  Ok, we have Chef/Puppet if we like overkill Disk cluDer §  §  Ever tried to roll back a copied .deb package in a hurry? Remember that SSL cer>ﬁcate we installed? What if we need to support Red Hat also? §  Hypothe>cal in the case of Novadex, but s>ll a valid problem. Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 8

9. Enter Zend Server deployment (and me) Cluster/Cloud ready, plaxorm agnos>c, PHP scrip>ng and version handling with integrated roll back. Yes, I used to work for Zend ;) Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 9

10. Tale of the liale known tool: zs-‐manage §  We all have heard about the Web API, right? §  §  We have had command line deployment tools since day one §  §  But nobody knew about them, which is a pity zs-‐manage has many advantages §  §  §  §  You may be forgiven for thinking it is a new thing Can leverage all deployment features Does not rely on SSL cer>ﬁcates for security Same commands for local/cluster/cloud deployment But beware, there are some caveats §  Pragma>sm to follow... Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 10

11. Let’s empower Jenkins… <!-‐-‐ retrieve the app.id for the applica>on, speciﬁed by app.name. may be empty if app hasn't already deployed -‐-‐> <exec executable="bash" dir="${basedir}" outputproperty="app.id" errorproperty="err" failonerror="true"> <arg value="-‐c" /> <arg value="/usr/local/zend/bin/zs-‐manage app-‐get-‐status -‐N ${zs.api.name} -‐U ${zs.url} -‐K ${zs.api.key} 2>&1 | grep "APPINFOs.*${app.name}.*" | awk '{print $2}'" /> </exec> <if> <equals arg1="${app.id}" arg2="" /> <then> <echo message="Couldn't determine app.id for app.name '${app.name}' -‐ preparing ini>al deployment of applica>on ..." /> <!-‐-‐ deploy applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐deploy -‐u "NDX_ENV=${zs.ndx.env}" -‐c -‐b hap://${app.name}/ -‐a ${app.name} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </then> <else> <echo message="app.name '${app.name}', app.id is ${app.id} -‐ preparing the update of applica>on ..." /> <!-‐-‐ update applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </else> </if> Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 11

12. Crea>ng the package to be deployed §  We have a choice §  §  §  §  Zend Studio -‐ a manual process zdpack -‐ Zend standard zip -‐ needs no introduc>on Novadex have used zip at ?mes §  But currently use the following code <exec executable="/usr/local/zend/bin/zdpack" dir="${working.basedir}" failonerror="true"> <!-‐-‐ NOTE: name of the created package is the name specified in deployment.xml name-‐tag -‐-‐> <arg line="pack -‐-‐src-‐dir=${package.contents.basedir} -‐-‐scripts-‐dir=${basedir}/resource/zs_deploy_new/scripts -‐-‐package-‐descriptor=${working.basedir}/deployment.xml -‐-‐output-‐dir=${working.basedir}" /> </exec> <!-‐-‐ rename created package to reflect version of package contents -‐-‐> <move file="${working.basedir}/${app.name}.zpk" tofile="${working.basedir}/${zs.package.name}" overwrite="true"/> <!-‐-‐ copy the new package from temp loca>on to package directory -‐-‐> <copy file="${working.basedir}/${zs.package.name}" todir="${zs.package.des>na>ondir}" overwrite="true"/> Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 12

13. Sadly, size does maaer §  You don’t leave your doors unlocked when you leave home §  §  Web API could be an aDack vector §  §  §  Most admins are equally careful with their servers Which is why Zend take steps to secure it Web API calls must be signed with, amongst other informa>on, a >mestamp To help prevent replay aDacks, calls must complete within 300 seconds §  §  This security feature can shoot us in the foot In short, the network speed determines our maximum packet size Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 13

14. Working around the size issue §  No perfect solu?ons, yet §  §  I am working on Zend ;) But we can apply some pragma?sm §  §  §  §  §  §  Package can be broken into sub-‐packages Zend Deployment has a new Library feature, use it Pull instead of push Upgrade the pipe Time travel Or, emulate the GUI Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 14

15. Our solu>on GUI emula?on: It works for us <echo message="copying ${zs.package.name} to ${remote.hostname}" /> <scp file="${zs.package.des>na>ondir}/${zs.package.name}" todir="${remote.user}@${remote.hostname}:${remote.dir}" keyfile="${remote.keyfile}" /> <echo message="app.name '${app.name}', app.id is '${app.id}' -‐ preparing the update of applica>on ..." /> <!-‐-‐ execute zs-‐manage locally on host where package should be deployed! -‐-‐> <sshexec host="${remote.hostname}" username="${remote.user}" keyfile="${remote.keyfile}" command="/usr/local/zend/bin/zs-‐manage app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${remote.dir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${remote.webapi.url}" /> ! Just remember: SCP/SSH may not be allowed by your opera>ons team. You may need another solu>on Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. ! 20/11/13 15

16. Memory and lots of it §  Ques?on: Where does all this deployment data go? §  §  The GUI allocates memory equivalent to three ?mes the packet size §  §  Hint: The WebAPI is part of the GUI Since the data is loaded, copied and ﬁnally unpacked (don’t ask) Where is the GUI’s memory limit conﬁgured? §  §  Not in the GUI, that would spoil the fun ;) Can you guess? Answer: The memory_limit variable in: /usr/local/zend/gui/lighapd/etc/php-‐fcgi.ini Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 16

17. Aaack of the cluster §  Zend deployment can be hard to synchronise §  §  An extreme case is the “run once” node §  §  In other words, you cannot guarantee when all nodes will run the same version of the applica>on Database migra>ons are usually performed here, so deployment is slower This caused problems with our “In maintenance” page §  Time for some extreme pragma>sm… Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 17

18. I am a Job Queue fan… so I wrote this // Sanity check if (!class_exists('ZendJobQueue')){ logger("Job Queue seems to be disabled!"); exit(1); } if (getenv("ZS_RUN_ONCE_NODE") == 1) { $name = DEPLOYMENT_JOB_BASENAME.$appCurrentVersion; $queue = new ZendJobQueue(); // Before scheduling our job, just make sure that it has not been scheduled before $total = null; $jobArray = $queue-‐>getJobsList(array('name' => $name), $total); if ($total != 0) { foreach($jobArray AS $job) { $queue-‐>removeJob($job['id']); } } // Schedule the job script to run a long >me in the future (since we don't really need it to run) $runAt = date('Y-‐m-‐d H:i:s', strto>me('+7 days')); $semaphoreJobId = $queue-‐>createHapJob('hap://www.google.de/search?q=dummy', array(), array('name' => $name, 'schedule_?me' => $runAt)); if (!$semaphoreJobId) { logger("Semphore job was not accepted in the queue!"); exit(1); } } Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 18

19. …and this at the end of the script if (getenv("ZS_RUN_ONCE_NODE") == 1) { // We are the RUN ONCE script so clear the semaphore $queue-‐>removeJob($semaphoreJobId); } else { // We are not the RUN ONCE script so wait for it to clear the semaphore $queue = new ZendJobQueue(); $total = 1; $status = ZendJobQueue::JOB_STATUS_PENDING | ZendJobQueue::JOB_STATUS_SCHEDULED; while ($total != 0) { $jobArray = $queue-‐>getJobsList(array('name' => $name, 'status' => $status), $total); sleep(1); } } We now plan to use an Nginx, that spontaneously appeared in our architecture, to set our maintenance page. Nevertheless, it was s>ll a cool idea ;) Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 19

20. Finding out what is happening §  So far so good, but we s?ll don’t have true integra?on §  §  Jenkins has no liale* visibility over the deployment process We can log the output of our deployment scripts §  §  §  But these are not available to Jenkins We can make them available to the GUI, at least, by adding the paths to the GUI_AVAILABLE_LOGS table in the Zend Server database More convenient, but we s>ll need to go check the GUI a‚er each deployment * Last minute slide hack Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 20

21. WebAPI to the rescue… again §  Features added to the WebAPI allow registered logs to be remotely read §  §  logsReadLines logsGetLogﬁle §  We can ﬁnally integrate log output into our Jenkins reports §  Just a few problems §  §  §  Finding the “run once” node These methods are not implemented in zs-‐manage (yet) Wri>ng the code (I haven’t had the >me yet) Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights. 20/11/13 21

22. Thank you very much for your aaen>on www.leaermaschine.com www.novadex.com Please scan this …and tell me where it goes Honestly, marke>ng wouldn’t tell me! Eric Ritchie Novadex GmbH DevOps Engineer hDps://joind.in/9926 Stuagarter Straße 59 74321 Bie>gheim-‐Bissingen +49 (0) 7142 / 788905-‐0 Germany eric.ritchie@novadex.com Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, 20/11/13 distribu>on, as well as in the event of applica>ons for industrial property rights. 22

Jenkins entwined with deployment and pragmatism

Du liest eine Vorschau.

Hier ansehen

Jenkins entwined with deployment and pragmatism

Weitere Verwandte Inhalte

Andere mochten auch (20)

Ähnlich wie Jenkins entwined with deployment and pragmatism (20)

Aktuellste (20)