Jenkins (to name just one CI tool) is a central pillar of agile development and does a terrific job of automating the project build process. Providing a reliable deployment path to a cluster of servers is something that it does less well and forms the challenge being faced by many today. In this talk Novadex GmbH, a leader in cloud based customer communication management, will demonstrate their pragmatic solution to this challenge which uses the toolkit provided by Zend Server

1. Jenkins
entwined
with
deployment
and
pragma>sm
Eric
Ritchie
DevOps
Engineer
Novadex
+49
(0)
7142
/
788905-­‐0
eric.ritchie@novadex.com
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.

2. Eric
Ritchie
bids
you
welcome!
Never
heard
of
him?
Aside
from
looking
like
this,
he
is:
§ currently
working
as
DevOps
Engineer
and
System
Administrator
at
Novadex
GmbH
§ a
Zend
Framework
and
PHP
ZCE
&
MySQL
DBA
§ a
Linux
System
Administrator
with
18
years
experience
§ a
PHP
developer
since
version
3.0.18
(or
so)
§ known
to
speak
at
a
conference
or
two
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
2

3. LeaerMaschine:
Custom-­‐tailored
Communica>on
Personalized
data
stream
Cross-­‐Channel
Output
Data
protec>on
Security
Content
management
E-­‐commerce
integra>on
CRM
integra>on
Portal
Op>on
Campaign
management
Collabora>on
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
3

4. What
I
read
in
a
book
once…
„DevOPs
should
aim
for
10
deployments
per
day“
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
4

5. Novadex’s
key
tool:
Jenkins
§
Great
for
Con?nuous
Integra?on
§
Can
hide
complex
processes
behind
a
nice
buDon
§
Deployment
remains
one
of
those
complex
processes
§
§
Requires
a
lot
of
scrip>ng/manual
work
This
risks
our
deployment
goal
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
5

6. An
example
of
that
scrip>ng
<target
name="package">
<echo
message="...using
UPSTREAM_BUILD_NUMBER
received
from
triggering
job
for
packagenaming.
UPSTREAM_BUILD_NUMBER
is
[${UPSTREAM_BUILD_NUMBER}]"/>
<!-­‐-­‐
leaermaschine-­‐complete
packaging
-­‐-­‐>
<!-­‐-­‐
wipe
the
dir
from
possible
previous
copy
opera>on
-­‐-­‐>
<delete
includeemptydirs="true">
<fileset
dir="${package.contents.basedir}"
includes="**/*"/>
<fileset
dir="${package.basedir}/DEBIAN"
includes="**/*"/>
</delete>
<!-­‐-­‐
prepare
debian
control
file
-­‐-­‐>
<copy
todir="${package.basedir}/DEBIAN"
overwrite="true">
<fileset
file="resource/package/test/control"/>
<filterset
begintoken="[["
endtoken="]]">
<filter
token="version"
value="${UPSTREAM_BUILD_NUMBER}-­‐1"/>
<filter
token="name"
value="leaermaschine-­‐complete"/>
<filter
token="depends"
value=""/>
<filter
token="descrip>on"
value="Complete
Leaermaschine
applica>on"/>
</filterset>
</copy>
<!-­‐-­‐
copy
pos>nstalla>on
script
-­‐-­‐>
<copy
file="${basedir}/resource/package/test/pos>nst_lm"
tofile="${package.basedir}/DEBIAN/pos>nst"/>
<chmod
file="${package.basedir}/DEBIAN/pos>nst"
perm="775"/>
<copy
todir="${package.contents.basedir}">
<fileset
dir="${export.basedir}">
<snip>
</fileset>
</copy>
<exec
executable="sudo"
dir="${working.basedir}"
failonerror="true">
<arg
line="sudo
dpkg
-­‐b
${package.basedir}
${dist.basedir}/leaermaschine-­‐complete_${UPSTREAM_BUILD_NUMBER}-­‐1_all.deb"/>
</exec>
</target>
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
6

7. Deployment
v1.x?
§
Applica?on
bundled
into
a
.deb
package
§
§
§
As
shown
on
the
previous
slide
(yeah,
that’s
what
it
did)
Shell
scrip>ng
available
for
in
deployment
manipula>ons
SCP’d
to
target
machine,
installed
via
SSH
exec
§
Need
to
set
up
SSL
cer>ficates
to
allow
password
free
access
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
7

8. Problems?
§
Who
is
comfortable
with
bash
shell
scrip?ng?
§
What
happens
when
something
goes
wrong?
§
§
Not
really
all
that
cluster
friendly
§
§
Each
version
of
the
.deb
package
is
cached
Need
to
grant
unlimited
access
to
live
servers,
argh!
§
§
Ok,
we
have
Chef/Puppet
if
we
like
overkill
Disk
cluDer
§
§
Ever
tried
to
roll
back
a
copied
.deb
package
in
a
hurry?
Remember
that
SSL
cer>ficate
we
installed?
What
if
we
need
to
support
Red
Hat
also?
§
Hypothe>cal
in
the
case
of
Novadex,
but
s>ll
a
valid
problem.
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
8

9. Enter
Zend
Server
deployment
(and
me)
Cluster/Cloud
ready,
plaxorm
agnos>c,
PHP
scrip>ng
and
version
handling
with
integrated
roll
back.
Yes,
I
used
to
work
for
Zend
;)
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
9

10. Tale
of
the
liale
known
tool:
zs-­‐manage
§
We
all
have
heard
about
the
Web
API,
right?
§
§
We
have
had
command
line
deployment
tools
since
day
one
§
§
But
nobody
knew
about
them,
which
is
a
pity
zs-­‐manage
has
many
advantages
§
§
§
§
You
may
be
forgiven
for
thinking
it
is
a
new
thing
Can
leverage
all
deployment
features
Does
not
rely
on
SSL
cer>ficates
for
security
Same
commands
for
local/cluster/cloud
deployment
But
beware,
there
are
some
caveats
§
Pragma>sm
to
follow...
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
10

11. Let’s
empower
Jenkins…
<!-­‐-­‐
retrieve
the
app.id
for
the
applica>on,
specified
by
app.name.
may
be
empty
if
app
hasn't
already
deployed
-­‐-­‐>
<exec
executable="bash"
dir="${basedir}"
outputproperty="app.id"
errorproperty="err"
failonerror="true">
<arg
value="-­‐c"
/>
<arg
value="/usr/local/zend/bin/zs-­‐manage
app-­‐get-­‐status
-­‐N
${zs.api.name}
-­‐U
${zs.url}
-­‐K
${zs.api.key}
2&gt;&amp;1
|
grep
&quot;APPINFOs.*${app.name}.*&quot;
|
awk
'{print
$2}'"
/>
</exec>
<if>
<equals
arg1="${app.id}"
arg2=""
/>
<then>
<echo
message="Couldn't
determine
app.id
for
app.name
'${app.name}'
-­‐
preparing
ini>al
deployment
of
applica>on
..."
/>
<!-­‐-­‐
deploy
applica>on
via
zend
server
command
-­‐-­‐>
<exec
executable="/usr/local/zend/bin/zs-­‐manage"
dir="/tmp"
failonerror="true">
<arg
line="app-­‐deploy
-­‐u
&quot;NDX_ENV=${zs.ndx.env}&quot;
-­‐c
-­‐b
hap://${app.name}/
-­‐a
${app.name}
-­‐p
${zs.package.des>na>ondir}/${zs.package.name}
-­‐N
${zs.api.name}
-­‐K
${zs.api.key}
-­‐U
${zs.url}"
/>
</exec>
</then>
<else>
<echo
message="app.name
'${app.name}',
app.id
is
${app.id}
-­‐
preparing
the
update
of
applica>on
..."
/>
<!-­‐-­‐
update
applica>on
via
zend
server
command
-­‐-­‐>
<exec
executable="/usr/local/zend/bin/zs-­‐manage"
dir="/tmp"
failonerror="true">
<arg
line="app-­‐update
-­‐u
&quot;NDX_ENV=${zs.ndx.env}&quot;
-­‐i
${app.id}
-­‐p
${zs.package.des>na>ondir}/${zs.package.name}
-­‐N
${zs.api.name}
-­‐K
${zs.api.key}
-­‐U
${zs.url}"
/>
</exec>
</else>
</if>
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
11

12. Crea>ng
the
package
to
be
deployed
§
We
have
a
choice
§
§
§
§
Zend
Studio
-­‐
a
manual
process
zdpack
-­‐
Zend
standard
zip
-­‐
needs
no
introduc>on
Novadex
have
used
zip
at
?mes
§
But
currently
use
the
following
code
<exec
executable="/usr/local/zend/bin/zdpack"
dir="${working.basedir}"
failonerror="true">
<!-­‐-­‐
NOTE:
name
of
the
created
package
is
the
name
specified
in
deployment.xml
name-­‐tag
-­‐-­‐>
<arg
line="pack
-­‐-­‐src-­‐dir=${package.contents.basedir}
-­‐-­‐scripts-­‐dir=${basedir}/resource/zs_deploy_new/scripts
-­‐-­‐package-­‐descriptor=${working.basedir}/deployment.xml
-­‐-­‐output-­‐dir=${working.basedir}"
/>
</exec>
<!-­‐-­‐
rename
created
package
to
reflect
version
of
package
contents
-­‐-­‐>
<move
file="${working.basedir}/${app.name}.zpk"
tofile="${working.basedir}/${zs.package.name}"
overwrite="true"/>
<!-­‐-­‐
copy
the
new
package
from
temp
loca>on
to
package
directory
-­‐-­‐>
<copy
file="${working.basedir}/${zs.package.name}"
todir="${zs.package.des>na>ondir}"
overwrite="true"/>
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
12

13. Sadly,
size
does
maaer
§
You
don’t
leave
your
doors
unlocked
when
you
leave
home
§
§
Web
API
could
be
an
aDack
vector
§
§
§
Most
admins
are
equally
careful
with
their
servers
Which
is
why
Zend
take
steps
to
secure
it
Web
API
calls
must
be
signed
with,
amongst
other
informa>on,
a
>mestamp
To
help
prevent
replay
aDacks,
calls
must
complete
within
300
seconds
§
§
This
security
feature
can
shoot
us
in
the
foot
In
short,
the
network
speed
determines
our
maximum
packet
size
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
13

14. Working
around
the
size
issue
§
No
perfect
solu?ons,
yet
§
§
I
am
working
on
Zend
;)
But
we
can
apply
some
pragma?sm
§
§
§
§
§
§
Package
can
be
broken
into
sub-­‐packages
Zend
Deployment
has
a
new
Library
feature,
use
it
Pull
instead
of
push
Upgrade
the
pipe
Time
travel
Or,
emulate
the
GUI
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
14

15. Our
solu>on
GUI
emula?on:
It
works
for
us
<echo
message="copying
${zs.package.name}
to
${remote.hostname}"
/>
<scp
file="${zs.package.des>na>ondir}/${zs.package.name}"
todir="${remote.user}@${remote.hostname}:${remote.dir}"
keyfile="${remote.keyfile}"
/>
<echo
message="app.name
'${app.name}',
app.id
is
'${app.id}'
-­‐
preparing
the
update
of
applica>on
..."
/>
<!-­‐-­‐
execute
zs-­‐manage
locally
on
host
where
package
should
be
deployed!
-­‐-­‐>
<sshexec
host="${remote.hostname}"
username="${remote.user}"
keyfile="${remote.keyfile}"
command="/usr/local/zend/bin/zs-­‐manage
app-­‐update
-­‐u
&quot;NDX_ENV=${zs.ndx.env}&quot;
-­‐i
${app.id}
-­‐p
${remote.dir}/${zs.package.name}
-­‐N
${zs.api.name}
-­‐K
${zs.api.key}
-­‐U
${remote.webapi.url}"
/>
!
Just
remember:
SCP/SSH
may
not
be
allowed
by
your
opera>ons
team.
You
may
need
another
solu>on
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
!
20/11/13
15

16. Memory
and
lots
of
it
§
Ques?on:
Where
does
all
this
deployment
data
go?
§
§
The
GUI
allocates
memory
equivalent
to
three
?mes
the
packet
size
§
§
Hint:
The
WebAPI
is
part
of
the
GUI
Since
the
data
is
loaded,
copied
and
finally
unpacked
(don’t
ask)
Where
is
the
GUI’s
memory
limit
configured?
§
§
Not
in
the
GUI,
that
would
spoil
the
fun
;)
Can
you
guess?
Answer:
The
memory_limit
variable
in:
/usr/local/zend/gui/lighapd/etc/php-­‐fcgi.ini
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
16

17. Aaack
of
the
cluster
§
Zend
deployment
can
be
hard
to
synchronise
§
§
An
extreme
case
is
the
“run
once”
node
§
§
In
other
words,
you
cannot
guarantee
when
all
nodes
will
run
the
same
version
of
the
applica>on
Database
migra>ons
are
usually
performed
here,
so
deployment
is
slower
This
caused
problems
with
our
“In
maintenance”
page
§
Time
for
some
extreme
pragma>sm…
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
17

18. I
am
a
Job
Queue
fan…
so
I
wrote
this
//
Sanity
check
if
(!class_exists('ZendJobQueue')){
logger("Job
Queue
seems
to
be
disabled!");
exit(1);
}
if
(getenv("ZS_RUN_ONCE_NODE")
==
1)
{
$name
=
DEPLOYMENT_JOB_BASENAME.$appCurrentVersion;
$queue
=
new
ZendJobQueue();
//
Before
scheduling
our
job,
just
make
sure
that
it
has
not
been
scheduled
before
$total
=
null;
$jobArray
=
$queue-­‐>getJobsList(array('name'
=>
$name),
$total);
if
($total
!=
0)
{
foreach($jobArray
AS
$job)
{
$queue-­‐>removeJob($job['id']);
}
}
//
Schedule
the
job
script
to
run
a
long
>me
in
the
future
(since
we
don't
really
need
it
to
run)
$runAt
=
date('Y-­‐m-­‐d
H:i:s',
strto>me('+7
days'));
$semaphoreJobId
=
$queue-­‐>createHapJob('hap://www.google.de/search?q=dummy',
array(),
array('name'
=>
$name,
'schedule_?me'
=>
$runAt));
if
(!$semaphoreJobId)
{
logger("Semphore
job
was
not
accepted
in
the
queue!");
exit(1);
}
}
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
18

19. …and
this
at
the
end
of
the
script
if
(getenv("ZS_RUN_ONCE_NODE")
==
1)
{
//
We
are
the
RUN
ONCE
script
so
clear
the
semaphore
$queue-­‐>removeJob($semaphoreJobId);
}
else
{
//
We
are
not
the
RUN
ONCE
script
so
wait
for
it
to
clear
the
semaphore
$queue
=
new
ZendJobQueue();
$total
=
1;
$status
=
ZendJobQueue::JOB_STATUS_PENDING
|
ZendJobQueue::JOB_STATUS_SCHEDULED;
while
($total
!=
0)
{
$jobArray
=
$queue-­‐>getJobsList(array('name'
=>
$name,
'status'
=>
$status),
$total);
sleep(1);
}
}
We
now
plan
to
use
an
Nginx,
that
spontaneously
appeared
in
our
architecture,
to
set
our
maintenance
page.
Nevertheless,
it
was
s>ll
a
cool
idea
;)
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
19

20. Finding
out
what
is
happening
§
So
far
so
good,
but
we
s?ll
don’t
have
true
integra?on
§
§
Jenkins
has
no
liale*
visibility
over
the
deployment
process
We
can
log
the
output
of
our
deployment
scripts
§
§
§
But
these
are
not
available
to
Jenkins
We
can
make
them
available
to
the
GUI,
at
least,
by
adding
the
paths
to
the
GUI_AVAILABLE_LOGS
table
in
the
Zend
Server
database
More
convenient,
but
we
s>ll
need
to
go
check
the
GUI
a‚er
each
deployment
*
Last
minute
slide
hack
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
20

21. WebAPI
to
the
rescue…
again
§
Features
added
to
the
WebAPI
allow
registered
logs
to
be
remotely
read
§
§
logsReadLines
logsGetLogfile
§
We
can
finally
integrate
log
output
into
our
Jenkins
reports
§
Just
a
few
problems
§
§
§
Finding
the
“run
once”
node
These
methods
are
not
implemented
in
zs-­‐manage
(yet)
Wri>ng
the
code
(I
haven’t
had
the
>me
yet)
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
20/11/13
21

22. Thank
you
very
much
for
your
aaen>on
www.leaermaschine.com
www.novadex.com
Please
scan
this
…and
tell
me
where
it
goes
Honestly,
marke>ng
wouldn’t
tell
me!
Eric
Ritchie
Novadex
GmbH
DevOps
Engineer
hDps://joind.in/9926
Stuagarter
Straße
59
74321
Bie>gheim-­‐Bissingen
+49
(0)
7142
/
788905-­‐0
Germany
eric.ritchie@novadex.com
Novadex
GmbH
2013.
All
rights
reserved,
also
regarding
any
disposal,
exploita>on,
reproduc>on,
edi>ng,
20/11/13
distribu>on,
as
well
as
in
the
event
of
applica>ons
for
industrial
property
rights.
22