Continuous Integration and Delivery of Research Applications
1. Continuous Integration and
Delivery of
Research Applications
Bruce Becker
bbecker@csir.co.za
Coordinator, AAROC
Meraka Institute, CSIR
Fanie Riekert
RiekertSJPK@ufs.ac.za
Senior Systems Engineer
HPC Unit, UFS
2. The problem
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
2
3. What users want
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Amazing infrastructure
Some users want highly
varied, modular
application selection
Vertically integrated
Highly specialised
applications
HHiigghhllyy ttrraaiinneedd ssuuppppoorrtt
4. What users get sometimes
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
5. Current Methods
Are you a legit
scientist ?
Ok, sounds good
what applications
are you running ?
We can run
almost anything !
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Hi, I heard about
AfricaGrid – sounds
grrrrreat ! Can I
Use it ?
I've got beakers
…
And glasses !
Awesome ! Well, first we do some
preprocessing with RaNDo, then we
run a massively parallel pipeline with
XBOT, then blah and blah then blah
blah blah and blah, we also need blah
blah and sometimes our colleagues
from FarAwayU need to access the
data with StrangeProtocolx1, and of
course blah blah, not to mention blah
…..
We'll see
what we
can do
6. Actual workflow
● Find the source code
● Unpack, ./configure && make … Error 1 missing x.h
● … hack hack hack … hm. Missing dependencies
● Recursively* determine the dependencies**
● *Yes, recursively means curse, curse then curse again.
● **The actual dependencies, not the ones that were written
on the web page from 5 years ago.
● … mutter mutter …
● ???? Why are you compiling now !?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
7. Actual communication
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Hey, how's it going ?
you done with my porting yet ?
You done yet ?
You done ?
…. bloody IT people !
Have you actually seen this code ?
I just...
It looks like Cthulhu's beard....
Ok, I got it to compile,
but only with if you
type with your left toes.
AAArrrgh, I'm a scientist,
not an IT people !
8. Let's give researchers a
”better deal”
There's gotta be a better way
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
9. Compuatational Infrastructure
What is it good for ?
• Hypothesis 1:
– It always comes down to an application
• At the end of all first-contact you get a list of applications
that users want
– Let's focus on that, shall we ?
• Hypothesis 2:
– No software is an island
• Everything has dependencies, even if it's just glibc or
libgfortran
– Let's make those explicit, shall we ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
9
10. Compuatational Infrastructure
What is it good for ?
• Hypothesis 3:
– Applications need an environment
• Compiler ?
• Architecture ?
• Dependencies ?
• Middleware ?
• Etc....
– Let's make those explicit shall we ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
10
11. Compuatational Infrastructure
What is it good for ?
• Hypothesis 4:
– there is more than one environment.
• Nothing is going to ever be really useful if it can only run
in one place.
• Nobody wants to be forced into a single environment
(grid/HPC/cloud, etc)
– Let's try to simulate as many target environments
(sites) as we can, shall we?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
11
12. Compuatational Infrastructure
What is it good for ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
12
● Hypothesis 5:
● Solutions decay
– The solution – as well as the dead ends – are part of the
process.
– If it's not in a version-controlled repository, it's no good.
– If it's not executable (in a wiki/document, etc), then you
have to convert it back into something a computer
understands – why not just leave it in executable format ?
● Let's recognise that solutions decay and code for
the future, shall we ?
13. Humans need not apply
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
13
● Hypothesis 6:
● Most of the work of porting and supporting
applications can be automated
– Reduce the load on the user and the operator, by
automating common tasks
● Let's:
– Integrate the various software, middleware, service and
infrastructure components as far as possible,
– Keep it Open
● … shall we ?
14. Compuatational Infrastructure
What is it good for ?
● Hypothesis 6 (Corollary) : You're in the way
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
14
15. Compuatational Infrastructure
What is it good for ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
15
● Hypothesis 7:
● This stuff's not hard
– Continuous Integration has been around for a long time
– If it can be done by hand, it can be scripted; if it can be
scripted, it can be automated properly.
● Let's
– adopt widely-used tools and methodologies, and
– hey, how about we use the web to communciate
(you know, like we do with EVERYTHING else ?)
16. Let's work on a better deal...
● How do we lower the barrier to entry to the grid or
cloud infrastructure ?
● How can the application expert prove to the resource
provider that the application will actually run on the
execution environment of the site ?
● How can we manage the lifecycle of applications
across multiple versions, architectures, configurations ?
● How can we ensure that once applications are
”certified”, they are actually available on as many sites
as possible ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
17. Go to Bootcamp
http://software-carpentry.org/bootcamps
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
18. What are we actually doing ?
● Put the ball in the developer/community/user's court
● You tell us how to build your application
● Provide a simple interface to porting pipeline
● Provide focussed, goal-oriented support when needed.
● Communication has a context and is directly linked to the issues
at hand
● Publish and preserve the entire toolchain
● It's a real collaboration now :)
● Collaborate with code
● The Big S word ? (not the 4-letter word - SUSTAINABLE)
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
18
19. Tools of the Trade (1)
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
19
Waffle.io
Project
planning
Slack.com
Team collab
21. What's the plan, Stan ?
https://waffle.io/aaroc/devops
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
21
22. Typical workflow
Application Dev/Stage env.
developer
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Testing matrix
Writes code to
pass required tests
Defines relevant
tests in Jenkins
Infrastructure
expert
Reads description
of execution environment tests
Promote a build
to CVMFS
23. Tests Fail
Tests Pass
Follow up
with the user Invoke Artifact
Fail Test
Open Issue
on Github
Stage to
Testing Repo
Update Repo
Version
Field Tests
Field Tests Pass
Field Tests Fail
Lock CVMFS
repo
Open Issue
on Github
DeployArtifact
Update Repo
Version
Tag sites
Application porting, testing, integration and deployment workflow
Tests Fail
Tests Pass
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
23
Create
Build Script
Pull Request
Create Jenkins
Job
Build
if first time
Invoke Test
if build
has passed
Fail Build
Build Fails Build Passes
Follow up
with the user Invoke Artifact
Fail Test
Open Issue
on Github
Stage to
Testing Repo
Update Repo
Version
The new pipeline
31. Generic build script.
# GADGET requires HDF5 FFTW2 ZLIB and
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
# GADGET requires HDF5 FFTW2 ZLIB and
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
Set up the
environment
make install DESTDIR=$WORKSPACE/build
mkdir p
$REPO_DIR
rm rf
$REPO_DIR/*
tar cvzf
$REPO_DIR/build.tar.gz C
$WORKSPACE/build apprepo Actually build...
Create the artifact
make install DESTDIR=$WORKSPACE/build
mkdir p
$REPO_DIR
rm rf
$REPO_DIR/*
tar cvzf
$REPO_DIR/build.tar.gz C
$WORKSPACE/build apprepo
Create the modulefile
puts stderr " This module does nothing but alert the user"
puts stderr " that the [moduleinfo
name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
modulewhatis
"$NAME $VERSION."
setenv GSL_VERSION $VERSION
setenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prependpath
LD_LIBRARY_PATH $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
puts stderr " This module does nothing but alert the user"
puts stderr " that the [moduleinfo
name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
modulewhatis
"$NAME $VERSION."
setenv GSL_VERSION $VERSION
setenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prependpath
LD_LIBRARY_PATH $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
32. There's gotta be a better way !
(thanks for the idea, Travis)
---
author: Bruce Becker
repo: https://github.com/SAGridOps/SoftwareInstallation
name: GCC
target:
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
- sites:
- generic
- mpi
- mpi_infiniband
- OS:
- sl6
- u1404
versions:
- 4.6.1
category: compilers
dependencies:
- [ name: mpfr, version: 3.1.2 ]
url: http://gnu.org/path/to/gcc.tar.gz
build_script: build.sh
33. Last Bit: Delivery
Executable Artifacts
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
33
go here
Every site
can mount
the CVMFS
repo
Even your
laptop...
34. Summary
● Making the best use of computational infrastructure
comes down to running applications
● Maintaining and porting them is hard and tricky.
Communication is the hardest part.
● We've built an automated porting system, which
● will deliver functional, tested, relevant software to your site.
● Can ease the communication blocks in collaborative work
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
34
● Oh and it works :)
● Come on in and help us build it, there's plenty to do !