SlideShare ist ein Scribd-Unternehmen logo
1 von 26
Downloaden Sie, um offline zu lesen
Jenkins + CVMFS :
Distributed Development,
Centralised Delivery
Bruce Becker | bbecker@csir.co.za
Coordinator: SAGrid
SANREN, Meraka Institute, CSIR
Stefanus Riekert | RiekertSJPK@ufs.ac.za
HPC Application Engineer
University of the Free State
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Outline
● What users want
● SAGrid VO – a catch-all VO with many applications
● Problem statements:
● Problem 1: ”the usual problem” – maintaining
applications in a distributed computing environment
● Problem 2: ”Another usual problem” - maintaining a
complex application inventory
● General solution : CVMFS + Jenkins
● Some specifics of SAGrid CI platform
● Outlook
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
SAGrid as a catch-all VO
● The South African National Grid operates a
catch-all VO which all South African researchers
can use to access computing and data
resources.
● SAGrid VO is not a domain-specific VO, so
● several widely-varying uses for the applications
supported by this VO
● Applications requested by users or communities
themselves
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
What users want
Amazing infrastructure
Some users want highly
varied, modular
application selection
Vertically integrated
Highly specialised
applications
Highly trained supportHighly trained support
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
What users get sometimes
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
The problem (1) -
”the usual problem”
● Software distribution was done mostly by hand”:
● Someone from the ops team develops script to install the application
● Apps installed via job submission
● Tags applied via script or by the job itself
● Issues:
● Major overhead of work
● Inconsistent installation procedures between applications and sites
● Bottleneck in porting applications (has to be done by someone in the
VO)
● Duplication of effort, especially in dependencies of applications
● Difficult to manage application lifecycles
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
The problem (2) -
what about the community ?
● Managing the inventory in a catch-all VO can be complex
when there are many applications
● Prioritising porting requests depends on the knowledge
of the export porting the application
●
Can lead to major delays in porting and deploying applications
● However, a user or community usually has an expert who
knows how to tune, port and configure the application
properly, as well as dependencies
●
Usually, ”they” have to conform to ”us” - learn grid tools and
terminology, etc
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Problem (3) :
Changes to the playing field
● New middleware stacks
● New architectures – GPGPU, ARM
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Questions to answer
● How do we lower the barrier to entry to the grid or
cloud infrastructure ?
● How can the application expert prove to the resource
provider that the application will actually run on the
execution environment of the site ?
● How can we manage the lifecycle of applications
across multiple versions, architectures, configurations ?
● How can we ensure that once applications are
”certified”, they are actually available on as many sites
as possible ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
General Solution:
Jenkins + CVMFS
● The issues outlined are ”typical” in a large
software project
● Usually solved by judicious use of Continuous
Integration system
● Once applications have been ”ported”, put them
into a trusted repository
● Previously – built RPMs, but required site-
admin intervention
● One-time configuration with CVMFS
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
First, some changes
● Distribute the effort, centralise the tools
● Move repository from ”closed” SVN repo
– https://ops.sagrid.ac.za/trac/svn/repo
● to git
– https://github.com/SAGridOps/SoftwareInstallation
● Don't have to give write access to a single repo, instead
accept pull requests
● Take advantage of all the Github infrastructure
● Expand possible contributors to those ”outside” the
infrastructure
● Recognise individuals' contribution
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Recognise individuals...
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Decentralise the team
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Collaborate with code
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Let the robots do the work
● Define what we want to deploy – let the experts
take care of how to deploy
● DevOps paradigm – same review/tag/release
mechanisms on operations code as we have for
scientific applications
● Teach a marketable skill
● Allow specialisation
● Enable remote management of complex services
● Ensure that published methodology is adopted
methodology
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Quality Control and feedback
● Ensure that
requested
applications are
included in the repo
● Provide testing and
QA infrastructure
● Self-serve to users
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
The CI environment
● Jenkins is extremely flexible... can do almost anything
● AuthN/AuthZ
● Currently using Github Oauth
● Take advantage of future Identity Federation
● We wanted to simulate different execution
environments
● Already in production
● Planned for future
● Track and re-use depedendencies
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Matrix-based builds
● Independent different builds and build statuses for
different configurations:
● Application name
● Version
● OS
● Architecture
● … can add specific tuning configurations...
● We can see exactly what's broken where – build
more resilient integration code.
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Typical workflow
Testingmatrix
Defines relevant
tests in Jenkins
Writes code to
pass required tests
Dev/Stage env.Application
developer
Infrastructure
expert
Reads description
of execution environment tests
Promote a build
to CVMFS
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Dependency management
simple case
● Common problem with applications :
need a specific version of a
compiler
● Compiling the compiler can itself be
tricky...
● Jenkins tests the full dependency
chain necessary
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Real-world application
● GADGET –
astrophysics
hydrodynamic
simulations
● Many (levels of)
dependencies
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Public Application Dashboard
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Authenticated view
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Generic build script
# GADGET requires HDF5 FFTW2 ZLIB and 
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
# GADGET requires HDF5 FFTW2 ZLIB and 
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
rm ­rf $FFTW_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/fftw/$FFTW_VERSION/build.tar.gz ­C /
rm ­rf $HDF5_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/hdf5/$HDF5_VERSION/build.tar.gz ­C /
rm ­rf $OPENMPI_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/openmpi/$OPENMPI_VERSION/build.tar.gz ­C /
rm ­rf $GSL_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/gsl/$GSL_VERSION/build.tar.gz ­C /
rm ­rf $FFTW_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/fftw/$FFTW_VERSION/build.tar.gz ­C /
rm ­rf $HDF5_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/hdf5/$HDF5_VERSION/build.tar.gz ­C /
rm ­rf $OPENMPI_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/openmpi/$OPENMPI_VERSION/build.tar.gz ­C /
rm ­rf $GSL_DIR
tar xvfz /repo/$SITE/$OS/$ARCH/gsl/$GSL_VERSION/build.tar.gz ­C /
Set up the
environment
Clean build, retrieve
dependency artifacts
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Generic build script
make install DESTDIR=$WORKSPACE/build
mkdir ­p $REPO_DIR
rm ­rf $REPO_DIR/*
tar ­cvzf $REPO_DIR/build.tar.gz ­C 
$WORKSPACE/build apprepo
make install DESTDIR=$WORKSPACE/build
mkdir ­p $REPO_DIR
rm ­rf $REPO_DIR/*
tar ­cvzf $REPO_DIR/build.tar.gz ­C 
$WORKSPACE/build apprepo
Actually build...
Create the artifact
cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
    puts stderr "       This module does nothing but alert the user"
    puts stderr "       that the [module­info name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
module­whatis   "$NAME $VERSION."
setenv       GSL_VERSION       $VERSION
setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prepend­path LD_LIBRARY_PATH   $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
    puts stderr "       This module does nothing but alert the user"
    puts stderr "       that the [module­info name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
module­whatis   "$NAME $VERSION."
setenv       GSL_VERSION       $VERSION
setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prepend­path LD_LIBRARY_PATH   $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
Create the modulefile
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
So, it works ! … almost
Next steps
● We have an open, collaborative, low-barrier platform for researchers
to bring applications to the grid
● Small technical tasks :
● Implement promoted builds mechanism to populate sagrid.ac.za CVMFS repo
● Implement SAML AuthN, integrate IdF
● Probes to check that CVMFS is mounted on sites (?)
● Operating in ”stealth mode” at the moment – not advertising, but open
to anyone who is interested to collect feedback
● Addressing specific user communities to test drive the system:
● Machine learning astro applications (rapid prototyping)
● Bioinformatics application suites (complex ecosystem)
● Present next phase of the project in November in Cape Town – move
to production

Weitere ähnliche Inhalte

Was ist angesagt?

Go for Operations
Go for OperationsGo for Operations
Go for OperationsQAware GmbH
 
Docker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupDocker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupWalid Shaari
 
Gradle.Enemy at the gates
Gradle.Enemy at the gatesGradle.Enemy at the gates
Gradle.Enemy at the gatesStrannik_2013
 
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)William Yeh
 
Thinking about Jenkins Security
Thinking about Jenkins SecurityThinking about Jenkins Security
Thinking about Jenkins SecurityMark Waite
 
Aws ug dxb 2021 container series iv
Aws ug dxb 2021 container series  ivAws ug dxb 2021 container series  iv
Aws ug dxb 2021 container series ivWalid Shaari
 
Modern Tools for Building Progressive Web Apps
Modern Tools for Building Progressive Web AppsModern Tools for Building Progressive Web Apps
Modern Tools for Building Progressive Web AppsAll Things Open
 
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!Crossing the Streams! Rollout Strategies to Keep Your Users Happy!
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!VMware Tanzu
 
Flux is incubating + the road ahead
Flux is incubating + the road aheadFlux is incubating + the road ahead
Flux is incubating + the road aheadLibbySchulze
 
Clean Infrastructure as Code
Clean Infrastructure as CodeClean Infrastructure as Code
Clean Infrastructure as CodeQAware GmbH
 
Configuration as Code in Jenkins. What's new? Nov 2016
Configuration as Code in Jenkins. What's new? Nov 2016Configuration as Code in Jenkins. What's new? Nov 2016
Configuration as Code in Jenkins. What's new? Nov 2016Oleg Nenashev
 
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"Daniel Bryant
 
Использование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложенийИспользование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложенийVitebsk Miniq
 
E bpf and profilers
E bpf and profilersE bpf and profilers
E bpf and profilersLibbySchulze
 
Mihai Criveti - PyCon Ireland - Automate Everything
Mihai Criveti - PyCon Ireland - Automate EverythingMihai Criveti - PyCon Ireland - Automate Everything
Mihai Criveti - PyCon Ireland - Automate EverythingMihai Criveti
 
A Hitchhiker's Guide to Cloud Native API Gateways
A Hitchhiker's Guide to Cloud Native API GatewaysA Hitchhiker's Guide to Cloud Native API Gateways
A Hitchhiker's Guide to Cloud Native API GatewaysQAware GmbH
 
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneySetting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneyWeaveworks
 
Node.js what's next (Index 2018)
Node.js what's next (Index 2018)Node.js what's next (Index 2018)
Node.js what's next (Index 2018)Gibson Fahnestock
 
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.All Things Open
 

Was ist angesagt? (20)

Go for Operations
Go for OperationsGo for Operations
Go for Operations
 
Docker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupDocker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetup
 
Gradle.Enemy at the gates
Gradle.Enemy at the gatesGradle.Enemy at the gates
Gradle.Enemy at the gates
 
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)
給 RD 的 Kubernetes 初體驗 (gcpug 2019-06 version)
 
Thinking about Jenkins Security
Thinking about Jenkins SecurityThinking about Jenkins Security
Thinking about Jenkins Security
 
Mini-training: Let’s Git It!
Mini-training: Let’s Git It!Mini-training: Let’s Git It!
Mini-training: Let’s Git It!
 
Aws ug dxb 2021 container series iv
Aws ug dxb 2021 container series  ivAws ug dxb 2021 container series  iv
Aws ug dxb 2021 container series iv
 
Modern Tools for Building Progressive Web Apps
Modern Tools for Building Progressive Web AppsModern Tools for Building Progressive Web Apps
Modern Tools for Building Progressive Web Apps
 
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!Crossing the Streams! Rollout Strategies to Keep Your Users Happy!
Crossing the Streams! Rollout Strategies to Keep Your Users Happy!
 
Flux is incubating + the road ahead
Flux is incubating + the road aheadFlux is incubating + the road ahead
Flux is incubating + the road ahead
 
Clean Infrastructure as Code
Clean Infrastructure as CodeClean Infrastructure as Code
Clean Infrastructure as Code
 
Configuration as Code in Jenkins. What's new? Nov 2016
Configuration as Code in Jenkins. What's new? Nov 2016Configuration as Code in Jenkins. What's new? Nov 2016
Configuration as Code in Jenkins. What's new? Nov 2016
 
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"
CNCF Webinar Series: "Creating an Effective Developer Experience on Kubernetes"
 
Использование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложенийИспользование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложений
 
E bpf and profilers
E bpf and profilersE bpf and profilers
E bpf and profilers
 
Mihai Criveti - PyCon Ireland - Automate Everything
Mihai Criveti - PyCon Ireland - Automate EverythingMihai Criveti - PyCon Ireland - Automate Everything
Mihai Criveti - PyCon Ireland - Automate Everything
 
A Hitchhiker's Guide to Cloud Native API Gateways
A Hitchhiker's Guide to Cloud Native API GatewaysA Hitchhiker's Guide to Cloud Native API Gateways
A Hitchhiker's Guide to Cloud Native API Gateways
 
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneySetting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
 
Node.js what's next (Index 2018)
Node.js what's next (Index 2018)Node.js what's next (Index 2018)
Node.js what's next (Index 2018)
 
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
 

Ähnlich wie Distributed Development, Centralised Delivery - SAGrid Jenkins + CVMFS

Microservices @ Work - A Practice Report of Developing Microservices
Microservices @ Work - A Practice Report of Developing MicroservicesMicroservices @ Work - A Practice Report of Developing Microservices
Microservices @ Work - A Practice Report of Developing MicroservicesQAware GmbH
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsSonja Schweigert
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsWeaveworks
 
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseContinuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseVMware Tanzu
 
Angular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraAngular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraLINAGORA
 
Developing Microservices Directly in AKS/Kubernetes
Developing Microservices Directly in AKS/KubernetesDeveloping Microservices Directly in AKS/Kubernetes
Developing Microservices Directly in AKS/KubernetesChakradhar Rao Jonagam
 
Modern Web-site Development Pipeline
Modern Web-site Development PipelineModern Web-site Development Pipeline
Modern Web-site Development PipelineGlobalLogic Ukraine
 
Devops automation using Docker, Kubernetes and Openshift
Devops automation using Docker, Kubernetes and OpenshiftDevops automation using Docker, Kubernetes and Openshift
Devops automation using Docker, Kubernetes and OpenshiftJonathan Tse
 
Best Practices - By Lofi Dewanto
Best Practices - By Lofi DewantoBest Practices - By Lofi Dewanto
Best Practices - By Lofi DewantoGWTcon
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...NETWAYS
 
Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015WaveMaker, Inc.
 
DevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatDevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatJessica DeVita
 
Network architecture design for microservices on GCP
Network architecture design for microservices on GCPNetwork architecture design for microservices on GCP
Network architecture design for microservices on GCPRaphaël FRAYSSE
 
Pragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecturePragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecturePiotr Pelczar
 
Webinar: End-to-End CI/CD with GitLab and DC/OS
Webinar: End-to-End CI/CD with GitLab and DC/OSWebinar: End-to-End CI/CD with GitLab and DC/OS
Webinar: End-to-End CI/CD with GitLab and DC/OSMesosphere Inc.
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendVlad Fedosov
 
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...Gilad Garon
 
Using Containers to More Effectively Manage DevOps Continuous Integration
Using Containers to More Effectively Manage DevOps Continuous IntegrationUsing Containers to More Effectively Manage DevOps Continuous Integration
Using Containers to More Effectively Manage DevOps Continuous IntegrationCognizant
 
OpenNfv Talk On Kubernetes and Network Function Virtualization
OpenNfv Talk On Kubernetes and Network Function VirtualizationOpenNfv Talk On Kubernetes and Network Function Virtualization
OpenNfv Talk On Kubernetes and Network Function VirtualizationGlenn West
 

Ähnlich wie Distributed Development, Centralised Delivery - SAGrid Jenkins + CVMFS (20)

Microservices @ Work - A Practice Report of Developing Microservices
Microservices @ Work - A Practice Report of Developing MicroservicesMicroservices @ Work - A Practice Report of Developing Microservices
Microservices @ Work - A Practice Report of Developing Microservices
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
 
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseContinuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
 
Angular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraAngular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - Linagora
 
Developing Microservices Directly in AKS/Kubernetes
Developing Microservices Directly in AKS/KubernetesDeveloping Microservices Directly in AKS/Kubernetes
Developing Microservices Directly in AKS/Kubernetes
 
Modern Web-site Development Pipeline
Modern Web-site Development PipelineModern Web-site Development Pipeline
Modern Web-site Development Pipeline
 
12-Factor Apps
12-Factor Apps12-Factor Apps
12-Factor Apps
 
Devops automation using Docker, Kubernetes and Openshift
Devops automation using Docker, Kubernetes and OpenshiftDevops automation using Docker, Kubernetes and Openshift
Devops automation using Docker, Kubernetes and Openshift
 
Best Practices - By Lofi Dewanto
Best Practices - By Lofi DewantoBest Practices - By Lofi Dewanto
Best Practices - By Lofi Dewanto
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
 
Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015
 
DevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatDevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to Habitat
 
Network architecture design for microservices on GCP
Network architecture design for microservices on GCPNetwork architecture design for microservices on GCP
Network architecture design for microservices on GCP
 
Pragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecturePragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecture
 
Webinar: End-to-End CI/CD with GitLab and DC/OS
Webinar: End-to-End CI/CD with GitLab and DC/OSWebinar: End-to-End CI/CD with GitLab and DC/OS
Webinar: End-to-End CI/CD with GitLab and DC/OS
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontend
 
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...
 
Using Containers to More Effectively Manage DevOps Continuous Integration
Using Containers to More Effectively Manage DevOps Continuous IntegrationUsing Containers to More Effectively Manage DevOps Continuous Integration
Using Containers to More Effectively Manage DevOps Continuous Integration
 
OpenNfv Talk On Kubernetes and Network Function Virtualization
OpenNfv Talk On Kubernetes and Network Function VirtualizationOpenNfv Talk On Kubernetes and Network Function Virtualization
OpenNfv Talk On Kubernetes and Network Function Virtualization
 

Mehr von Bruce Becker

Trusted Configurations for e-Infrastructure Deployment
Trusted Configurations for e-Infrastructure DeploymentTrusted Configurations for e-Infrastructure Deployment
Trusted Configurations for e-Infrastructure DeploymentBruce Becker
 
The Sci-GaIA project
The Sci-GaIA projectThe Sci-GaIA project
The Sci-GaIA projectBruce Becker
 
NICIS: Stepping Stones to a Cyberinfrastructure Commons
NICIS: Stepping Stones to a Cyberinfrastructure CommonsNICIS: Stepping Stones to a Cyberinfrastructure Commons
NICIS: Stepping Stones to a Cyberinfrastructure CommonsBruce Becker
 
Continuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsContinuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsBruce Becker
 
Continuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsContinuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsBruce Becker
 
Now you can cite APHRC's data sets (CHAIN-REDS)
Now you can cite APHRC's data sets (CHAIN-REDS)Now you can cite APHRC's data sets (CHAIN-REDS)
Now you can cite APHRC's data sets (CHAIN-REDS)Bruce Becker
 
Gadget2 Science Gateway Development
Gadget2 Science Gateway DevelopmentGadget2 Science Gateway Development
Gadget2 Science Gateway DevelopmentBruce Becker
 
Continuous integration
Continuous integrationContinuous integration
Continuous integrationBruce Becker
 
Sa grid application inventory 36
Sa grid application inventory 36Sa grid application inventory 36
Sa grid application inventory 36Bruce Becker
 
Paerip chain-becker-10-11-2011
Paerip chain-becker-10-11-2011Paerip chain-becker-10-11-2011
Paerip chain-becker-10-11-2011Bruce Becker
 

Mehr von Bruce Becker (10)

Trusted Configurations for e-Infrastructure Deployment
Trusted Configurations for e-Infrastructure DeploymentTrusted Configurations for e-Infrastructure Deployment
Trusted Configurations for e-Infrastructure Deployment
 
The Sci-GaIA project
The Sci-GaIA projectThe Sci-GaIA project
The Sci-GaIA project
 
NICIS: Stepping Stones to a Cyberinfrastructure Commons
NICIS: Stepping Stones to a Cyberinfrastructure CommonsNICIS: Stepping Stones to a Cyberinfrastructure Commons
NICIS: Stepping Stones to a Cyberinfrastructure Commons
 
Continuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsContinuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research Applications
 
Continuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research ApplicationsContinuous Integration and Delivery of Research Applications
Continuous Integration and Delivery of Research Applications
 
Now you can cite APHRC's data sets (CHAIN-REDS)
Now you can cite APHRC's data sets (CHAIN-REDS)Now you can cite APHRC's data sets (CHAIN-REDS)
Now you can cite APHRC's data sets (CHAIN-REDS)
 
Gadget2 Science Gateway Development
Gadget2 Science Gateway DevelopmentGadget2 Science Gateway Development
Gadget2 Science Gateway Development
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Sa grid application inventory 36
Sa grid application inventory 36Sa grid application inventory 36
Sa grid application inventory 36
 
Paerip chain-becker-10-11-2011
Paerip chain-becker-10-11-2011Paerip chain-becker-10-11-2011
Paerip chain-becker-10-11-2011
 

Kürzlich hochgeladen

RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024suelcarter1
 
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Sérgio Sacani
 
MARSILEA notes in detail for II year Botany.ppt
MARSILEA  notes in detail for II year Botany.pptMARSILEA  notes in detail for II year Botany.ppt
MARSILEA notes in detail for II year Botany.pptaigil2
 
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...PirithiRaju
 
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Sérgio Sacani
 
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTMARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTjipexe1248
 
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Sérgio Sacani
 
Bureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxBureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxkastureyashashree
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...marwaahmad357
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfsantiagojoderickdoma
 
Pests of tenai_Identification,Binomics_Dr.UPR
Pests of tenai_Identification,Binomics_Dr.UPRPests of tenai_Identification,Binomics_Dr.UPR
Pests of tenai_Identification,Binomics_Dr.UPRPirithiRaju
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptxVaishnaviAware
 
Pests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPirithiRaju
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsHassan Jolany
 
Controlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentControlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentRahulVishwakarma71547
 
KeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceKeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceLayne Sadler
 
CW marking grid Analytical BS - M Ahmad.docx
CW  marking grid Analytical BS - M Ahmad.docxCW  marking grid Analytical BS - M Ahmad.docx
CW marking grid Analytical BS - M Ahmad.docxmarwaahmad357
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docxmarwaahmad357
 

Kürzlich hochgeladen (20)

RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024
 
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
 
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
 
MARSILEA notes in detail for II year Botany.ppt
MARSILEA  notes in detail for II year Botany.pptMARSILEA  notes in detail for II year Botany.ppt
MARSILEA notes in detail for II year Botany.ppt
 
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...
3.2 Pests of Sorghum_Identification, Symptoms and nature of damage, Binomics,...
 
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
 
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTMARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
 
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
 
Bureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxBureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptx
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
 
Pests of tenai_Identification,Binomics_Dr.UPR
Pests of tenai_Identification,Binomics_Dr.UPRPests of tenai_Identification,Binomics_Dr.UPR
Pests of tenai_Identification,Binomics_Dr.UPR
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptx
 
Pests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPR
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbits
 
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
 
Controlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentControlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform Environment
 
KeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceKeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data science
 
CW marking grid Analytical BS - M Ahmad.docx
CW  marking grid Analytical BS - M Ahmad.docxCW  marking grid Analytical BS - M Ahmad.docx
CW marking grid Analytical BS - M Ahmad.docx
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docx
 

Distributed Development, Centralised Delivery - SAGrid Jenkins + CVMFS

  • 1. Jenkins + CVMFS : Distributed Development, Centralised Delivery Bruce Becker | bbecker@csir.co.za Coordinator: SAGrid SANREN, Meraka Institute, CSIR Stefanus Riekert | RiekertSJPK@ufs.ac.za HPC Application Engineer University of the Free State
  • 2. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Outline ● What users want ● SAGrid VO – a catch-all VO with many applications ● Problem statements: ● Problem 1: ”the usual problem” – maintaining applications in a distributed computing environment ● Problem 2: ”Another usual problem” - maintaining a complex application inventory ● General solution : CVMFS + Jenkins ● Some specifics of SAGrid CI platform ● Outlook
  • 3. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za SAGrid as a catch-all VO ● The South African National Grid operates a catch-all VO which all South African researchers can use to access computing and data resources. ● SAGrid VO is not a domain-specific VO, so ● several widely-varying uses for the applications supported by this VO ● Applications requested by users or communities themselves
  • 4. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za What users want Amazing infrastructure Some users want highly varied, modular application selection Vertically integrated Highly specialised applications Highly trained supportHighly trained support
  • 5. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za What users get sometimes
  • 6. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za The problem (1) - ”the usual problem” ● Software distribution was done mostly by hand”: ● Someone from the ops team develops script to install the application ● Apps installed via job submission ● Tags applied via script or by the job itself ● Issues: ● Major overhead of work ● Inconsistent installation procedures between applications and sites ● Bottleneck in porting applications (has to be done by someone in the VO) ● Duplication of effort, especially in dependencies of applications ● Difficult to manage application lifecycles
  • 7. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za The problem (2) - what about the community ? ● Managing the inventory in a catch-all VO can be complex when there are many applications ● Prioritising porting requests depends on the knowledge of the export porting the application ● Can lead to major delays in porting and deploying applications ● However, a user or community usually has an expert who knows how to tune, port and configure the application properly, as well as dependencies ● Usually, ”they” have to conform to ”us” - learn grid tools and terminology, etc
  • 8. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Problem (3) : Changes to the playing field ● New middleware stacks ● New architectures – GPGPU, ARM
  • 9. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Questions to answer ● How do we lower the barrier to entry to the grid or cloud infrastructure ? ● How can the application expert prove to the resource provider that the application will actually run on the execution environment of the site ? ● How can we manage the lifecycle of applications across multiple versions, architectures, configurations ? ● How can we ensure that once applications are ”certified”, they are actually available on as many sites as possible ?
  • 10. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za General Solution: Jenkins + CVMFS ● The issues outlined are ”typical” in a large software project ● Usually solved by judicious use of Continuous Integration system ● Once applications have been ”ported”, put them into a trusted repository ● Previously – built RPMs, but required site- admin intervention ● One-time configuration with CVMFS
  • 11. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za First, some changes ● Distribute the effort, centralise the tools ● Move repository from ”closed” SVN repo – https://ops.sagrid.ac.za/trac/svn/repo ● to git – https://github.com/SAGridOps/SoftwareInstallation ● Don't have to give write access to a single repo, instead accept pull requests ● Take advantage of all the Github infrastructure ● Expand possible contributors to those ”outside” the infrastructure ● Recognise individuals' contribution
  • 12. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Recognise individuals...
  • 13. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Decentralise the team
  • 14. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Collaborate with code
  • 15. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Let the robots do the work ● Define what we want to deploy – let the experts take care of how to deploy ● DevOps paradigm – same review/tag/release mechanisms on operations code as we have for scientific applications ● Teach a marketable skill ● Allow specialisation ● Enable remote management of complex services ● Ensure that published methodology is adopted methodology
  • 16. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Quality Control and feedback ● Ensure that requested applications are included in the repo ● Provide testing and QA infrastructure ● Self-serve to users
  • 17. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za The CI environment ● Jenkins is extremely flexible... can do almost anything ● AuthN/AuthZ ● Currently using Github Oauth ● Take advantage of future Identity Federation ● We wanted to simulate different execution environments ● Already in production ● Planned for future ● Track and re-use depedendencies
  • 18. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Matrix-based builds ● Independent different builds and build statuses for different configurations: ● Application name ● Version ● OS ● Architecture ● … can add specific tuning configurations... ● We can see exactly what's broken where – build more resilient integration code.
  • 19. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Typical workflow Testingmatrix Defines relevant tests in Jenkins Writes code to pass required tests Dev/Stage env.Application developer Infrastructure expert Reads description of execution environment tests Promote a build to CVMFS
  • 20. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Dependency management simple case ● Common problem with applications : need a specific version of a compiler ● Compiling the compiler can itself be tricky... ● Jenkins tests the full dependency chain necessary
  • 21. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Real-world application ● GADGET – astrophysics hydrodynamic simulations ● Many (levels of) dependencies
  • 22. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Public Application Dashboard
  • 23. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Authenticated view
  • 24. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Generic build script # GADGET requires HDF5 FFTW2 ZLIB and  openmpi module add ci module add fftw/2.1.5 module add hdf5 module add openmpi module add gsl # GADGET requires HDF5 FFTW2 ZLIB and  openmpi module add ci module add fftw/2.1.5 module add hdf5 module add openmpi module add gsl rm ­rf $FFTW_DIR tar xvfz /repo/$SITE/$OS/$ARCH/fftw/$FFTW_VERSION/build.tar.gz ­C / rm ­rf $HDF5_DIR tar xvfz /repo/$SITE/$OS/$ARCH/hdf5/$HDF5_VERSION/build.tar.gz ­C / rm ­rf $OPENMPI_DIR tar xvfz /repo/$SITE/$OS/$ARCH/openmpi/$OPENMPI_VERSION/build.tar.gz ­C / rm ­rf $GSL_DIR tar xvfz /repo/$SITE/$OS/$ARCH/gsl/$GSL_VERSION/build.tar.gz ­C / rm ­rf $FFTW_DIR tar xvfz /repo/$SITE/$OS/$ARCH/fftw/$FFTW_VERSION/build.tar.gz ­C / rm ­rf $HDF5_DIR tar xvfz /repo/$SITE/$OS/$ARCH/hdf5/$HDF5_VERSION/build.tar.gz ­C / rm ­rf $OPENMPI_DIR tar xvfz /repo/$SITE/$OS/$ARCH/openmpi/$OPENMPI_VERSION/build.tar.gz ­C / rm ­rf $GSL_DIR tar xvfz /repo/$SITE/$OS/$ARCH/gsl/$GSL_VERSION/build.tar.gz ­C / Set up the environment Clean build, retrieve dependency artifacts
  • 25. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za Generic build script make install DESTDIR=$WORKSPACE/build mkdir ­p $REPO_DIR rm ­rf $REPO_DIR/* tar ­cvzf $REPO_DIR/build.tar.gz ­C  $WORKSPACE/build apprepo make install DESTDIR=$WORKSPACE/build mkdir ­p $REPO_DIR rm ­rf $REPO_DIR/* tar ­cvzf $REPO_DIR/build.tar.gz ­C  $WORKSPACE/build apprepo Actually build... Create the artifact cat <<MODULE_FILE #%Module1.0 ## $NAME modulefile ## proc ModulesHelp { } {     puts stderr "       This module does nothing but alert the user"     puts stderr "       that the [module­info name] module is not available" } preqreq("gsl","fftw/2.1.5","hdf5") module­whatis   "$NAME $VERSION." setenv       GSL_VERSION       $VERSION setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION prepend­path LD_LIBRARY_PATH   $::env(GSL_DIR)/lib MODULE_FILE ) > modules/$VERSION cat <<MODULE_FILE #%Module1.0 ## $NAME modulefile ## proc ModulesHelp { } {     puts stderr "       This module does nothing but alert the user"     puts stderr "       that the [module­info name] module is not available" } preqreq("gsl","fftw/2.1.5","hdf5") module­whatis   "$NAME $VERSION." setenv       GSL_VERSION       $VERSION setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION prepend­path LD_LIBRARY_PATH   $::env(GSL_DIR)/lib MODULE_FILE ) > modules/$VERSION Create the modulefile
  • 26. Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za So, it works ! … almost Next steps ● We have an open, collaborative, low-barrier platform for researchers to bring applications to the grid ● Small technical tasks : ● Implement promoted builds mechanism to populate sagrid.ac.za CVMFS repo ● Implement SAML AuthN, integrate IdF ● Probes to check that CVMFS is mounted on sites (?) ● Operating in ”stealth mode” at the moment – not advertising, but open to anyone who is interested to collect feedback ● Addressing specific user communities to test drive the system: ● Machine learning astro applications (rapid prototyping) ● Bioinformatics application suites (complex ecosystem) ● Present next phase of the project in November in Cape Town – move to production