By Christian Lipphardt, Camunda Services Camunda is an open source, Java-based framework process/business process automation. As a middleware technology, Camunda integrates with six different Java application servers (in different versions) and supports six different database products. The team at Camunda maintains five supported versions of Camunda itself, adding two versions every year. Maintaining the necessary continuous integration (CI) infrastructure based on virtual machines became increasingly problematic, with poor build reproducibility and limited scalability. Feedback cycles for developers were unacceptable. Recently Camunda switched from the virtual machine model to a container model based on Docker. The Camunda team now develops infrastructure as code and applies microservice-like separation of concerns. In the talk, Daniel will share the new CI architecture and present lessons learned.

1. From Virtual Machines to Containers
Achieving Continuous Integration, Build Reproducibility, Isolation and Scalability
London, UK, June 2015
1

2. #jenkinsconf
Agenda
• Introduction
• The Dark Age
• The Promising Present
• Lessons Learned
• The Bright Future

3. #jenkinsconf
2014201320122011201020092008
Incorporation
Camunda Services GmbH
Berlin, Germany
Incorporation
Camunda Inc.
San Francisco, US
BPM Consulting BPM Software Vendor
● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year
Introduction - About Camunda
@camundaBPM
Introduced
Hudson
Adapted
Jenkins

4. #jenkinsconf
Introduction - Camunda BPM

5. #jenkinsconf
Introduction - What is Camunda BPM
• Camunda BPM is an open source platform
for workflow and business process automation
• Integrates with:
• 7 Application Server (11 different versions)
• 6 Databases (17 different versions)
• 1 Development & 4 Maintained Versions

6. #jenkinsconf
• Every Camunda BPM version is tested against:
• 187 combinations of DBs and App Servers
• 11 JDKs
• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version
• Support for Enterprise Customer (24/7 + Fix Time)
Introduction - Why CI is Important

7. #jenkinsconf
The Dark Age
Why our CI had to change
7

8. #jenkinsconf
The Dark Age - The Numbers
• 1 Jenkins Master with alot of plugins
• 8 Jenkins Slaves VMs
• ~1000 Jobs total configured / manually managed:
• 4 Camunda Versions
• Community Projects
• Websites
• Maintenance

9. #jenkinsconf
The Dark Age - Isolation Problem
• Unit and Integration Tests need a
database/application server
• Only 1 Instance per Database
• All Jobs use the same Databases
• Every half-year a new Version (~400 Jobs) using
same Databases

10. #jenkinsconf
The Dark Age - No Scalability
• Total Executors: 12
• 1 Jenkins -> 4 Executors
• 8 static heterogenous slave VMs, each with 1
Executor
• Jobs tied to slaves through labels
• Slaves “restrict” database access by allowing no
other build to run

11. #jenkinsconf
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin
• Supporting a new Database vendor / version
• Supporting a new App Server version
• Creating jobs for new Camunda BPM version
• Disaster recovery

12. #jenkinsconf
The Dark Age - The Other Problems
• Slow feedback cycle for developers
• Developers cannot reproduce CI environments
• QA engineers use and maintain separate bloated
test build setup

13. #jenkinsconf
So ...

14. #jenkinsconf
The Promising Present
How we solved our Problem
14

15. #jenkinsconf
The Present - What we achieved
• Configuration & Infrastructure as Code
• Isolated and Reproducible Jobs
• Scalable CI Infrastructure

16. #jenkinsconf
The Present - Infrastructure as Code
1. Every configuration is checked into SCM
2. Every application/test runs in a Docker
Container
3. Every Docker image is build
automatically

17. #jenkinsconf
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for
• Applications
• Test Environments
• JobDSL for
• Jenkins Jobs

18. #jenkinsconf
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:
• Application (Jenkins, Nexus …)
• Test Env. Images (DB + SSH)
• Build Env. Images
• DIND, QEMU + Packer.io

19. #jenkinsconf
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images
• KVM Images build in Docker Container
with Packer + QEMU
• KVM Images bundled in Docker Image

20. #jenkinsconf
The Present - The Current Flow
camunda-ci
camunda
Camunda BPM
Platform
Infrastructure
Jenkins
CI Jenkins

21. #jenkinsconf
The Present - Isolation
One Jenkins per Concern:
• CI
• Release
• Infrastructure
• Community and other Projects
• Marketing

22. #jenkinsconf
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container
• No Interference between Jobs
• The Database Settings are well documented
• Every Docker Image is stored in a private registry
• Developers/QA can use the Docker Images for local
testing

23. #jenkinsconf
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud
running on Docker Swarm
• Docker images are added through Groovy scripting
• Running on Commodity Hardware
• 1 Infrastructure Host (Jenkins, Nexus, …)
• 6 Docker Hosts as 1 Swarm

24. #jenkinsconf
The Present - Scalability

25. #jenkinsconf
The Present - Advantages
• Easy to add new Databases/Test Environments
• New Release = New Branch of JobDSL Repository
• Fully parallelized Job Execution
• Accountable Configuration History
• Testable Infrastructure
• Minimize Administration Overhead

26. #jenkinsconf
The Present - Conclusion
• 2 People + 3 Months of Work
• A fully scalable, isolated and reproducible CI
Infrastructure
• Faster Feedback
• Happy Developers and Product Owner

27. #jenkinsconf
Lessons learned
27
Some advice from us to you

28. #jenkinsconf
• Automate as much as you can
• Jenkins config
• Jobs config
• Environment creation
• Design to scale to support the business agility
Lessons learned - Architecture

29. #jenkinsconf
Lessons learned - Testing
Test everything:
• Jenkins configuration
• required plugin features
• Job generation
• Docker images
• Scalability
• Disaster recovery

30. #jenkinsconf
Lessons learned - Job DSL
• Unit-test the job generation
• Write JobGenerator classes to abstract the
common build logic of most jobs out of the box
• Use XML diffing to compare previously generated
jobs with new ones

31. #jenkinsconf
Lessons learned - Docker
https://github.com/rancherio/vm/

32. #jenkinsconf
• Pin your plugin versions
• Be prepared to contribute to plugin development or
maintain a branch yourself
• Choose the right plugin for the job
Our Top 3 plugins:
JobDSL, Docker-Plugin, Build-Failure-Analyzer
Lessons learned - Plugins

33. #jenkinsconf
Lessons learned - Control
• Control as much as possible
• Third party binaries vs package manager
• explicit versions
• own mirrors for important packages

34. #jenkinsconf
The Bright Future
What comes next
34

35. #jenkinsconf
The Future
• Public Community Jenkins
• Internal Webapp for Developers and QA to start
Environments
• Jenkins is deploying Jenkins
• Back to the Datacenter
• Centralized Logging and Monitoring (ELK)

36. #jenkinsconf

37. #jenkinsconf
37

38. #jenkinsconf
Resources
• Camunda CI repository
https://github.com/camunda-ci
• Job DSL gradle example
https://github.com/sheehan/job-dsl-gradle-example
• Camunda BPM Docker image
https://registry.hub.docker.
com/u/camunda/camunda-bpm-platform/

39. #jenkinsconf
We Are Hiring
Come Join Us
@camundaBPM
http://camunda.com/jobs

40. #jenkinsconf
Please Share Your Feedback
• Did you find this session valuable?
• Please share your thoughts in the
Jenkins User Conference Mobile App.
• Find the session in the app and click
on the feedback area.
40