5. www.camunda.com
Introduction - What is Camunda BPM
• Camunda BPM is an open source platform
for workflow and business process automation
• Integrates with:
• 7 Application Server (11 different versions)
• 6 Databases (17 different versions)
• 1 Development & 4 Maintained Versions
-5-
6. www.camunda.com
Introduction - Why CI is Important
• Every Camunda BPM version is tested against:
• 187 combinations of DBs and App Servers
• 11 JDKs
• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version
• Support for Enterprise Customer (24/7 + Fix Time)
-6-
8. www.camunda.com
The Dark Age - The Numbers
• 1 Jenkins Master with lots of plugins
• 8 Jenkins Slaves VMs
• ~1000 Jobs total configured /
manually managed:
• 4 Camunda Versions
• Community Projects
• Websites
• Maintenance
-8-
9. www.camunda.com
The Dark Age - Isolation Problem
• Unit and Integration Tests need a database/application
server
• Only 1 Instance per Database
• All Jobs use the same Databases
• Every half-year a new Version (~400 Jobs) using same
Databases
-9-
10. www.camunda.com
The Dark Age - No Scalability
• Total Executors: 12
• 1 Jenkins -> 4 Executors
• 8 static heterogenous slave VMs, each with 1 Executor
• Jobs tied to slaves through labels
• Slaves “restrict” database access by allowing no other build
to run
-10-
11. www.camunda.com
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin
• Supporting a new Database vendor / version
• Supporting a new App Server version
• Creating jobs for new Camunda BPM version
• Disaster recovery
-11-
12. www.camunda.com
The Dark Age - The Other Problems
• Slow feedback cycle for developers
• Developers cannot reproduce CI environments
• QA engineers use and maintain separate bloated test build
setup
-12-
15. www.camunda.com
The Present - What we achieved
• Configuration & Infrastructure as Code
• Isolated and Reproducible Jobs
• Scalable CI Infrastructure
-15-
16. www.camunda.com
The Present - Infrastructure as Code
1. Every configuration is checked into SCM
2. Every application/test runs in a Docker Container
3. Every Docker image is build automatically
-16-
17. www.camunda.com
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for
• Applications
• Test Environments
• JobDSL for
• Jenkins Jobs
-17-
19. www.camunda.com
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:
• Application (Jenkins, Nexus …)
• Test Env. Images (DB + SSH)
• Build Env. Images
• DIND, QEMU + Packer.io
-19-
20. www.camunda.com
The Present - Example - DB2
FROM ci1.camunda.loc:5000/camunda-ci-base-centos
ENV DB2_VERSION=10.5 DB_USERNAME=camunda DB_PASSWORD=camunda DB_NAME=engine DB2_HOME=/opt/ibm/db2/V10.5
DB_DIR=/db2_databases
RUN save-env.sh DB2_VERSION DB_USERNAME DB_PASSWORD DB_NAME DB2_HOME
RUN install-packages.sh libaio
ADD etc/db2/db2expc.rsp /tmp/db2expc.rsp
RUN curl https://nginx.service.consul/ci/binaries/ibm/db2/v10.5_linuxx64_expc.tar.gz > /tmp/db2.tar.gz &&
mkdir -p /tmp/db2 && tar xzf /tmp/db2.tar.gz -C /tmp/db2 --strip 1 &&
cd /tmp/db2 && ./db2setup -r /tmp/db2expc.rsp && rm -rf /tmp/db2.tar.gz /tmp/db2 &&
mkdir ${DB_DIR} && chown db2inst1:db2iadm1 ${DB_DIR} && chmod 775 /${DB_DIR} &&
su -l db2inst1 -c "mkdir ${DB_DIR}/engine"
ADD bin/* /usr/local/bin/
ADD etc/supervisor.d/db2.conf.ini /etc/supervisord.d/
EXPOSE 50000
-20-
21. www.camunda.com
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images
• KVM Images build in Docker Container
with Packer + QEMU
• KVM Images bundled in Docker Image
-21-
22. www.camunda.com
The Present - Example - SQL Server 2012
FROM ci1.camunda.loc:5000/ubuntu:14.04.2
ENV IMAGE_NAME=sqlserver-2012.qcow2 IMAGE_DIR=/qemu/ RUN_DIR=/qemu/run
RUN mkdir -p $IMAGE_DIR $RUN_DIR
VOLUME $RUN_DIR
ADD bin/* /usr/local/bin/
RUN apt-get update &&
apt-get -y install --no-install-recommends curl qemu-system-x86 qemu-utils &&
apt-get clean && rm -rf /var/cache/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN curl https://nginx.service.consul/ci/binaries/microsoft/${IMAGE_NAME} > ${IMAGE_DIR}
/${IMAGE_NAME}
EXPOSE 22 1433 5900
CMD ["/usr/local/bin/start-qemu.sh"]
-22-
23. www.camunda.com
The Present - The Current Flow
-23-
camunda-ci
camunda
Camunda BPM
Platform
Infrastructure
Jenkins
CI Jenkins
24. www.camunda.com
The Present - Isolation
One Jenkins per Concern:
• CI
• Release
• Infrastructure
• Community and other Projects
• Marketing
-24-
25. www.camunda.com
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container
• No Interference between Jobs
• The Database Settings are well documented
• Every Docker Image is stored in a private registry
• Developers/QA can use the Docker Images for local testing
-25-
26. www.camunda.com
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud running
on Docker Swarm
• Docker images are added through Groovy scripting
• Running on Commodity Hardware
• 3 Infrastructure Hosts (Jenkins, Nexus, …)
• 4 Docker Hosts as 1 Swarm
-26-
31. www.camunda.com
The Present - Advantages
• Easy to add new Databases/Test Environments
• New Release = New Branch of JobDSL Repository
• Fully parallelized Job Execution
• Accountable Configuration History
• Testable Infrastructure
• Minimize Administration Overhead
-31-
32. www.camunda.com
The Present - Conclusion
• 2 People + 3 Months of Work
• A fully scalable, isolated and reproducible CI Infrastructure
• Faster Feedback
• Happy Developers and Product Owner
-32-
34. www.camunda.com
Lessons learned - Architecture
• Automate as much as you can
• Jenkins config
• Jobs config
• Environment creation
• Design to scale to support the business agility
-34-
36. www.camunda.com
Lessons learned - Job DSL
• Unit-test the job generation
• Write JobGenerator classes to abstract the common build
logic of most jobs out of the box
• Use XML diffing to compare previously generated jobs with
new ones
-36-
38. www.camunda.com
Lessons learned - Plugins
• Pin your plugin versions
• Be prepared to contribute to plugin development or
maintain a branch yourself
• Choose the right plugin for the job
Our Top 3 plugins:
JobDSL, Docker-Plugin, Build-Failure-Analyzer
-38-
39. www.camunda.com
Lessons learned - Control
• Control as much as possible
• Third party binaries vs package manager
• explicit versions
• own mirrors for important packages
-39-
41. www.camunda.com
The Future
-41-
• Public Community Jenkins
• Internal Webapp for Developers and QA to start
Environments (Dattln)
• Continuous Deployment for Infrastructure Container
• Back to the Datacenter
• Centralized Logging and Monitoring (ELK)
• Secret management