Openstack is one of the largest OSS projects today with hundreds of commits flowing in daily. This high rate of change requires an advanced CI infrastructure. The purpose of the talk is to provide an overview of this infrastructure, explaining the role of each tool and the pipelines along which changes have to travel before they find their way into the approved Openstack codebase.

1. How Openstack is
Built
Ant(on) Weiss - Otomato
http://otomato.link
1

2. How Big is OpenStack?
265,850 commits
5,744 developers
54 project teams
585 git repos
2

3. OpenStack CI System
3
• Developed and managed by the OpenStack Infra team

4. Ansible+Puppet - two invisible hands.
■ All configuration of CI infra nodes is managed by Puppet manifests
■ The ‘Puppetmaster’ is actually an Ansible control machine
■ Puppetmaster holds playbook + hiera data
■ Ansible playbooks trigger ‘puppet apply’ on the nodes
4
+

5. Git - the mighty archivist
■ the most widely used distributed
open-source VCS
■ Served by ‘cgit’
■ Changes submitted with the help of
‘git-review’ tool
5

6. Gerrit – the gatekeeper
■ Originally developed by Shawn
Pearce at Google for Android
development.
■ Manages git repositories
■ Provides code review and approval
rules
■ Collaborates with human and non-
human gatekeepers
6

7. Zuul – the dispatcher
■ Defines and enforces the gating
process
■ Listens to Gerrit events
■ Manages the queue and
dependencies
■ Dispatches the verification process
to Jenkins (through Gearman)
7

8. Gearman - the outsourcer
■ a framework to farm out work to
other machines or processes
8
■ sends jobs to Jenkins through
jenkins gearman plugin

9. Jenkins - the butler :)
■ Continuous Integration Server
■ Executes testing jobs, records
executions results
■ Jobs performed on slaves
■ Jobs created and managed by
Jenkins Job Builder
■ Slaves provisioned by Nodepool
9

10. Jenkins Job Builder
■ Configures Jenkins jobs using YAML files stored in git
■ Defaults, macros and job configs
- job:
name: example-docs
node: node-label
triggers:
- zuul
builders:
- git-prep
- docs
publishers:
- scp:
site: 'scp-server'
files:
- target: 'dir/ectory'
source: 'build/html/foo'
keep-hierarchy: true
- console-log 10

11. Nodepool - the provisioner
■ Manages a pool of Devstack images to use in project testing.
■ Provisions nodes on HP and Rackspace clouds.
■ Makes sure there are always available nodes for each provider type.
■ Tears down old instances when tests are completed.
11

12. DevStack Gate
■ The integration test that verifies that all projects still work together after the
changes
■ Provides a full OpenStack installation on a single machine with Devstack
■ Installs Devstack and runs Tempest
12

13. Logstash
■ All jenkins job logs are indexed
and stored by Logstash
■ Logs pushing also handled by
Gearman jobs
13
■ Elastic Search helps analyze the
test results

14. Wrap Up
■ Git
■ Gerrit
■ Zuul
■ Gearman
■ Jenkins Job Builder
■ Jenkins
■ Devstack
■ Logstash
14
Only 9 out of 32 projects/tools
managed by the
Openstack Infra
Team