Tempest is an Openstack test suite which runs against all the OpenStack service endpoints. It makes sure that all the OpenStack components work together properly and that no APIs are changed. Tempest is a "gate" for all commits to OpenStack repositories and will prevent merges if tests fail.

1. OpenStack QA and Tempest
David Kranz
May 6, 2013
Monday, May 06, 2013

2. Outline
 OpenStack QA
 Tempest
 Future directions
 How to contribute
Monday, May 06, 2013

3. Current OpenStack QA Landscape
 Upstream (OpenStack QA project)
 Tempest (functional and integration tests)
 Grenade (upgrade testing)
 Gerrit/Jenkins (continuous integration and gating)
 Projects
 Unit and functional tests
 Packagers and Deployers
 Testing in real multi-node environments
 Stress and performance testing
Monday, May 06, 2013

4. What is Tempest?
 Comprehensive test suite for OpenStack
 API validation
 Integration testing
 Can be configured to run against real
deployments
 Runs as part of the gating process or on a
periodic basis
 Written in Python
Monday, May 06, 2013

5. What is Devstack?
 Opinionated single-node deployment
 Creates complete OpenStack system
 All components configurable
 Used by developers, testers and by jenkins
 The jenkins “gate” uses devstack and Tempest
 https://github.com/openstack-dev/devstack.git
Monday, May 06, 2013

6. What is “the gate”?
 Tests that run on every commit before merging
 Includes per-project unit tests
 Tempest runs on all commits to all projects
 Integration tests help make sure commits do
not break other projects
 Blocks incompatible API changes
 https://wiki.openstack.org/wiki/Governance/Appro
ved/APIStability
Monday, May 06, 2013

7. Running Tempest with Devstack
 Stack.sh runs devstack and creates
/opt/stack/etc/tempest.conf
 Tempest will spin up images, servers, volumes,
etc., do things to them, bring them down.
 Takes about 45 minutes to run all tests
 Edit tempest.conf to run against real deployment
Monday, May 06, 2013
git clone git://github.com/openstack-dev/devstack.git
cd devstack
./stack.sh
cd /opt/stack/tempest
./run_tests.sh [tempest/tests/...]

8. Example of Tempest Test Case
Monday, May 06, 2013
def test_create_with_existing_server_name(self):
# Creating a server with a name that already exists is allowed
server_name = rand_name('server')
resp, server = self.create_server(name=server_name, wait_until='ACTIVE')
id1 = server['id']
resp, server = self.create_server(name=server_name, wait_until='ACTIVE')
id2 = server['id']
self.assertNotEqual(id1, id2, "Did not create a new server")
resp, server = self.client.get_server(id1)
name1 = server['name']
resp, server = self.client.get_server(id2)
name2 = server['name']
self.assertEqual(name1, name2)
 Test cases use unittest2
 From tempest/tests/compute/servers/test_servers.py

9. Tempest in the Gate
 Three tempest runs per commit
 MySQL
 PostGresSQL
 Quantum
 Full tempest run
 Spins up blank Ubuntu 12.04 node
 Installs with devstack
 Runs 700+ tests
 Takes about 40 minutes in gate
Monday, May 06, 2013

10. Opentack QA Roadmap
 Move more testing upstream (shared effort!)
 Support multi-node testing in jenkins (triple-o)
 Integrate reports from downstream tempest runs using
various real hardware configurations
 Performance testing
 Better upgrade testing
 Add newly incubated projects to Tempest
 Use Tempest as validation suite for the OpenStack
trademark
 Improve Tempest performance by parallel execution
 Make stress testing reliable
Monday, May 06, 2013

11. Where Tempest needs Help
 New tests to cover rest of API
 Strong on Nova, weaker on other projects
 Lots of extensions untested
 New integrated projects: ceilometer and heat
 Improving performance
 Upgrade testing
 Stress testing
Monday, May 06, 2013

12. Getting Involved with Tempest
 Hang out with us in #openstack-qa
 Weekly QA meetings Thursday 17:00 UTC
 Get on openstack-qa mailing list
 http://lists.openstack.org/cgi-
bin/mailman/listinfo/openstack-qa
 Always looking for help filling test gaps
Monday, May 06, 2013

13. Questions?
Monday, May 06, 2013