by Shashidhar Soppin, WIPRO

1. OpenStack Tempest and
REST API Testing

2. Agenda
One
• Introduction
• OpenStack & DevStack
Two
• Tempest & Architecture
• DevStack config and H/w required
Three
• Various tests & Test Results
• Key Findings
Four
• Summary & Conclusion
• Tech-Scenarios & Citations

3.  To demonstrate
starting/running
of OpenStack
services
 Specifically for
development
work on
OpenStack
 We can
download the
latest version of
DevStack from
”git” repo.
 Cloud operating
system/platform
that controls large
pools of compute,
storage, and
networking
resources
 Used for
developing private
clouds
• Keystone
• Cinder
• Glance
• Neutron
• Horizon
• Nova
OpenStack & DevStack
OpenStack
DevStack

4. OpenStack Testing ??
OpenStack
integration Test
Suite
• Tempest Test Suite
• Ready to deploy
Unit Tests in each
project
• Nova, Cinder, Swift
• Neutron etc
• OpenStack Code
base is written in
Python
• So how to
use/write tests?
• Isolate the
working
Environment
• Quick testing
• Automated
configuration
• Ease of use

5. Tempest
• Tempest is a
set of
integration
tests
• To be run
against any
live
OpenStack/
Dev Stack
cluster
F
e
a
t
u
r
e
s
Official OpenStack
Integration Test Suite
Interacts with majority
ReSTful API’s
Around 3000+ tests
covering all projects
Used as gating tests for all
new projects
Runs on all OpenStack
releases
Self Testing/Self Cleaning
Uses a unified tempest
REST client
Latest version of “Tempest
12.1.1.dev37”
A
t
t
r
i
b
u
t
e
s
Scenario based Tests
CLI Tests Stress tests
Ensure that OpenStack cloud
works with the OpenStack API

6. Basic Tempest Reference Architecture
OpenStack/ DevStack
deployment
Service
Clients
Python
Clients Boto Tests
Rest Client
API Tests
Scenario
Tests
Third party
tests
CLI based
Tests
Library CLI
OpenStack ReST API’s EC2 API’s
OpenStack ReST API’s
1

7. Basic Tempest Reference Architecture
Compute
Node
Compute
Node
Controller
Node
Test Server
(Tempest Node)
HTTP Request HTTP Response
OpenStack configuration
{Tempest tests executed
from controller node}
2

8. Tempest in a “Nutshell” 1

9. Tempest in a “Nutshell” 2

10. Hardware configuration required and OS used 1

11. Hardware configuration required and OS used 2

12. Tempest Execution….steps

13. Tempest Execution….steps

14. Tempest Execution….steps

15. Tempest Execution….steps

16. Tempest Execution….steps

17. Tempest Execution….steps

18. Tempest Execution….steps

19. Tempest Execution….steps

20. Test Results-Kilo
968
724
962
22
76
13
141 126 141
1131
926
1116
0
200
400
600
800
1000
1200
CentOS Ubuntu Fedora
toxefull
Passed Failed Skipped Total TC

21. Test Results-Liberty
999 1023
944
10 5 24
136 136 138
1145 1164
1106
0
200
400
600
800
1000
1200
CentOS Ubuntu Fedora
toxefull
Passed Failed skipped Total TC

22. Test Results-Kilo
41
32
43
3
9
1
39 38 39
83
79
83
0
10
20
30
40
50
60
70
80
90
CentOS Ubuntu Fedora
toxesmoke
Passed Failed Skipped Total TC

23. Test Results-Liberty
44 45
43
1 0
2
39 39 39
83 84 84
0
10
20
30
40
50
60
70
80
90
CentOS Ubuntu Fedora
toxesmoke
Passed Failed skipped Total TC

24. Test Results-Kilo
867
700
1015
55
102
15
102 96 107
1024
898
1137
0
200
400
600
800
1000
1200
CentOS Ubuntu Fedora
run_tempest
Passed Failed Skipped Total TC

25. Test Results-Liberty
894
955
984
51
26 15
138 139 139
1024
898
1137
0
200
400
600
800
1000
1200
CentOS Ubuntu Fedora
run_tempest
Passed Failed skipped Total TC

26. Key Findings-Failed Tests analysis
OS/Tests toxefull toxesmoke run_tempest
CentOS 7.0 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related
to Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
Most of the Tempest test case
failures are
• Volume API’s related
Most of the Tempest test case failures are
• Scenario tests, snapshot pattern related,
Volume API’s and third party boto(this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS)
ReST API failures.
Ubuntu 14.04 Most of the Tempest test case
failures are
• Compute, scenario and Cinder
Volume related
• Third party boto (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
Most of the Tempest test case
failures are
• Compute ,scenario and
Volume Rest API failures.
Most of the Tempest test case failures are
• Compute, scenario, Volume and third
party boto (this is related to Elastic
Compute (EC2) and Object storage-S3
features of AWS) ReST API failures.
Fedora 21 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related
to Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
• Only one of the test case
failed here and is related to
Volume ReST API failure.
Most of the Tempest test case failures are
• Volume API’s and third party boto (this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS)
ReST API failures.
Kilo

27. Key Findings-Failed Tests analysis
OS/Tests toxefull toxesmoke run_tempest
CentOS 7.0 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS)
• Server multi-node ReST API
failures.
Tempest test case failure is
related to following
• Volume API’s
Most of the Tempest test case failures are
• Compute, Volume API’s, basic-Ops test
server, multi-node
• third party boto(this is related to Elastic
Compute (EC2) and Object storage-S3
features of AWS) ReST API failures.
Ubuntu 14.04
Most of the Tempest test case
failures are
• third party boto” (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
No failures found. Most of the Tempest test case failures are
• Compute, scenario, Volume and third party
boto (this is related to Elastic Compute
(EC2) and Object storage-S3 features of
AWS) ReST API failures.
Fedora 21 Most of the Tempest test case
failures are
• scenario tests, compute,
snapshot deleting, Volume API’s
and third party boto (this is
related to Elastic Compute (EC2)
and Object storage-S3 features
of AWS) ReST API failures.
Found two failures which are
• Volume ReST API failure.
Most of the Tempest test case failures are
• Volume API’s and third party boto (this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS) ReST
API failures.
Liberty

28. Key Findings-Skipped Tests analysis
Some of the major reasons for “Skipped” tests are,
 Pending bugs need to be resolved
 Live block migration not supported
 Instance validation and multiple instance support not available
 SSH & VNC support not available in some cases
 Some of the features not included in Dev Stack
 Ephemeral disk verification could not be done
 Multiple policy feature not supported
 Instance validation and multiple instance support not available
Note:- These findings are across OpenStack releases and OS flavors
Liberty
Kilo

29. Key Findings-Summary
KILO LIBERTY
toxefull toxefull
Passed Failed Skipped Total TC Passed Failed skipped Total TC
Centos 968 22 141 1131 CentOS 999 10 136 1145
Ubuntu 724 76 126 926 Ubuntu 1023 5 136 1164
Fedora 962 13 141 1116 Fedora 944 24 138 1106
toxesmoke toxesmoke
Passed Failed Skipped Total TC Passed Failed skipped Total TC
Centos 41 3 39 83 CentOS 44 1 39 83
Ubuntu 32 9 38 79 Ubuntu 45 0 39 84
Fedora 43 1 39 83 Fedora 43 2 39 84
run_tempest run_tempest
Passed Failed Skipped Total TC Passed Failed skipped Total TC
Centos 867 55 102 1024 CentOS 894 51 138 1024
Ubuntu 700 102 96 898 Ubuntu 955 26 139 898
Fedora 1015 15 107 1137 Fedora 984 15 139 1137

30. Conclusion
• Tempest test results across different OpenStack releases and OS flavors is distinct
• One of the primary reasons of this ambiguity is each OS flavor having different
kernel versions.
• The more latest the Kernel version used, the more stable are the test results
• The other reason is because of different OpenStack releases on different hardware
platforms
• Tempest execution on Liberty release on various Linux OS flavors is yielding more
stable test results as compared to Kilo release.
• The more latest the OpenStack version, the more stable are the test results.
• These test results can be used for future reference (based on h/w architecture used)

31. Troubleshooting…..
1. We have observed some times that when “localrc” file from root user and try to run
stack.sh command from the stack user, following error will be encountered.
Solution:- When such error is encountered, change the ownership of devstack
directory to stack user using the following command

32. Troubleshooting…..
2. If you are using proxy, While running stack.sh command you may get the
following error
Solution:- Best practice is to have internet connection without proxy, which will
avoid majority of the issues.
3. When we clone “nova” manually to /opt/stack/nova from git repository and try to
execute the command “stack.sh”, following exception will be encountered.
Solution:- To avoid the above exception, copy .git directory from cloned devstack
directory to /opt/stack/nova directory. This will help us to overcome the above
mentioned exception.

33. Tech-Scenarios

34. Tech-Scenario-I Validation of OpenStack workload management tool
Context
 Validation and functionality test of
OpenStack & management tool together
was required
 Validation of RESTful API’s for supported
OpenStack releases was required
 Certification testing on different OS
platforms supporting OpenStack was
required.
Solution/approach used
 Used Tempest test suite (along with some
customization where required) for the
validating stability of ReSTful API’s
 Using Tempest test suite and some home
grown test cases, we were able to verify the
management tool and various OpenStack
features
 Triaging with multiple combinatorial tests
and identifying gaps and helping respective
customer to fix the issues on time.
Outcomes & Learning
 Since OpenStack was niche technology,
understanding the requirement clearly was a
challenge
 State of the art certification of RHEL and
Ubuntu on OpenStack was a challenge.
 Support for Continuous OpenStack testing
for supported releases was a challenge
Technical Challenges
Technology Ubuntu, SuSE, RHEL, Cent OS , OpenStack releases Havana, Icehouse and Juno
Test Suite used Tempest
 Validation of complete workload management
tool on OpenStack with various test scenarios
 Validating Nova, Keystone, AMQP message
bus protocol and Horizon interfaces.
 Evaluation of the management tool for
robustness and stability of usage.
 Triaging issues that are related to OpenStack
product
 Certification of various OpenStack releases
done for RHEL and Ubuntu.
 Ready test suite for future releases of
OpenStack was evolved

35. Tech-Scenario-II: Validation of OpenStack API’s using Tempest suite
Context
 Complete coverage of OpenStack API
testing required
 Reliability and Serviceability of OpenStack
on a specified hardware configuration was
required
Solution/approach used
 Validation of complete ReST API suite for
various OpenStack releases and for specified
h/w platforms.
 Complete test failure analyses with log
reporting on time.
 Parallel Tempest configurations kick started
with test executions
Outcomes & Learning
 Running Tempest test suite on multiple
Linux platforms
 Debugging and finding failures and causes
 Rally configuration and execution
Technical Challenges
Technology Ubuntu, RHEL, Cent OS , OpenStack releases Juno and Kilo,
Test Suite used Tempest
Tools Rally
 Tempest configuration on multiple
platforms and OpenStack releases.
 Validated complete basic OpenStack API
suite and reported failures
 Suggested fixes where possible

36. Citations
http://docs.openstack.org/
https://github.com/openstack/
http://docs.openstack.org/developer/tempest/field_guide/
https://wiki.openstack.org
https://wiki.openstack.org/wiki/devstack
http://docs.openstack.org/developer/devstack/overview.html
http://devstack.org
http://devstack.org/faq.html
https://github.com/openstack-dev/devstack
http://docs.openstack.org/developer/tempest/configuration.html

37. Thank You
Shashidhar Soppin
Senior Architect
Reachable @
“shashidhar.soppin@wipro.com”

38. Back-up Slides

39. What is Tox?
tox is a generic virtual environment management and test command line tool
that we can use for the following purposes.
 We can check our package installed correctly with different Python
versions and interpreters or not.
 Running our tests in each of the environments, configuring our test tool
of choice is possible with using tox.
 tox acts as a frontend to Continuous Integration servers.
tox will come default along with DevStack package. But if you want to
manually install and configure it, install tox with pip install tox. Then put
basic information about the test environments you want to carry out
into a tox.ini file and follow the tox execution sequences as explained
earlier.

40. How to download latest Devstack?