Testing in a multi-cloud environment presents unique challenges. This presentation reviews some of these challenges and includes some examples of how they can be addressed

1. Testing in a Multi-Cloud
Environment
Seema Jethani

2. Who am I?
Seema Jethani @seemaj
Senior Product Manager, Dell
( formerly Enstratius / enStratus )
Blog
★http://seemaj.wordpress.com
★http://cloudfieldnotes.com/author/seema/
Slideshare
http://www.slideshare.net/svjethani/
presentations

3. CLOUDS ARE LIKE
BABIES

4. Bringing joy

5. But also presenting...
UNIQUE CHALLENGES

6. Speaking their own language

7. Capability Terminology
Account
AWS = Account
Terremark = Organization
Region
AWS = Region
Terremark = Environment
Data center
AWS = Availability Zone
Terremark = Compute Pool
Machine Image
AWS = Machine Image
VMware = Template vApp
Joyent = Datasets
OpenStack = Image
CloudStack, Terremark = Template
Firewall
AWS, OpenStack, CloudStack = Security Group
Terremark = Firewall
Volume
AWS = Elastic Block Storage
VMware, IBM, Terremark = Disk
OpenStack = Cinder or Nova Volume
Partial
List

8. And are sometimes hard to
work with

9. Examples of challenging cloud behavior
1. Requiring paperwork / manual processing
2. Requiring calling support to create private images
3. Requiring a new account for new API versions, thus forgetting
backward compatibility
4. Using same API version, but continuing to make changes
5. Taking hours to launch a server or launching server requests
serially
6. Having a ridiculously low cut-off for throttling API requests e.g.
dozens of API calls per second
7. Not providing a way for the VM to discover its own ID

10. Each is able to do different
things in different ways

11. Partial Cloud Feature Matrix
Feature AWS Rackspace Terremark Joyent
Launch Server S S S S
Pause Server S NS S S
Make Image S S S NS
Create Snapshot S S NS NS
Create Volume S S NS NS
Create Loadbalancer S S NS NS
Create/ Reserve IP S NS S NS

12. Examples of the cloud nuances
1.Make Image
CloudSigma requires server to be stopped before making an image
2.Attaching a volume
e.g. Terremark requires the server to be stopped for for volume to
be attached
3.Creating a snapshot
e.g. Make a snapshot from the volume fails in CloudCentral if the
Volume is not attached to a server
4.Connecting via ssh
CloudSigma requires users to access a centos server via vnc first
to enable ssh

13. Cloud requires us to
rethink our testing
approach

14. We need to learn “cloud speak”
Understand differences in behavior
But also importantly

15. Plan for failure

16. How is this different?
Customers expect cloud failure
but also
Expect recovery from cloud behavior to be transparent. They
don’t care about root cause infrastructure issues
Test for failure , auto-recovery , MTTR

17. How is this different?
Customers expect cloud failure
But don’t expect it to impact their business
Test for cloud failure and its impact on customer application
performance.

18. Enstratius testing journey

19. Enstratius Cloud Management Platform

20. The release process

21. build
Today
Artifacts Installer API tests
Github
Artifactory
Vagrant
AWS

22. Jenkins
ReportNG Report
Automated
Test
Framework
(TestNG)
GitHub
Enstratius
API Server
Cloud A
Cloud Z
estrov
ant
API Test Framework

23. Cloud account
access keys, test
configs
HeaderConfig.xml
Cloud specific
resource/method
exclusions
JSON/XML Inputs for
REST Calls common
to all clouds
Setup Test bed on Cloud under test
Tear Down Test bed on Cloud under test
UpdateAdd Delete List
Atomic Tests
Cloud Specific Inputs
for REST Calls (if any)
estrov

24. API test workflow
<?xml&version="1.0"&encoding="UTF98"?>!
<cloudService!name="AWS">!
<excluded>!
<resource!name="kvdb"!/>!
<resource!name="customer">!
<method!name="add"!/>!
<method!name="changeCurrency"!/>!
<method!name="revertCurrency"!/>!
<method!name="setTimeZone"!/>!
<method!name="revertTimeZone"!/>!
</resource>!
</excluded>!
</cloudService>
{"addLoadBalancer":!
[!
{!
"region":{"regionId":REGION_ID},!
"budget":BILLINGCODE_ID,!
"description":"TestNG_Trial_LB",!
"name":DYNAMIC,!
"label":"red",!
"customer":{"customerId":CUSTOMER_ID},!
"dataCenters":[{"dataCenterId":DATACENTER_ID}],!
"owningGroups":[{"groupId":GROUP_ID}],!
"listeners":!
[{"protocol":"HTTP","publicPort":80,"privatePort":80}]!
}!
]!
}
start
Read the cloud
access keys & test
configs
Iterate through the
test resources
Exclude
Resource
?
Run atomic tests
Generate report
Headerconfig.xml
AWS.xml
Read resource/
method inputs
<?xml&version="1.0"&encoding="UTF98"&standalone="no"?>!
<CONFIG>!
<BILLINGCODE_ID>403</BILLINGCODE_ID>!
<USER_ID>50950</USER_ID>!!
<DUPLICATE_BILLING_ID>402</DUPLICATE_BILLING_ID>!
<DUPLICATE_GRP_ID>501</DUPLICATE_GRP_ID>!
<ROLE_ID>55501</ROLE_ID>!
<GROUP_ID>500</GROUP_ID>!!
<ACCOUNT_ID>200</ACCOUNT_ID>!
<CLOUD_ID>1</CLOUD_ID>!
<CUSTOMER_ID>200</CUSTOMER_ID>!
</CONFIG>
Generate
Dependency
resources
<?xml&version="1.0"&encoding="UTF98"&standalone="yes"?>!
<header>!
<csp>AWS</csp>!
<resource>all</resource>!
<accessKey>aws_accesskey</accessKey>!
<secretKey>aws_secretkey</secretKey>!
<apiServer>enstratius_apiserver_ip</apiServer>!
<apiServerPort>enstratius_apiserver_port</apiServerPort>!
<version>enstratius_apiserver_version</version>!
<testForFailure>false</testForFailure>!
</header>
stop
Loadbalancer/add.json
Config_200.xml
Finished iterating
Yes
No

25. The extra steps
1. Not all functions are supported by all clouds. An exclusion list
needs to be maintained.
2. Cloud specific behavior needs to be coded
e.g. Terremark requires the server to be stopped for for volume to be
attached
3. Sanity check - Issues found must be verified directly using cloud
console / cloud API

26. Build
Near-term updates
Artifacts Launch
API testsGithub
Artifactory
AWS
Jenkins
Installer
Checkout Teardown

27. Build
Future
Artifacts Launch
API testsGithub
Artifactory
AWS
Jenkins
Installer
1. Checkout 6. Teardown
Selenium

28. Other tools on the radar

29. Visualizing performance
http://twitter.github.io/zipkin/

30. Vulnerability scans