More Related Content Similar to Cloud testing: challenges and opportunities, TaaS, Integration Testing (20) More from Dr Ganesh Iyer (20) Cloud testing: challenges and opportunities, TaaS, Integration Testing1. Test challenges and methodologies
with Cloud
Dr Ganesh Iyer, Progress Software
Ref: Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
2. Introduction
Cloud Computing and Web 2.0 technologies
• Web-driven applications
• Driven by browser and do not need any installation
• On-demand resource availability
• Faster time to market
• Reduced capital and operational expenses
2
© 2013 Progress Software Corporation. All rights reserved.
3. Introduction
Paradigm
shift
Seamless
upgrades
Sharing of
resources:
Multitenancy
Traditionally,
applications are
designed first,
then developed,
tested and
distributed
Traditionally,
software upgrade
needs to get new
version of the
software, down
the system and
upgrade
Traditionally,
software
development &
deployment is
mostly in
dedicated
resources
3
But with Cloud,
upgrade should
happen live with
minimal or no
down time E.g.
Gmail
© 2013 Progress Software Corporation. All rights reserved.
With Cloud, these
resources are
often being shared
among multiple
customers
Auto-elasticity:
Resources ondemand
Performance
Disaster recovery
SaaS Applications;
Browser based
configuration and
usage
New test
dimensions
Changes in
existing
dimensions
Compatibility
Availability and
Business
Continuity
Security: Multitenant penetration,
Identity federation
management
4. Common Cloud Testing Dimensions
Elasticity
Testing
Security
Testing
Performance
Testing
Resource
acquisition/
Release
Time
Traversal
vulnerability
Compatibility
Testing
Time to
deploy
Accessibility
testing
User access/
Roles
Multi-tenancy
Time to
Genesis
Provisionin
g on-the-go
Load
Testing for
ELB
4
Identity
federation
management
Communicati
on latency
over SSL
Multi-tenant
penetration
Connectivity
and reliability
with 3rd
parties
Reliability
and
Availability
Latency
© 2013 Progress Software Corporation. All rights reserved.
Globalization
and
Localization
testing
API
Integration
Testing
Live
Testing
Connectivity
and
invocation
testing
Disaster
recovery
API load
testing
Live
upgrade
testing
API security
Compatibility
under
different
situations
Self-healing
ability
Multitenancy
Availability
and
business
continuity
Multitenancy
Testing
Multi-tenant
penetration
Rigid failure
containment
Availability
and
Business
continuity
Risk of
correlated
behaviors
Service
transition
activity
analysis
5. Elasticity Testing
Based on subscription plans, check the maximum vertical/horizontal limit
Auto provisioning/freeing on-the-go
Testing for Load Balancing
Performance
5
Test for the impact while auto scaling
Response time/Release time for provisioning of resources
Load Testing of Different subscriptions
© 2013 Progress Software Corporation. All rights reserved.
6. Elasticity and Scalability
• Limitations on max objects/applications at a time
• Number of applications that can be developed per
platform instance
Load
requirements
Cloud App Development
Platform
Load
requirements
Elasticity Load Testing for different usage scenarios
• 100’s of administrators accessing the management
Time
Time
6
Load
requirements
Load
requirements
console
Time
Unpredictable/Predictable burst: Some tenants have a specific pattern in
their usage and test for sudden expected/unexpected variations in the
usage
© 2013 Progress Software Corporation. All rights reserved.
Time
7. Security Testing Implications
Outsourced + Insourced
External (Data Storage)
Sensitive data management in the Cloud
S3 Storage or any other cloud vendor storage
DBaaS (OE Database)
Third party services to be tested only for connectivity
with over services
Functionality testing for our services
Single sign on account for all the services
Application Security – using both the ProPaaS and
third party services
Multi-tenant penetration testing
Proprietary (API’s)
7
De-perimeterised
Traversal vulnerability
Communication latency over SSL
API Level and Application Security
User access/Roles testing
Access from different Clients to the ProPaaS
platform testing
From VPN, Firewall settings, Antivirus software
Authentication/Authorization
Identity federation management testing
© 2013 Progress Software Corporation. All rights reserved.
https://collaboration.opengroup.org/jericho/cloud_cube_model_v1.0.pdf
8. Security
User access/roles
Web UI
Vulnerabilities and Attacks
• Authentication and authorization
• Identity federation management Single Sign-On
• Access from different clients to the
platform
• VPN, firewall settings, anti-virus
• User privileges
• SQL Injection
• URL Manipulation
• Cross-site scripting
• Password cracking
• Hidden-field manipulation
• Multi-tenancy penetration testing
• Traversal vulnerability
• DDoS attacks
Multi-tenant Database
• Data management at DB (Encryption
security)
• Data retention and destruction for DB:
Erase and sanitize when space is
reallocated
8
Other Security Concerns
• Fault-injection-based testing for web-driven services (Including verification for all
input fields, network interface, environment variables etc)
• Fuzzy testing for web-driven services (Injecting random data into application to
determine whether it can run normally under the jumbled input)
• Data privacy: Custom SLA capabilities
© 2013 Progress Software Corporation. All rights reserved.
9. Performance Testing
Time to deploy
Density (multi-tenancy)
Reliability Testing
Availability Testing
To facilitate Follow the Sun advantage model
Connectivity and reliability with 3rd party components from our Cloud
• Reliability of the data sent between two systems
9
© 2013 Progress Software Corporation. All rights reserved.
10. Latency under different conditions
Network
• The round-trip time between the browser and the server
• The number of round-trip times it takes to completely load a web
page
• The protocol’s flow & congestion control properties, and
• Competing traffic, unreliable network
Processing
Client side
10
• The time it takes to the server to prepare the content that will be sent
to the user.
• Resource sharing introduces contention risks, increased recovery
latencies
• The time the web browser needs to prepare the received content to
be presented
• Latency when accessed from different unreliable sources (e.g.
handheld devices, PDAs etc..)
© 2013 Progress Software Corporation. All rights reserved.
11. Live Testing and Failover Testing
Robustness of infrastructure against failures
Live Upgrade Testing - Managing customer applications for maintenance/upgrade
Recovery time in case of product/platform crash
Self healing ability in case of product/platform
Availability and Business continuity in multi-tenant environment
11
© 2013 Progress Software Corporation. All rights reserved.
12. Multi-Tenancy Testing
Verifiable resource accounting for Billing
•
In case, multiple tenant’s data need to be captured simultaneously by the billing agent
Multi-tenancy penetration testing
•
Tenant level access and their boundaries based on authentication and authorization
Rigid failure containment between tenants
•
Failure because of one tenant’s action shouldn’t stall other tenant’s activities or bring down the complete system down
Testing for the risk of correlated behaviors
•
Example: multiple application instances execute the same recovery action or periodic maintenance actions
simultaneously
Service transition activity analysis
•
12
No service impact on other tenant instances when each and every tenant-specific configuration parameter is changed.
© 2013 Progress Software Corporation. All rights reserved.
14. PCTF: Progress Cloud Test Framework
PCTF
Injection strings
Test Logger
Input
Crawler
Error
parameters
Injector
patterns
Error Pattern
Manager
Security
Testing
Test manager
Injection strings & Results
Pages/hyperlinks
Client shell
Output
Repository
• Parameter
Test Repository:
• Test suite
Plug-ins
configuration
• Libraries: SQL string library, Error
•
Test suite
selection
Result
Analyzer
• Result/Log
collection
API Integration
Testing
SLA
patterns library etc
System Under Test (Cloud
Platform/Application)
Test DB
(MySQL)
Synthetic Load
Generator
• SLA mapping table
• SLA metrics database
SLA information
Stubs for 3rd parties
SLA Monitor
Testing
14
Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
© 2013 Progress Software Corporation. All rights reserved.
16. What is Integration Testing?
Often, many Cloud-based systems will be integrated to each other for delivering a
cloud-based offering.
For example, for a SaaS application, the SSO (single-sign-on) mechanism may be
handled by a 3rd party system and payment mechanisms might be handled by another
3rd party system such as PayPal.
Hence, we need to get sandbox environments for such payment gateways which are
identitical to its production environment in order to have the systems behave same in
both test as well as live environments.
16
© 2013 Progress Software Corporation. All rights reserved.
17. Typical issues
Multiple systems in an integrated Cloud-based product behave differently and
verification process for different systems differ each other.
Some of the systems do not allow automated deletion of data created for testing. So it
imposes a unique requirement to have unique users created every time we perform
such test automation
Unpredictable delays in updating various systems.
Different types of environments for testing. For example, presence of web UI testing
and runtime testing in one test scenario poses its own unique challenges
17
© 2013 Progress Software Corporation. All rights reserved.
18. Integration test automation challenges
Unique requirements:
• UI components
• Runtime components
Need to either develop a test framework that can efficiently test both run time and UI
components or use a combination of two frameworks for runtime/UI testing
Needs to have mechanisms to initiate the test written in one framework from the other
one and to generate a combined test results
Using the exposed APIs provided by other third party systems integrated as part of the
product
18
© 2013 Progress Software Corporation. All rights reserved.
21. Overview of Testing as a Service TaaS
Wikipedia - Testing as a Service (TaaS, typically pronounced 'tass') is a model of
software testing whereby a provider undertakes the activity of software testing
applications/solutions for customers as a service on demand. …involves the ondemand test execution of well-defined suites of test material, generally on an
outsourced basis.
Shared Services delivery model
Pay per by drink and not by Glass
Standardised, Repeatable services
21
© 2013 Progress Software Corporation. All rights reserved.
Courtesy: http://tinyurl.com/taasmphasis
22. TaaS: Conceptual Model
Customer 1
Customer …n
Customer 2
Fixed price
per product
Commercial
Models
SLAs
Customer Service Management
Pre-defined
Automation Offering
Service
Products
Move towards
standardised services
Service Catalogue for business
Web
Performance Offering
Perf .Test
Web/ERP
Functional Test
Offering
Customer
Interface
Activities and
deliverables
Predefined
Manual Testing
Demand Management
Service Catalogue for Operations
Owned by service
provider.
Continuous
Improvement
Service
n…
Service 3
Test
Assets
Test
Assets
Service 2
Test
Assets
Global Delivery
Model
Testing
Framewor
k
Test
Assets
TAAS
Engine
Internal Service Management
Owned by
service
provider
22
HAAS/
Cloud
In house
Tools /
External Tools
Test
Analyst
Test
Process
High sharing
of resources
Courtesy: http://tinyurl.com/taasmphasis
© 2013 Progress Software Corporation. All rights reserved.
23. Advantages of testing in the Cloud
Traditional Testing
Testing in Cloud
Low asset utilization
Scalability: Long time to increase
capacity
Less time (instantaneous) increase
and reduction in capacity
Long time to build datacenters
Purchased as a service from cloud
providers
Difficult to manage
Better management and increased
productivity
Duplicate test systems
Aggregated system
Creates unnecessary wastes
23
Improved asset utilization
Cleaner, greener testing, saving in
CO2 emissions
© 2013 Progress Software Corporation. All rights reserved.
24. Conclusions
Various Cloud test dimensions
• Elasticity, Multi-tenancy, Security, Live Upgrade, Performance
Integration Testing: A unique requirement with Cloud
Automation challenges and possible approaches
TaaS
24
© 2013 Progress Software Corporation. All rights reserved.