This presentation explores the latest service virtualization research and shares firsthand best practices and benefits of service virtualization from Comcast’s Director of Performance Test. Discover how to: enable more complete testing earlier in each iteration, streamline lean processes with more reliable test environments, and manage complex tests in a dynamic development environment.

1. Service Virtualization
Delivering Complex Test Environments on Demand
Frank Jennings; Director, Performance Test & Engineering, TQM, Comcast
2014-11-12
Mark Lambert; VP Products and Support, Parasoft Corporation
Parasoft Proprietary and Confidential 1

2. Drivers of the SDLC
SDLC Speed is the
difference between a
first mover and a
follower
Damages associated
with software failure
are increasing and
very real
Brand equity is critical
and Quality drives
brand loyalty
Parasoft Proprietary and Confidential 2

3. Current Pressures of the SDLC
 Adoption of Agile to meet Acceleration
 Requires Continuous Infrastructure
 Need to assess Business Risk of rapidly changing
code
 Needs Continuous Testing with automated Quality Gates
 Unit Test is not enough
 Testing complexity grows further into the SDLC
 DevOps requires infrastructure to “shift left” On-demand
Environment access
Parasoft Proprietary and Confidential 3

4. Complexity is a Barrier to Innovation
Scheduled Access
3rd Party System
Evolving Component
Mainframe
Parasoft Proprietary and Confidential 4

5. An application is not just “one application” …
Parasoft Proprietary and Confidential 5

6. … it’s made up of „several components” …
Parasoft Proprietary and Confidential 6

7. … and they are not all ready at the same time.
Parasoft Proprietary and Confidential 7

8. How do we ensure quality of the components?
Static Analysis,
Unit Testing,
Code Review
Parasoft Proprietary and Confidential 8

9. How to test without dependant components?
API
testing
Reusable
Virtual
Assets
Parasoft Proprietary and Confidential 9

10. Test as soon as the components are ready
Parasoft Proprietary and Confidential 10

11. Service Virtualization
Service Virtualization delivers
simulated application behavior
in dev / test environments
allowing an organization to test
on-demand
Parasoft Proprietary and Confidential 11

12. How to apply Service Virtualization
1. Identify the assets;
 Which types of asset are good to simulate?
2. Model behaviour;
 How much is enough?
3. Deploy and reuse;
 Provide easy ad-hock access and sharing
4. Automate Provisioning;
 Integrated into a Continuous Test Environment
Parasoft Proprietary and Confidential 12

13. When to use SV? Access vs Configuration
 Access
 Dependent applications difficult
 Scheduling conflicts
 High access fees
 Geo-political boundaries
 3rd party or partner applications
 Configuration
 Complex to configure
 No control
 Limited variability
 Consumes test time
Complex
Configuration
Cloud
App
Web
Server
Internal
Service
ERP
Mainframe
External
Application
External
Database
Message
Queue
Internal
Database
ESB
Internal
Application 3rd Party
Service
Simple Difficult
Test Access
Parasoft Proprietary and Confidential 13

14. When to use SV? Access vs Configuration
 Access
 Dependent applications difficult
 Scheduling conflicts
 High access fees
 Geo-political boundaries
 3rd party or partner applications
 Configuration
 Complex to configure
 No control
 Limited variability
 Consumes test time
Complex
Configuration
Service Virtualization
Cloud
App
Web
Server
Virtual Lab /
Server
Virtualization
(i.e. VMWare)
Internal
Service
ERP
Mainframe
External
Application
External
Database
Message
Queue
Internal
Database
ESB
Internal
Application 3rd Party
Service
Simple Difficult
Test Access
Parasoft Proprietary and Confidential 14

15. The Total Cost of System Access
Service Virtualization dramatically reduces the cost of
delivering a dev / test environment
Hardware Savings
Staged Application Instance
Staged w/ Virtual Test Lab Access Efficiency
Service Virtualization
System Behavior
Cost
 Only manage the functionality
you need
 Only manage data you need
 Low-Risk “disposable instance”
that can be accessed at anytime
 Dramatically reduce test
environment setup and
management
Parasoft Proprietary and Confidential 15

16. Setting up Service Virtualization
Define &
Capture
Initiated from the system under test,
the user has the ability to capture
detail from a live monitor that
analyzes system traffic, from
analyzing transaction logs or by
modeling virtual behavior within the
Parasoft Virtualized interface.
Model &
Deploy
After the virtualized artifact has been
captured, users can now instruct the
details of the virtualized asset
behavior. This includes: performance,
data sources and conditional response
criteria. The virtualized asset is then
provisioned for simplified uniform
access across teams and business
partners.
Provision &
Consume
The virtualized asset can now be called
for unit, functional and performance tests.
The virtualized asset can be leveraged by
any test suite – including Parasoft Test.
Parasoft Proprietary and Confidential 16

17. Example: Capturing current behavior
Application
Under Test
Service
Test
2 Capture
4 Deploy
Development
LoadRunner
Performance Test
3 Create
Database
Application
Mainframe
Service
QA and Test
Engineer
1 Define Monitors
Parasoft Proprietary and Confidential 17

18. Provision the Environment on Demand
Application
Under Test
Database
Application
Mainframe
Service
6 Consume
5 Manage
Environment
Management
QA and Test
Development
Performance Test
Engineer
QC/ALM
Rational
Parasoft Proprietary and Confidential 18

19. Rapid Environment Access
Parasoft Proprietary and Confidential 19

20. Reduced wait time for test team by 60% for a
major media conglomerate
Agile/Parallel development limited by system dependencies
 The Challenge
 Large agile development effort to adopt Service Oriented Architecture (SOA)
 High risk project but the Test team “stuck waiting for the first build”
 Development of functionality was not easy to coordinate as different teams
had different schedules; not all finished at the same time
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
Current
Development/Testing
dependencies
Parasoft Proprietary and Confidential 20

21. Reduced wait time for test team by 60% for a
major media conglomerate
Agile/Parallel development limited by system dependencies
 The Solution
 Use descriptions of the new services (WSDL, XSD, example JSON payloads) to
build Virtual assets prototyping the new functionality.
 Test team builds tests with against the prototypes with SOAtest and the
independent development tests use the prototypes to perform early stage
Integration Testing
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Virtual Asset Iteration
Virtual Asset Iteration
Current
Development/Testing
dependencies
Parasoft Proprietary and Confidential 21

22. Reduced wait time for test team by 60% for a
major media conglomerate
Agile/Parallel development limited by system dependencies
 The Business Benefits
 Met business goals and timelines, were able to test functionality “as soon as”
it was available. Practiced TDD against prototypes to get a head-start on ‘full
system testing’
Iteration
✔
✖
Iteration
Iteration
Team A
Team B
Team C
✔
✔ ✖
✖
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
Parasoft Proprietary and Confidential 22

23. Provisioning of test environments went from weeks to
minutes from large government organization
Limited access to mainframes, ERPs, or 3rd party systems
 The Challenge
 Test results did not reflect actual performance in production
 Inability to test multiple projects at once
 Testing environments were unstable and unreliable
Parasoft Proprietary and Confidential 23

24. Provisioning of test environments went from weeks to
minutes from large government organization
Limited access to mainframes, ERPs, or 3rd party systems
 The Solution
 Combining Service Virtualization and VMWare technology to build a
testing environment that properly replicates the production
environment while allowing multiple virtual copies of that
environment to be provisioned on-demand.
PV
PV
PV
PV
PV
PV
PV
PV
Parasoft Proprietary and Confidential 24

25. Provisioning of test environments went from weeks to
minutes from large government organization
Limited access to mainframes, ERPs, or 3rd party systems
 The Business Benefit
 Developers empowered to provision their own test environments.
 Provisioning delays reduced from weeks to minutes.
 Project reliability enhanced.
&
Parasoft Proprietary and Confidential 25

26. Reduced cost of creation and management of Test
Environments by 85% for a major international bank
Mobile Application development and extension of current systems
 The Challenge
 Complex test environments including; ATM, website and mobile front-end
systems with WebSphere, Mainframe, MQ, SAP and DataPower back-end
systems
 Each test system costs >$2m, therefore only two environments available
 Active development at every level of the system resulting in full end-to-end
testing requiring careful coordination and sub-system teams often offline or
unavailable due to upgrades
Application
Database
Mainframe
SAP
Parasoft Proprietary and Confidential 26

27. Reduced cost of creation and management of Test
Environments by 85% for a major international bank
Mobile Application development and extension of current systems
 The Solution
 Create a library of virtual assets for teams to share and use Environment
Manager to replicate and manage Virtual test labs using different performance
charactistics for different load scenarios.
 Use automated test solution to test and validate at each layer independantly
Environment
Management
Virtual
AVssirettual
AVssirettual
Asset
Application
Virtual
Asset Virtual
Virtual
Asset
Database
Virtual
AVssirettual
Asset
Mainframe
Virtual
AVssirettual
AVssirettual
AVssirettual
Asset
SAP
Virtual
Asset
Asset
✔
✖
Parasoft Proprietary and Confidential 27

28. Reduced cost of creation and management of Test
Environments by 85% for a major international bank
Mobile Application development and extension of current systems
 The Business Benefit
 Improved efficiency, enabling Agile development, with the teams able to test
in isolation without effecting full end-to-end system tests.
Environment
Management
✔
✖
✔
✖
✔
✖
✔
✖
Parasoft Proprietary and Confidential 28

29. Comcast Case Study: Applying
Service Virtualization to
Performance Testing
Parasoft Proprietary and Confidential 29

30. Performance Team Test Coverage Test Target
Commercial Residential
Virtualized
Test with Virtualize
Self Service eCommerce
Knowledge
Management
Authentication
Event
Mgmt
Order
Mgmt
Enterprise Services Portal
Business
Process
Mgmt
Self Install
Process
Provisioning Location CDV Services
Location DB
Customer
Customer &
Account
IMS
Network Provising
Customer
Self
Service
Order Entry
Management
UI Widget
Warehouse
Inventory Billing
Frontend OSS / BSS Middleware Network
Order
Entry
Care
Care
Voice
Response
System
Order
Fullfillment
Order
Commercial Services Platform
Ticketing
Inventory
Offer Mgmt
Customer &
Account
Activation
Billing
3rd Party
Network
Services
Customer DB Offer DB
eMTA
STB
Modems
Wifi
CMTS
Network
Element
Status
Networks
Services
Commercial
DBs
TN Inventory
Managment
E911
Services
Parasoft Proprietary and Confidential 30

31. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 31

32. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 32

33. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 33

34. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 34

35. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 35

36. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 36

37. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 37

38. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 38

39. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 39

40. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 40

41. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 41

42. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 42

43. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 43

44. System Under Test
QA Stage Production
Test Automation
Services & DBs
Virtualized Services
Real End Points
Development
Intelligent Proxy
Deployment Unit
Services Regression Performance Manual Test
Deployment
Automation
Continuous
Deployment Engine
Code
Deploy
Dev
Regression
Deployment
Validation
Code
Quality
Regression
Baseline
Performance
Deploy
QA
Deploy
Stage
Deploy
Prod
Deployment
Validation
Functional
Deployment
Validation
Performance
Deployment
Validation
Parasoft Proprietary and Confidential 44

45. Service Virtualization
 Emulates dependencies for the Test Environment
 Reduces the complexity for early stage testing
 Increases predictability
 On-Demand “Test Anytime, Anywhere, Anyway”
 Automated Provisioning for different use-cases
 Automated Test Data Management/Simulation
 Remember: Does not eliminate the need for
System/Integration Testing
Parasoft Proprietary and Confidential 45

46. Thank you
Questions?
2014-11-12
Parasoft Proprietary and Confidential 46