Presentation by Richard Bishop and Gordon Appleby at HP Discover 2014 in Barcelona. In the presentation, Richard and Gordon described their experiences in cloud-based performance testing. They discussed the increased adoption of the cloud as an application-testing platform as well as the evolution of HP’s cloud-based testing products including LoadRunner, Performance Center and StormRunner.

1. B6424
Cloud Performance Testing
with LoadRunner
Richard Bishop & Gordon Appleby
3rd December 2014

2. Cloud Performance Testing with LoadRunner
Richard Bishop & Gordon Appleby
© Copyright 2014 Vivit Worldwide

3. Trust IV
• Richard Bishop
• Vivit Leader in UK, Board Member
• Performance Tester since 2000
• Background:
IT Infrastructure > MCSE >
Automated Software Testing
• Gordon Appleby
• Vivit Co-Leader in UK
• Performance Tester since 2000
• Background:
Ops > Dev > Infrastructure >
Messaging / Collaborative
Technology > Consultancy >
Automated Software Testing

4. Agenda
 Introductions
 Cloud timeline and hype
 LoadRunner history and development
 Choosing the right test tool
 LoadRunner in the Cloud
 Testing from the Cloud
• Use Cases
• Case Study
 Cloud testing tips
 Demo
 Q&A / Discussion

5. What is the Cloud?
“Cloud is a style of computing where scalable and elastic IT-related capabilities are provided
as a service to external customers using Internet technologies”
Gartner 2010
1968 1990 1998
Last 10 years
1962 1989 1992 1999 2006 2014

6. Cloud Hype

7. LoadRunner and related products
LoadRunner
6.0
2008 2010 2014
Performance
Center
9.0
1990s
On-Premise
In-cloud
2006
Performance
Center
LoadRunner
in the Cloud
V2
StormRunner
LoadRunner
8.0
LoadRunner
9.0
2012
LoadRunner
11.0
LoadRunner
12.0
Performance
Center
11.0
Performance
Center
12.0
LoadRunner
in the Cloud
V1

8. What does LoadRunner offer ?
Features such as TruClient make scripting easier.
Simple record / replay
Multi-protocol support, unlike most competition.
Single tool for Mobile, Ajax, Flex, HTML / HTML5, .Net, Java,
Silverlight, SOAP, Citrix, ERP and legacy
Browser and mobile device emulation built in.
Options to extend testing with network and service virtualisation.
Various cloud options:
Integration with Amazon & HP Cloud or other third parties.
SaaS options with StormRunner and Performance Centre.
Integration with multiple IDEs as well as CI and automated build systems.
Real time analytics built in, options to extend with SiteScope.
Diagnostic tool support, HP Diagnostics & third party tools.
Excellent reporting.

9. LoadRunner and related products
LoadRunner
12.0
LoadRunner
in the Cloud
V2
StormRunner
Performance
Center
12.0
Best of breed performance test tool
Integrates with other ALM products
Multi-protocol support, excellent reporting capabilities
Alternative to LoadRunner (on-premise)
On-demand licensing and hardware
Detailed reporting
Multi-protocol support
Cloud only
On demand usage
Uses standard LoadRunner scripts (web only)
Less detailed reporting but great for “burst testing”
Enterprise testing
Better integration with HP ALM suite
Dashboards, shared assets, ideal for global “follow the sun” operations

10. Which tool should I choose?
Decision factors
• Application Under Test
• Comms protocols
• Virtual user numbers
• Virtual user locations
• Tester locations
• Tester skillset
• Existing test assets
• Script complexity
• Test data requirements
• Security
• Budget
• Timescales
• Cloud?

11. Choosing a cloud-based test tool
StormRunner
LoadRunner
in the Cloud
LoadRunner
12.0
Performance
Center
SaaS
Load Generators
Performance
Center
On-premise
Load Generators

12. LoadRunner on-premise
Load Generators
Load Generators
LoadRunner
12.0

13. LoadRunner in the Cloud
LoadRunner
in the Cloud
Load Generators
Load Generators

14. Performance Center on-premise
Load Generators
Performance
Center
On-premise
Load Generators

15. Performance Center SaaS
Performance
Center
SaaS
Load Generators
Load Generators

16. StormRunner
StormRunner
Load Generators

17. So which product should I choose?
StormRunner
LoadRunner
in the Cloud
LoadRunner
Performance
Center
Quick, high volume tests
Web only
Remote load generation
Short-term engagement (s)
Any protocol
Remote and local load generation
Advanced use e.g. NV, diagnostics etc.
More frequent testing
C of E approach
Remote and local load generation
Detailed reporting and analysis
Very frequent testing
Larger C of E / “follow the sun”
Dashboard for stakeholders
Integration with ALM etc.

18. LoadRunner in the Cloud
Web-based VM admin console
• Pre-configured servers, flexible license options.
• 100, 200 & 500+ vUsers on-demand
• Support for any other number
• All LR protocols supported
• Controllers and Load Generators
• CPU / RAM / HDD configurable
• With or without firewall
• Remote or local LGs
• All HP ALM integrations supported
• ALM / QC / UFT
• NV software pre-installed
• UFT and other software available
• On-demand availability and pricing
• PayPal or Account
• Pay for what you use
• Efficient license use

19. Is “The Cloud” any good for testing?
“Cloud computing is a model for enabling convenient, on-demandnetwork access to a shared
pool of configurable computing resources that can be rapidly provisioned and released with
minimal management effort or service provider interaction.”
Source: The National Institute of Standards and Technology
BUT….
• Unproven technology for testing:
reliability/repeatability
• Steep learning curve
• Costs difficult to quantify
• Security concerns
• Performance

20. But does it work?

21. Use Case 1
Variable / unanticipated demand
Ad-hoc testing at short notice

22. Use Case 2
Small organisation, limited budget
Possibly considering open-source
Performance
Center
LoadRunner
LoadRunner
in the Cloud
StormRunner

23. Use Case 3
Users outside firewall
Better end to end tests



24. Use Case 4
Agile Dev/Ops
Developers do own testing, limited budget, pay-per-use desirable
Multi-protocol support.
SOA testing
Component-level testing
Service Virtualisation etc.
Integration with multiple IDEs as well
Integration with CI and automated build systems.

25. Use Case 5
High Virtual User Numbers
Lots of load generators needed
Virtual User
Type
vUsers per Load
Generator
Web (HTTP / HTML) 2000
TruClient 50
RDP 200
Citrix ICA 20
SAP GUI 20

26. Anon
Case Study
UK Financial organisation
250 vUser tests
Citrix and Web
Needed “outside firewall testing”
Needed remote testing



27. Tips when testing from cloud
Valid concerns
1 - “You don’t know what you’re getting with virtual machines”
2 - “Performance of load generators is too variable”
3 - Understand your application
4 - Time synchronisation
Avoid pitfalls
Know your platform’s capabilities
Over provision
Smoke test AUT and test platform
Re-validate tests, test and re-test (best practice)
Use diagnostics software to simplify performance engineering
Monitor all infrastructure closely
Optimise test tool for virtualisation platform
Use benefits of the cloud/virtualisation
Low cost
On-demand
Flex up & down
Over provision
Research the subject

28. Demonstration
LoadRunner in the Cloud

29. Thank you
Please ask / get in touch if you have any more questions:
Richard Bishop
@richardbishop
richard.bishop@trustiv.co.uk
http://uk.linkedin.com/in/richardbishop
Gordon Appleby
@trustiv
gordon.appleby@trustiv.co.uk
http://uk.linkedin.com/in/gordonappleby
www.vivit-worldwide.org
blog.trustiv.co.uk
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change
without notice.