Two key challenges to continuous load testing are provisioning a test system to handle the load and accessing load generators to drive the traffic. In this webinar from SOASTA & CloudBees, you will learn how to: Build realistic automated web performance tests and run them in Jenkins Architect and launch a test environment that auto-provisions in the cloud Manage a load generation grid to drive load tests in a lights-out mode Establish a performance baseline in your daily Jenkins reports

1. Copyright © 2014 SOASTA, Inc. - All rights reserved.Webinar
and
Present
Automating
the hard
parts, made
easier

2. 2Copyright © 2014 SOASTA, Inc. - All rights reserved.
Continuous Load Testing
TODAY’S PRESENTERS
Mark Prichard: CloudBees, Dir. Product Management
Mike Ostenberg: SOASTA, Dir. Systems Engineering
Brad Johnson: SOASTA, VP Product
TWO COMPANIES, A SINGLE SOLUTION
- the leader in Mobile and Cloud Testing
- the home of Jenkins in the Cloud
Together, we offer a complete cloud-based service:
Continuous Delivery for Web & Mobile

3. 3Copyright © 2014 SOASTA, Inc. - All rights reserved.

4. 4Copyright © 2014 SOASTA, Inc. - All rights reserved.
• Why should we load test earlier?
• Building tests and preparing the test environment
• Connecting your load tests to your CI (Jenkins)
• Establishing and managing to a performance
baseline
• Q&A
Questions:
Submit in the question box during event

5. 5Copyright © 2014 SOASTA, Inc. - All rights reserved.

6. 6Copyright © 2014 SOASTA, Inc. - All rights reserved.

7. 7Copyright © 2014 SOASTA, Inc. - All rights reserved.Copyright © 2014 SOASTA, Inc. - All rights reserved.

8. 8Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do

9. 9Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile

10. 10Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile
“Because no one ever told us
to.”
oThere are no “performance coverage” requirements
oThere isn’t a performance baseline for reference
oThere isn’t a defined performance target
oNo one is managing to a performance metric

11. 11Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile
“Because no one ever told us
to.”
oThere are no “performance coverage” requirements
oThere isn’t a performance baseline for reference
oThere isn’t a defined performance target
oNo one is managing to a performance metric
Until Now.

12. 12Copyright © 2014 SOASTA, Inc. - All rights reserved.

13. 13Copyright © 2014 SOASTA, Inc. - All rights reserved.
Iteration 1
Automated unit tests
The Issue:
• High expectations from tests run with diminishing time
windows
• Code-level performance bugs found late set entire project
back
• Separate team and schedule conflict with delivery goals
• Major issues escape to production
Iteration 2
Automated unit &
functional tests
Iteration 3
Regression test suite
Iteration n
Regression test suite
Load &
Performan
ce Testing
Deploy
Issues
?

14. 14Copyright © 2014 SOASTA, Inc. - All rights reserved.
Iteration 1
Automated unit tests
Deploy
Production perf tests
Iteration 2
Automated unit &
functional tests
Iteration 3
Regression test suite
Incl:
Iteration n
Regression test suite
Incl: End to End
Production Scale Test
Small perf test Small perf tests
Perf smoke
tests
Perf smoke
tests
The Solution:
• Run incremental performance tests automatically with each iteration
• Establish a trend line for performance early
• Developers find code and architecture issues early
• Performance teams focuse on end-to-end infrastructure and user
related testing
• Minimize production escapes and implement production testing
Development/Test team Performance/Ops teamDevOps

15. 15Copyright © 2014 SOASTA, Inc. - All rights reserved.
I JUST MADE A
CODE CHANGE
CloudTest runs
load tests via
Jenkins
Daily (or more frequent) updates
keep teams “performance focused”
Identify
bottlenecks
& inform
teams, just
like unit and
functional
tests in
Jenkins
Jenkins launches
suitable cloud-
based test
environment
Manage with a new metric!

16. 16Copyright © 2014 SOASTA, Inc. - All rights reserved.

17. 17Copyright © 2014 SOASTA, Inc. - All rights reserved.
Mark
Prichard
@mqprichard@mqprichard

18. o #1 OSS CI server
o Written in Java
o Highly Extensible
o About 8 years old
o Easy to install/use
o Extensible via 1000+ plugins
o Widely adopted
o 70K+ installations
o CloudBees knows Jenkins
oOn-premise, cloud and hybrid
oExpert support
oEnterprise plugins
Have you met Jenkins?

19. oLet developers focus on developing
o Leave building, packaging, testing, distributing to a computer
o Make the right thing the easiest thing – don’t think about it
oMany, widely-adopted plugins
o Safety in numbers – hugely active community
o Dashboard view of the state of your builds
o You don’t have to be an expert at everything
oTest Anywhere
o Deploy on-premise or in the cloud for testing
o Spin up VMs or EC2 Instances as build executors
o We can do it for you, if you like…
Continuous Integration and Automated
Testing

20. • CloudBees ClickStarts – A complete project in just two clicks!
• Bees Shop Demo
• Git repo, database, build and deploy in seconds
• Online examples: https://partnerdemo.ci.cloudbees.com
Setting up the Test Envrionment

21. 21Copyright © 2014 SOASTA, Inc. - All rights reserved.
o Demo:
• Bees Shop Overview
• SOASTA Cloud Test: Setting up a load test up for unattended runs
• Jenkins Performance Data:
• Reviewing Performance Trending Data: Track performance of each build
• Reviewing Performance Pass/Fail Data: Failing
• Setting up Continuous Load Testing
• Install Plot Plugin
• Download GitHub: https://github.com/mostenberg/SOASTAPERFPERL
• Add steps to plot build data for any metrics of interest.
• Publish Junit test results to PASS/FAIL based on performance thresholds:
SOASTA/5_PERF_THRESHOLD_RESULTS.xml

22. 22Copyright © 2014 SOASTA, Inc. - All rights reserved.
Mike Ostenberg

23. 23Copyright © 2014 SOASTA, Inc. - All rights reserved.
TouchTest
Continuous
Testing
CloudTest
Load &
Performance
mPulse
Real User
Monitoring

24. • Open Source: online at jenkins-ci.org
• Automatically available on CloudBees DEV@cloud
• Complete control over CloudTest/TouchTest Automation from within your
Jenkins builds:
• Totally integrated into Jenkins environment
SOASTA Jenkins Plug-in

25. • CloudBees ClickStarts – A complete project in just two clicks!
• Example configurations all online:
• https://partnerdemo.ci.cloudbees.com/
• Bees-Shop-PerformanceTest
Try it out yourself!

26. 26Copyright © 2014 SOASTA, Inc. - All rights reserved.
Thanks & Q&A
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtesttwitter.com/cloudtest
facebook.com/cloudtestfacebook.com/cloudtest
Knowledge Center
•White Papers
•Webinar
Recordings
•Case Studies
CloudLink
Community
•Support
•Tutorials
•Video
RESOURCES
Contact CloudBees
www.cloudbees.com
info@cloudbees.com
1.781.404.5100
Follow us:
twitter.com/cloudbeestwitter.com/cloudbees
facebook.com/CloudBeesfacebook.com/CloudBees
www.SOASTA.com www.CloudBees.com
blog.cloudbees.com
•Look for Mark’s post on this
topic
developer.cloudbees.com
•Platform Services
•Frameworks
•ClickStarts