This webinar discusses enabling continuous load testing with Jenkins and SOASTA CloudTest. Tom Chavez and Andy Pemberton will present on integrating load testing into continuous delivery pipelines. The webinar will cover why load testing is important, challenges with traditional load testing approaches, how continuous load testing works, and a demo of setting up load tests in Jenkins with the CloudTest plugin. Attendees will learn how to establish a performance baseline, detect issues early, and ensure applications perform under load throughout development and deployment.

1. 1 © 2015 SOASTA Inc.- All rights reserved.
In This Webinar
TODAY’S PRESENTERS
Tom Chavez: SOASTA, Sr. Product MarkeEng Manager
Andy Pemberton: CloudBees, Senior Director, SoluEons Architecture & ConsulEng
TWO COMPANIES, A SINGLE SOLUTION
SOASTA: The leader in Cloud and Mobile TesEng
CloudBees: The Enterprise Jenkins Company
Together, we offer a complete cloud-based service:
ConEnuous Load TesEng & ConEnuous Delivery with Jenkins

2. QuesEons?
Ask in QuesEon window

3. 3 © 2015 SOASTA Inc.- All rights reserved.
Poll QuesEon
What best describes your status with ConEnuous IntegraEon / ConEnuous
Delivery (CI/CD)? (single answer)
a) Determining if CI/CD is a good opEon
a) ImplemenEng CI/CD
a) PracEcing CI
b) PracEcing CD
c) What is CI/CD?

4. 4 © 2015 SOASTA Inc.- All rights reserved.
Agenda
• Load tesEng: why?
• Why don’t we load test earlier?
• ConEnuous Load and Performance TesEng
• CloudBees and the Enterprise Jenkins PlaYorm
• ConnecEng load tests to your CD Pipeline
• Q&A
QuesEons:
Submit in the quesEon box during event

5. 5 © 2015 SOASTA Inc.- All rights reserved.
Why Should We Load Test?
o Ensure that site will perform well at load
• Test hardware configuraEons
• Test applicaEon, database, components
• Test third party partners
o To be ready for peak season
• Black Friday, Cyber Monday, Christmas
• Tax filing, Insurance filing, SuperBowl commercial

6. 6 © 2015 SOASTA Inc.- All rights reserved.
Why Should We Load Test?

7. 7 © 2015 SOASTA, Inc.-All rights reserved.
Why Don’t We Load Test Earlier?
Copyright © 2014 SOASTA, Inc. - All rights reserved.

8. 8 © 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”
o Tools require coding skills we don’t have (C, VB?)
o We don’t have a scalable test environment
o We don’t have hardware for load generation
o It’s a different tool
o There are too many other tests to do

9. 9 © 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”
o Tools require coding skills we don’t have (C, VB?)
o We don’t have a scalable test environment
o We don’t have hardware for load generation
o It’s a different tool
o There are too many other tests to do
“Because it takes too long.”
o Scripting takes forever
o Results are slow
o Fast setup of environments is impossible
o We’re dependent on IT or other teams
o We’re too agile

10. 10 © 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”
o Tools require coding skills we don’t have (C, VB?)
o We don’t have a scalable test environment
o We don’t have hardware for load generation
o It’s a different tool
o There are too many other tests to do
“Because it takes too long.”
o Scripting takes forever
o Results are slow
o Fast setup of environments is impossible
o We’re dependent on IT or other teams
o We’re too agile
“Because no one ever told us to.”
o There are no “performance coverage” requirements
o There isn’t a performance baseline for reference
o There isn’t a defined performance target
o No one is managing to a performance metric

11. 11 © 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”
o Tools require coding skills we don’t have (C, VB?)
o We don’t have a scalable test environment
o We don’t have hardware for load generation
o It’s a different tool
o There are too many other tests to do
“Because it takes too long.”
o Scripting takes forever
o Results are slow
o Fast setup of environments is impossible
o We’re dependent on IT or other teams
o We’re too agile
“Because no one ever told us to.”
o There are no “performance coverage” requirements
o There isn’t a performance baseline for reference
o There isn’t a defined performance target
o No one is managing to a performance metric
Until Now.

12. 12 © 2015 SOASTA Inc.- All rights reserved.
Poll QuesEon
What do you see holding back a more iteraEve load tesEng approach? (check all
that apply)
a) Skills gaps
b) Lack of test environment(s)
c) Slow test development
d) Test tools limitaEons
e) 3rd Party Dependencies
f) Undefined Performance Requirements
g) Lack of Management Push
h) Other

13. 13 © 2015 SOASTA Inc.- All rights reserved.
Load TesEng Has Been an Elite Event
SThe Issue:
• High expectaEons from tests run with diminishing Eme windows
• Code-level performance bugs found late set enEre project back
• Separate team and schedule conflict with delivery goals
• Major issues escape to producEon
Iteration 1
Automated unit
tests
Iteration 2
Automated unit &
functional tests
Iteration 3
Regression test
suite
Iteration n
Regression test
suite
Load &
Performance
Testing
Deploy
Issues?

14. 14 © 2015 SOASTA Inc.- All rights reserved.
ConEnuous Load TesEng
Validate performance piece by piece
Iteration 1
Automated unit
tests
Deploy
Production perf
tests
Iteration 2
Automated unit &
functional tests
Iteration 3
Regression test
suite
Incl:
Iteration n
Regression test
suite
Incl:
End to End
Production Scale
Test
Small perf test Small perf tests
Perf smoke tests Perf smoke tests
The SoluEon:
• Run incremental performance tests automaEcally with each iteraEon
• Establish a trend line for performance early
• Developers find code and architecture issues early
• Performance teams focus on end-to-end infrastructure and user related tesEng
• Minimize producEon escapes and implement producEon tesEng
Development/Test team Performance/Ops teamDevOps

15. 15 © 2015 SOASTA Inc.- All rights reserved.
ConEnuous Performance (for Web & Mobile)
The Path to a Performance Culture
I JUST MADE A
CODE CHANGE
CloudTest runs load
tests via JenkinsDaily (or more frequent) updates keep
teams “performance focused”
Identify
bottlenecks &
inform teams,
just like unit
and functional
tests in
Jenkins
Jenkins launches
suitable cloud-
based test
environment
Manage with a new metric!

16. 16 © 2015 SOASTA Inc.- All rights reserved.
Poll QuesEon
What types of Load TesEng are you doing today? (Single Answer)
a) Not doing any load tesEng
b) Non-Web Protocols (not mobile)
c) Primarily web protocols
d) Primarily mobile web traffic
e) Both web and mobile
f) All web, mobile & non-web protocols

17. 17 © 2015 SOASTA, Inc. All rights reserved.
Enabling ConEnuous Performance
TesEng with Jenkins Pipeline
Andy Pemberton
Twitter: @apemberton
Email: apemberton@cloudbees.com

18. 18 © 2015 SOASTA Inc.- All rights reserved.
Have you met Jenkins?
#1 CI/CD Automation Engine
• Open Source
• Highly Extensible
About 8 years old
Easy to install/use
1000+ plugins
Widely adopted
• 100K+ installations

19. 19 © 2015 SOASTA Inc.- All rights reserved.
Jenkins is the #1 Automation Platform Used for CI
Continuous Integration
Developer
Source
Code
Control
System
Compile
/BuildCode
Scan TestsCode
Commit
Results
Report
BuildCommit Test

20. 20 © 2015 SOASTA Inc.- All rights reserved.
Jenkins is Also Your Key to Continuous Delivery
Developer
Source
Code
Control
System
Compile
/BuildCode
Scan TestsCode
Commit
Results
Report
BuildCommit Test Stage Deploy
Deploy
Prod
Deploy
Stage
Complex Delivery Pipelines
Delivery of App and Config
Continuous Delivery
ProdDev

21. 21 © 2015 SOASTA Inc.- All rights reserved.
Over 1000 Jenkins Plugins for Tool Integration
Integrated Development Environments
(IDEs), editors/compilers/debuggers,
code review and collaboration tools.
Source code control
systems and related asset
management tools.
Automated API-based
testing frameworks.
Release
management and
release automation
software
Automated API-based
testing frameworks, static
code analysis tools.
Load Testing Tools
and Security Scanning
technology
Automated and
manual UAT tools
Release management,
release automation, and
change control software.
Environment and Configuration
Management SoftwareDevelop
Commit
Build &
Integrate
Unit Tests
Release
(to Test)
Automated
Regression
Tests
Release
(to Prod)
User
Acceptance
Tests
Load &
Security
Tests

22. 22 © 2015 SOASTA Inc.- All rights reserved.
Jenkins is the Hub of the CD Ecosystem
On-premise or
in the cloud
Plug-ins for all
your tools
RunDeployStageTestBuildCommit

23. 23 © 2015 SOASTA Inc.- All rights reserved.
Are You Using Today?
23
Chances are someone in your
organization is.

24. 24 © 2015 SOASTA Inc.- All rights reserved.
24

25. 25 © 2015 SOASTA Inc.- All rights reserved.
25
How Do You Manage Jenkins and Testing at
Enterprise Scale?

26. 26 © 2015 SOASTA Inc.- All rights reserved.
About CloudBees
Jenkins founder, Kohsuke Kawaguchi, CloudBees CTO
We optimize Jenkins for the enterprise
• Jenkins Operations Center
• Jenkins OSS Support
• Enterprise-class plugins
• Commit code and support the Jenkins Community
• Jenkins training and consulting
• Jenkins User Conferences (JUC) and newsletters
…we are the Jenkins Enterprise experts!

27. 27 © 2015 SOASTA Inc.- All rights reserved.
CloudBees Jenkins Enterprise
27
Professional support
High availability
Large installations
Security
Optimized utilization
Continuous Delivery

28. 28 © 2015 SOASTA Inc.- All rights reserved.
Additional Jenkins Services from CloudBees
28
Jenkins Certified Training
• Delivered via CloudBees partners
Jenkins Consulting
• Delivered via CloudBees Professional Services Team
Jenkins Newsletter
• Community news and expert tips
Jenkins User Conferences
• Learn from other community members

29. 29 © 2015 SOASTA Inc.- All rights reserved.
How Can CloudBees Jenkins Enterprise Help with Your CD
Strategy?
29
PipelineStage View
Provides deep insight into the execution of your workflow stages so
you can detect problems and performance issues
Manage complex “level-of-detail” testing pipelines
Jenkins Pipeline
Created by CloudBees to provide Jenkins with a way to define
flexible delivery pipelines that allow for a rich set of new
delivery scenarios
Manage and share “testing-as-code”
Dynamic Environment Creation
Spin up new test/deployment environments automatically as part of
your delivery pipeline
Deployment Tracking and Verfiication
Track the history of your deployments – including who kicked
off a deployment and what code changes were included in it
Create pipelines that leverage verification tests to determine
the quality of a new deployment
Jenkins Analytics
Monitor and analyze performance across all your Jenkins instances
Monitor, measure and manage adoption of testing, CI and CD best
practicesRestart builds, tests, and deployments where they
break
Leverage pipeline checkpoints to restart delivery pipelines
without a full (expensive) rebuild
...central source of orchestration to bridge dev, test, and ops alike
And more…

30. 30 © 2015 SOASTA Inc.- All rights reserved.
#Jenkins2
Greatly Enhanced UI/UX for easier startup and beOer security

31. 31 © 2015 SOASTA Inc.- All rights reserved.
#Jenkins2
Jenkins Pipeline-as-Code front and center

32. 32 © 2015 SOASTA Inc.- All rights reserved.
Jenkins Pipeline and Pipeline Stage View Enable Full
Continuous Delivery Pipelines
Continuous Delivery
ProdDev
Developer
Source
Code
Control
System
Code
Commit

33. 33 © 2015 SOASTA Inc.- All rights reserved.
33
CloudBees Jenkins Pipeline Stage View
Restartable Checkpoints
Artifact Downloads

34. 34 © 2015 SOASTA Inc.- All rights reserved.
Monitoring, Visualization and Analytics
Jenkins Pipeline Stage View
Visibility into Pipeline status
Jenkins Analytics
Cross-Team Build Analytics
… among many other features, tailored for enterprise
organizations using Jenkins @ scale

35. 35 © 2015 SOASTA Inc.- All rights reserved.

36. 36 © 2015 SOASTA Inc.- All rights reserved.
ConEnuous Load TesEng
o SOASTA Cloud Test Components / Architecture
o Demo:
• CloudTest UI Overview
• Jenkins Setup
• Downloading and installing CloudTest plug-in
• CreaEng a Performance Job in Jenkins
• CreaEng a Pipeline in Jenkins
• Reviewing Results
What are we going to show

37. 37 © 2015 SOASTA, Inc. All rights reserved.
Architecture:
Jenkins
Jill
SriRaj
DevOps Team 1
CloudTest
Manager
Always up at:
Cloudtestmanager.soasta.com
CloudTestserver
any IP Address/ Domainname
Up ‘as needed’ at
“US Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
“WW Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
CloudTest“Main” server
any IP Address/ Domainname
Up ‘as needed’ at
dev.acme.com
CPU,DISK,
Memory,I/O
Etc.

38. 38 © 2015 SOASTA, Inc. All rights reserved.
Jenkins Enterprise on
CloudBees
Jill
SriRaj
DevOps Team 1
CloudTest
Manager
Always up at:
Cloudtestmanager.soasta.com
dev.acme.com
“US Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
CloudTest“Main” server
any IP Address/ Domainname
Up ‘as needed’ at
Demo Preview:

39. 39 © 2015 SOASTA, Inc. All rights reserved.
Demo

40. QuesEons?
Ask in QuesEon window

41. 41 © 2015 SOASTA Inc.- All rights reserved.
Resources
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
twiOer.com/SoastaInc
facebook.com/cloudtest
Knowledge Center
• White Papers
• Webinar Recordings
• Case Studies
CloudLink Community
• Support
• Tutorials
• Video
Contact CloudBees
www.cloudbees.com
info@cloudbees.com
1.781.404.5100
Follow us:
twiOer.com/cloudbees
facebook.com/CloudBees
www.SOASTA.com
www.CloudBees.com
Try it @ www.cloudbees.com/get-started
Documenta]on
• CloudBees Jenkins Platform
• Workflow in Jenkins
White Papers:
• Making the Leap from Continuous Integration to
Continuous Delivery
• The Business Value of Continuous Delivery

42. How to Introduce Cloud Based Load TesEng
to your Jenkins ConEnuous Delivery Pipeline
April 20, 2016
Tom Chavez, SOASTA @TomChavez
Andy Pemberton, CloudBees @apemberton