- Steve Thair presented on whether the current model of load/performance testing is broken for modern web applications.
- He discussed how Betfair separated load injection from performance measurement due to the complexity of their system.
- The current model of load testing with waterfalls, single reports, and scripted user journeys is insufficient for continuous delivery and real user monitoring needs.
- Thair advocated for cheaper and more continuous methods like session replay from logs and APM tools to align with modern development practices.
4. 4
The Killer Comment…
“We’ve had to look at separating load
injection from performance
measurement”.
- Andrew Harding, Betfair
http://www.seriticonsulting.com/blog/2011
/12/9/is-the-current-model-of-
loadperformance-testing-broken.html
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
5. So why are you paying all 5
that money for that
What are key expensive brand-name
reasons behind load testing tool then?
that decision?
Surely there are
open source or
cheap cloud
tools if all you
want is load?
What other
issues might we
have with
What are you
“traditional” load
using to measure
testing models
Performance
in a Web 2.0
then?
world?
@TheOpsMgr #ukcmg #webperf
6. 6
Defining the “current model”
“Stephen didn’t explicitly define
what the current model of
load/performance testing is…”
Alex Podelko
http://applicationperformanceengineeringhub.com/is-the-current-model-
of-loadperformance-testing-broken/
@TheOpsMgr #ukcmg #webperf
7. 7
“Current Model Straw Man”
(1) Waterfall Development Cycle
(2) Load Tool compiles the report
(3) Reporting at the end of the test
(4) Request / Response paradigm
(5) Well-defined customer journeys
(c) Seriti Consulting, 2011
8. 8
Testing in a CI/Agile world
Performance testing initiated every time code is committed to
the CM repository (e.g. SVN, GIT etc)
Betfair issue was that their system was complex with many
layers of caching etc so it took longer to “warm up” the
environment (to achieve a steady performance state) than
they had between check-ins…
So they needed “continuous injection” to keep the
environment constantly warm…
The test tool never “stopped” to “compile the report”…
So any tool that “reported at the end” wasn’t as useful…
@TheOpsMgr #ukcmg #webperf
9. 9
Test tools in a APM & RUM world
“traditional test tools” generally have the ability to deploy
agents to gather metrics from the target environment
But the depth of analysis and correlation falls well below that
of modern Application Performance Management tools e.g.
AppDynamics
APM tools offer a deeper insight and better event correlation
across tiers…
And they are getting (much) cheaper…
@TheOpsMgr #ukcmg #webperf
12. 12
WebSockets
HTTP 1.1
Start Timer
GET /index.html
Response 200 OK
Stop Timer
WebSockets
Start Timer
Socket “upgrade”
N many frames…
Stop??? Bi-directional socket channel
@TheOpsMgr #ukcmg #webperf
13. 13
HTTP 2.0
HTTP 1.1
Sequential
ordered
HTTP 2.0 (& SPDY)
Multiplexed over a
single connection
Responses
returned out of
sequence…
Hard to time!
http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy
@TheOpsMgr #ukcmg #webperf
15. 15
Visitor Flow
How many paths thru a
website?
Classic script-driven
approaches can’t hope to
address the complexity
Network and log file
replay solutions?
@TheOpsMgr #ukcmg #webperf
16. 16
A PCAP Solution to Replay?
POC solution based on Cloudmeter
Pion + custom scripts
Read a PCAP (network capture)
Identify the HTTP traffic
Filter it (based on your requirements)
Parameterise it (query strings, POST
parameters etc)
Randomise inputs from SQL, CSV etc
Replay it against a test environment
i.e. change the base URL
Amplify & rate throttle req/sec as
required
http://www.cloudmeter.com/pion/data-processing.php
@TheOpsMgr #ukcmg #webperf
17. 17
Personal Opinion!
YMMV…
So what’s the Answer?
Use the cheapest method to generate load that you can
find…
Move away from scripting-based approaches towards using
real-user session replay (if possible!)
Generate load continuously
Measure continuously
using APM & RUM type tools – FOSS or Commercial
Look for changes in histograms, averages, standard dev etc
Protocol and Framework aware instrumentation
AFAIK this currently doesn’t exist…
@TheOpsMgr #ukcmg #webperf
18. 18
@LDNWebPerf User Group!
Join our London Web Performance Meetup
http://www.meetup.com/London-Web-Performance-Group/
Next Wednesday 17th Oct – 7pm – Central London
Follow us on Twitter @LDNWebPerf
#LDNWebPerf & #WebPerf
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
19. 19
About Me
21yrs IT experience.
Started with www in 1998 (IIS3! Site Server 3!).
Web Architect @ BNP Paribas, CSFB etc
Web Operations Manager for www.totaljobs.com, www.tes.co.uk
Professional Services Manager @ www.siteconfidence.com
Seriti Consulting – specialising in web operations, management and
Performance
e:stephen.thair@seriticonsulting.com
m:+44 7971 815 940
Twitter: http://twitter.com/TheOpsMgr
Blog: http://www.seriticonsulting.com/blog/
LinkedIn: http://uk.linkedin.com/in/stephenthair
Skype: seriti-steve
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
Thanks for coming to my talk… I know it must have been hard to tear yourself away from all about “Workload License charges in IBM System Z” so I appreciate your trust! We have lot of really interesting stuff to talk about…You will get a lot of food for thought and I will confess right now that I don’t know all the answers to this as yet, but hopefully we might find some out along the way!
I run a monthly Meetup group on Web Performance… and back in December 2011 we had a presentation from the Performance team at Betfair about performance testing in their continuous integration environment.
And they presented an environment like this…
So I am like what, huh, that’s that? You separated load injection from performance measurement? Doesn’t that sort of destroy half the value proposition of the all those expensive load testing tools? Why did you do that?How are you measuring it then? And then all sorts of other issues came out of that…
So I had all these questions in my head… but before I get to that I need to address Alex Podelko’s objection raised in a comment on my blog…
True, I didn’t…. So say hello to my little friend… the Straw Man!
How do you get “results” when the testing never “stops”…
APM tools offer more insight…. I mean, that’s what they are designed to do so it’s hardly a surprise. So increasing in my load testing I search
Some RUM tools are even free – like Google SiteSpeedSo why do I need expensive load tools if I all I am doing is measuring load?
WebSockets is a new HTML 5 API protocol for bi-directional real-time communication between browser (client) and server.But the key here is that there isn’t a nice request/response “round-trip” any more – the very thing that most of the current generation of test tools rely on (especially the HTTP 1.1 level protocol tools like JmeterAnd in a HTTP 2.0 world it gets even worse… because you have HTTP channel multiplexing…
Websites are getting more and more complex… especially as we add in new functionality like AJAX and HTML5 I have been playing with a solution
I created a proof of concept
DB2 was the reason I moved into local area networking…