The document provides guidance on developing reliable load test scripts and scenarios. It discusses test data requirements, parameterizing dynamic values, proper use of HTTP protocol versus GUI-level scripting, handling asynchronous requests, implementing think time and pacing, validating scripts, and best practices for load scenario configuration. The goal is to explore main points around scripting best practices, validating load scripts thoroughly, and configuration best practices to build effective performance tests.
Introduction to IEEE STANDARDS and its different types.pptx
PAC 2019 virtual Arjan Van Den Berg
1. Productive way of load test development
for any system using any tool
Arjan van den Berg
www.performancetestexpert.nl
2. Generic approach to develop reliable load
script(s) and scenario(s)
www.performancetestexpert.nl
3. Context presentation (1/2)
Presentation is focused on developing reliable HTTP protocol load scripts and scenarios
Advice when to script on Browser / GUI level
It’s related to performance test development & configuration so that my humble knowledge
and mistakes, from the past becomes knowledge for the future
…..a reliable load test script / scenario is crucial within every stage of the software
development cycle……and is often underestimated….
www.performancetestexpert.nl
5. To explore the following main points:
1. “Scripting best practices”, for developing load scripts
Test data needed for scripting,
Web (2.0) UI user interface
Introduction
Using HTTP protocol
Script on GUI-level
Web services scripting
2. How to validate a load scripts/scenario thoroughly.
3. Load script/scenario configuration best practices.
7. Test data (1/3)
Identify Test scenarios and get functional understanding (happy) test flow(s)
Identify test data for all Test scenarios
Unique user ID’s / accounts, sufficient to handle all load profiles !!
Create/Get production like test data
Of course we can also use mock/stubs, so responses are based on a given request
Creating test data for your load test is IMPORTANT
Test data should be “linked” to user ID, so that we know that each user “sees” is own
data….
8. Test data (2/3)
Examples:
#1 User has unique shopping articles , “correlated” to userID
#2 session replication , is session (data) handled correct under(high) load
#1 #2
9. Test data (3/3)
Testdata should be releated to userIds / Account.
So for instance don’t create a script like:
User A: Order a smart watch
User B: Order a smart watch
User C: Order a smart watch
But , like:
User A: Order a smart watch , shoes, <prod x>,
User B: Order a laptop, smartphone, <prod y>,
User C: Order a wireless mouse , <prod z>
10. Test data (3/3)
Releated to session data, an other example .
A user search action:
Usecase:
User A: Search for <birds>
User B: Search for <cars>
User C: Search for <product X>
Result:
User A: Search results for birds
User B: Search results for cars
User C: Search for <product X>
An example of what can happen in real life…
Usecase:
User A: Search for <birds>
User B: Search for <cars>
User C: Serach for <product X>
Result:
User A: Search results for cars
User B: Search results for birds
User C: Serach for <product X>
“Session
Data
issue””
11. Scripting best practices, for developing load scripts
Web (2.0) UI Interface
www.performancetestexpert.nl
12. Web (2.0 ) UI Interface - Introduction
A lot of GUI frameworks today Angular , React, …
Asynchronous website changes are common
Sometimes also a persistent , real time bi-directional web-socket channel is used…
Polling requests
A lot off java script client-side
You may think how to record this on http protocol level, but bottom-line still a get/post
request/response is used to communicate with the server so for technical reasons you have
rarely to use GUI-level scripting ….
13. Scripting best practices, for developing load scripts
Using HTTP protocol
www.performancetestexpert.nl
14. Web (2.0 ) UI Interface – HTTP Protocol
Start with simple user flows first
Make sure the script doesn’t record unneeded URLs / get / post requests
Identify ALL dynamic data (as the response from server) and correlate it !!
Recording scripts 2 times and compare in detail all request or use commercial tools
features
Get first the login session(Ids) / correlate the login sequence
Parameterize also all dynamic data; dates , GUIDs, (epoch) timestamps, …
Dynamic data is fetched with “expressions” from server response;
Some load test tools supporting xpath , jsonpath and so on…my personal flavor is
regular expressions (with grouping) to get the correct response into a parameter,
because this works always
15. Web (2.0 ) UI Interface – HTTP Protocol
Parameterize scripts to support dynamic data set.
Dynamic data, every simulated user get different data; avoids responses from cache …
Use also with a few transactions wrong test data values e.g.
wrong phone number format;
forget to fill an edit field; and so on…,
Test will be more realistic…else your test will be to clean…..
Use proper checks / assertions for all steps involved in the transaction.
Check on errors
Check also on expected results
Use also parameters for assertions, get also from test data table
Absence of assertions might result in misleading results….
16. Web (2.0 ) UI Interface – HTTP Protocol
Make sure that script can be executed against multiple environments
Users rarely log out , so don’t assume always “clean logout” and design scripts like that ..
Headers and cookies are handled automatically(almost) within any tool
Configure the think - and pacing time in the script.
Don’t use a constant think time value.
Distributing think time values in a range .(e.g 5 sec +/- 50%)
Implement think time and pacing time within your script with care, pay attention to
timing!!
Pacing is a must ! Sometimes not implemented at all….
17. Pacing in more detail
Think time and Pacing, both are a kind of “wait” or “delay” in terms of time but still both
differ from each other
Pacing can be the delay between two iterations of test
Responsetimealways 10 sec ?
What happens with TPS when response time is 5 sec ?
Yellow bar == pacing Time
So use pacing to get “steady” load
To compare load test results SUT, release x, y, z….. always use pacing!!
Thinktime + Round trip time <= Pacing !
18. Keep load scripts simple
Avoid custom flow logic within the script, it’s better to split it up into 2 separate
scripts
Don’t try to write custom error handling within the scripts, e.g. retry save
All tests scripts should be independent,
If data between scripts needs to be shared , use a temp file /queue which all
scripts can access
Avoid checks which not occur in real user scenario as much as possible , e.g
query with a script to check the database for “expected content”
19. Web (2.0 ) UI Interface – HTTP Protocol
Take Special attention to repeated / similar looking requests
• If repeated / intermittent requests are observed “like polling/heartbeat pattern”
• delete all except one and put this in a loop;
• use a variable extractor to check the state/response
Solution above can block the test flow, so if needed fork a separate thread
If you observe /expect asynchronously requests in your recorded script
Are maybe recorded sequential; but within a normal browser executed at the same
time
Check async requests, with browser developer tools, e.g In Chrome Network TAB, XHR
At least group such XMLHttpRequest together in your load script , or send them in
parallel
20. Web (2.0 ) UI Interface – HTTP Protocol
Anyhow, even if the requests are send synchronously within your load script , it will still
work (almost everytime), but is related to timings less accurate….
Pay attention to the transactions within your loadscript, should cover the right http
requests…
If a websocket channel is used…
Try to understand the (async) message patterns within this websocket channel
Measuring response times accurate, can be challenging
All requests/responses are send to one, channel, to take special attention to:
Assertions & Extracting parameters needs to be (extreme) generic
To get scripts up & running very often Multiple threads are needed,
Debug / Maintenance websocket based tests can be (very) difficult…
23. Scripting best practices, for developing load scripts
Script on GUI level
www.performancetestexpert.nl
24. Web (2.0 ) UI Interface – Script on GUI level
Scripting on GUI level remarks / some personal experiences
GUI level scripting is for sure not record and playback !
GUI level tests are used for backend loadtesting, frontend end-to-end user experience
testing is a different test !!
GUI Object identification needs also a learning curve/ gain experience with it..
GUI scripts also need maintenance…
Popup windows can be an issue
Assertions are also GUI objects, so needs attention !
Errors needs also to be identified as a GUI object
25. Web (2.0 ) UI Interface – Script on GUI level
Scripting on GUI level vs. HTTP
1. If possible use HTTP recording, and focus on fast script generation
Framework parameters in Neoload
Automatic correlation in other commercial tools
2. If the GUI is using Ajax “on average” still possible to use HTTP protocol
3. If the application is full websocket based, and the majority of the calls async. switch to
browser / GUI level
4. But to get GUI scripts stable takes also (development) time..
5. GUI Script flow is less technical, and more easy to explain to other stake holders
6. GUI script can reduce maintenance effort,
27. Webservices
Webservices script
Pretty straight forward
Often a subset of existing functional tests can be used / converted to a performance
test (work in this area close together with the functional testautomation engineer)
Use WSDL import within the load test tool, if possible, soapActions then available
Test also error paths; wrong data format in request (small percentage e.g. 1%)
Of course use assertions to check expected response
Use also response size check (in bytes) to check expected response
Importance of test data
Size message , large included attachments (Important for parsing, memory usage,
Database request(s),…)
Make sure that all operations / transformations are covered
Test security impact in a separate test
28. How to validate a load scripts/scenario thoroughly.
www.performancetestexpert.nl
29. Validate loadscript and scenario
Verify the scripts during design time
Validate with one iteration with one user
Validate with multiple iterations (#10) with one user
Validate with multiple iterations (#10) with multiple concurrent users (#10)
DONT START TOO QUICKLY WITH A FULL LOADTEST
30. Validate loadscript and scenario
Important best practices during validation script
Execute verification with full log in load test tool
Checking parameter/test data values during execution
Check response against recorded session
Check server logs during replay, or if possible check sessions in APM tools ; check
together with developers that what you see in the log is correct / valid
Check if pacing / think time works as expected
Check application server (e.g jvm) and resource usage SUT
If test with one user is already (very) slow, report a performance problem….
32. loadscenario
Load scenario configuration (1/2)
Group user scenarios / configurate the ratio between scripts
User standard naming conventions for the different load profiles e.g. Baseline, Stress,
Load, Spike test
Keep the settings for the load profiles fixed during the different test cycles, else you
loose the possibility to compare results…, because you have a lot of options there
Rampup time, ramp down, duration;
browser cache settings;
static content on/of;
rendezvous points;
Pacing time, Think time(s);
Option X,Y,Z
33. loadscenario
Load scenario configuration (2/2)
In case of (script) error, stop de Virtual user and start a new one,
To avoid collateral damage
If one step fails, there is a high risk that all other steps fail
Does not appy for webservices…
If possible make notes within the testrun results settings
what the run conditions are;
Even better automate this with a CI/CD pipeline
Store at least SUT software version(s);
And e.g Threadpool settings DB pool, log level SUT ,
34. A Quick Recap
Test data must be in place before developing load scripts and scenarios
Correlate unique test data to each user(ID) /session
Pay strong attention to any detail of your load script and scenario, you have to think of
more things than you initially aware of….
Use HTTP protocol if possible, until it appears that this becomes technically too complex,
then switch to GUI protocol
Automate the correlation , to detect dynamic values in the script , so you can replace
dynamic data automaticallywith parameters
until it appears that this becomes technically too complex
until it appears that this becomes technically too complex
www.performancetestexpert.nl
35. This is what you should have just learned
A Productive way of load test development
Building effective tests !
….And you are now aware of (all) pieces:
(session) testdata,HTTP protocol,GUI level scripting,dynamic
data , parameterization,correlation,web 2.0, (a) sync requests,
websockets,think time,pacing,
assertions,transactions,validating scripts, configure scenario
(best practices), ….