Neotys organized its first Performance Advisory Council in Scotland, the 14th & 15th of November. With 15 Load Testing experts from several countries (UK, France, New-Zeland, Germany, USA, Australia, India…) we explored several theme around Load Testing such as DevOps, Shift Right, AI etc. By discussing around their experience, the methods they used, their data analysis and their interpretation, we created a lot of high-value added content that you can use to discover what will be the future of Load Testing. You want to know more about this event ? https://www.neotys.com/performance-advisory-council

1. Historical Performance Testing Requirement
Tooling Evolution in Response to Change

2. Tooling Evolution

3. HistoricalTooling Requirement..
• Protocol Support
• Many and varied
• Citrix – ICA
• SAPGUI
• Oracle NCA
• ODBC
• Native Database, MS SQL, SIEBEL, Oracle
• Winsock, WAP, HTTP(s)
• Tuxedo…
• Client Support
• Fat Client
• Early Browser
• Testing Client -> Server Model

4. Reasons for Change..
• Business
• Digital Revolution and the WWW
• Explosion of eCommerce
• Cloud
• Mobile
• Technology
• Demise of the client /server model
• Move to distributed and increasingly modular architecture
• SOA
• Services (Web, RESTApi)
• Micro-services
• Containers
• Move to web enablement and cloud deployment

5. Current Tooling Requirement..
• Far Fewer Protocols
• HTTP / HTTPS
• Websockets
• Citrix – ICA
• Some packaged applications (Remedy, SIEBEL)
• Rarely a requirement for anything else
• Legacy tooling vendors continue to deprecate protocol support
• More Client Options
• Browser
• Mobile Device
• Service / API
• Client Choice and Expectation
• SaaS, Self-Service Offerings
• Opensource tooling resurgent – attempt to make it more of an Enterprise solution.
• Licensing cost much more of an issue
• Free Community versions of licensed tooling becoming the norm.

6. Impact on Tooling Functionality

7. Scripting
• Change
• Move from coding to DOM scripting
• Support for multiple scripting languages
• Improved auto-correlation
• Easier to learn
• Reduced scripting effort
• Reduced dependencies – No compiler required
• Real-World
• Must still offer a scripting option
• One well known vendor recently deprecates click and script support
• Script in one tool, convert to another.

8. Load Injection
• Change
• On-premise and Cloud support
• Windows and LINUX O/S support
• Bandwidth simulation
• Real World
• Cloud support is mandatory
• Cloud injector management sometimes clumsy
• Much less requirement to test on-premise
• Much easier to provision injectors

9. Test Management
• Change
• Increased sophistication in test design options
• Options to integrate with test automation servers and frameworks
• Jenkins, HP Quality / Performance Centre. HP ALM
• Collaborative project options
• Real World
• Important to easily control VU weighting and injection profile
• Reality is most tests are simple ramp to target load or break point
• Little uptake of collaborative performance testing
• Management of test assets still a mix of native and 3rd-party source control
• GitHub

10. Analysis
• Change
• Increased sophistication of reporting.
• Real-time graphical analysis the norm.
• Greater selection of auto-generated graphs and tables.
• Real World.
• Test reporting still largely a manual exercise.
• Manual extraction of graphs and tables into MS Word / pdf document.
• MS Excel still a more flexible analysis option in many instances.
• Many tools still don’t support X,Y and Z axis series in same graph.
• Difficult to set / extract Nth percentiles (90th,95th).
• Difficult to compare results between test runs
• Native management of test results not as slick as it could be.

11. Integration
• Change
• Increased native integration with point monitoring solutions.
• Integration with leading Application Performance Management (APM).
• DynaTrace, Appdynamics, New Relic
• Simplification of Application and Infrastructure metric correlation.
• Integration with CI automation and build servers.
• Jenkins, Cucumber
• Real World
• Monitoring Integration often troublesome to leverage effectively –
• Pushback from SecOps
• Relatively little uptake of single tool solution for NF automated CI integration as part of
DevOps.
• Clients prefer to build their own custom integrations as part of CI,TDD,BDD

12. Future Trends

13. FutureTrends?
• Protocols
• IOT support although use case still unclear
• Continued deprecation of legacy protocols
• Scripting
• Maintain flexibility in language choice
• DOM and code continue to co-exist
• Load Injection
• Better management of injector deployment
• Test Management
• Centralise test management across Dev (QA) Ops.
• Ability to leverage non-native test assets
• Analysis
• Improved summary reporting
• Auto comparison of KPI delta across test runs
• Better integration with Big Data (ELK , Grafana)
• Integration
• Single vendor solution unlikely for CI
• Maintain flexible API and leverage community plugin development

14. Thank you - Q & A