Combining Automated Functional And Load Testing
•Als PPTX, PDF herunterladen•
5 gefällt mir•4,576 views
This document discusses integrating Ranorex automated functional testing with NeoLoad load testing. It begins with introducing the companies and tools, then discusses how performance impacts user experience. Automated functional testing benefits like speed and coverage are reviewed. While functional and load testing are typically separate, combining them allows validating critical workflows under real load and finding issues earlier. A workflow is proposed where automated tests and exploratory testing are augmented with load testing. A live demonstration then follows.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (16)
Ähnlich wie Combining Automated Functional And Load Testing
Ähnlich wie Combining Automated Functional And Load Testing (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Combining Automated Functional And Load Testing
- 1. Integrating Ranorex and NeoLoad Time to make peace between functional and non-functional testing
- 2. Welcome to the Ranorex-Neotys Webinar Christian Breitwieser, Ranorex Innovation Manager Henrik Rexed, Neotys Technical Evangelist
- 3. Agenda • Quick Company and Tool Intro • Performance Matters in User Experience • Benefits of Automated Functional Testing • Benefits of Combining Automated Functional and Load Testing • Live Demonstration
- 4. • Functional test automation for desktop, web and mobile applications • Cross-technology end-to-end testing • Tool for testers and developers
- 5. • Functional test automation for desktop, web and mobile apps • Cross-technology end-to-end testing
- 6. Tool for testers and developers Code Module Action Based Modules Ranorex Studio
- 7. • Deliver faster applications faster through continuous performance validation - Who We Are
- 8. - Who We Are Mission: Deliver faster applications faster through continuous performance validation
- 9. - Who We Are Mission: Deliver faster applications faster through continuous performance validation
- 10. User Demands Process Revolutions We’re Driven by Your Challenges Technology Transitions
- 11. User Demands Process Revolutions We’re Driven by Your Challenges Technology Transitions
- 12. User Demands Process Revolutions We’re Driven by Your Challenges Technology Transitions
- 13. User Demands Process Revolutions We’re Driven by Your Challenges Technology Transitions Docker
- 14. Load and Performance Testing 24x7 Performance Monitoring Continuous Performance Validation Solution
- 15. Load and Performance Testing 24x7 Performance Monitoring Continuous Performance Validation Solution
- 16. Continuous Performance Validation Solution Development Continuous Integration & Component Testing
- 17. Continuous Performance Validation Solution Development Continuous Integration & Component Testing
- 18. Continuous Performance Validation Solution Pre-production System-wide Testing Development Continuous Integration & Component Testing
- 19. Continuous Performance Validation Solution Pre-production System-wide Testing Development Continuous Integration & Component Testing
- 20. Continuous Performance Validation Solution Pre-production System-wide Testing Development Continuous Integration & Component Testing
- 21. Continuous Performance Validation Solution Pre-production System-wide Testing Production Proactive Monitoring Development Continuous Integration & Component Testing
- 22. Continuous Performance Validation Solution Pre-production System-wide Testing Production Proactive Monitoring Development Continuous Integration & Component Testing
- 23. Continuous Performance Validation Solution Pre-production System-wide Testing Production Proactive Monitoring Development Continuous Integration & Component Testing
- 24. Agenda • Quick Company and Tool Intro • Performance Matters in User Experience • Benefits of Automated Functional Testing • Benefits of Combining Automated Functional and Load Testing • Live Demonstration
- 25. User Experience is Crucial • 57% of users use their mobile devices to access applications • 40% of users move towards a competitor after a bad user experience • 44-61% of users share their feelings on social networks
- 27. User Experience Impacts the Business
- 28. Case Study Impact of the response time on the conversion rate:
- 30. Agenda • Quick Company and Tool Intro • Performance Matters in User Experience • Benefits of Automated Functional Testing • Benefits of Combining Automated Functional and Load Testing • Live Demonstration
- 32. Otherwise…
- 34. Manual & Integration Testing Manual testing • Time-consuming • Error-prone • Cannot „run“ overnight Integration testing • No black box • Hard to cross technology borders • Good coding skill required
- 35. Benefits of Automated Functional Testing • Quick feedback • Parallel test execution • Scalable • Codeless functional testing • Increased test coverage
- 36. Numbers & Facts TomTom – Case Study: • Testing time: from 10 days down to 4h • Over 90% of time saved with Ranorex automated tests Siemens – Case Study: • Due to the use of C#, new test cases can be implemented more effectively • Only 60% of the time needed with Ranorex compared to previously used tool
- 37. Agenda • Quick Company and Tool Intro • Performance Matters in User Experience • Benefits of Automated Functional Testing • Benefits of Combining Automated Functional and Load Testing • Live Demonstration
- 38. Separated Functional & Performance Testing • Functional testing ensures functionality … but not when the AUT is under load • Load/Performance testing ensures a fluent user experience … but does not guarantee a functional UI
- 39. Ranorex Customer Experience Survey No Idea Yes No 54% 37% 9% Do you combine Ranorex test automation with non-functional tests? In addition to functional test automation you perform… 4% 19% 22% 27% 31% 33% Other Load Testing Database Testing Web-Service-Testing Performance Testing Unit Testing
- 40. Functional – Performance Tests Golden Rule: Automated performance testing should not take place if functional regression tests failed. Performance tests require that the application or part of the system is: • Operational: No systematic errors or exceptions • Representative : Enough data available in the test environment to determine the behavior under load conditions
- 41. How Can We Increase Risk Coverage? 100 test cases 40 test cases covering 80% of the risk not necessarily selected for performance testing We prioritize test cases based on risk. These test cases are validated through : - Automated regression testing - Exploratory testing
- 42. Validate Critical Business Case Under Load This guarantees: • The stability of the crucial use cases under real usage conditions • High user experience • The identification of potential bottle-necks, even across technology borders
- 43. The Initial Workflow… Automated functional testing Exploratory testing Load testing Exploratory testing under load …
- 44. Applying the Ranorex-NeoLoad Integration… 00:00:00 00:00:30 00:01:00 00:01:30 Load NeoLoad Application Server App Error Rate Virtual Users Load Automated functional tests User Experience
- 45. … You‘ll Find Bugs Earlier! …
- 46. Agenda • Quick Company and Tool Intro • Performance Matters in User Experience • Benefits of Automated Functional Testing • Benefits of Combining Automated Functional and Load Testing • Live Demonstration
- 47. Further Resources Combining Automated Functional and Load Testing http://www.ranorex.com/blog/combining-functional-and-load-testing How to Combine Ranorex and NeoLoad Tests http://www.ranorex.com/blog/combine-ranorex-and-neoload-tests
- 48. Questions? Henrik Rexed: henrik.rexed@neotys.com Download NeoLoad Free Edition Visit our neotys.com Follow me on Twitter @hrexed Christian Breitwieser: cbreitwieser@ranorex.com Download free Ranorex trial Visit www.ranorex.com Follow us on Twitter @Ranorex
Hinweis der Redaktion
- Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation {click} Offices around the globe {click} Over 1600 customers Nearly every industry Big name organizations trust Neotys [Transition] Choose Neotys because of focus on key performance challenges they face… {click}
- Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation {click} Offices around the globe {click} Over 1600 customers Nearly every industry Big name organizations trust Neotys [Transition] Choose Neotys because of focus on key performance challenges they face… {click}
- Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation {click} Offices around the globe {click} Over 1600 customers Nearly every industry Big name organizations trust Neotys [Transition] Choose Neotys because of focus on key performance challenges they face… {click}
- 3 categories of challenges: User Demands Application Development Process Revolutions Technology Transitions {click} [User Demands] High expectations for performance As fast as google search Mobile as fast as desktop The results of bad performance: Lost revenue Lose users to competitor apps Brand damage Lost productivity for internal apps FASTER APPLICATIONS {click} [Process Revolutions] Agile DevOps Continuous Integration and Continuous Delivery Force testing to become more automated QA being squeezed out QA needs to keep up with development DELIVER FASTER {click} [Technology Transitions] More and more interfaces: Web Mobile APIs IoT New technologies: WebSocket Docker HTML5 HTTP/2 Need to test whatever technologies developers send your way [Transition] And this is why we developed our Continuous Performance Validation solution… {click}
- 3 categories of challenges: User Demands Application Development Process Revolutions Technology Transitions {click} [User Demands] High expectations for performance As fast as google search Mobile as fast as desktop The results of bad performance: Lost revenue Lose users to competitor apps Brand damage Lost productivity for internal apps FASTER APPLICATIONS {click} [Process Revolutions] Agile DevOps Continuous Integration and Continuous Delivery Force testing to become more automated QA being squeezed out QA needs to keep up with development DELIVER FASTER {click} [Technology Transitions] More and more interfaces: Web Mobile APIs IoT New technologies: WebSocket Docker HTML5 HTTP/2 Need to test whatever technologies developers send your way [Transition] And this is why we developed our Continuous Performance Validation solution… {click}
- 3 categories of challenges: User Demands Application Development Process Revolutions Technology Transitions {click} [User Demands] High expectations for performance As fast as google search Mobile as fast as desktop The results of bad performance: Lost revenue Lose users to competitor apps Brand damage Lost productivity for internal apps FASTER APPLICATIONS {click} [Process Revolutions] Agile DevOps Continuous Integration and Continuous Delivery Force testing to become more automated QA being squeezed out QA needs to keep up with development DELIVER FASTER {click} [Technology Transitions] More and more interfaces: Web Mobile APIs IoT New technologies: WebSocket Docker HTML5 HTTP/2 Need to test whatever technologies developers send your way [Transition] And this is why we developed our Continuous Performance Validation solution… {click}
- 3 categories of challenges: User Demands Application Development Process Revolutions Technology Transitions {click} [User Demands] High expectations for performance As fast as google search Mobile as fast as desktop The results of bad performance: Lost revenue Lose users to competitor apps Brand damage Lost productivity for internal apps FASTER APPLICATIONS {click} [Process Revolutions] Agile DevOps Continuous Integration and Continuous Delivery Force testing to become more automated QA being squeezed out QA needs to keep up with development DELIVER FASTER {click} [Technology Transitions] More and more interfaces: Web Mobile APIs IoT New technologies: WebSocket Docker HTML5 HTTP/2 Need to test whatever technologies developers send your way [Transition] And this is why we developed our Continuous Performance Validation solution… {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- 2main products: NeoLoad for Load and Performance Testing NeoSense for 24x7 Performance Monitoring {click} Tightly integrated to deliver a Continuous Performance Validation Solution {click} [Development] Validate the performance of individual components API or web services level {click} NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo Automated performance tests on every build against SLAs Regression Testing for performance [Pre-production] Test designs from development can be passed on to pre-production Build more realistic business use cases In pre-production, system-wide load testing Complex user paths Mobile device and web browser simulation Network Virtualization High volume cloud load generation {click} Integrate with functional testing tools to capture end user experience under load APM tools for deep dive diagnostics {click} Results passed back to Dev for fixes {click} [Production] Same user used by NeoSense in production to proactively monitor SLA = email and SMS notifications go out immediately {click} Send notifications to 3rd party dashboards and IT search tools Integrate with APM tools for RUM and Code-level diagnostics {click} dashboards and reports can be shared with Dev and pre-prod to fix issues quickly {click}
- Automated functional testing: Exploratory testing: Identify issues not covered by automation Load testing: Validate the stability/availability of the application under load Exploratory testing under load: Validate crucial business cases and user experience under load
- Automated functional testing: Automated functional testing under load: Validate the stability/availability of the application under load Exploratory testing: Identify issues not covered by test automation Exploratory testing under load: Validate crucial business case and the user experience under load