Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

ConFoo: Moving web performance testing to the left

429 Aufrufe

Veröffentlicht am

Learn the why what and how of performance testing and moving it left

Veröffentlicht in: Software
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

ConFoo: Moving web performance testing to the left

  1. 1. © 2014 SOASTA. All rights reserved. 1
  2. 2. © 2014 SOASTA. All rights reserved. 2 o What is the impact of Web Performance? o Your Web Site and the need for Performance Testing o Typical Performance Testing o Issues found in Performance Testing o Making Performance Testing Continuous
  3. 3. What do users care about? Survey Says:
  4. 4. Source: http://www.soasta.com/blog/web-performance-mothers-day/ Fast: they buy more. Slow: they buy less.
  5. 5. Performance affects every business KPI • Revenue • Conversions • User satisfaction • User retention • Time on site • Page views • Bounce rate • Organic search traffic • Brand perception
  6. 6. Impact of a 1-second delay A site that earns $100K per day could lose $2.5M in sales in one year
  7. 7. …website slowdowns occur 10X more often than outages. Source: TRAC Research
  8. 8. Free download: soasta.io/timeismoneybook
  9. 9. 12 Anatomy of Your Web App Responsive web site Web Servers • Desktop Users • Mobile Users • Same or similar content o Adapting to different screen sizes Mobile optimized • “m-dot” site for mobile • Custom built for mobile devices All have html, images, JavaScript, css, plus third-party content
  10. 10. 13 Native App • Android – Java • iOS – Objective C/Swift • Mobile app framework o Same source code Hybrid App • Mix of native code + html All have backend calls for login/authentication, account settings, database calls, transactions, plus third-party SDK calls Mobile Apps and Back End Services WebSite/Mobile Back end
  11. 11. 14 The Need for Load Testing • Everyone runs load testing on their website, right? o Load and Performance testing is often not done or too late o Black Friday, Cyber Monday, Holiday failures • What’s the need for mobile web load testing? o Mobile is growing faster than desktop o Responsive sites and mobile sites deliver different content to mobile devices over mobile network • What about mobile app load testing? o Mobile app back ends serve different content to mobile apps o Remember Pokemon Go? It wasn’t the app that failed.
  12. 12. 15 When is your Peak Season? • Peak is year-round o Super Bowl Sunday – 110 million viewers o Bi-weekly or monthly payroll processing o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day o Quarterly processing and filing o U.S. Tax filing o Spring, Summer, Fall & Winter season sales o July 4th, Christmas in July o Back to School o Olympics o Halloween o Election Day o Singles Day – world’s biggest shopping day, $9B in 12 hours o Black Friday, Cyber Monday and Holidays Back to top to repeat again next year!
  13. 13. © 2014 SOASTA. All rights reserved. 16 s Memory leaks Scale of Test Stage/Team Dev & Test Release & Deploy Network & Ops CDN file placement Load Balancer configuration Network bandwidth Network configuration DNS routing Inadequate server resources Default configuration settings Unbalanced web serversAuto-scaling failures Latency between systems Slow third-party plug-ins Garbage collection Database thread counts Inefficient database queriesSlow pages Conflict with other apps Test Lab Staging Production (100% +++) Search technology limits Method-level tuning Max sockets exceeded Firewall max capacity Global latency variance Security bottlenecks Tuning for full-scale…and well beyond
  14. 14. © 2014 SOASTA. All rights reserved. 17 Testing In General Linear approaches include delays between testing Ut Unit Testing Ft Functional Testing Development Testing Operations Pt Performance Testing Mo Monitoring Legacy Approach: Silos and waiting TIME
  15. 15. © 2014 SOASTA. All rights reserved. 18 Eliminating Delays Compresses Time Enabling more testing to be done earlier and more often Ut Unit Testing Ft Functional Testing Testing Pt Performance Testing Mo Monitoring Legacy Approach: Silos and waiting Agile Approach: Automation and Continuous Testing OperationsDevelopment Dev Ops Testing TIME
  16. 16. © 2014 SOASTA. All rights reserved. 19 Load Testing with Specific Objectives Tests at different stages deliver higher coverage Component scalability Performance baseline Integration testing End to End w/ Service Virtualization Unit scalability Database testing 3rd party services Live E2E tests Stress testing Soak test (reliability) Failover Capacity Latency Performance Regression (smoke tests) Network Testing (staging) Operations (prod)Development (sandbox)
  17. 17. © 2014 SOASTA. All rights reserved. 20 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments o Developer – Tester Collaboration
  18. 18. 21 Shifting Left • Agile means shorter development cycles o Used to be months, now weeks o Quality sign-off within sprint • Developers and QA working together o Developers write unit tests; some teams swap roles • Performance testing is moving to left o Meet performance SLA in all new code • Automation with CI tools including Jenkins o Code that meets quality goals gets promoted automatically Testing is moving to the left, including unit, functional, and performance
  19. 19. 22 Shifting Right • Performance testing in development o Development integrating performance tests into SDLC o Strong adoption of open source test tools, such as JMeter • Performance engineers using commercial tools o Execute performance tests at scale, from multiple geos o Testing not just software but infrastructure • Testing tools addressing both sets of users o Adding JMeter support, Continuous Integration automation Moving performance tests from development to production
  20. 20. © 2014 SOASTA. All rights reserved. 23 • Scalability issues • Capacity issues • Concurrency issues • Reliability issues
  21. 21. © 2014 SOASTA. All rights reserved. 24 • What does a problem look like? o Longer response times is a clue o ”High” CPU / Memory / Storage / Network utilization • Solutions? o Ops Problem? Can often be “solved” with hardware o Tall stacks -> Wirth’s Law: “Software is getting slower more rapidly than hardware gets faster” • Testing in development o Test at loads greater than 1 user o Monitor resource usage across builds and compare to baseline / acceptable
  22. 22. © 2014 SOASTA. All rights reserved. 25 • What does a problem look like? o Response times vary related to load on system o Increasing CPU / Memory / Network utilization o Increasing I/O latency • Solutions? o Cannot be “solved” with hardware o Re-architect solution for high-capacity • Testing in development o Test at loads greater than 1 user, compare as load increases o Monitor resource usage across builds and compare to baseline / acceptable
  23. 23. © 2014 SOASTA. All rights reserved. 26 • What does a problem look like? o Infrequent functional issues that seem only to occur under load o Process crashes and restarts o Not easily reproducible • Solutions? o Cannot be “solved” with hardware o Re-architect solution for high-concurrency • Testing in development o Test at loads greater than 1 user o Monitor functional results while testing at increasing loads o Monitor processes for crashes and restarts
  24. 24. © 2014 SOASTA. All rights reserved. 27 • What does a problem look like? o Process crashes and restarts o Memory or object leaks o More frequent garbage collection o Decaying response times • Solutions? o Cannot be “solved” with hardware o Find resource leaks based on failure points • Testing in development o Test at loads greater than 1 user for increasing amounts of time o Monitor processes for crashes and restarts o Compare behavior and resource usage as time increases
  25. 25. © 2014 SOASTA. All rights reserved. 28 • Additional measurements and aggregation o Response Time Average / 90th (92/95/98th) Percentile o Minimum / Maximum Response Times o Completion / Error Rate % o Transactions per second o Size of Response o Network Throughput o Measurement against previous run or baseline o Use more than one of the above!
  26. 26. © 2014 SOASTA. All rights reserved. 29 • The Path to a Performance Culture Run load tests via JenkinsDaily (or more frequent) updates keep teams “performance focused” Identify bottlenecks & inform teams, just like unit and functional tests in Jenkins Jenkins launches suitable cloud- based test environment Manage with a new metric!
  27. 27. 30 Test Continuously at Different Load Test from Development to Integration to Staging to Production
  28. 28. 31 JMeter • Hosted by Apache Software Foundation o http://jmeter.apache.org • Designed to load test functional behavior and measure performance o Web – HTTP, HTTPS; SOAP / REST; FTP; Database, more • GUI design • Plug-ins • Tests as xml Popular Open Source performance testing tool
  29. 29. © 2014 SOASTA. All rights reserved. 32 Have you met Jenkins? #1 CI/CD Automation Engine • Open Source • Highly Extensible About 8 years old Easy to install/use 1000+ plugins Widely adopted • 100K+ installations
  30. 30. © 2014 SOASTA. All rights reserved. 33 Over 1000 Jenkins Plugins for Tool Integration Integrated Development Environments (IDEs), editors/compilers/debuggers, code review and collaboration tools. Source code control systems and related asset management tools. Automated API-based testing frameworks. Release management and release automation software Automated API-based testing frameworks, static code analysis tools. Load Testing Tools and Security Scanning technology Automated and manual UAT tools Release management, release automation, and change control software. Environment and Configuration Management SoftwareDevelop Commit Build & Integrate Unit Tests Release (to Test) Automated Regression Tests Release (to Prod) User Acceptance Tests Load & Security Tests
  31. 31. © 2014 SOASTA. All rights reserved. 34 Jenkins is the Hub of the CD Ecosystem On-premise or in the cloud Plug-ins for all your tools RunDeployStageTestBuildCommit
  32. 32. © 2014 SOASTA. All rights reserved. 35 SOASTA Jenkins Plug-in o Open Source: online at jenkins-ci.org o Automatically available on CloudBees DEV@cloud o Complete control over test automation from within your Jenkins builds: o Totally integrated into Jenkins environment o Similar plug-ins available for HP LoadRunner, BlazeMeter, JMeter, Load.io, etc.
  33. 33. 36 Automate Your Testing and Promotion Use Continuous Integration tools to automate testing
  34. 34. 37 Results of Shifting Performance Testing Left Using Continuous Integration tools to automate testing • Run incremental performance tests automatically with each iteration • Establish a trend line for performance early • Developers find code and architecture issues early • Performance teams focus on end-to-end infrastructure and user related testing • Minimize production escapes and implement production testing
  35. 35. 38 Questions, Resources & Next Steps • Holiday Readiness o soasta.com/holiday-readiness • The Performance Beacon o soasta.com/blog o soasta.com/blog/2015-popular-posts-web-performance • Webinars o soasta.com/webinars o Load Testing for Peak; RUM-Based Testing • CloudTest Lite o soasta.com/cloudtest: Free community version for 100 VUs Resources
  36. 36. © 2014 SOASTA. All rights reserved. 39
  37. 37. © 2014 SOASTA. All rights reserved. 40 How fast should your site be?
  38. 38. © 2014 SOASTA. All rights reserved. 41 Speed costs money…
  39. 39. © 2014 SOASTA. All rights reserved. 42 …and how much load should it handle?
  40. 40. © 2014 SOASTA. All rights reserved. 43 Your customers will be your guide…
  41. 41. © 2014 SOASTA. All rights reserved. 44 RUM How fast is my site? What locations? From where? Which pages? Which flows? Test Full scale load From any location Going through real user flows With real-time analytics Real-Time Analytics with Real User Measurement
  42. 42. © 2014 SOASTA. All rights reserved. 45 • Tying Business Metrics to (ROI) with website performance metrics and real user data
  43. 43. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 Building performance tests that model user behavior on your site First you need to determine the workload for the test. How are companies doing that today? • Guessing. • Logs. • 3rd Party tools. What if I have a new system?
  44. 44. © 2014 SOASTA. All rights reserved. 47 o What information I need? ● When is your peak second/minute/hour/day? ● What is your peak volume? (active sessions, unique users, page visits, orders, ...) ● What is the median session length? ● What is the distribution of user paths? ● What is the distribution of devices? ● What is the distribution of locations? o Good Test Data ● Search Terms ● Product Distribution ● Browse Distribution ● Cart Size ● Order Size How to get a realistic load test Factors to consider
  45. 45. © 2014 SOASTA. All rights reserved. 48 Use real user data to model real user tests
  46. 46. ROI of Performance
  47. 47. Focus on the ROI of Performance
  48. 48. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 “What If” : Why we test Predict the future with mPulse!
  49. 49. © 2014 SOASTA. All rights reserved. 52 Test how you want, when you want, where you want
  50. 50. © 2014 SOASTA. All rights reserved. 53 • Real Users (RUM) provide deep data about your visitors o No other way to know everything about your users on your site, from desktop and mobile • Virtual Users tell you how your site or app will do at scale o You can’t get a million friends to visit your site or app at the same time! JMeter, Load.io, LoadRunner, SOASTA CloudTest let you build and execute load tests at scales from hundreds to millions of virtual users
  51. 51. © 2014 SOASTA. All rights reserved. 54 • Session paths o How are your users going through your site • Mobile App Performance o How are your mobile apps performing? o What are users doing in your mobile apps? • Revenue o How much money are you making from each user and where?
  52. 52. © 2014 SOASTA. All rights reserved. 55
  53. 53. © 2014 SOASTA. All rights reserved. 56
  54. 54. © 2014 SOASTA. All rights reserved. 57 What was the busiest Day, Hour, Minute, Second?
  55. 55. © 2014 SOASTA. All rights reserved. 58 What were most popular page flows with think times? Which APIs?
  56. 56. © 2014 SOASTA. All rights reserved. 59 • Testing to match realistic situations o Users are not all from one location on a fast network o Packets get delayed, lost, re-ordered • Apps perform differently as network conditions worsen o Reduced throughput o Timeouts occur, connections are lost o Transactions take longer, longer sessions Adding real network modeling to load and performance testing
  57. 57. © 2014 SOASTA. All rights reserved. 60
  58. 58. Walmart found… Converted shoppers were served pages that were 2X faster than pages served to non-converted shoppers. Non-buyers were served category pages that were 2-3 seconds slower than category pages served to buyers. For every 100 milliseconds of improvement, incremental revenue increased by up to 1%.
  59. 59. http://www.soasta.com/blog/mobile-web-performance-monitoring-conversion-rate/
  60. 60. Conversion rate barely decreases when load times for “checkout” pages degrade
  61. 61. Conversion rate shrinks by about 50% when load time for “browse” pages increases from 1 to 6 seconds
  62. 62. What is the Conversion Impact Score? The Conversion Impact Score (CIS) is a relative score that ranks page groups by their propensity to negatively impact conversions due to high load times. For each page group, the Conversion Impact Score is calculated using the proportion of overall requests that are associated with that group, along with the Spearman Ranked Correlation between its load times and number of conversions. The Conversion Impact Score will always be a number between -1 and 1, though scores much greater than zero should be very rare. The more negative the score, the more detrimental to conversions that high load times for that page group are, relative to the other page groups.
  63. 63. How much impact does the performance of this page have on conversions? 1. Product 2. Category 3. Shopping cart 4. Home 5. Search results
  64. 64. http://www.soasta.com/blog/website-monitoring-conversion-impact-score/
  65. 65. 69 Test in Production Some issues will only be found at scale load on production systems Don’t forget your third party providers: CDNs, external scripts
  66. 66. 70 Measure Mobile App Performance Under Load All mobile apps perform well when there is no load
  67. 67. 71 Now That the Test is Over • Analyze results for future areas to address o Weak or stress areas that aren’t likely this time but … • Identify third parties that were affected by load o Coordinate with suppliers to address issues o Align schedules or at least be aware of their changes Even if the test passed, you’re not done
  68. 68. 72 Issues Will Happen • “I am 100% confident that everything we tested will work just fine.” – Nordstrom Performance Leader o When loads are different than what you modeled, may have problems o E.g. At Nordstrom, during a recent anniversary sale, mobile sign-ins were 9x greater than anticipated o Used RUM to get to the root cause of the issue Realize that Issues Will Happen Where You Can’t Predict Them
  69. 69. Q & A Resources Next Steps
  70. 70. 74 CloudTest – Load Testing for mobile web, apps • Load testing from development to production o Continuous testing at any scale, on-prem or in the cloud • Real-time analytics with customizable dashboards o Safely test in production with full visibility • Run tests live or automatically o Full hands-on control or hands-off automation • Full end-to-end view o Front-end performance to back-end systems Fast, scalable, affordable load testing for performance optimization
  71. 71. 75 Questions, Resources & Next Steps • Holiday Readiness o soasta.com/holiday-readiness • The Performance Beacon o soasta.com/blog o soasta.com/blog/2015-popular-posts-web-performance • Webinars o soasta.com/webinars o Load Testing for Peak; RUM-Based Testing • CloudTest Lite o soasta.com/cloudtest: Free community version for 100 VUs Resources

×