In this presentation we will look at what web performance testing is and the various types of testing that can be performed. We will then dig into Visual Studio 2013 Ultimate to see that the Visual Studio platform is now a real contender in performance testing automation. And we will see how the Visual Studio integration with Visual Studio Online and Azure can take your web performance tests and spin up impressive load tests in a truly useful way.

1. Performance testing with
Visual Studio & Azure
How to test your application using the tools
you already know!
Andrew Siemer | Clear Measure
andrew@clear-measure.com
@asiemer

2. Andrew Siemer
http://about.me/andrewsiemer
ASP Insider
MS v-TSP (Azure)
Azure Advisor Program
Father of 6. Jack of all trades, master of some.

4.  We are hiring!!!

5. Introduction

6. Agenda
• What is performance testing?
• How can Visual Studio help?
• How can Azure make load testing easy?

7. What is
performance
testing?

8. What is it?
• A type of testing intended to determine
• Responsiveness
• Throughput
• Reliability
• Scalability
…under a given workload

9. Why would I care?
• Are you ready to go to production?
• Can you handle the expected load?
• Do you have edge case bugs?
• How much traffic causes your app to tip over?
• Do you need to optimize something?
• Have new features caused the app to support less traffic?
• Do you have enough hardware to support known load?
• What causes your application to fail spectacularly?

10. Three types of performance tests
• Performance testing
• Load testing
• Stress testing

11. Performance testing
• Speed
• Scalability
• Stability
• Achieves expected performance in general use case
• Can be run often
• Cheap to run with low volume

12. Load testing
• Expected performance remains steady under production load
• Assumes everything is normal
• Does the network handle this load
• Can the database handle it
• Is the application still meeting SLA and usable

13. Stress testing
• Pushing the application beyond expected limits
• Identifies ceilings in capacity
• Tests low memory, disk space limitations, or dead server
• Helps see how and when an application will fail
• What happens when the network gets clogged

14. Many other types of tests
• Capacity (system capacity meets business volume)
• Component (component meeting expectations)
• Endurance (is performance consistent over time)
• Investigation (performance trending over time)
• Smoke (build ready for perf testing)
• Spike (temporarily exceeds expected load)
• Unit (segment of code reasonably efficient)
• Validation (faster or slower)

15. Baselines
• A baseline tells you where the app is now
• Allows you to see change over time
• Target to demonstrates improvements
• Baselines can be created for
• System
• Component
• Application

16. Benchmarking
• The comparison of a current test with a baseline
• Looking for changes in the results
• Or a comparison against industry standards
• Are my commerce pages as fast as the industry expects

17. When to shift from testing to tuning?
• Once testing has found unacceptable results
• When results are acceptable, but servers are running extra hot
• When an SLA is breached that we know we need to fix
• Especially when it impacts a large test surface

18. Test planning

19. Before we test
• Define acceptance criteria
• Write down key scenarios
• Create workload model
• Target load levels
• Determine metrics to capture
• Design tests

20. Acceptance criteria
• Write down your objectives
• Response time – page load speed
• Throughput - number of concurrent users
• Resource utilization – processor, memory, disk I/O, network I/O
• Maximum user load
• Business metrics – number of orders to handle

21. User scenarios
• “Happy paths” through your application
• Touches many components in your system
• Commonly used paths through the application
• Highest business value paths through the application

22. User scenarios - examples
• Log in
• Browse catalog
• Add to cart
• Check out
• Register
• Search
• Faceted navigation

23. User scenarios – specific example
• For a given scenario – determine activities in the test
• The scenario “add to cart” may include these activities
• View home page
• View dirt bike category page
• Choose grips
• Add a grip to the cart
• Go to the cart
• Validate that the item is in the cart

24. User scenarios – most common/intensive
• Be sure to include highest use scenarios
• And include most resource intensive scenarios
• Look at your existing applications log files for top use case

25. Define the workload
• Know and attempt to simulate existing traffic patterns
• Understand user delay, or “think” time
• A load test should not be a denial of service attack!
• Plan scenarios around average session times
• Not too short or too long
• Not every scenario can be a new user, returning user, or either
• Plan around the reality of your application

26. Target load levels
• Load levels are applied to “workload”
• Understand business volume as it relates to your objectives
• Common load, vs. a big marketing push, or black Friday
• Key scenarios
• Distribution of work
• Average session times

27. Define what metrics to capture - business
• Too many metrics can make the results hard to read
• Ask questions related to performance that have specific answers
• How many orders are place per minute
• What is the response time of the cart page
• Identify what metrics to capture based on your questions
• Looks for lower level metrics that help answer your questions
• Reevaluate this often – applications change – so should the metrics

28. Define what metrics to capture - application
• Network – hardware, switches, routers, gateways, load balancers
• System – disk, processor, memory, network
• Platform – the host of your app: IIS, worker role, web role, VM
• Application – perf counters, instrumentation, file locks, db locks,
queue

29. Design tests
• Using your thought so far design specific tests to meet your needs
• Don’t change the tests because it is hard to write it as designed
• Test expected data as well as unexpected data (form validation, bad
credit card, etc.)
• Be sure to include think time
• Best tests data is collected from production data (where applicable)
• Think about spiders, batch processes, while real users are browsing
• Don’t over simplify your tests!

30. How can Visual
Studio help?

31. Visual Studio 2013 Ultimate
• VS feature matrix: http://goo.gl/3VtY01
• Ultimate gets you web load and performance testing
• Test Pro doesn’t get you this functionality!

33. Web performance
tests
The beginnings of our load testing!

34. The test project

35. Recording a test

36. Recording a test – uh oh!
• The first time you try to record,
IE likely won’t work
• Basically, you need to
• Disable Enhanced Protection Mode
• Enable Web Test Recorder
• Go here for full steps to fix this:
http://goo.gl/LcFNSJ

37. Recording a test – uh oh!

38. Recording a test

39. Recording a test

40. Recording a test

41. Verbose recording…likely too much!

42. Let’s clean up the recording

43. Pruned tests

44. Recorded tests cleaned up

45. Run the tests locally

46. Run the tests locally

47. Modifying test run settings

48. Making dynamic
tests
Way more useful!

49. Once you start seeing repetition

50. Consider investigating patterns in the tests
• Look for patterns in the requests
• Create a data source with variable data
• Sku’s
• Makes
• Models
• Year
• Dynamically create a wide set of tests

51. Adding a data source
• Once tests need to cover more…
• Or you need to create representative load…
• Types of data source
• Database – “select * from tablename”
• CSV
• XML

52. Create a shippable datasource

53. Add the data source to your project

54. Point at the file then magic

55. Choose the table to include

56. And then we have a data source

57. Update recording to use data source

58. Update recording to use data source

59. If the test only runs once…

60. View the dynamic results

61. Conditional & loop steps
• Context parameter exists (test
context)
• Cookie exists
• Cookie value comparisons
• Last request outcome
• Pass or fail
• Last response code
• http response codes
• Number comparison
• String comparison
• Probability rules
• Return true some times
• Loops

62. Plug ins
• Web test plug ins
• Web test request plugins
• Should be in a separate assembly
• Allows you to run code pre or post a test/request execution
• Microsoft.VisualStudio.TestTools.WebTesting
• WebTestPlugin
• WebTestRequestPlugin

63. Load tests

64. Web tests to the max!!!
• Use the web test scenarios you create
• But put the workload, load, metrics, browser mix, and other points
around them

65. Load test wizard
• Running the load test wizard makes life easy!

68. When matching transaction percentages

69. When matching percentage of users per test

70. When matching pace of user per test

71. When matching pace of user per test

72. Choose to tests to include

73. Set distribution for each test

74. Define network mix

75. Define browser mix

76. Capture computer metrics

77. Configure the run frequency

78. Ready to run!

82. Errors?

83. But wait
there’s more!

84. How can Azure
let you go big?

85. Your laptop can’t tip over production!
• Local tests are great to capture big pain points
• Or to capture simple content missing errors
• Or to run as a smoke test

86. Test farm in Azure is magic!
• With Visual Studio Online
• And Azure
• We can go big!

87. Open local test settings

88. Choose to run in VSO

89. Connect to your VSO account

90. Choose your team project

91. And run the test!

96. Ok, that rocks, HOW MUCH?
• Based on “Virtual Users Minutes” – VUM
• Azure calculator for load testing:
• http://goo.gl/XBrkY1
• $.0004/VUM for 20,001-2M VUM
• $.0002/VUM for 2,000,001-10M VUM
• $.0001/VUM for usage
above 10M VUM/MO

97. Questions?
Andrew Siemer - Clear Measure
andrew@clear-measure.com
(512) 387-1976
@asiemer