Доклад Евгения Чигиринского, SQA Days-13, 26-27 апреля 2013, www.sqadays.com

1. Методология и практический опыт
тестирования быстродействия приложений,
сервисов и сайтов с высокой нагрузкой с
помощью Visual Studio 2012
Евгений Чигиринский Microsoft Corp.

2. Содержание
• Методологии тестирования
качества и нагрузки
– пробы и ошибки
• От методологии – к практике на
примере msn.com
– Visual Studio Profiler
– Тестирование нагрузки в условиях
автоматического управления датацентрами
– Тестирование быстродействия на
клиентской части

3. Why MSN?
• 19 лет online
• Траффик главной страницы (www.msn.com) – больше чем 4.3
миллиарда просмотров за месяц
• Постоянно в Top 20 сайтов мира
• Очень высокие требования к производительности сайта
• Присутствует во многих странах мира
– Порталы (www.msn.ru)
– Тематические сайты (http://cars.uk.msn.com/)

5. • What does High Quality mean?
• Resilience Quality
• Environment Quality
• Diagnostics & Monitoring Quality
• Configurability
• Maintainability

6. What does High Quality mean?
Quality is not just
working Functionality
Contributors to High Quality

7. Service Resilience Quality
Fault Tolerance
Recover from Fault conditions
Degrade Gracefully
How to Test? – Fault Injection!
“Chaos Monkey”
Handling Human Errors
Configuration Management
File & backup Management
Performance thresholds
If you don’t play well, we don’t play with you!
Throttling Capabilities
Dependency mapping
Avoids potential failures
Reduces Mitigation Time
Better Prediction of Impacts
Lowers Maintenance Risk and Increases Efficiency
“Design for failure and nothing will fail!”
“The best way to avoid failure is to fail constantly”
Software should protect humans from making mistakes

8. Environment Quality
• Test Environments in-sync with Production Environment
• Design For Roll back
• Solid Deployment testing in place
• Test Rollback mechanisms thoroughly
• Design for Human Error
• Quality Gates and Checks in place to prevent Corruption of Environment
• Configuration Changes should be treated like full fledged deployments
• Operational Excellence
• Mandatory Peer Reviews for Configuration changes & Script Executions

9. Diagnostics & Monitoring Quality
• Monitoring
 Being Proactive Vs. Reactive
 Lower MTTD [Mean Time to Detect]
 Lower Downtime & Higher Availability
 Add Monitoring Capability when you are building the service - not after!
 Monitoring Testing should be part of Test plan
 Plan for Multiple levels of Monitoring – Watch dogs, FTIP, etc
• Diagnostics
• Ability to collect required data
• More effective debugging and troubleshooting
• Lower time to restore
• How soon can we get the service back up and running

10. Maintainability
• The ease with which the system can be:
• Modified to meet new requirements
• Modified to make future maintenance easier
• Modified to correct defects
• Adapted to a changed environment
• Maintainability testing:
• Maintainability Index
• Cyclomatic Complexity
• Class Inheritance
• Class Coupling
• Lines of Code

11. Methodology
• Min Install
Component Level Testing
• Cluster Level Testing

12. Methodology : Minimal Installation
• One Box Setup
• Build the baseline (expectations)
• Tuning for scenarios
• See what comes from experiment (run)

13. Min Install - Performance Load and
Stress Testing
• Identify Production Scenarios (BVTs)
– Confirm meeting original expectations
– Define Baselines
• Identify Max Throughput
– Identify Bottlenecks (always start one to many)
– Identify Ceiling (how much work you can do with the system – can be memory, IO, CPU)
– Identify Capacity
– How it behaves beyond the limit
• Failure Scenarios (stress testing)
– Finding failure conditions, pushing the app to failure scenarios
– Examples:
• Too much traffic
• Too much memory/cache items
• Too many connections (resource starvation)
• Network IO/Disk IO
– Failing gracefully
• Endurance Testing
– Issues/Concerns happening over time (usually over 72 hours) – Leaks, Disk space, Timed events

14. Methodology : Cluster Level Testing
• Integration Testing
• Prod simulation (IIS Playback, SQL playback, etc)
• Testing new behavior for new features
• Failing gracefully
• Load Test In Production (LTIP)

15. Performance is important
• How to improve it?
– Measure
– Fix
– Measure again
• How to measure it with VS?
– VS Profiler

16. What is VS Profiler?
• Performance measurement tool
• Process oriented
Ultimate Premium Professional Express

17. Common Performance Issue
• High CPU utilization
• I/O bottleneck
• Tiers interaction
• Resource contention, Poor core utilization
• Memory issues

18. High CPU utilization
• Sampling: statistical form of CPU profiling
• Choose Sampling when
– CPU is the critical resource
– Low overhead is required
• Non-intrusive
• Samples != Time

19. Sampling Profiling
• void Alpha()
• {
•
• Beta();
• }
• void Beta()
• {
• }
30 samples
50 samples
Exclusive Inclusive
Alpha
Beta 50 50
30 80

20. Sampling Profiling Call Tree

21. Q&A

22. I/O bottleneck
• Instrumentation: method timing profiling
• Choose Instrumentation when
– CPU usage is low
– Exact function call counts are necessary
– Exact timing is necessary
• Higher overhead
• Intrusive

23. Instrumentation Profiling
• void MyFunction()
• {
• FUNC_ENTER(MyFunction);
• . . .
• CALL_ENTER(WriteFile);
• // Call another function
• WriteFile(...);
• CALL_EXIT(WriteFile);
• . . .
• FUNC_EXIT(MyFunction);
• }
Elapsed Time
Application
Time
Application
Time

24. Instrumentation Profiling Call Tree

25. Managed memory profiling
• Allocation data
– Allocated type
– Allocating call stack
• Lifetime data
– GC generations

26. More power to VS Profiler
• Data collection
– Remote profiling
– ETW based collection
– Various performance counters
– Command line tools
– APIs
Multiproc collection
Standalone profiler
Collect what you need
Attach/Detach, Pause/Resume

27. VS Profiler - Limitations
• Move to ETW in Visual Studio 2012
• Windows 8 Limitations with CPU sampling
– Tier Interaction Profiling data cannot be collected
– The “Sampling” performance session cannot be configured
– Windows performance counters cannot be collected while CPU sampling
– NGEN-ed methods will not show real method names

28. Тестирование нагрузки в условиях автоматического
управления датацентрами

29. Automatic DC Management
• Like a cloud, but with more control
• Testing the deployment process
• Monitoring

30. Monitoring
• Deployment monitoring
– Automated rollbacks
• Watchdogs
– Simple Watchdogs (disk space, CPU, etc)
– User Watchdogs
• service that monitors the production service
– Set machine properties with centralized service
• Alerts
– Actions based on machine properties set by watchdog

31. Тестирование быстродействия на
клиентской части

32. Что будет рассмотрено
TTI
PLT
TTV
PLT2
TTG
TTLB
TTO
TTFR
TTFB
VRTA
MSN
PIE
Keynote
YSlow
WebPageTest
Gomez
WebRunner
Что это за метрики?
Как измерять
производительность?
Что можно сделать?

33. Метрики для оценки производительности
Запрос
страницы
Сервер посылает
HTML браузеру
Браузер парсирует
HTML, загружает JS и
CSS, строит DOM,
запрашивает
изображения.
Отображение
первого
видимого
элемента
Отображение
видимой
страницы
Загрузка последнего
видимого элемента
страницы
(чаще всего за экраном)
Вся ресурсы
страницы
загружены.
Browser fires
on-load event
Beacons
are fired
Прием
последнего
байта
страницы.
Производительность
«сервера»
TTLB
Time to
last byte
TTFB
Time to
first byte
TTFR
Time to
first render
TTV
Time to
last visual
TTO
Time to
OnLoad
TAFR
Above fold
render
Page Load Time - PLT
perceived approximated

34. Какие метрики производительности
нужно использовать?
TTLB
Time to
last byte
TTFB
Time to
first byte
TTFR
Time to
first render
TTV
Time to
last visual
TTO
Time to
OnLoad
TAFR
Above fold
render
Perceived Approximated

35. Performance Optimizations
• Images loaded optimally in phases
• Async script loading
• JS execution in multiple phases
• Testing on real mobile devices
• Optimized ad loading
• Other optimizations

36. Optimal Image Loading
Define and maintain Performance budget
– PLT goal permits only 400KB weight at 100% bandwidth efficiency (less in
practice)
Responsive design
– Need to load different images based on view mode (e.g. Snap view);
determined client-side
– Load images in phases

37. CPU Analysis Methodology
• Capture event traces (ETW) on the device
• Use XPerf trace analysis tool to visualize CPU cost across IE subsystems
– JavaScript, CSS, Formatting, Layout, Display etc.
• Identify bottlenecks and JS code paths responsible for them

38. Example: Jerky TOC Snapping
Situation
• TOC snapping animation is jerky (12fps on Surface RT)
Opportunity
• Cycle of measuring and updating CSS position of TOC as user
scrolls down results in costly recalculation of entire layout tree with
every loop iteration
• 80ms CPU time for each cycle results in jerky animation

39. Goal: Ads do not block Onload
• Onload has impact on perceived performance, such as the ‘Done’ indicator
• Onload is used to trigger secondary functionality
Example: Optimized Ad Load
Onload Ad request blocks Onload
Solution:
• Modify ads handling code to avoid blocking of Onload
Onload Ad request does not block Onload

40. Key Findings for MSN
• Define and maintain perf budgets
• Test as near to Prod as possible
• Know your dependencies
– Avoid 3rd party libraries that are not designed with performance in mind
• Make sure that your sample data is valid
• Monitoring is the key
• Visual Studio Profiler helps finding issues quickly!

41. References
• Visual Studio ALM + Team Foundation Server Blog
http://blogs.msdn.com/b/visualstudioalm/
• Profiling Windows 8 and Windows Server 2012 applications
http://msdn.microsoft.com/en-us/library/hh974575.aspx
• Configuring the profiler as part of your load testing in Visual Studio
http://msdn.microsoft.com/en-us/library/dd504817.aspx
• Load Testing in Visual Studio 2012
http://blogs.msdn.com/b/visualstudioalm/archive/2012/06/04/getting-started-with-load-testing-in-visual-studio-
2012.aspx
• VS Profiler – CPU Sampling
http://blogs.msdn.com/b/visualstudioalm/archive/2013/02/27/how-to-profile-a-xaml-windows-store-app.aspx
• Profiling .NET Memory Allocation
http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio-
2012.aspx

42. Q&A