Performance Optimization in Large Systems - Cusec 2019

1. Performance Optimization in Large Systems
CUSEC 2019

2. About me
Senior Developer at PBSC
Professional experience: VoIP, cloud hosting, e-commerce, bike sharing
Editor at InfoQ.com, writing about .NET
Twitter: @plmaheu

3. “Large” system?
- Multiple servers
- Multiple services
- Many types of users
- > 100,000 LOC

4. The start of the journey
- “This page is too slow!”
- Some server is at maximum capacity
- Alerts fire by the dozen
- The system crashes, nobody knows why

5. What does “performance” means?
- Generic term, depends on how it is measured.
- Varies from system to system: latency, CPU usage, memory usage, memory
fragmentation, GPU usage, etc
- Performance in larger systems depends much more about how subsystems
interact with each other. (server with databases)

6. The method everybody knows: Guessing
- Tempting to optimize code out of a gut feeling
- Unlikely to get it right more than 5-10% of the time
- The possible causes are endless: code, third party library, web server, CDN,
DNS, bandwitdh, ISP, hardware, etc.

7. Ahmdal’s law
Regardless the magnitude of an improvement, the theoretical speedup of a task is
always limited by the part of the task that cannot benefit from the improvement.
TL;DR: Optimize the right thing.

8. Different issues at different scales.
In one second you can do:
- 1 file upload to a cloud storage provider
- 100 SQL queries
- 10,000 regex matches
- 10,000,000 property get through reflection
- 1,000,000,000 additions

9. Tooling

10. Code profilers
- Variable level of detail, from a few chosen methods to *everything*
- Stacktrace with CPU usage
- External service calls
- Highlights hot spots in the stacktrace
- From light to heavy overhead, depending on detail resolution

11. Heavy artillery: Redgate ANTS (.NET)

12. Lighter, casual: MiniProfiler

13. An optimized page

14. Application Performance Monitoring
- Provides a high level view of performance on a system, usually web site or
service
- Always running, meant for production use
- Slight overhead, usually < 5% CPU
- Variable side features such as error logging

15. Stackify
Retrace

16. Takeaways
- Always measure before and after optimizing.
- Understand the root cause before changing something.
- Establish your goal(s) before an optimizing session.
- Get familiar with the tooling of your environnement.