Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Cheat Codes for Managing Application Performance Kwame Thomison Magnetic Inc.
About Me Meebo: Web based instant messaging Facebook: Social network Asana: Collaboration software
What makes managing app performance so challenging?
What makes managing app performance so challenging? It’s not most engineers’ first choice No true north star for the team R...
Risks of failing to address those challenges Business metrics suffer Burnout Wasted time Endless cycle of improvement and ...
Agenda Telemetry and Tooling Building Alignment North Star
Telemetry and Tooling Metrics Reasoning About Data Building the Right Tools
Your architecture
What your API expert sees
What your product expert sees
What your product expert sees
Your architecture
Telemetry and Tooling: Metrics
Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
Two Popular Perf Metrics Time to Interact (TTI) End to End (E2E)
Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
Telemetry and Tooling: Reasoning About Data
A quick refresher on graphs
Time Series
Histogram
Cumulative Distribution Function
Time Series: Slow and Steady Degradation • Gradual increase over day/ weeks • No corresponding increase in system load • N...
Time Series: The “blip” • Regression that looks like noise • w/w shows a periodic regression • No correlation with app usa...
Histogram: Bimodal distribution • Histogram reveals multiple modes •No pattern when slicing by demographic •Indicates a re...
CDF: Extreme outliers • The CDF levels off sharply at p99 • Increases even more at p99.9 • What’s broken? • Big fish or cel...
Telemetry and Tooling: Building the Right Tools
Building the Right Tools: Tool Type Static Analyzers Profilers Predictive Modelers
Building the Right Tools: Develop Workflow Example • Dev in sandbox • Run test suite • Push to staging • Push to prod
Building the Right Tools: Render Pipeline Example • DB Query • Server Processing • Server Response • Client CPU
3 Tool Types x 4 Steps in Dev Workflow x 4 Steps in Rendering Pipeline x = up to 48 tools So what do you build first?
Prioritizing Tools • Sort by time on the critical path for rendering pipeline • Sort by stages of dev workflow • Sort by to...
Building Alignment A Common Language Demonstrating Impact Performance Culture
Building Alignment: A Common Language • Perf experts speak a different language • Others often uses neologisms • Loss of t...
Building Alignment: A Common Language What do you call this? • JS file • JS bundle • Static asset
Building Alignment: A Common Language Primary concerns for perf • Why has perf regressed? • What impact does a given chang...
How do you get people speaking the same language?
Define a perf grammar!
Example {magnitude} {sequence} {percentile} {stage of work}
-100ms p75 critical path client CPU
Building Alignment: Demonstrating Your Impact • Display measured TTI on Every Page Load • Share Experiments From Other Com...
Conducting Your Own Study: Bucketing Users • Bucket sessions into perf classes (e.g. < 2s, > 2s) • Examine business metric...
Conducting Your Own Study: Artificial Perf Manipulation • Introduce an artificial perf regression/improvement • Measure diff...
Conducting Your Own Study: A Longterm Holdout Group • For a portion of your users, don’t ship any optimizations • Review b...
Building Alignment: Cultivating a Performance Culture
What does a healthy culture of managing app performance look like?
People have a shared definition for and approach to achieving excellence
Cultivating a Performance Culture • Clear ownership • Prioritization • Tooling and Documentation • Levers
You’re out to build allies!
Unclear Priorities or Lack of Education or Lack of Tooling = Adversaries
Shared Priorities and Performance Education and Data, Tools, and APIs = Allies
The North Star
North Star • Architecture Strategy • Team Mission
North Star: Architecture Strategy • What’s your perf cost per incremental feature? • Use that lens to guide technology dec...
North Star: Architecture Strategy • What’s your perf cost per incremental feature? • Use that lens to guide technology dec...
North Star: Team Mission • Empower other engineers • Work on the hardest problems • Minimize perf cost per incremental fea...
Summing it all up • Tips for Telemetry and Tooling • Aligning the Company • Defining a North Star
Let’s make the entire web faster!
Questions? kwame@magnetic-inc.com
Managing application performance by Kwame Thomison
Managing application performance by Kwame Thomison
Nächste SlideShare
Wird geladen in …5
×

Managing application performance by Kwame Thomison

180 Aufrufe

Veröffentlicht am

Managing application performance is a huge challenge for most engineering organizations. The most difficult questions we have to answer often only indirectly related to shortening the critical path. How do you align stakeholders with competing priorities and agendas? How do you drum up interest, persuade your best engineers to focus on performance, and keep the team motivated? How should the the team be positioned within the larger organization? Where do you even start? You can't find good answers to any of those questions with a simple web search.

In this talk, Kwame Thomison shares tips for creating a perf team with a clear, compelling mission, a few frameworks he’s devised for thinking about web performance, and some of the lessons he’s learned working on perf teams at companies like Meebo, Facebook, and Asana.

Veröffentlicht in: Technologie
no profile picture user

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

  • Gehören Sie zu den Ersten, denen das gefällt!

Managing application performance by Kwame Thomison

  1. 1. Cheat Codes for Managing Application Performance Kwame Thomison Magnetic Inc.
  2. 2. About Me Meebo: Web based instant messaging Facebook: Social network Asana: Collaboration software
  3. 3. What makes managing app performance so challenging?
  4. 4. What makes managing app performance so challenging? It’s not most engineers’ first choice No true north star for the team Requires your very best engineers Unique skillset Perf work isn’t prioritized early enough
  5. 5. Risks of failing to address those challenges Business metrics suffer Burnout Wasted time Endless cycle of improvement and regression
  6. 6. Agenda Telemetry and Tooling Building Alignment North Star
  7. 7. Telemetry and Tooling Metrics Reasoning About Data Building the Right Tools
  8. 8. Your architecture
  9. 9. What your API expert sees
  10. 10. What your product expert sees
  11. 11. What your product expert sees
  12. 12. Your architecture
  13. 13. Telemetry and Tooling: Metrics
  14. 14. Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
  15. 15. Two Popular Perf Metrics Time to Interact (TTI) End to End (E2E)
  16. 16. Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
  17. 17. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
  18. 18. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
  19. 19. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/
  20. 20. Choosing the right metrics Approximate User Value Use Percentiles Correlative Metrics
  21. 21. Telemetry and Tooling: Reasoning About Data
  22. 22. A quick refresher on graphs
  23. 23. Time Series
  24. 24. Histogram
  25. 25. Cumulative Distribution Function
  26. 26. Time Series: Slow and Steady Degradation • Gradual increase over day/ weeks • No corresponding increase in system load • No discernible pattern when slicing by demographic • Slow rollout of a new service or feature
  27. 27. Time Series: The “blip” • Regression that looks like noise • w/w shows a periodic regression • No correlation with app usage • Cold/warm start mechanic
  28. 28. Histogram: Bimodal distribution • Histogram reveals multiple modes •No pattern when slicing by demographic •Indicates a relatively expensive code path being executed part of the time
  29. 29. CDF: Extreme outliers • The CDF levels off sharply at p99 • Increases even more at p99.9 • What’s broken? • Big fish or celebrities
  30. 30. Telemetry and Tooling: Building the Right Tools
  31. 31. Building the Right Tools: Tool Type Static Analyzers Profilers Predictive Modelers
  32. 32. Building the Right Tools: Develop Workflow Example • Dev in sandbox • Run test suite • Push to staging • Push to prod
  33. 33. Building the Right Tools: Render Pipeline Example • DB Query • Server Processing • Server Response • Client CPU
  34. 34. 3 Tool Types x 4 Steps in Dev Workflow x 4 Steps in Rendering Pipeline x = up to 48 tools So what do you build first?
  35. 35. Prioritizing Tools • Sort by time on the critical path for rendering pipeline • Sort by stages of dev workflow • Sort by tool type • Throw away moonshots
  36. 36. Building Alignment A Common Language Demonstrating Impact Performance Culture
  37. 37. Building Alignment: A Common Language • Perf experts speak a different language • Others often uses neologisms • Loss of trust over time
  38. 38. Building Alignment: A Common Language What do you call this? • JS file • JS bundle • Static asset
  39. 39. Building Alignment: A Common Language Primary concerns for perf • Why has perf regressed? • What impact does a given change have? • How is perf work prioritized?
  40. 40. How do you get people speaking the same language?
  41. 41. Define a perf grammar!
  42. 42. Example {magnitude} {sequence} {percentile} {stage of work}
  43. 43. -100ms p75 critical path client CPU
  44. 44. Building Alignment: Demonstrating Your Impact • Display measured TTI on Every Page Load • Share Experiments From Other Companies • AB Test Your Optimizations • Conduct Your Own Study
  45. 45. Conducting Your Own Study: Bucketing Users • Bucket sessions into perf classes (e.g. < 2s, > 2s) • Examine business metrics for each perf class • Infer causal relationship between perf and those business metrics • You may be misled into inferring a causal relationship where one doesn’t exist
  46. 46. Conducting Your Own Study: Artificial Perf Manipulation • Introduce an artificial perf regression/improvement • Measure differences between control and experiment groups • Infer causal relationship between perf and business metrics • You’re hurting the UX intentionally • You might infer a linear relationship where one doesn’t exist
  47. 47. Conducting Your Own Study: A Longterm Holdout Group • For a portion of your users, don’t ship any optimizations • Review business metrics after several weeks/months • Infer causal relationship between perf and business metrics • Potentially very complex to implement
  48. 48. Building Alignment: Cultivating a Performance Culture
  49. 49. What does a healthy culture of managing app performance look like?
  50. 50. People have a shared definition for and approach to achieving excellence
  51. 51. Cultivating a Performance Culture • Clear ownership • Prioritization • Tooling and Documentation • Levers
  52. 52. You’re out to build allies!
  53. 53. Unclear Priorities or Lack of Education or Lack of Tooling = Adversaries
  54. 54. Shared Priorities and Performance Education and Data, Tools, and APIs = Allies
  55. 55. The North Star
  56. 56. North Star • Architecture Strategy • Team Mission
  57. 57. North Star: Architecture Strategy • What’s your perf cost per incremental feature? • Use that lens to guide technology decisions • Give engineers control over rendering pipeline • Pay or Parallelize
  58. 58. North Star: Architecture Strategy • What’s your perf cost per incremental feature? • Use that lens to guide technology decisions • Give engineers control over rendering pipeline • Pay or Parallelize
  59. 59. North Star: Team Mission • Empower other engineers • Work on the hardest problems • Minimize perf cost per incremental feature • Focus on the big picture
  60. 60. Summing it all up • Tips for Telemetry and Tooling • Aligning the Company • Defining a North Star
  61. 61. Let’s make the entire web faster!
  62. 62. Questions? kwame@magnetic-inc.com

×