Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
KPIs for APIs 
or, 
why API Calls are the new Web Hits 
and you may be measuring all wrong 
John 
Musser 
/ 
@johnmusser 
...
Why do we need 
a Web site? 
1995 
Of course we have 
a Web site 
Of course we 
have an API 
Why do we need 
an API? 
2000...
Why do we need 
a Web site? 
1995 
Of course we have 
a Web site 
You are here 
Of course we 
have an API 
Why do we need ...
Remember hit counters?
Evolution of web site metrics 
Hits 
Page 
Views 
Visitors 
Conversions
Evolution of API metrics? 
Calls 
? 
? 
?
Great APIs get measured
But measure 
what?
Well, 
it depends…
Why 
do you have an API?
Marketing channel BizDev / LeadGen 
New line of business 
Extend product 
Drive innovation 
User acquisition 
Upsell oppor...
Who 
is your API for?
API consumer continuum 
You 
Your partners 
& customers 
Everyone 
else 
Internal Private Open
The answers to 
who & why 
should drive your KPIs
The answers to 
who & why 
should drive your KPIs
Many ways to think about API KPIs… 
Traffic 
Total calls 
Top methods 
Call chains 
Quota faults 
Developers 
Total develo...
More ways to think about API KPIs… 
Customer 
Satisfaction 
NPS 
Churn 
Community 
Forum activity 
Social media 
(both for...
There are a lot of 
API KPIs 
you could measure
Don’t drown in API KPIs
KPI SECRET #1 
Great APIs 
prioritize 
what to measure
Choose your “KPI lens” 
CFO lens 
CMO lens 
CEO lens 
Ops lens 
QA lens 
CRO lens 
PM lens 
PR lens
CFO lens: financial metrics 
Total direct and indirect revenue 
Margin 
ROI 
Marginal cost 
Capex / Opex investment 
CLTV ...
DevOps lens: operational metrics 
Performance 
Availability 
Error rates 
Faults by API type 
Defect backlog 
API traffic ...
CMO lens: marketing metrics 
Developer acquisition cost 
Developer demographics 
Portal visitor count & traffic sources 
S...
KPI SECRET #2 
Know your 
Developer Funnel
KPI SECRET #2 
Know your 
Developer Funnel 
(for public APIs)
Web sites have funnels 
Acquisi9on 
Ac9va9on 
Reten9on 
Revenue 
Referral 
How 
do 
users 
find 
you? 
Does 
the 
user 
ha...
APIs have funnels too 
Acquisi9on 
Ac9va9on 
Reten9on 
Revenue 
Referral 
How 
do 
developers 
find 
you? 
Does 
the 
deve...
Potential KPIs 
Acquisi9on 
Ac9va9on 
Reten9on 
Revenue 
Referral 
Dev 
portal 
site: 
sources, 
traffic 
vol, 
uniques 
#...
KPI SECRET #3 
Measure your 
API ecosystem
App Users Apps Developers 
Your APIs You 
API 
1 
API 
2 
API 
3
App Users Apps Developers 
Your APIs You 
API 
1 
API 
2 
API 
3 
Total app users 
Revenue per user 
User growth rate 
Use...
App Users Apps Developers 
Your APIs You 
API 
1 
API 
2 
API 
3 
Total apps 
Trending apps 
Revenue per app 
Channel: mob...
App Users Apps Developers 
Your APIs You 
API 
1 
API 
2 
API 
3 
Total developers 
Active developers 
Revenue per develop...
App Users Apps Developers 
Your APIs You 
Total API calls 
Top API calls 
API 
1 
Top API packages 
Internal usage 
API 
2...
App Users Apps Developers 
Your APIs You 
API 
1 
API 
2 
API 
3 
Total revenue 
Partnerships 
Market share 
Innovation
KPI SECRET #4 
Most APIs 
obey the 
Power Law
The Power Law of APIs 
Developers 
API call volume
The Power Law of APIs 
Most of your API 
traffic, revenue, 
success, etc…. 
Developers 
API call volume
Think quality, not just quantity 
Use your API KPIs, to 
find who, how, where, 
when. 
Developers 
API call volume
KPI SECRET #5 
API KPIs 
need buy-in
Don’t let API KPIs live in a vacuum 
Who are your stakeholders? 
Who’s your sponsor? 
Get executive buy-in on your top 3-5...
KPI SECRET #6 
Measuring 
internal API usage often just 
as important as external
Guardian: 70% internal calls 
API usage from 
internal apps 
External 
usage
Evernote: 99% internal calls 
API usage from 
internal apps 
External 
usage
Netflix: 99.9% internal calls 
API usage from 
devices 
External 
usage
Netflix: 99.9% internal calls 
Why 
Enable Netflix experience 
Who 
Netflix device and UI teams
KPI WARNING #1 
Beware of 
API vanity metrics
API Billionaire Club 
13 billion API calls / day (May 2011) 
5 billion API calls / day (April 2010) 
5 billion API calls /...
5 Billion 
API calls/day
“What if, all things being equal, 
instead of 30 billion API calls a month, 
we served 5 billion?” 
Daniel Jacobson, Netfl...
KPI WARNING #2 
Beware of 
API KPI traps
3 common API KPI traps 
Not having a metrics process 
(have a metrics plan & a feedback loop) 
Using the wrong metrics 
(d...
One important metric 
you may have missed...
What’s your TTFHW? 
Time To First “Hello World” 
aka: how long from zero to 60?
What’s your TTFHW? 
Developer experience (DX) 
can make or break your API.
What’s your TTFHW? 
Too subjective to measure?
What’s your TTFHW? 
Don’t forget about 
qualitative metrics
What’s your TTFHW? 
User testing. 
Surveys. 
Forums. 
Social media…
Real world 
API KPIs
KPI: Conversions 
“We are seeing a trend where a person 
who uses another app with Evernote is 
50% more likely to move on...
KPI: Engagement 
“Apps that have an API integration with 
Evernote are seeing much more usage: 
Pocket readers normally re...
KPI: Revenue 
API now accounts 
for 20-25% 
of the charity’s 
annual revenue
KPI: Photo Edits
KPI: Apps 
3,500 Apps 
Aviary, 
March, 
2013
Goal: “more eyeballs on our content” 
API calls (requests) 
Response volume (ex: # stories) 
Impressions (via image beacon...
KPI: Recipe delivery 
“We sell Vodka. 
But people buy drinks.” 
Goal: “ensure top quality drinks 
consistently delivered t...
KPI: Recipe delivery 
APIs let you be where 
your customers are. 
Measure how. 
“We sell Vodka. 
But people buy drinks.” 
...
Evolution of API metrics? 
Calls 
? 
? 
?
Who & why drives KPIs 
Have a KPI lens & process 
Prioritize & repeat
Thank You 
john@apiscience.com 
@johnmusser
Photo Credits 
Race 
car: 
hCp://www.flickr.com/photos/lim_lik_wei/3270522646/ 
Stopwatch: 
hCp://www.flickr.com/photos/pu...
KPIs for APIs (and how API Calls are the new Web Hits, and you may be measuring all wrong)
Nächste SlideShare
Wird geladen in …5
×

KPIs for APIs (and how API Calls are the new Web Hits, and you may be measuring all wrong)

61.241 Aufrufe

Veröffentlicht am

How do you measure API success? What KPIs do APIs need? What mistakes should I avoid? Find out what you should, and shouldn't, be measuring as part of your API program in this Business of APIs Conference NYC talk. Dive into a breadth of API metrics, the 6 keys to better API metrics, and the traps to beware of (the important do's and don'ts). Also real-world API case studies show who measures what.

Veröffentlicht in: Internet

KPIs for APIs (and how API Calls are the new Web Hits, and you may be measuring all wrong)

  1. 1. KPIs for APIs or, why API Calls are the new Web Hits and you may be measuring all wrong John Musser / @johnmusser / API Science / apiscience.com Business of APIs Conference, NYC, 2014
  2. 2. Why do we need a Web site? 1995 Of course we have a Web site Of course we have an API Why do we need an API? 2000 2005 2010
  3. 3. Why do we need a Web site? 1995 Of course we have a Web site You are here Of course we have an API Why do we need an API? 2000 2005 2010
  4. 4. Remember hit counters?
  5. 5. Evolution of web site metrics Hits Page Views Visitors Conversions
  6. 6. Evolution of API metrics? Calls ? ? ?
  7. 7. Great APIs get measured
  8. 8. But measure what?
  9. 9. Well, it depends…
  10. 10. Why do you have an API?
  11. 11. Marketing channel BizDev / LeadGen New line of business Extend product Drive innovation User acquisition Upsell opportunity Device and mobile support Content acquisition API as Product Distribution channel Increase stickiness Accelerate internal projects Partner opportunities Drive traffic Increase footprint
  12. 12. Who is your API for?
  13. 13. API consumer continuum You Your partners & customers Everyone else Internal Private Open
  14. 14. The answers to who & why should drive your KPIs
  15. 15. The answers to who & why should drive your KPIs
  16. 16. Many ways to think about API KPIs… Traffic Total calls Top methods Call chains Quota faults Developers Total developers Active developers Top developers Trending apps Retention Service Performance Availability Error rates Code defects Marketing Dev registrations Dev portal funnel Traffic sources Event metrics Support Support tickets Response times Business Direct revenue Indirect revenue Market share Costs
  17. 17. More ways to think about API KPIs… Customer Satisfaction NPS Churn Community Forum activity Social media (both for API and apps built on it) Quality Defects Error prone APIs Repair rate API Maturity Stability Maturity metric Change rate Innovation Number of products Number of apps Speed to market Channel Call volume: Mobile, web, others Revenue by channel
  18. 18. There are a lot of API KPIs you could measure
  19. 19. Don’t drown in API KPIs
  20. 20. KPI SECRET #1 Great APIs prioritize what to measure
  21. 21. Choose your “KPI lens” CFO lens CMO lens CEO lens Ops lens QA lens CRO lens PM lens PR lens
  22. 22. CFO lens: financial metrics Total direct and indirect revenue Margin ROI Marginal cost Capex / Opex investment CLTV per developer
  23. 23. DevOps lens: operational metrics Performance Availability Error rates Faults by API type Defect backlog API traffic trends
  24. 24. CMO lens: marketing metrics Developer acquisition cost Developer demographics Portal visitor count & traffic sources SEO / SEM metrics Social media metrics Evangelism and event metrics
  25. 25. KPI SECRET #2 Know your Developer Funnel
  26. 26. KPI SECRET #2 Know your Developer Funnel (for public APIs)
  27. 27. Web sites have funnels Acquisi9on Ac9va9on Reten9on Revenue Referral How do users find you? Does the user have a good first experience? Do users return? How do you make money? Do users tell others? Dave McClure’s “Startup Metrics for Pirates”
  28. 28. APIs have funnels too Acquisi9on Ac9va9on Reten9on Revenue Referral How do developers find you? Does the developer have a good first experience? Do developers return? How do you and developers make money? Do developers tell others?
  29. 29. Potential KPIs Acquisi9on Ac9va9on Reten9on Revenue Referral Dev portal site: sources, traffic vol, uniques # dev registraCons, API keys issued # acCve devs, API call volume by dev, # apps $ per dev, $ per app, # end users # dev referrals, dev/app growth, NPS
  30. 30. KPI SECRET #3 Measure your API ecosystem
  31. 31. App Users Apps Developers Your APIs You API 1 API 2 API 3
  32. 32. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total app users Revenue per user User growth rate User churn
  33. 33. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total apps Trending apps Revenue per app Channel: mobile/web
  34. 34. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total developers Active developers Revenue per developer Top developers Stuck developers
  35. 35. App Users Apps Developers Your APIs You Total API calls Top API calls API 1 Top API packages Internal usage API 2 API 3
  36. 36. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total revenue Partnerships Market share Innovation
  37. 37. KPI SECRET #4 Most APIs obey the Power Law
  38. 38. The Power Law of APIs Developers API call volume
  39. 39. The Power Law of APIs Most of your API traffic, revenue, success, etc…. Developers API call volume
  40. 40. Think quality, not just quantity Use your API KPIs, to find who, how, where, when. Developers API call volume
  41. 41. KPI SECRET #5 API KPIs need buy-in
  42. 42. Don’t let API KPIs live in a vacuum Who are your stakeholders? Who’s your sponsor? Get executive buy-in on your top 3-5 KPIs
  43. 43. KPI SECRET #6 Measuring internal API usage often just as important as external
  44. 44. Guardian: 70% internal calls API usage from internal apps External usage
  45. 45. Evernote: 99% internal calls API usage from internal apps External usage
  46. 46. Netflix: 99.9% internal calls API usage from devices External usage
  47. 47. Netflix: 99.9% internal calls Why Enable Netflix experience Who Netflix device and UI teams
  48. 48. KPI WARNING #1 Beware of API vanity metrics
  49. 49. API Billionaire Club 13 billion API calls / day (May 2011) 5 billion API calls / day (April 2010) 5 billion API calls / day (October 2009) 1.1 billion API calls / day (April 2011) 1 billion API calls / day (May 2012) 1 billion API calls / day (Q1 2012) 1 billion API calls / day (January 2012)
  50. 50. 5 Billion API calls/day
  51. 51. “What if, all things being equal, instead of 30 billion API calls a month, we served 5 billion?” Daniel Jacobson, Netflix
  52. 52. KPI WARNING #2 Beware of API KPI traps
  53. 53. 3 common API KPI traps Not having a metrics process (have a metrics plan & a feedback loop) Using the wrong metrics (don’t be afraid to change what you track) Unrealistic expectations (“Absolutely, we’ll have X developers on-board by X date”)
  54. 54. One important metric you may have missed...
  55. 55. What’s your TTFHW? Time To First “Hello World” aka: how long from zero to 60?
  56. 56. What’s your TTFHW? Developer experience (DX) can make or break your API.
  57. 57. What’s your TTFHW? Too subjective to measure?
  58. 58. What’s your TTFHW? Don’t forget about qualitative metrics
  59. 59. What’s your TTFHW? User testing. Surveys. Forums. Social media…
  60. 60. Real world API KPIs
  61. 61. KPI: Conversions “We are seeing a trend where a person who uses another app with Evernote is 50% more likely to move on to our premium (paid) service”
  62. 62. KPI: Engagement “Apps that have an API integration with Evernote are seeing much more usage: Pocket readers normally read 15% of what they save, but Evernote users read 80% of what they save in Pocket.”
  63. 63. KPI: Revenue API now accounts for 20-25% of the charity’s annual revenue
  64. 64. KPI: Photo Edits
  65. 65. KPI: Apps 3,500 Apps Aviary, March, 2013
  66. 66. Goal: “more eyeballs on our content” API calls (requests) Response volume (ex: # stories) Impressions (via image beacon) Loyalty More KPIs KPI: Impressions
  67. 67. KPI: Recipe delivery “We sell Vodka. But people buy drinks.” Goal: “ensure top quality drinks consistently delivered to people regardless of where people encounter Absolut”
  68. 68. KPI: Recipe delivery APIs let you be where your customers are. Measure how. “We sell Vodka. But people buy drinks.” Goal: “ensure top quality drinks consistently delivered to people regardless of where people encounter where people encounter Absolut”
  69. 69. Evolution of API metrics? Calls ? ? ?
  70. 70. Who & why drives KPIs Have a KPI lens & process Prioritize & repeat
  71. 71. Thank You john@apiscience.com @johnmusser
  72. 72. Photo Credits Race car: hCp://www.flickr.com/photos/lim_lik_wei/3270522646/ Stopwatch: hCp://www.flickr.com/photos/purplemaMish/3020016417/ Stop sign: hCps://www.flickr.com/photos/9toperez/5791194358 Power law: hCp://en.wikipedia.org/wiki/Power_law

×