27. G. Scandens C. Pallidus G. Magnirostris G. Conirostris C. Heliobates G. Fortis G. Fulginosa C. Psittacula C. Pauper C. Olivacea C. Parvulus Certhidea Camarhynchus Geospiza
28. “Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.” Charles Darwin On Origin of Species
29. So while it may look slow and gradual in hindsight
33. Navigation Navigation While it’s nice to think that evolutionary change happens in a planned way, this is not the case – it only looks smooth in hindsight. Not even the voyage of Darwin’s HMS Beagle was smoothly planned. Looking back he seems to have been predestined to arrive at the Galapagos islands, but this is not the case. Planning is hard enough and at best “no plan survives contact with the enemy” (Patton) – so navigation becomes a crucial skill. Knowing where you mean to go is one thing. Knowing where you are and how to move forward is quite another. Realizing that you may in fact be making progress towards a different and better goal, rather than failing to meet your original goal, can be the key to success. Columbus may have set out to find India but as a result Spain fared extremely well through the discovery of the Americas. Dead reckoning – the art of knowing where you are based observing your speed and direction over time. Very difficult and subject to cascading error. “coasting” like the ancient Greeks Word of mouth and story-based navigation “books” - portolani Blue-water sailing – takes instruments, maps, goals. Necessary when there are no landmarks in sight. Shaping of the hull, multiple masts, the rudder – enabling the ship to sail closer to the wind and cross the oceans. (AKA Technology). But this was not enough – a way to manage this was needed in order to achieve the goal: compass and sea-chart navigation. Pole Star – Stella Maris Astrolabe Columbus – set out to make landfall at 24 degrees north and succeeded, although in America and not China Started with a map like p18 and p20, but we’ll give you one like p35 or p52 Sea charts transformed the world view APIs Are a bit like sailing off to find new lands and opportunity. But while the first to travel may have had to suffer through early tools, poor maps, and hostile natives, you can learn from their experience, borrow their maps and the tools they had to invent to succeed. Military advantage came from percentage-point improvements in navigational accuracy – and time became a military technology. The more precisely you could measure time itself – and keep the measuring devices in sync across your fleet – the more effectively you could compete and win.
101. App Developer Device App Big Box Retailer Mobile App National Chain Web Retail Department Store Web Catalog Specialty Store Indirect Sales Direct Sales
103. Hardt’s Theorem: The Internet Power Law The first wave of the Internet demonstrated the economic impact of web-based business models. 99:1 95:5 The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software. The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits. 80:20 Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom. The next wave of the Internet is demonstrating the economic impact of API-based business models.
104. But you need to tackle it in a way that fits your business
180. Controller Model View iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer
181. Stable Flexible Agile iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer
183. Instance Architecture Based on Specialization List Creation Dependency Service ELASTIC INSTANCE LAYER API METADATA CACHING LAYER Meta Data Dependency Service METADATA SERVICE
184. Instance Architecture Based on Specialization Handles Request/Response List Creation Dependency Service ELASTIC INSTANCE LAYER Caches Dependency Data API METADATA CACHING LAYER Meta Data Dependency Service Populates and Manages Cache METADATA SERVICE
185. Map out your usage patterns and cache your data accordingly
199. Extend your API by extending data types Addition is not version-worthy Better to be incomplete than inaccurate Withhold implementation if you are unsure
200. Don’t forget to support your API With technology, people and management
203. Service Level Agreements (SLA) Set reasonable SLAs Will vary based on API Communicate them to customers Set up monitors Monitor average latencies at various percentiles Monitor error rates and types Trigger alerts when SLAs get broken Establish escalation and response policies Who responds? What is the escalation plan?
204.
205. Use metrics to guide your decisions They provide better navigation than anything else
206. Metrics / Business Intelligence Request-Based Metrics (default) What endpoints were called? What parameters were passed in? Response-Based Metrics What was delivered? If applicable, what was presented? Was the response valid? Request Trace What underlying systems were called? How did they respond? System Health Are the systems meeting uptime SLAs? Are they performing with reasonable latencies?
207. Metrics (continued) They can affect the future design of the API They can inform strategic business decisions They can reveal rogue usage They can reveal errors in the system
209. DataFrom Silos to Social Silos Better than a random collection of flat files! At least it’s all in one place! Shared (in the enterprise) History: Data warehousing and insights (extra-enterprise) Now: OData shared with communities according to business model Pubsub - Events & Async – better for the provider, better for the app Relevance/Collaborative Filtering/Community Modeling From aggregating data or mixing data to produce a contextual recommendation or view Domain-specific database monopolies Social Bootstrapping social context within your own data via the Facebook Graph Surround your customer by understanding their context and delight them Understand Domain-specific database monopolies – location, messaging, more Look for experts in other domains that you need Is the verb “enrich” instead of deliver? From content to experiences
210. Domain-specific Data APIs Private Cloud DBs Caching DBs Data API Data Warehousing Mainframe RDBMS Flat file Shared Silos Data
215. optimize data delivery enable larger downloads for fewer queries more liberal retention policies fewer refreshes, less breakage push incremental updates limited access to richer queries
216.
217.
218. Highest rated titles from the 80's and their awards: http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 and AverageRating gt 4&$expand=Awards
219. [{ "id": null, "name": null, "type": "/dining/restaurant", "/business/business_location/address": [{ "street_address": [], "citytown": { "id": "/en/toronto" } }], "cuisine": [{ "/dining/cuisine/region_of_origin": [{ "!/film/film/featured_film_locations": [{ "id": "/en/the_italian_job" }] }] Find Toronto restaurants with cuisine from a filming location of "The Italian Job"
225. Open your database via an API and let others access + download RPC-like access to a database is not a great idea SQL-like access makes more sense – which indicates OData (but beware being tied to a Microsoft infrastructure)
232. Private, partner vs. public API audience continuum Start with partner - minimize work partner needs to do - security IT->Strategic Partners->Platform Modular vs. one-off, scenario APIs What API team can do vs. what others can do Diversity of implementation can have costs Mainframe to modern internet evolution Internet vs. API-enabled Internet (HTML 5 + AJAX + API) - Efficiency in development (well defined model) - Best UI experience - Why is this better than an IFRAME - IFRAME on top of API - Optimize once scenario and UI platform considerations are well understood Platform angle can create big opps for businesses, but you have to commit. Big resource commitment.
http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.htmlThere are two models for the flow of evolution, each with its adherents. Phyletic gradualism says that things change slowly over time, day by day, until one day you realize there has been enough change that something new has arisen.The opposing model is Punctuated Equilibrium. This model says that species tend to remain in a homeostatic mode of stable equilibrium – but that every so often some significant new stress affects the environment, prompting rapid, dramatic changes that settle down into equilibrium again.For our purposes, punctuated equilibrium serves as a powerful model for viewing the changes in technology, business models, and data over the last few decades and shows us that the rapid, dramatic changes part of the cycle is coming again.
http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
Cherokee-class 10-gun brig-sloop of the Royal Navy downgraded to a 6-gun barquehttp://www.theora.com/msdb.php/OriginOfSpecies/orderBy/subject/msdbCat/subject/12%20-%20Geographical%20Distribution%20--%20continuedExploration - like evolutionary change - only looks smooth in hindsight. Not even the voyage of Darwin’s HMS Beagle was smoothly planned. Looking back he seems to have been predestined to arrive at the Galapagos islands, but this is not the case.
http://darwin-online.org.uk/people/chancellor.htmlAt some points the trip got pretty rough.
http://thebeagleproject.blogspot.com/2008_07_01_archive.htmlAnd the voyage itself was anything but linear.
http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
Credit to Dick Hardt who formulated this in December 2010.
http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
Also in 2010, Twitter announced at Chrip Conference that 75% of its total traffic comes from sources other than Twitter.com. This is part of a growing trend where new, emerging companies gain relevance on mobile and connected platforms other than the web.
The Netflix API is a good example of how metrics can influence the design of the API.As Mike talked about earlier, the Netflix API was originally launched to let 1000 flowers bloom.Over time, we started adding our own devices to run off the API.
Also in 2010, Twitter announced at Chrip Conference that 75% of its total traffic comes from sources other than Twitter.com. This is part of a growing trend where new, emerging companies gain relevance on mobile and connected platforms other than the web.
The Netflix API is a good example of how metrics can influence the design of the API.As Mike talked about earlier, the Netflix API was originally launched to let 1000 flowers bloom.Over time, we started adding our own devices to run off the API.
As a result, we are in the process now of redesigning the API with an emphasis towards supporting our devices, while trickling down API features to the 1000 flowers.
Another key metric going into our redesign is total requests per month. The growth of the API skyrocketed around October/November. Scale is one of our biggest issues moving forward and continuing on this trajectory will create more challenges.
REST is dominating APIs – expanding share of APIs even as the number of APIs is exploding (from just over 1000 in 2008 to over 3000 in 2010).
Netflix: Groups of entire movies will be stored together
Netflix: The same part of all movies will be stored together – such as frequently-used data about the movies such as actors, director, synopsis, first few minutes of video, etc.
Netflix: The same part of groups of movies will be stored together
And don’t just design the API based on your current audience. Dream big and design the system for the audience that you want. That doesn’t mean you need to implement the system for the dream audience. It just means your designs should allow for the system to scale to that audience.
Similar to the rapid changes we've seen in API-enabled business models and related architectures, we've witnessed four major shifts in how data is shared.Back in the mainframe days that Daniel spoke of, flat file data storage ruled and data sharing was fairly non-existent. Allowing multiple applications to access the same file was impractical.Databases with concurrency controls through locks made sharing more possible, as you could now more easily let multiple apps manipulate the same data. However, having applications work directly against data in the format with which it is stored introduced a very tight dependency that could be hard to manage.Middleware introduced another layer that allowed applications to share data via messages to applications, insulating the apps from the actual storage format of the data, effectively making those apps immune to changes in data storage format and schema.Web services improved data sharing even further. The adoption of the internet-friendly HTTP protocol for exposing data now lets sharing occur beyond the boundaries of your business, while also lowering the technical expertise and software required for consumers of the data.
So technology has evolved to enable unprecedented levels of data sharing. How can a business successfully navigate the opportunities and threats of this new world?[Honestly assess the cost, benefits and risks of sharing your data]Sharing your data via an API entails a different set of considerations than other forms of APIs. Why?You lose significant control once data passes out of your API- Unlike non-data APIs, you can't technically restrict access to data once it's been retrieved, although you can restrict access to data in the future- At that point left mostly with legal recourse as defined by any restrictions in your TOU, which is a very blunt enforcement instrument- Enforcement of TOU can be expensive, in terms of your time, money, and reputation with developers, esp. if you seek legal recourse (even if you're right)That being said, it may be time to get over your control issues- Be honest about the value you might get from sharing your data as well as the costs and risks of sharing it
Case study: opening up Netflix movie metadata- Value of sharing: others build great movie discovery experiences -> enables Netflix as a fulfillment option- Costs: - on par with anyother API for allbut - Risk: competitor uses the data to bootstrap their business - data islargely public domain or otherwisereproducible with a small to moderate degree of effort - with controls: can easilycut off a developer'saccess to new data, whichisfairlycritical - TOU - enforcelimits and dev account revocation - nuclear option: legalrecourseCounter example: Hoovers, which has contact information for executives of top companies, their data is their asset
[Optimize your data delivery strategies]1. Consider allowing the download of larger datasets, providing an efficient way to get updates, and allowing liberal retention policies to improve availability, scalability and query-ability- Only allowing bite-sized data requests will cause developers to make many more queries which will affect your scalability- Retention restrictions, such as a one-day limit on data retention, will encourage developers to come back often to refresh their data- even if the data hasn't changed- Both result in higher scalability requirements for your API, which costs money- More frequent requests means an increase likelihood you'll break an app dependent on your data, assuming your service is anything less than 100% available- Push or publish incremental updates to consumers so they don't have to download the world regularly2. Consider limited access to richer queries to determine what new APIs to provide
Basics:[Netflix service screen shots? Create two slides if so]- Lists (a user's queue, recommendations, watching histories)- Text search on entity names (Movie titles, actor names, etc.)- Not super-flexible, but how customers use the business. Limits innovation space.[Netflix movie title API output?]Inevitably, esp. for larger and more interesting data sets, developers will want to slice-and-dice the data- creating queries combining data fields to get very specific subsets of large data sets- minimizing data returned in results to optimize performance- writing a query engine is probably not your core competency, but there are a few options available, albeit with some limitations
Two emerging web-based query engines can provide rich query semantics for your data without requiring you to make large data sets available for downloadAPI limits can be tough with open query access. Consider limiting access (OData) or offloading the work (Freebase).Caveat structor: We haven't seen anyone solve this problem at massive scale, however.Once you use platforms like OData or Freebase and find successful and/or common access patterns to your data, make it a new data API.
Open standard, backed by MS under their Open Specification PromiseThe describe it as “a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today”They provide implementation help for enabling OData on your API servers. .NET producer libraries most mature, other libraries (e.g. Java) starting to emergeIn simpler terms, it enables rich query semantics for your data APIHighest rated titles from the 80's and their awards:http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 and AverageRating gt 4&$expand=Awards [json]
Wikipedia of structured dataYou push your data into their serviceVery robust query language and APICan query across your data as well as other sources of data with common entities (e.g. an actor, a screenplay, etc.) - Example: Actors that have co-starred in a movie with Kevin Bacon that have also been in a movie with a screenplay adapted from a Steven King novelLicensing of hosted data is creative commonsFreebase is in control of the data, their rules, limits, availability, etc.
API limits can be tough with open query access. Consider limiting access (OData) or offloading the work (Freebase).Caveat structor: We haven't seen anyone solve this problem at massive scale, however.Once you use platforms like OData or Freebase and find successful and/or common access patterns to your data, make it a new data API.
While most of our API discussion today has focused on how to open up your business to others, we'd be remiss if we didn't call attention to the value that can be attained by pulling data from other API providers into your business. Think about whether it would be valuable for your business to understand:
1. Would it be useful for your business to know which of your customers are friends? Facebook friends[clipart of friends]What does that logo mean to you today? Maybe it's time to think about it differently.
2. What businesses do your customers frequent? Foursquare and Facebook check-ins[clipart:high scale shop vs. low-rent shop]
3. What is the gender or age of your customers? Facebook core profile [clipart: old man and young girl]The next wave of data competition is not only about what data you can provide, but what outside data can you integration into your business?