apidays LIVE Australia 2020 - Building Business Ecosystems
Growing Domain APIs - "T'aint what you do..."
Liz Douglass, Partner at Deloitte & Saul Caganoff, Principal at Deloitte
apidays LIVE Australia 2020 - Growing an API Culture by Liz Douglass & Saul Caganoff
1. Growing an
API Culture
âTâaint what you doâŠâ
Liz Douglass, Partner at Deloitte Platform Engineering
@lizdouglass
Saul Caganoff, Principal & CTO at Deloitte Platform Engineering
@scaganoff
2. Agenda
Part 1: Common problems and anti-
patterns
Part 2: What does better look like?
Part 3: Growing applications into domains
4. Bottlenecks
â· Consumers waiting on APIs
â· Waiting on central integration team
â· Waiting on shared infrastructure
â· System inertia due to technical debt
4
Silos
â· Lots of similar APIs
â· No benefits from reuse
Contention
â· Version proliferation â APIs donât get
retired
â· Version churn â APIs change rapidly
12. The problem with (only) projects
â· Compartmentalised scope and time
ⶠâJust need to get it doneâ
ⶠSometimes this is unavoidable â but
technical debt is OK if you pay it back
â· Short-term incentives with no consequences
for shortcuts
â· More convenient to look at things as
evergreen
ⶠEasier to start from scratch than to re-
use someone elseâs stuff
â· Projects are hampered in their ability to
promote durable, re-usable assets such as13
13. Product thinking
â· An antidote to some of the problems of
projects
â· Product thinking looks like:
ⶠA roadmap of features extending into
the future
ⶠFeatures driven by customer demand
ⶠThe ability to push back (or say ânoâ)
on customer demand â e.g. to maintain
the integrity of the product
â· Typically manifest in a product manager
who has resources, rights and longevity to
deliver on a product roadmap
14
15. Many organisations struggle with
product thinking
â· 50 years of the project paradigm:
ⶠConcepts
ⶠProcesses
ⶠFunding models
ⶠOrganisational structure
â· Procurement models based around COTS
ⶠProjects commission COTS âboxesâ
and join them up (integration)
ⶠPick the product then pick an
implementation partner (system
integrator)
â· Attention centres on the box rather than the
business capability
16
16. Integration
becomes a thing
we âdoâ
17
â·Delivered via projects
â·Lines between boxes
â·An obligation
â·Inherits the problems of projects
â¶difficult to create durable, re-usable
assets
Application 1
Asset boundary
Application 2
Asset boundary
Integration
Design
Build
Run
Endpoints
Data structures
Mapping rules
17. Can we flip this
model?
18
â·Most organisations already have durable asset
teams
â¶Centred on COTS applications
â¶Stopping at the application boundaries
â·What if we move the APIs into those asset
boundaries?
â·Asset teams become the durable owners of their
domain APIs
Application 1
Asset boundary
Application 2
Asset boundary
API Provider
Design
Build
Run
Endpoints
Data structures
API Consumer
Mapping rules
24. 25
Disconnected
Projects
Sweet Spot
âą Domain autonomy& durability
âą Governanceat the edges
đ
đ
HighAlignment
âą Discovery
âą Consistency
High Autonomy
âą Durability
âą Self-service
LowAutonomy
âą Order-takers
Delivery Model
Governance
Model
Integration
Factory
đ
Slide template courtesy of slidescarnival.com â licensed under Creative Commons
Hinweis der Redaktion
Introductions â Liz first
Saul:
Our talk today is split into 3 parts
- part 1 looks at common problems and anti-patterns that we see today
- part 2 looks at what a better model might look like
- part 3 presents a technique for achieving that better model within enterprises today
Saul: so first â whatâs the problem?
Saul:
In Platform Engineering we specialize in enterprise integration and we work with dozens of different clients across a range of industries and with different digital maturity.
The common thread we see is that organisations want to deliver technology faster and at reduced cost.Â
Weâre in the position where we can see many different responses to this challenge and there are common anti-patterns that emerge from these responses
Liz: Bottlenecks
- Teams waiting for their APIs to be implemented
often because of a dependency on a single centralised team. Alternatively this can occur when you're constrained by shared infrastructure, orÂ
because you're weighed down by technical debt.Â
Saul: Silos
We often see duplicate APIs â lots of slightly different copies of APIs doing essentially the same thing
We also see organisations struggling with achieving time and cost savings through reuse of APIs
Liz:
The third class of anti-pattern we see is contention
And in this context we're talking about contention between versions -Â
- Either you have the domain expessed in lots of APIs, or
The expression of the API is changing quickly the versions are churning.
Saul:
And it's important to note that these anti-patterns arise regardless of the technologies used in API programs:Â
- API gateways,Â
- developer portals,Â
- microservices,Â
- CI/CD etc.Â
These things are helpful but not necessarily sufficient for success.Â
Liz:
Each of these problems can be traced back to intermediate causes:
Ownership â not possible because of dependencies on tasks or infrastructure that belong to a different team
Durability â or âattention spanâ â the ability of a team to evolve their âproductâ over a period of time
Liz
prioritized roadmap that allows different teams to plan and collaborate over shared asset
And finally having visibility of those shared assets â both the ones in production and the ones which are planned or under development
Liz:
And then we thought about it more we decided....Â
Ownership and Durability are driven by the level of Autonomy that teams have,
which in turn is driven by the delivery model used.
Roadmap and visibility are driven by how different teams
achieve alignment across the broader organisation
and ultimately that comes down to your governance model
Liz:
Each of these problems can be traced back to intermediate causes:
Ownership â not possible because of dependencies on tasks or infrastructure that belong to a different team
Durability â or âattention spanâ â the ability of a team to evolve their âproductâ over a period of time
prioritized roadmap that allows different teams to plan and collaborate over shared asset
And finally having visibility of those shared assets â both the ones in production and the ones which are planned or under development
And then we thought about it more we decided....Â
Ownership and Durability are driven by the level of Autonomy that teams have,
which in turn is driven by the delivery model used.
Roadmap and visibility are driven by how different teams
achieve alignment across the broader organisation
and ultimately that comes down to your governance model
Saul:
This is reminiscent of the way that Spotify have talked about the relationship between team autonomy and alignmentÂ
and how technology teams need both in order to move faster in the same direction.
Liz:
We can overlay this diagram with an integration perspective and we see similar patterns of use occurring in organisations....Â
Liz:
We see most organisations that we speak to are in the top left hand quadrant
Organisations in the quardrant have traditional approach to integration
and a centralised team that does all the integration work.
They have high level of alignment by virtue of the fact that they sit together terms of:Â
- discovery, and
- consistency
But the team often have the symptoms of being a bottleneck that we spoke about at the start of this presentation.Â
Saul:Â
Similarly, if they have multiple project teams and a poor governance model, then
we see many disconnected projects leading to the silo and contention antipatterns that we also spoke about at the start of this presentation.
Liz:Â
Now this being a good 2x2 means that we always want to get to the top right quadrant. In this state the people working on the APIs have:Â
- domain autonomy and durabilityÂ
- and a high level of governance at the domain boundaries
So how do can we get to the top right without falling into the chaos state at the bottom right where everyone just does whatever they feel like?Â
Saul:
Liz:
Let's start by going into some more detail into what good looks like.Â
One of our favourite books at the moment is "Project to Product" by Mik Kersten.Â
In the book he describes how teams need to balance four flow types
There's the feature flows where you are building new capabilityÂ
Defect flow where you are correcting errorsÂ
Risk mitigation flow where you are proactively preventing issues like a compliance or security incidents, andÂ
Technical debt flow where you are paying down the debt that you might have accumulated whilst working on one of the other flow types.Â
The point that Mik makes in the book is that you need to constantly balance the four flow types.Â
All are required andÂ
you must make tradeoffs
so as to prevent any of them from stagnating,Â
Though the balance of them changes at each stage of the product lifecycle.Â
Saul:
The problem is that projects generally only concentrate on two types of flow â features and defects
Projects are particularly bad at dealing with technical debt which is an important flow type which leads to systems inertia if not addressed
Projects are often built around short-term incentives with no consequences for short-cuts - it is the following projects or the enterprise at large that suffer for short-cuts
Projects also find it difficult to re-use because often that adds risk and dependencies rather than taking them away.
So overall, projects are hampered in their ability to promote durable, re-usable assets such as APIs
Liz:
- This is especially true in the API space because you can keep building them with impunityâŠ
Liz:Â
The way to get to that top right quadrant is to take a product thinking approach.Â
Teams that do that:Â
- Have durability: a roadmap of what they are going to build
that takes into consideration how to evolve what they already haveÂ
- They know their customers and therefore their domain well
and the domain is expressed in the API design.Â
- And importantly, the team, and in particular the product manager, is in a position to maintain the integrity of the productÂ
Saul:Â
<EXAMPLE â doing some logic in the integration layer that should be in in the app. Don't corrupt your domain>
Liz:
So to summarise what we've established so far:Â
API anti-patterns are driven for the most part by the governance model and the delivery model
To avoid these anti-patterns we're aiming for a governance model that promotes high alignment, and a delivery model that allows for high autonomy
...the top right quadrant
And we've made the case that that an essential element of getting there is to adopt product thinking....
âŠ. So what does that practically look like?Â
I'm sure some of you are thinking, well that's great, but it would be easier to cycle to the moon than make that happen in my organisation....
Saul: transition
Saul:
âŠ. And yes we acknowledge that the project paradigm is deeply entrenched in many organisations.Â
The project paradigm is deeply entrenched in the way we do things and it ripples through lots of dimensions:Â
- Concepts: our mental model make it hard to believe there are different ways <LANGUAGE, SKILLS>
- Processes: including the processes for addressing (or not addressing) the four flow types, like risk management processesÂ
- Funding models, as well asÂ
- The organisational structureÂ
Liz:
We often find that organisations where the project paradigm is most ingrainedÂ
also tend to typically buy software.Â
They buy boxes and then customise themÂ
The enterprise capability has a hard boundary at the edge of the box
And you go out and get a 2 for 1 deal: you get a box from a vendor and a 'systems integrator' to wire it in â which is not exactly a durable team approach.Â
This procurement approach also impacts how organisations also think about integration.....Â
Saul:
As a result of the fact that we orient around applications, integration has become a specific thing that we do
Integration is delivered within projects
Itâs concerned with drawing lines between boxes
And itâs often regarded as an obligation rather than something with inherent value
Delivering integration within projects makes it difficult to create durable, reusable APIs as assets
Liz:
- Yes and design , build and run etc all fall between asset boundaries
Liz:
So what to change?Â
The best way we've found is to give roadmap responsibilityÂ
and ownership to the asset teamsÂ
for the APIs that integrate their systems.Â
The asset teams are more likely to have at least a core team that is consistently funded to be able to take up this responsibilityÂ
And they have the best knowledge of the end-users and the business domain.Â
This has the effect of expanding the asset boundary outside the COTs application boundary
And has numerous benefits including:Â
- Durability of the APIs, andÂ
- A better likelihood of visibilityÂ
and therefore re-use for the APIs.
Saul:
- The first thing we want to do is expand application boundaries to form business domains
Saul: domain capabilities (services) are expressed via business APIs and event streams
Saul: our original applications still reside at the core of our domains â but they become augmented by auxilliary applications and microservices which take care of concerns like customisation or internal integrations
Saul: and these tightly controlled domain boundaries become a scaffold for modernisation of our applications within the domain with minimal impact outside the domain
Saul: Zooming out to the bigger picture we see digital channels are built on top of the underlying domain capabilities â delivering our business capabilities out to customers, employees and partners
Liz:
Integration is a byproductÂ
Liz:
To finish up,Â
We've covered some common anti-patterns associated with operating in the left hand side of the matrix, specifically:Â
- Bottlenecks, silos and contention
We've looked at the underlying causes of these issues
and proposed that the aim is to achieve both high alignment and autonomy.Â
We've spoken about product thinking
and how it better positions teams to achieve the top right quadrant than project thinking.
And we've proposed that that an achievable way to adopt a product approach is for asset teams to become the durable owners of their domain APIs.