Presentation on how we work at Auto Trader UK from #PINK16 IT Service Management conference in Las Vegas.
Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK
7. Our Technology Platform
1.2 billion page views per month
70 million peak page views per day
15 million unique visitors per month
Supported by 100 live applications
12. Getting our platform under control
Reporting and measurement
Continuous Improvement
Pragmatic, Focused on Business Value
Our ITIL foundation
13. Challenges we faced
Started to hit limitations
Too bureaucratic
Safety through control (change freeze?)
Un-scalable
Focussed on process over relationships
16. What is Agile? - Values
Individuals and
interactions
over Processes and tools
Working software over
Comprehensive
documentation
Customer collaboration over Contract negotiation
Responding to change over Following a plan
http://www.agilemanifesto.org/
17. What is Agile? - Practices
Incrementally Instead of all at once
33. Spotify Engineering model
Small, multi-disciplinary
team
Autonomous
Based around a Purpose
Led by Tech and Product experts
https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
40. Long Automated Release migration
0
10
20
30
40
50
60
70
Mar
2014
Apr
2014
May
2014
Jun
2014
Jul 2014 Aug
2014
Sep
2014
Oct
2014
Nov
2014
Dec
2014
Jan
2015
Feb
2015
Mar
2015
Apr
2015
May
2015
Jun
2015
Auto Deployments Manual Deployments
Deployments by week:
Auto vs Manual
41. Changing Release process again
Gradually given Delivery Squads the ability
to LIVE release
Start to push Operational responsibility
back to Delivery Squads
Our purpose: “To lead Continuous Delivery
and innovation of the Auto Trader platform”
Operations teams then Re-organized as
Squads.
42. Effect on our Service Transition Process
0
500
1000
1500
2000
2500
3000
FY11 FY12 FY13 FY14 FY15 FY16
Release Totals
Overall Success Total Releases
43. Effect on our Service Transition Process
93.0%
94.0%
95.0%
96.0%
97.0%
98.0%
99.0%
FY11 FY12 FY13 FY14 FY15 FY16
Release Success
44. Impact on working culture
End of Project
Culture
End of Contractor
Army
Operational and
Technical issues highly
valued
Improved operational
responsibility within
Squads
46. Still some Organisational Problems
Starting to build the things right… but are we
building the right things?
How do we build the right things right?
48. Change in approach
New Requirements Become Assumptions
We know Becomes We believe
Let’s build it Becomes Lets test it!
http://www.slideshare.net/jgothelf/better-product-definition-with-lean-ux-and-design-thinking
50. Product Discovery – effects on Service Transition
Squads need to
prototype and
experiment more
with their customers
All ideas and
decisions are
validated with Data
Another step change
in the numbers of
changes
Needs new levels of
of automation and
self service
51. Our Business is continuing to grow and change
http://www.londonstockexchange.com/
Editor's Notes
Intro to me:
Why I’m here
Introduce Jono and Pete
What we’re going to cover
Tell you a bit about Auto Trader – bit of context
Then talk about how Service Transition has changed as our organisation has changed.
DevOps is not mentioned much here. But hopefully you’ll see the themes of cultural change, collaboration, automation , lean thinking coming through really strongly.
AutoTrader grew as a publishing institution in the UK
Selling classified magazines with private and trade cars and motorbikes for sale
Bikes, Commercial vehicles etc.
400,000 magazines per week at it’s heyday
It’s a really well known brand- 92% of the UK population know what we do and what our brand stands for.
Last magazine printed in 2013
We’re also a great story about moving a business from print to digital
A journey that started for us in 1996
Rapidly that part of the the business expanded until the vast majority of revenue was generated from digital services and the magazine part of our business slowly reduced in size.
POINT IS
We are a digital business and aspire to be one of the best digital businesses in the world
BUT We are not a start-up
20 years of digital history and decades of print history
http://www.autotrader.co.uk/digital
http://about-us.autotrader.co.uk/
Private Sellers: us selling our Cars
Trade Car Dealers 15,000 - Independent dealers, Franchise dealers, Car Supermarkets
Availability at 99.99%
Availability at 99.99%
supporting products for Consumers, Private Sellers and Trade Retailers.
Supporting access across multiple platforms
Supporting Commercial and International Autotrader sites
e.g. Dealer Websites
Automotive leader for dealer websites with just under 5000 dealers’ sites hosted
So I’m going to talk about Service Transition – how we safely build and release new services and changes to services.
It’s where I’ve worked for the past few years and its really interesting because it’s affected by loads of factors – how our organqisatio is structured, out culture, how we deiliver work, the technology we use, architecture etc.
Service Transistion is a key enabler for the business as well – something that will either drive your business forward, or hold you back.
Hopefully it will be useful to look at the changes, and how we’ve adapted and improved our Service to our Customers
What factors can affect Service Transition
Organisational structure / Changes
Financial model of organisation
Project Methodologies
Approach to product development
Physical location
People / Culture / Habits
Risk aversion,
Regulatory pressure
Tech stack
Architectural decisions
Service Transition
Why it’s cool / important
Getting an idea out of someone’s head, to delivering value for a customer
(in a way that is supportable and reliable)
Touches loads of people, skillsets, departments
Can be a technical/logistical challenge
Should be quick and painless
Our jobs are to make it speedy , painless and safe
SO I’m going to talk through a few aspects of how our Service Transition process has changed
Although this is a timeline – and this has been a journey – not one stage has replaced the previous – it might be better to think of this as layers on top of eachother that all contribute to the way we work at the moment.
Hopefully it will be useful to look at the changes, and how we’ve adapted and improved our Service to our Customers
When we started implementing our ITIL processes, our Digital Services were pretty unreliable
I’d like to tell you how unreliable, but we didn’t measure too much!!
We had major incidents weekly (some periods daily) - our main website down for hours at a time.
A lot of our working practices were uncontrolled and inconsistent
We had a strong core team of Service Managers
Initially in dedicated Service Management team to implement ITIL based processes.
Then distributed as leads throughout our Operations team
Positives – that we keep TO THIS DAY – and still really value.
reporting and measurement key to management
Continuous Improvement is part of our day to day roles
we are pragmatic – we adopt what’s useful and adapt to our business
Focussed on Customers and Business Value
As our business started to change more rapidly,
Too bureaucratic –
2 hours CAB meetings for each development department -Insisting all product changes are documented for CAB approval.
unnecessary documentation/approvals
Safety through control Big Change Freezes - taking access/responsibility away
Un-scalable e.g. Focussed on planning, implementing and recovering from infrequent complicated, risky releases
Focussed on process over relationships- handovers, tickets and documentation between teams and departments.
Out of hours implementation
Frequent failures
Complicated manual processes
Focussed on detailed planning
Extensive documentation and approval process
Releases of software were infrequent, complicated and costly
Big batch size – INFREQUENT releases
2009 started to change how we deliver our projects
1 project scaling out to all projects
Need to still be responsive
Cannot have more and more change meetings
AGILE is an iterative approach to software delivery that builds software incrementally from the start of the project, instead of trying to deliver it all at once near the end.
Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
It works by breaking projects down into little bits of user functionality called user stories, prioritizing them, and then continuously delivering them in short two week cycles called iterations.
The most common complaint I heard from Ops during the “agile migration” was that the devs would no longer document new services.
In reality, most of that documentation was useless anyway as no-one maintained it.
What agile actually did was break down barriers for communication, through standups, retros etc. Through bringing stakeholders into the project. Through making project walls/backlogs visible.
Documentation largely replaced by standups, showcases, retro’s = better/more immediate conversation
Ditch pointless paperwork. But only the pointless stuff. Anything with a point can stay.
Basically, for releases we trimmed out all the fat and were left with a calendar entry & release note
The documentation dev used to produce for ops was replaced with a wiki page, maintained by the release team
The devs also create documentation, for their own use.
Daily stand-ups, RETROS, SHowcases.
CAB as a 5-10min standup focussing on the next few weeks, instead of a lengthy meeting
CAB just for service-affecting changes
Kanban for small changes, allowing the Ops teams to be open about the prioritisation of work etc..
Change in build practices for test environments.
Many builds in non-live per day.
Automated tests!!!!
Continuous Integration
But it stopped there
Smaller, more frequent releases, to de-risk the act of releasing
With the greater volume of releases, and the improvement in safety,
No more out of hours deployment (very few – maybe 1%_2%))
Change freezes become much reduced (maybe not on Christmas Day eh?)
it was not only necessary to widen the release window, but finally possible.
Scaled using Release Team as go - betweens
Release team link Development and Operations teams
Attending standups and being the public face of Ops
Ensuring documentation is done
Preparing the way with Infrastructure teams
Scheduling and managing releases
Fixing issues
We started to see interest from Developers who were keen to know how their apps were performing. Firstly in Test environments then quickly in LIVE
GRAPHITE became the common language to interact with Development teams.
Development and Operations teams starting working together a lot more
Releasing less (smaller change) more frequently has the obvious effect on volume.
It also allows QA’s to focus on fewer changes, which increases the chance of catching something before it goes live.
“The implementation of Release Management within AutoTrader is one of the most mature that Pink Elephant has ever witnessed and AutoTrader is firmly within the top 10 organisations in terms of Release maturity scoring.”
- Pink Elephant
Releasing less (smaller change) more frequently has the obvious effect on volume.
It also allows QA’s to focus on fewer changes, which increases the chance of catching something before it goes live.
Pink Elephant quote
The implementation of Release Management within TMG is one of the most mature that Pink Elephant has ever witnessed and TMG is firmly within the top 10 organisations in terms of Release maturity scoring.
More importantly, the process reports show a clear year on year improvement since 2010 in terms of real business benefit, despite a year on year increase in workload as the business model of TMG has changed and more agility and flexibility has been required of the IT department.
The Release management process has seen its workload increase 300% in terms of numbers of Releases handled, since 2010 but has also increased the success rate from 85% to 93% during that timeframe.
Different Reporting Lines for Product and Technology
Many physical locations
Focus on PROJECTS - New agile shiny products
Big bursts of activity in one area while other products get neglected.
Capital vs Operational budget lines
Manual Processes for deployment – can’t hire any more people!
Lack of ‘Operational Mindset’ within teams – no way to get issues addressed
Technology roles not ‘decision makers’
Still little focus on quality after launch – developers move off the project on launch day
Lack of ownership - developers moving from one project to the next
Large number of contractors – 40%
We got a new CEO
Stopped producing a Magazine.
Now a DIGITAL BUSINESS
Product and Technology Teams brought together
Consolidate into 2 offices, all tech teams in Manchester
Every team has a Product Lead and a Tech Lead
Stopped producing a Magazine.
Now a DIGITAL BUSINESS
Product and Technology Teams brought together
Consolidate into 2 offices, all tech teams in Manchester
Every team has a Product Lead and a Tech Lead
Each squad will have a purpose around a particular service or product
Squads aligned to our customers
With freedom (***) to carry out that purpose as best they can
Where squads are working towards a similar market or customer base they are grouped together into a TRIBE
Tribes then have leaders who set a wider strategic goal
We’ve got about 4 tribes across our organisation
WE DID NOT RE-STRUCTURE OPERATIONS
We created a Squad within Operations
#Continuous Delivery Squad
Ops Engineers and Developers
Chance to treat our Operational tools and processes properly!
Focussing on
Automated Deployment
Monitoring
Provisioning of environments
Auto Deploy
Bit of a stop-gap
Single Click deployment to LIVE environments
Use Continuous Integration pipelines and deployment scripts
Triggered by Release Team
Reduce the time to release and manual effort
No more constraints on Deploy Engineers’ time
Less need to schedule
Squads Release software within a few minutes of passing QA
We start treating Operational tools the same as External Services
Single Click deployment to LIVE environments
Mostly goverened by the same Release team
Reduce the manual effort of releasing
Remove constraint around scheduling and massively improve release bandwidth
We have up to 5+ releases ongoing at the same time
We kept Calendar entries
Recording and monitoring more automated
Admin less automated
Improved Monitoring and Visualisation
AGILE – visibility of information
Competative advantage: Changing Data provider
Gone from 40 manual releases a week to 4
Gone from 4 Auto Releases to 45
Note the 41 auto releases for CAP Switchover
Start to give Delivery teams the ability to LIVE release!
Start to push responsibility back to Delivery teams
Restricted by
Access to live systems
Complexity of fixing anything
Interdependent services (some too tightly coupled)
Delivery Teams not aware of wider platform/operational issues
Real desire within our Business to experiment and try things out
Product Discovery,
So that gets us almost up today
But what challenges do we face now - and how do we want to improve?
We’ve focused a lot on how to build things RIGHT
Product discovery is about making sure we build the right thing
Hands up:
Who works on projects in your job?
Keep your hands up if you’ve worked hard on a project that never saw the light of day? (never got released to a customer?)
EXAMPLE – Damien working on 2 projects for 18 months that didn’t go live.
An endemic problem for everyone in a digital world is knowing what the right thing is to build
Lean Enterprise p47
Long product development cycles dramatically reduce the potential return
on investment we can achieve from successful new products.
• Long development cycles delay the time it takes to get
customer feedback on whether we are building something valuable.
• Typical market research activities are poor at predicting a product/market
fit, especially in new product categories. Research said that minivans and
iPods would not be successful.
• In the absence of good data, people tend to get their pet projects funded.
Particularly in enterprise IT, we often see spectacular amounts of money
poured down the drain on systems replacement
http://www.slideshare.net/jgothelf/better-product-definition-with-lean-ux-and-design-thinking?next_slideshow=1
Key questions:
How long do we wait for launch
How do we define the right requirements for our product
What signals are we looking for from the market
REQUIREMENTS are actually ASSUMPOTIONS
e,.g early assumptions
Who is our customer
AMAZON
Run thousands of experiments per day
Roughly on third will add some value, one third will do nothing and one third will be detrimental
They have not found a way of predicting which experiment will delivery which outcome.
Lean Enterprise p32
Data gathered from evolving web-based systems reveals that the plan-based
approach to feature development is very poor at creating value for customers
and the organization. Amazon and Microsoft (along with many startups) use a
technique called A/B testing to gather data on whether a feature will actually
deliver value to users before it gets built in full. Ronny Kohavi, who directed
Amazon’s Data Mining and Personalization group before joining Microsoft as
General Manager of its Experimentation Platform, reveals the “humbling statistics”:
60%–90% of ideas do not improve the metrics they were intended to
improve. Based on experiments at Microsoft, 1/3 of ideas created a statistically
significant positive change, 1/3 produced no statistically significant difference
and 1/3 created a statistically significant negative change.12 All of the ideas tested
were thought to be good ones—but neither intuition nor expert opinion are
good gauges of the value our ideas have for users.
Starting to build the things right… but are we building the right things?
EXAMPLE – Damien working on 2 projects for 18 months that didn’t go live.
Call what we think will deliver value to our customers ASSUMPTIONS (not requirements)
Start with the highest risk assumptions
Design a testable hypothesis so we can run an experiment with a measurable result.
Do the minimum amount of work required to measure success of that experiment
Outcomes are important – it’s not what we want to build but what outcome we want from the problem we solve.
Outcomes not outputs – examples.
Creation of prototypes and experiments
Getting to a world where change is continual.
Each of these changes is likely to be a release
A/B testing, MVT, changes to logging , metrics,
Leading Continuous Delivery
Watching for big batches! (Big batch police)
Forcing changes to be easier and safer.
e.g More reliance on signals/metrics/data to quickly interpret
Identifying constraints within that value stream and working with Squads to fix them
Limiting factors for Release process
Culture within delivery teams – manual testing, admin,
How quickly we can get feedback and change again
Automated recording of releases
Still measure success/ failure and take actions to improve
So our business is continuing to grow and change
We floated on the London Stock exchange last year and I naively thought this would be a good slide to demonstrate how we are changing.
(If anyone knows what’s happended to the stock markets since December then you’ll know this was a good time to take a picture)
But we’re outperforming the market and we’re outperforming our competition
When our CEO talks to investors about Auto Trader as a company - he will talk about how we work, our culture, our agility and responsiveness to market changes – our ability to know what delivers value to
customers and focus on that and not waste time and money.
Service Transition is a great indicator about how our company works, but also been a key enabler to delivering value to our customers.
