Slides from STV Tech Talks (Glasgow, UK) - 27 August 2013
--
Based on the experience of leading the current initiative to move towards Continuous Delivery at Skyscanner, I presented a view on 5 key focus areas that must be considered to make Continuous Delivery an achievable objective within a very complex Service Oriented Architecture.
4. Continuous Delivery
“How long would it take your organisation to deploy a
change that involved just a single line of code?”
Do you do this on a repeatable, reliable basis?”
…is a set of practices and principles aimed at building,
testing and releasing software reliably and repeatably
at a necessary frequency.
Keeping systems production-ready throughout
development, so that they can be released to users at
any time
8. Continuous Integration
Everyone Commits to the Mainline Every
Day
Automate the Build
Make Build Self Testing
Every Commit triggers build on Integration
server
Fail Fast, Feedback Faster
http://martinfowler.com/articles/continuousIntegration.html
10. Build Quality In
“Cease dependence on mass inspection to
achieve quality. Improve the process and build
quality into the product in the first place”
W. Edwards Deming
13. Configuration Management
Stuff we make is valuable
Not just the source code :
Application configuration
OS setup
Machine Configuration
Tests
Documentation
Where possible should live with the code
16. 8 Principles / Patterns
Reliable & Repeatable delivery process
Automate (almost) everything
Take the pain early and more often
Source control is king
Done or Done, Done = Released,
Build quality in
Everybody is responsible
Continuous improvement
17. Service Oriented Architecture
Definition:
- software architecture design pattern
- structured collections of discrete software
modules (the services)
- collectively provide the complete functionality
of a large software application
19. Considerations for CD in SOA
Focus on 5 key areas
- Partitioning the SOA into governed
Deployable Units
- Consistent Deployment Pipelines
- Just Enough Test Automation Coverage
- Real Time Metrics
- Aligning the Delivery Conditions
20. Deployable Units
1+ Services to be deployed
Single, built once Package Comprising:
Releasable Software
Tests
Application Configuration
“Install” / Rollback Scripts
Assets
One mainline (Trunk) per Deployable Unit
Decoupled Deployments and Rollbacks
Fits naturally with S.O.A.
But not necessarily 1:1 ratio
22. Service Governance
a.k.a. Making sure people do the right things –
Anne Thomas Manes, OOP2007
SLAs
Versioning
Fault Tolerance and Resilient Engineering
24. Databases
Services may have own DB
Versioned with
Some DBs act as integration points
Legacy
Must be treated as any other Service
Same levels of governance,
Same SLAs
Same Tolerance to failures
25. Consistent Deployment Pipelines
Early Integration not Late Integration
Consistency, Consistency and Consistency
Build Scripts
Tools of choice,
Test Frameworks and Usage
Everything where possible
Canary Release Patterns
Aligning to a single flow for all components
26. Just Enough Test Automation
Cost of
Execution
Time
Number of Tests
Unit Tests
Integration Tests
Acceptance Tests
UI Tests
100% Automated
100% Manual
28. Real Time Metrics
Safety Net for canary release
Coupled with Operational Metrics
Individual Service defined
Asynchronous Data Loaders – throughput
Failing Fast and Hard
31. Culture
Agile Practices First
Breaking down Silos
Getting over the fear of collaboration
Trunk Based Development over Feature
Branching
Discipline in Continuous Integration
Dealing with breaks
32. Summary
Architecture has a huge impact on the
ability to Continuously Deliver
Comes with its own set of challenges and
It is the right thing for modern web-based
businesses.
As Michael has pointed out. Skyscanner – I am sure most of you know Skyscanner – even those few of you that work there… But for those wanting some idea of the scale.:#1 flight search engine across Europe, #3 in the world for traffic. Fastest Growing international travel search site outside of the US.50-75million visits per month40+ languages (some of which are in process of going live) in 30+ local markets and in 70+ currencies200 members of staff globally (closer to 300 now I think) – across 5 offices – Edinburgh, Galsgow, Singapore, Beijing and Miami. +150 by end of the year.Engineering is centred in Scotland, primarily Edinburgh but also Glasgow. Overall 120+ in Development + Test Engineering, 20+ in Service Management (ops). I’ll come on to a few technical implementations.Topic:I am currently leading the current project to bring Continuous Delivery practices to Skyscanner. I am going to talk about the lessons learned so far in doing that, the compromises we’ve made an how it applies to our complex S.O.A. We could talk all day on the topic and as I have an hour (and I am getting back to Edinburgh on the last train) – we are going to skirt some of the detail and keep it high level.Quickly introduce the topic – How many people understand the principles of Continuous Delivery? We’ll do a recap but I’ll be keeping it very brief, and my interpretation of it. First off I’d like to make a statement – IMHO the value of Agile isn’t in the prescription of the SCRUM or XP flavours. Agile development is about sticking to the principles but finding out exactly what the What is Continuous DeliveryWhat is SOAWhat is that at Skyscanner? Why bother ? Perfect Storm (Test team building up, defining release operations; too much pain)5 key challenges and how we are planning to solve itGet business buy in…Deployable Units (Monolithic vs component)Automating TestsService GovernanceLate integrationConsistency
50+ logical components 30+ “physical” services + web apps and Both:Message based (RabbitMQ)Restful Web ServicesMixed Technology Stack.Net (C#)Python JS / CSS / Mobile applicationsHA and Resiliance2 DCs (4 by Q2 next year)
Literally playing through the pain…Some of our core applications and services are propped up by levels of effort every 4-6 weeks that they really shouldn’t be. Yes we have a strong SOA, component based architecture but we struggle to release – particualrly now we are at a scale of multiple DCs worldwide; hundreds of virtual machines.
Mary & Tom Popendieck, Implementing Lean Software Development
Martin Fowler:“Everyone should commit to the mainline at least once every day”Check in daily – checkRuns through builds. The values of the feature branching etc.
Detecting when things go wrong – Building Quality in…Unit Testing from the developers
Statistician said these words.. Fixing issues when they are created is by far cheaper than finding them at the end.The example is - In building quality at each stage of the German car production line, the quality teams at the end of the process had nothing to do. And if there’s one thing the Germans do well its penality taking and building cars. Essentially the same for Software Engineering.And the way we do that is through Automated Testing – at multiple levels right around the entire system.
Version everythingWhere ever possible we should be automating that. Build Once – Deploy manyIntroduce the concept of a deployable unit.Data is a challenge – whether you look at something like DB Deploy.
Ubiquitous Pipeline Image for Continuous Delivery Presentations….What does it mean?A deployment (or Build) pipeline is an automated manifestation of the process for getting the software out of versions control and in front of users.
Particularlly simple example and know where near good enough.
Loose Coupling between product systems enables small batchesServices have a single Responsibility – yes.Not the same as a deployable unit…
Pretty simple statement – a key – value.Own heartbeat…I think the CD guys got it wrong when they talked about “Fast” – I believe in Efficient. Continuous Delviery is about having the mechanisms to “Once a product developer realises that small batches are desirable, the start adopting product architectures that permit work to flow in small, decoupled batches.”Principles of product Development Flow, Don Reinertson.
Plannign carefullyIf 2 services move in direct lock step, then you don’t need to explode them into indeividual services. You do need to be very careful of integration testing nightmare of O to the NDelivering the elements
MakingMassively Important for Services to be governed correctly. For many SOAs an Enterprise Service Bus is uesed to enforce and delvier the governance. That is one approach. But in its essence it is an abstract se
Non-breaking changes are required. Any breaking changes are required to be Versioned and delivered.--Server Backward Compatability & Client Forward Compatibilty- New version of service must support v1 of client written to old interfaceClient continues to function against v2 of a service written to the new interdace.Here is the extra bits in this diagram. Took me a long time to get my head around this.Client Backward CompatibilityNew version of client – should continue to support v1 of Service as well as v2Server Forward compatibilityExisting Services must work with newer clients that written to newer interface.A non breaking change to this might be the addition of some optional parameters to a service call or some extra fields in the JSON response. This should be handled gracefully by the client with no changes required. Having the service also ignore unnecessary parameters rather than fail also delivers this level of compatibility.Or in a breaking change then the client must be aware that All of this supports releasing canaries and test elements.
Late integration vs. Early integration
Revisit the test pyramid.As part of our exapnded growth – massively increased our test automation capabilities in house. Those guys are delivering a shift towards the elements.Using a variety of integration tests against services dricectly and BDD techniques.They are applying it within the areas of expertise.Employ a Testing StrategyIdentify and Prioritise based on RiskDecide What Actions To TakeTests = Executable SpecificationsIncreasing ConfidenceAllows manual testing to be focussed on Exploratory rather than the mundane strategic elements
Initially each Service is aware of the services on which it depends on and healthchecks are necessary. Looking at this diagram there are 3 healthchecks running at the outer layer. Each in turn bases it health, not just on itself but also the state of the healthcheck below.This takes effort to maintain on behalf of the component.sIn some SOA’s the ESB will take care of this. But an ESB is a large wieldy piece of kit – and not essentail for SOA.
Me – Well I am me - WTF is an architect – well I would summarise it – a technical leads who has a broader field of vision when it comes to the application both in the systems space and in time. See byond the delviery of the current task and sets out the standards. Skyscanner – Who knows about Skyscanner? We are based in Edinburgh, have new glasgow office (+ singapore, Beijing and Miami) – all engineering is done in Scotland. – and yes we are hiring good engineers. Uve been there almost 4 years and therefore am a veteran. Topic:I am currently leading an underway project to bring Continuous Delivery practices to Skyscanner. I am going to talk about the lessons learned so far in doing that, the compromises we’ve made an how it applies to our complex S.O.A. We could talk all day on the topic and as I have an hour (and I am getting back to Edinburgh on the last train) – we are going to skirt some of the detail and keep it high level.Quickly introduce the topic – How many people understand the principles of Continuous Delivery? We’ll do a recap but I’ll be keeping it very brief, and my interpretation of it. First off I’d like to make a statement – IMHO the value of Agile isn’t in the prescription of the SCRUM or XP flavours. Agile development is about sticking to the principles but finding out exactly what the What is Continuous DeliveryWhat is SOAWhat is that at Skyscanner? Why bother ? Perfect Storm (Test team building up, defining release operations; too much pain)5 key challenges and how we are planning to solve itGet business buy in…Deployable Units (Monolithic vs component)Automating TestsService GovernanceLate integrationConsistency