SlideShare a Scribd company logo
1 of 21
Download to read offline
Chrome Release Cycle




 Author: Anthony Laforge (laforge@chromium.org)
Our Philosophy

 Think of any major website, even the fancy Web 2.HTML5
 ones... do they have version numbers?
 We took the same approach to our client software as an
 online web service.
 That is... we treat releases as a means of getting features
 out to users and not goals in and of themselves.
 It's about flow.
How Our Users Work

 Users sign up for Chrome release channels based on the
 level of stability they want.
 Updates happen automatically in the background.
 New features and functionality just flow in without any effort
 on the user's part.
How the Chrome Team Works


 Developers work almost exclusively on a single
 central trunk (possible due to our try bots and continuous
 build infrastructure)
 Our branches stem from points on the trunk
 We stabilize branches by pulling in changes from trunk (i.e.
 everything lands on trunk first)
Where we started (v1-4) - On Paper
In practice, for that final beta...we'd

  Cut a branch (when we thought we had everything)
  Merge about ~500 patches (since we didn't have
  everything)
  Spend weeks stabilizing and re-stabilizing issues.
  Ended up working 1-3 months to get a release out the door,
  always certainly missing our 13 week plan.
  And at the end finally shipping something we were happy
  with... but that left us pretty drained (i.e. the bad flow)
The chain of events

  We'd hold releases, sometimes our branch points, until
  features were done, which meant...
  Lots of changes got queued up on the trunk, which
  caused...
  Engineers merging lots of changes to the branch, which led
  to...
  Instability on the branch that we had to deal with and that
  meant...
That chain of events...

  Long lived branches, which...
  Made supporting the branch (i.e. merging changes) more
  difficult as we drifted further from the trunk...
  All of which led to...
       Unpredictable release cycles, with date targets we could
       never hit, and....
       Engineers who were always in a rush to get their features
       into "this" release since we couldn't make any promises
       about the schedule
Which led me to think...

  I need a long vacation.
  There has to be a better way.
  What's causing things not to flow smoothly?




                             ...Yes...In that order.
Primary Goals
 Shorten the release cycle and reduce the life span of a
 branch (make merges easier)
 Make releases more predictable and easier to scope
 Reduce the strain on the entire team




  Good                             Bad
Goal to Plan - Getting down the cycle

  Originally set out a plan to simplify the 12 week cycle, 6
  weeks on dev, 6 weeks on beta, and then to stable.
  Once diagrammed, using blocks to represent the weeks,
  realized that with 3 channels you could have two
  overlapping releases running at once, which could get us a
  stable release ~ every 6 weeks.
Block diagrams
It was a start, but it wasn't the full answer


   Sure, turning the wheel faster would make some things, like
   merges, easier.
   But without addressing the scope problems, the flow
   wouldn't be any better and would continue
   to interrupt releases.
   Things wouldn't be any more predictable and the whole
   cycle would just fall apart (more puddles, less of a stream).
Goal to Plan - Controlling Scope




 The pace of the schedule sets the boundaries for the amount
 of work that can be completed.
 It's important to have specific points in the schedule to
 review features and cut scope.
 Establish clear expectations (and engineering practice) to
 developers that any features not ready to ship at branch will
 be disabled (i.e. we only cut post branch, never add).
Plan to Pitch
  Reached out to the various cross functional groups on the
  Chrome team, who would all be impacted, before
  approaching our executives.
       Engineering, QA, Product, Marketing, Support,
       Localization, Security, etc...
  There were a lot of concerns to address, but the exercise
  forced a lot of thought on the implications of the schedule
  change.
  It took time, but it made the pitch easier to present to the
  leadership and the rest of the team.
Concerns
 Would large feature development still be possible?

 "Yes, engineers would have to work behind flags, however
 they can work for as many releases as they need to and can
 remove the flag when they are done."
 Can the engineers keep up?

 "Their pace won't need to change, since features can be
 disabled there should be no milestone pressure, things ship
 when they are ready."
 What would a world look like where we didn't base our
 marketing on releases?

 "We market features, not releases."
And so we implemented it

       11 week overlapping schedules.
Our General Rules

 The branch point is the end of our development cycle
 Features only get disabled post branch point
 Features should be engineered so that they can be disabled
 easily (1 patch)
 Only stability, security, and critical regressions can ever
 block a release
Things we struggled with
  Overcoming team inertia (particularly when we cut scope)
  Saying no consistently to the team
  Fighting the urge to return to date driven management
  decisions
  We initially didn't solve the trunk->dev->branch time to
  patch problem (daily channel helped).
  Setting the right burn down point (i.e. branch point).
  Disabling features.
Key Lessons


 Having clear and concise rules is important.
 Predictable schedules cut down on communication costs
 and team confusion.
 It takes work and fore-thought to disable features.
 Diligent feature tracking is important.
Conclusion
 Speed alone isn't always the right answer, it's about keeping
 things running smoothly.
 For us, scope was getting in the way of that goal.




We basically wanted to operate more like trains leaving Grand Central Station
(regularly scheduled and always on time), and less like taxis leaving the Bronx
                         (ad hoc and unpredictable).

More Related Content

Similar to Chrome Release Cycle Gets Faster and More Predictable with Overlapping Schedules

Chrome release cycle
Chrome release cycleChrome release cycle
Chrome release cycleJolicloud
 
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)Manuel Padilha
 
White Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release CycleWhite Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release CyclePerforce
 
The Architect's Two Hats
The Architect's Two HatsThe Architect's Two Hats
The Architect's Two HatsBen Stopford
 
How do we drive tech changes
How do we drive tech changesHow do we drive tech changes
How do we drive tech changesJaewoo Ahn
 
2012 - A Release Odyssey
2012 - A Release Odyssey2012 - A Release Odyssey
2012 - A Release OdysseyErnest Mueller
 
From Monolith to Microservices - What Could Go Wrong?
From Monolith to Microservices - What Could Go Wrong?From Monolith to Microservices - What Could Go Wrong?
From Monolith to Microservices - What Could Go Wrong?Phuong Mai Nguyen
 
DevOPs Transformation Workshop
DevOPs Transformation WorkshopDevOPs Transformation Workshop
DevOPs Transformation WorkshopJules Pierre-Louis
 
Questions Log: Dynamic Cubes – Set to Retire Transformer?
Questions Log: Dynamic Cubes – Set to Retire Transformer?Questions Log: Dynamic Cubes – Set to Retire Transformer?
Questions Log: Dynamic Cubes – Set to Retire Transformer?Senturus
 
A Tale of Two Workflows - ChefConf 2014
A Tale of Two Workflows - ChefConf 2014A Tale of Two Workflows - ChefConf 2014
A Tale of Two Workflows - ChefConf 2014Pete Cheslock
 
Scaling Up Lookout
Scaling Up LookoutScaling Up Lookout
Scaling Up LookoutLookout
 
Working with Agile technologies and SCRUM
Working with Agile technologies and SCRUMWorking with Agile technologies and SCRUM
Working with Agile technologies and SCRUMAndrea Tino
 
Agile Development Brown Bag Lunches Slides
Agile Development Brown Bag Lunches SlidesAgile Development Brown Bag Lunches Slides
Agile Development Brown Bag Lunches Slidesguesta1c5d7
 
Agile Delivery Methods And Leadership
Agile Delivery Methods And LeadershipAgile Delivery Methods And Leadership
Agile Delivery Methods And LeadershipRanjith Varghese
 

Similar to Chrome Release Cycle Gets Faster and More Predictable with Overlapping Schedules (20)

Chrome release cycle
Chrome release cycleChrome release cycle
Chrome release cycle
 
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)
 
White Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release CycleWhite Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release Cycle
 
The Architect's Two Hats
The Architect's Two HatsThe Architect's Two Hats
The Architect's Two Hats
 
How do we drive tech changes
How do we drive tech changesHow do we drive tech changes
How do we drive tech changes
 
Kanban Primer
Kanban PrimerKanban Primer
Kanban Primer
 
2012 - A Release Odyssey
2012 - A Release Odyssey2012 - A Release Odyssey
2012 - A Release Odyssey
 
Egov Projects For Fun Profit
Egov Projects For Fun Profit Egov Projects For Fun Profit
Egov Projects For Fun Profit
 
ETCA_8
ETCA_8ETCA_8
ETCA_8
 
From Monolith to Microservices - What Could Go Wrong?
From Monolith to Microservices - What Could Go Wrong?From Monolith to Microservices - What Could Go Wrong?
From Monolith to Microservices - What Could Go Wrong?
 
DevOPs Transformation Workshop
DevOPs Transformation WorkshopDevOPs Transformation Workshop
DevOPs Transformation Workshop
 
Scrum overview
Scrum overviewScrum overview
Scrum overview
 
Questions Log: Dynamic Cubes – Set to Retire Transformer?
Questions Log: Dynamic Cubes – Set to Retire Transformer?Questions Log: Dynamic Cubes – Set to Retire Transformer?
Questions Log: Dynamic Cubes – Set to Retire Transformer?
 
A Tale of Two Workflows - ChefConf 2014
A Tale of Two Workflows - ChefConf 2014A Tale of Two Workflows - ChefConf 2014
A Tale of Two Workflows - ChefConf 2014
 
Intro to Gitflow
Intro to GitflowIntro to Gitflow
Intro to Gitflow
 
Scaling Up Lookout
Scaling Up LookoutScaling Up Lookout
Scaling Up Lookout
 
Working with Agile technologies and SCRUM
Working with Agile technologies and SCRUMWorking with Agile technologies and SCRUM
Working with Agile technologies and SCRUM
 
Agile Development Brown Bag Lunches Slides
Agile Development Brown Bag Lunches SlidesAgile Development Brown Bag Lunches Slides
Agile Development Brown Bag Lunches Slides
 
Agile Delivery Methods And Leadership
Agile Delivery Methods And LeadershipAgile Delivery Methods And Leadership
Agile Delivery Methods And Leadership
 
Kanban Methodology
Kanban MethodologyKanban Methodology
Kanban Methodology
 

More from 36Kr.com

《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)36Kr.com
 
《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)36Kr.com
 
《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)36Kr.com
 
《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)36Kr.com
 
《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)36Kr.com
 
《氪月报》六月刊
《氪月报》六月刊《氪月报》六月刊
《氪月报》六月刊36Kr.com
 
微网介绍36
微网介绍36微网介绍36
微网介绍3636Kr.com
 
图吧新产品介绍 随行 20120613
图吧新产品介绍 随行 20120613图吧新产品介绍 随行 20120613
图吧新产品介绍 随行 2012061336Kr.com
 
益趣课程 Study.33iq.com demo final
益趣课程 Study.33iq.com demo final益趣课程 Study.33iq.com demo final
益趣课程 Study.33iq.com demo final36Kr.com
 
优伴户外旅行(0706)
优伴户外旅行(0706)优伴户外旅行(0706)
优伴户外旅行(0706)36Kr.com
 
1折店简介 --to36氪7.7
1折店简介 --to36氪7.71折店简介 --to36氪7.7
1折店简介 --to36氪7.736Kr.com
 
1折店简介 --to36氪7.7
1折店简介 --to36氪7.71折店简介 --to36氪7.7
1折店简介 --to36氪7.736Kr.com
 
36kr jing fm-0701
36kr jing fm-070136kr jing fm-0701
36kr jing fm-070136Kr.com
 
《氪周刊:互联网创业必读》(第83期)
《氪周刊:互联网创业必读》(第83期)《氪周刊:互联网创业必读》(第83期)
《氪周刊:互联网创业必读》(第83期)36Kr.com
 
4 七维网-chengdu
4 七维网-chengdu4 七维网-chengdu
4 七维网-chengdu36Kr.com
 
09心愿fm-chengdu
09心愿fm-chengdu09心愿fm-chengdu
09心愿fm-chengdu36Kr.com
 
01 扑扑网
01 扑扑网01 扑扑网
01 扑扑网36Kr.com
 
3 咕咚运动--chengdu
3 咕咚运动--chengdu3 咕咚运动--chengdu
3 咕咚运动--chengdu36Kr.com
 

More from 36Kr.com (20)

《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)
 
《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)
 
《氪周刊》(第85期)
《氪周刊》(第85期)《氪周刊》(第85期)
《氪周刊》(第85期)
 
《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)
 
《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)《氪周刊:互联网创业必读》(第84期)
《氪周刊:互联网创业必读》(第84期)
 
《氪月报》六月刊
《氪月报》六月刊《氪月报》六月刊
《氪月报》六月刊
 
微网介绍36
微网介绍36微网介绍36
微网介绍36
 
图吧新产品介绍 随行 20120613
图吧新产品介绍 随行 20120613图吧新产品介绍 随行 20120613
图吧新产品介绍 随行 20120613
 
益趣课程 Study.33iq.com demo final
益趣课程 Study.33iq.com demo final益趣课程 Study.33iq.com demo final
益趣课程 Study.33iq.com demo final
 
魔豆
魔豆魔豆
魔豆
 
优伴户外旅行(0706)
优伴户外旅行(0706)优伴户外旅行(0706)
优伴户外旅行(0706)
 
1折店简介 --to36氪7.7
1折店简介 --to36氪7.71折店简介 --to36氪7.7
1折店简介 --to36氪7.7
 
1折店简介 --to36氪7.7
1折店简介 --to36氪7.71折店简介 --to36氪7.7
1折店简介 --to36氪7.7
 
36kr jing fm-0701
36kr jing fm-070136kr jing fm-0701
36kr jing fm-0701
 
豆果
豆果豆果
豆果
 
《氪周刊:互联网创业必读》(第83期)
《氪周刊:互联网创业必读》(第83期)《氪周刊:互联网创业必读》(第83期)
《氪周刊:互联网创业必读》(第83期)
 
4 七维网-chengdu
4 七维网-chengdu4 七维网-chengdu
4 七维网-chengdu
 
09心愿fm-chengdu
09心愿fm-chengdu09心愿fm-chengdu
09心愿fm-chengdu
 
01 扑扑网
01 扑扑网01 扑扑网
01 扑扑网
 
3 咕咚运动--chengdu
3 咕咚运动--chengdu3 咕咚运动--chengdu
3 咕咚运动--chengdu
 

Recently uploaded

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 

Recently uploaded (20)

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 

Chrome Release Cycle Gets Faster and More Predictable with Overlapping Schedules

  • 1. Chrome Release Cycle Author: Anthony Laforge (laforge@chromium.org)
  • 2. Our Philosophy Think of any major website, even the fancy Web 2.HTML5 ones... do they have version numbers? We took the same approach to our client software as an online web service. That is... we treat releases as a means of getting features out to users and not goals in and of themselves. It's about flow.
  • 3. How Our Users Work Users sign up for Chrome release channels based on the level of stability they want. Updates happen automatically in the background. New features and functionality just flow in without any effort on the user's part.
  • 4. How the Chrome Team Works Developers work almost exclusively on a single central trunk (possible due to our try bots and continuous build infrastructure) Our branches stem from points on the trunk We stabilize branches by pulling in changes from trunk (i.e. everything lands on trunk first)
  • 5. Where we started (v1-4) - On Paper
  • 6. In practice, for that final beta...we'd Cut a branch (when we thought we had everything) Merge about ~500 patches (since we didn't have everything) Spend weeks stabilizing and re-stabilizing issues. Ended up working 1-3 months to get a release out the door, always certainly missing our 13 week plan. And at the end finally shipping something we were happy with... but that left us pretty drained (i.e. the bad flow)
  • 7. The chain of events We'd hold releases, sometimes our branch points, until features were done, which meant... Lots of changes got queued up on the trunk, which caused... Engineers merging lots of changes to the branch, which led to... Instability on the branch that we had to deal with and that meant...
  • 8. That chain of events... Long lived branches, which... Made supporting the branch (i.e. merging changes) more difficult as we drifted further from the trunk... All of which led to... Unpredictable release cycles, with date targets we could never hit, and.... Engineers who were always in a rush to get their features into "this" release since we couldn't make any promises about the schedule
  • 9. Which led me to think... I need a long vacation. There has to be a better way. What's causing things not to flow smoothly? ...Yes...In that order.
  • 10. Primary Goals Shorten the release cycle and reduce the life span of a branch (make merges easier) Make releases more predictable and easier to scope Reduce the strain on the entire team Good Bad
  • 11. Goal to Plan - Getting down the cycle Originally set out a plan to simplify the 12 week cycle, 6 weeks on dev, 6 weeks on beta, and then to stable. Once diagrammed, using blocks to represent the weeks, realized that with 3 channels you could have two overlapping releases running at once, which could get us a stable release ~ every 6 weeks.
  • 13. It was a start, but it wasn't the full answer Sure, turning the wheel faster would make some things, like merges, easier. But without addressing the scope problems, the flow wouldn't be any better and would continue to interrupt releases. Things wouldn't be any more predictable and the whole cycle would just fall apart (more puddles, less of a stream).
  • 14. Goal to Plan - Controlling Scope The pace of the schedule sets the boundaries for the amount of work that can be completed. It's important to have specific points in the schedule to review features and cut scope. Establish clear expectations (and engineering practice) to developers that any features not ready to ship at branch will be disabled (i.e. we only cut post branch, never add).
  • 15. Plan to Pitch Reached out to the various cross functional groups on the Chrome team, who would all be impacted, before approaching our executives. Engineering, QA, Product, Marketing, Support, Localization, Security, etc... There were a lot of concerns to address, but the exercise forced a lot of thought on the implications of the schedule change. It took time, but it made the pitch easier to present to the leadership and the rest of the team.
  • 16. Concerns Would large feature development still be possible? "Yes, engineers would have to work behind flags, however they can work for as many releases as they need to and can remove the flag when they are done." Can the engineers keep up? "Their pace won't need to change, since features can be disabled there should be no milestone pressure, things ship when they are ready." What would a world look like where we didn't base our marketing on releases? "We market features, not releases."
  • 17. And so we implemented it 11 week overlapping schedules.
  • 18. Our General Rules The branch point is the end of our development cycle Features only get disabled post branch point Features should be engineered so that they can be disabled easily (1 patch) Only stability, security, and critical regressions can ever block a release
  • 19. Things we struggled with Overcoming team inertia (particularly when we cut scope) Saying no consistently to the team Fighting the urge to return to date driven management decisions We initially didn't solve the trunk->dev->branch time to patch problem (daily channel helped). Setting the right burn down point (i.e. branch point). Disabling features.
  • 20. Key Lessons Having clear and concise rules is important. Predictable schedules cut down on communication costs and team confusion. It takes work and fore-thought to disable features. Diligent feature tracking is important.
  • 21. Conclusion Speed alone isn't always the right answer, it's about keeping things running smoothly. For us, scope was getting in the way of that goal. We basically wanted to operate more like trains leaving Grand Central Station (regularly scheduled and always on time), and less like taxis leaving the Bronx (ad hoc and unpredictable).