Continuous Deployment - Lean LA

Editor's Notes

1. Hi - I am Ash with WiredReach and I'm going to be presenting a case-study on how I transitioned from a traditional development process to continuous deployment. Just to get a sense of who is in the room, How many people here know what Continuous Deployment is? And how many people practice Continuous Deployment today?
2. If I were to ask you “How do you maximize progress in a lean startup?”, most of you would probably say:
3. If I were to ask you “How do you maximize progress in a lean startup?”, most of you would probably say:
4. The “lean” in lean startup is not about being cheap but about speed. It’s about having a strong initial vision and then systematically testing it through tightly defined experiments.
5. When you first hear: “get out of the building and talk to customers”, it’s easy to think that all you have to do is run a bunch of surveys or focus groups to gather feature requests from customers. It’s NOT. A Lean Startup is NOT about asking customers what they want, but measuring what they do.
6. But most important of all, a Lean startup is about focus. Startups are inherently chaotic but at any given point in time, there are only a few actions that truly matter. A Lean Startup focuses on those and ignores the rest.
7. If I were to ask you “How do you maximize progress in a lean startup?”, most of you would probably say:
8. So lets take a look at where in the development process we learn about customers. Some of this learning happens during the requirements stage in the form of customer discovery when we’re out talking to customers and figuring and out what to build. But most of this learning happens only after we get the product into customers’ hands in the form of customer validation. There is very little learning during development and QA. Sure we learn about other things, just not about customers.
9. So lets take a look at where in the development process we learn about customers. Some of this learning happens during the requirements stage in the form of customer discovery when we’re out talking to customers and figuring and out what to build. But most of this learning happens only after we get the product into customers’ hands in the form of customer validation. There is very little learning during development and QA. Sure we learn about other things, just not about customers.
10. So lets take a look at where in the development process we learn about customers. Some of this learning happens during the requirements stage in the form of customer discovery when we’re out talking to customers and figuring and out what to build. But most of this learning happens only after we get the product into customers’ hands in the form of customer validation. There is very little learning during development and QA. Sure we learn about other things, just not about customers.
11. Even though building a product is the purpose of a startup, you could say that Product Development actually gets in the way of learning about customers.
12. While we obviously can’t eliminate development and QA, we can shorten the cycle time from requirements to release so we can get to the learning parts faster. That is exactly what Continuous Deployment does. Continuous Deployment shortens the cycle time it takes to build, test, and deploy features. Shorter cycle times mean we get feedback faster and learn faster.
13. While we obviously can’t eliminate development and QA, we can shorten the cycle time from requirements to release so we can get to the learning parts faster. That is exactly what Continuous Deployment does. Continuous Deployment shortens the cycle time it takes to build, test, and deploy features. Shorter cycle times mean we get feedback faster and learn faster.
14. While we obviously can’t eliminate development and QA, we can shorten the cycle time from requirements to release so we can get to the learning parts faster. That is exactly what Continuous Deployment does. Continuous Deployment shortens the cycle time it takes to build, test, and deploy features. Shorter cycle times mean we get feedback faster and learn faster.
15. While we obviously can’t eliminate development and QA, we can shorten the cycle time from requirements to release so we can get to the learning parts faster. That is exactly what Continuous Deployment does. Continuous Deployment shortens the cycle time it takes to build, test, and deploy features. Shorter cycle times mean we get feedback faster and learn faster.
16. While we obviously can’t eliminate development and QA, Lean Startups shorten the cycle time from requirements to release by continuously deploying features and involving customers throughout the development process. These shorter cycle times translate to faster feedback and faster learning for the startup.
17. Here’s an overview of what I’ll be talking about today. I’ll describe what my development process looked like before and after continuous deployment. I’ll talk about how I got started and how I build features now.
18. So, before Continuous Deployment, we used to release on a 2 week cycle which is fairly fast by most standards. We used to spend about a week in development and then QA tested for another 2-3 days before we deployed to customers. Now we release multiple times a day.
19. Before Continuous Deployment, we had a common staging area that mirrored production. This is what we used for development and QA. It worked fine in the early days but as we grew, coordinating around a single staging area became a problem. Now we build complete standalone sandboxes for development and QA. Each developer has the complete system on their workstation which gives them a lot more freedom to experiment. Each QA machine also runs on a standalone sandbox which makes it easier for us to do isolated testing as well as scale the testing infrastructure horizontally.
20. Before Continuous Deployment, releases were all day events. We would do a code freeze typically on a Thursday and spend most of Friday building, testing and packaging a release. Now a release is triggered automatically every time we commit code. The system is run against a battery of tests and only deployed into production if it passes all the tests. We constantly monitor the production environment and can tell good changes from bad changes quickly and revert a release if we need to. Our release process currently takes about 20 minutes.
21. Before Continuous Deployment, the average size of a release was measured in hundreds of lines of code. Now a typical release is under 25 lines of code.
22. When you are committing 25 lines of code versus hundreds per release, the impact on the system is much more localized. This has led to fewer production emergencies and less firefighting for us.
23. But most important of all, if you are a technical founder like me, you constantly have to trade-off outside the building activities like customer development against inside the building activities like product development. Before Continuous Deployment, I had to schedule my week for coding days and customer days. Now I can do both in the same day. I schedule my coding in 2 hour blocks usually early in the day. That leaves the rest of the day open for everything else.
24. So, Continuous Deployment sounds great in theory.
25. But taking the plunge was still scary. The biggest reason for us was having the feeling of there being no safety net. With a traditional development process, there is a QA cycle before deployment which provides a safety net of time and there is some comfort in sharing testing responsibility with someone else.
26. But taking more time in QA was not always optimal for us: - despite our best testing efforts, bugs still crept into production - bugs got more expensive and harder to fix the longer we waited - but most important of all, we felt we weren’t learning anything about customers during that time The answer isn’t taking more time in QA but less through test automation and getting better at detecting and fixing issues in production.
27. But taking more time in QA was not always optimal for us: - despite our best testing efforts, bugs still crept into production - bugs got more expensive and harder to fix the longer we waited - but most important of all, we felt we weren’t learning anything about customers during that time The answer isn’t taking more time in QA but less through test automation and getting better at detecting and fixing issues in production.
28. But taking more time in QA was not always optimal for us: - despite our best testing efforts, bugs still crept into production - bugs got more expensive and harder to fix the longer we waited - but most important of all, we felt we weren’t learning anything about customers during that time The answer isn’t taking more time in QA but less through test automation and getting better at detecting and fixing issues in production.
29. But taking more time in QA was not always optimal for us: - despite our best testing efforts, bugs still crept into production - bugs got more expensive and harder to fix the longer we waited - but most important of all, we felt we weren’t learning anything about customers during that time The answer isn’t taking more time in QA but less through test automation and getting better at detecting and fixing issues in production.
30. But taking more time in QA was not always optimal for us: - despite our best testing efforts, bugs still crept into production - bugs got more expensive and harder to fix the longer we waited - but most important of all, we felt we weren’t learning anything about customers during that time The answer isn’t taking more time in QA but less through test automation and getting better at detecting and fixing issues in production.
31. So lets take a look at how we got started.
32. Here’s some background on WiredReach and the type of products we build. We’ve been in business for 7 years and have launched 2 products - BoxCloud and CloudFire. The first product BoxCloud was built using a release early, release often methodology while CloudFire was built using Lean Startup techniques. Both products are hybrid web/desktop applications. What I mean by that is they have both a web component and a downloaded client component which runs on mac and windows.
33. Here’s some background on WiredReach and the type of products we build. We’ve been in business for 7 years and have launched 2 products - BoxCloud and CloudFire. The first product BoxCloud was built using a release early, release often methodology while CloudFire was built using Lean Startup techniques. Both products are hybrid web/desktop applications. What I mean by that is they have both a web component and a downloaded client component which runs on mac and windows.
36. Another problem with tests is that over time, they get out of date and start failing. I’ve worked in places where developers start ignoring these which is a slippery slope as the problem only keeps getting worse. In Continuous Deployment, these tests are your only line of defense before deploying code, so you have to take failing tests very seriously. We only deploy a release if it passes all the tests. Otherwise, we stop and fix the tests first.
43. The first and most important practice we adopted was fitting releases into small batch sizes. Coding in small batches is the key concept in continuous deployment and can directly be attributed back to shorter cycle times, faster feedback, and a better work flow. For me, a small batch is the output of a 2 hour work block. We can not always build a full feature in 2 hours but we have gotten good at deploying features incrementally. We start with non-user facing changes first, like api updates and database changes, before building user-facing changes. Even deploying these changes early greatly help to lower the integration risk of a feature.
44. Before Continuous Deployment, the average size of a release was measured in hundreds of lines of code. Now a typical release is under 25 lines of code.
45. When you are committing 25 lines of code versus hundreds per release, the impact on the system is much more localized. This has led to fewer production emergencies and less firefighting for us.
53. We now have a practice of writing a new functional test with every user facing change, but we didn’t start that way. We got started by writing a single test for the user activation flow first. This is the path users take when they first signup, download and interact with our product. If something goes wrong here, nothing else matters after that.
55. The next thing we did was not try to achieve 100% automation from the start. We kept deploying these small batch releases manually for a while and audited everything we did. This helped us build confidence in the process and overcome some of the initial fear of loosing control. We already had a continuous integration server that ran a large collection of unit tests but once we took out the formal QA step, we found ourselves preferring functional tests over unit tests since they were much better at reflecting what users actually did with the system.
58. You’ll eventually want to build one of these cluster immune systems (just hopefully not that one) that can automatically tell good changes from bad ones and do something about it. But here too, it’s important to build it out incrementally. We got started simply monitoring the health of production servers using off-the-shelf tools like ganglia and nagios. Over time we built other application and business level monitoring into it. We did this mostly reactively to production issues. 5Whys.
66. Lastly, I want to spend some time talking about how I build features. Continuous Deployment shortens the cycle time it takes to deploy features but how do you make sure you’re actually building what customers want and not simply cranking out new features faster. Here are some rules we follow.
67. Features must be pulled not pushed. If you’ve followed a customer discovery process, identified your top 3 problems, and defined a mvp, you do not need to push more features until you have validated your mvp. This doesn’t mean you stop development, but most of your time should be spent measuring and improving existing features versus chasing after new problems to solve. From experience, I know this can be a hard rule to enforce and the next rule helps with that.
68. A good practice for ensuring the 80/20 rule is constraining the features pipeline. This is a common practice from Agile and Kanban, but with the addition of a validated learning state. Here’s how it works: Ideally, a new feature must be pulled or tested with more than one customer for it to show up in the backlog. The number of features in-progress is constrained by the number of developers and so is the number of features waiting for validation. This ensures that you cannot deploy a new feature until a previously deployed feature has been validated.
69. So how do you validate a feature? Unless you have a lot of traffic, quantitative metrics can take some time to collect. For this reason, I prefer to validate a feature qualitatively first. Once a feature goes live, I directly contact customers that expressed interest in that feature and ask them for feedback. It’s important not just to test the coolness factor of a feature but that it actually solves a customer problem and more importantly makes or keeps the sale. If I don’t get a strong initial signal, I try and figure out why and either improve or kill the feature. We use google website optimizer, KISSmetrics, Mixpanel, and homegrown scripts to collect quantitative data. It’s important here too, to focus on the macro and track key metrics over time, like revenue and retention, versus just clicks.
70. So when is the best time to adopt Continuous Deployment? I believe the ideal time is as early in the product development cycle as possible - when you are small and have a few or even no customers. Continuous Deployment is an incremental process that you have to practice to get really good at. But even adopting simple practices like “coding in small batch sizes” paid off for us very quickly. The biggest benefit we have derived from Continuous Deployment is the ability to integrate Customer Development with Product Development. The fundamental call to action from Continuous Deployment is to “Ship More Frequently”. Once you take that first step, what you need to do next becomes clearer. Thank you.