2. Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: triha74@gmail.com
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
3. Agenda
Why do we want to do Continuous Delivery
Intro to Continuous Delivery
Principles of Continuous Delivery
Look at a Pipe
Test Automation for Continuous Delivery
Test Driven Development and Continuous Delivery
Exploratory Testing and Continuous Delivery
Some words on Tooling
Areas not covered
4. Why do we want to do Continuous Delivery
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
5. Why do we want to do Continuous Delivery
Decreased lead time not only improves our time to market
but also our ability to learn and improve through a faster
feedback loop
6. Why do we want to do Continuous Delivery
As soon as we have added value to the application it needs
to be in the hands of the customer or else we have just
added cost.
7. Why do we want to do Continuous Delivery
Reduced risk through delivery of small changes.
8. Are we really delivering as soon as possible?
Pre
Planning
Dev Sys Test Reg Test
Pre
Planning
Dev Sys Test Reg Test
Scrummerfall happens because its hard for developers to have
something deployable for the testers to test.
System testing on something that has not been regression tested is
fundamentally flawed.
Cost of regression test tends to bloat sprint content.
Sprint 2-4 weeks
9. Continuous Delivery
Build Release Deploy Test Prod
Your application always builds, passes its unit tests,
deploys and passes system tests.
10. Continuous Integration/Delivery/Deploy
Continuous Integration
Your application always builds and passes its unit tests.
Continuous Delivery
Your application always builds, passes its unit tests, deploys and passes its
system tests making it always ready to deploy to production through an
automated process
Continuous Deployment
Your application automatically deploys to production through a fully
automated process
11. Continuously as in all the time?
Yes!
End of sprint isn't continuously.
Weekly isn't continuously.
Nightly isn't continuously.
Every code commit is continuously!
12. What Tests? Unit? Component? System?
Yes!
Unit
Component
System
Integration
Rollback
Load
Failover
13. Principles of Continuous Delivery
1. The process MUST be repeatable, reliable and
independent of key personnel
22. The pipe - Build once!
Build Release
Build and Unit test then release it!
23. The pipe - Release everything!
Build Release
Build Release
Build Release
Build Release
Code
DB Scripts
Server Config
Deploy script
Feedback Feedback
24. The pipe -Bundle released artifacts
Build Release
Build Release
Build Release
Build Release
Assembly
Feedback
25. The pipe - Use Same Deploy Mechanism
Function TestAssembly Deploy
DB
Server
Function Test
Function Test
Pipe Status
Feedback
Feedback Feedback
26. The pipe - Use Same Deploy Mechanism
Deploy
DB
Server
Pipe Status
Server
DB
Server Server
DB
Server Server
Deploy
Pre Prod Prod
UAT
FeedbackFeedback Feedback
27. Continuous Delivery
Build Release Assemble Deploy
Deploy
PreProd/Prod
Test
Summary
We build once
Release everything
Automate everything
Fast feedback
Automate all tests except UAT
28. Obvious benefits
Continuous Regression Testing gives instant feedback.
Continuously deploying to test servers tests deploy mechanism several
hundred times per release.
Always ready to push new release into UAT
We can get customer feedback as soon as value has been added
31. Testing in the past
What we have done
100s of manual tests that..
... verify our requirements
... ensures bad input doesn't break system
... data is stored in the right place
... find bugs
32. Testing in the past
We have automated it!
We have tried to automate the 100s of manual tests that..
... verify our requirements
... ensures bad input doesn't break system
... data is stored in the right place
... find bugs
We do it through instrumenting the GUI and verifying
Database, GUI and System resources.
With very bad results....
33. Robust and Fast Test Automation
Tests that execute 100s of times per day have to be...
... executable without any human interaction
... robust and give the same response every single time
... robust and have to survive refactoring
... fast and parallelizable as adding 10 tests should not slow
down the feedback
34. We need to change the way we look at Test
On System Level we do verification of Use Case
Acceptance Criterias. Requirement Verification.
On Component Level we do testing of Component
Functionality. Testing.
35. We need to change the way we look at
Architecture
Remember “Build quality in”.
We need to improve testability of our applications through...
...isolation of functionality into independent services with
life cycles of their own.
...building interfaces for defined test points.
40. Continuous Delivery
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
100s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified
41. Test Driven Development
Pre
Planning
Dev Sys Test Reg Test
Pre
Planning
Dev Sys Test Reg Test
Scrummerfall happens because its hard for developers to have
something deployable for the testers to test.
System testing on something that has not been regression tested is
fundamentally flawed.
Cost of regression test tends to bloat sprint content.
Sprint 2-4 weeks
42. Test Driven Development
Pre
Planning
Dev
Test
Automation
Reg Test
Design Requirement Verification and Tests on multiple levels with the feature.
Develop test automation in parallel with feature development.
Cross functional pairing to improve quality of Tests, Test Architecture and Feature
Testability
Pre
Planning
Reg Test Reg Test
Verification
Verification
43. Test Driven Development
Who should automate the tests?
Test Design - Done by Testers
Test Automation - Done by ????
Coding - Done by Developers
44. Test Driven Development
Tests as Code
Test Automation IS Code and needs to be treated as first
class application.
Test Architecture is super important and needs to be
worked with in parallel with Application Testability
Architecture.
45. Test Driven Development
Tests as Code
Internal
Service
DSL
Glue Code
SpecificationText or Code based Specification
Implementation of Specification
Domain Specific Language packaging
service testability
46. Test Driven Development
Internal
Service
DSL
Glue Code
Specification
Give a valid user name and password
when login is called
then user is authorized
boolean authorized = loginService.authorizeUser(“triha”,”cdrockz”);
assert(authorized, true);
public boolean authorizeUser(String userName, String password){
//Implementation of REST call to internal service
return true;
}
47. Test Driven Development
Tests as Code
Specification & Glue Code IS the Test
DSL is testability
Test Automation is Code
48. Test Driven Development
Pre
Planning
Dev
Test
Automation
Reg Test
Specifications of Requirement Verification written during Pre-Planning.
Specification of Component Tests written during Pre-Planning.
Implementation of DSL and Glue Code in parallel with feature development.
Implementation of Component Tests in parallel with feature development.
Pre
Planning
Reg Test Reg Test
Verification
Verification
49. Test Driven Development
Tomorrow Agile Team Member
Developers with Testing skills
Testers with Development skills
WILL BE SUPER HOT AND EARN SUPER NICE $$$$
Will be the only ones who get assignments
53. Exploratory Testing
Exploratory Testing should be...
...done without knowledge of requirements.
...in time based sessions with specific focus area.
...outside the release lifecycle of the application.
54. Benefits
Continuous Regression testing gives instant feedback.
Continuously deploying to test servers tests deploy mechanism several
hundred times per release.
Always ready to push new release into UAT
We can get customer feedback as soon as value has been added
Test Driven Development
Everyone is responsible for Quality
Sustainable Quality
55. Tools Used
We love to talk about what tools are best and spend ages arguing about it.
We need to understand what we must do before we pick the tools.
Still which ever tool we pick it will be insufficient and crap.
56. Tools Used
Don't use one tool to rule them all.
Cucumber and Fitnesse are ok for system level testing
RESTAssured, JUnit or pure Groovy, Ruby or Python scripts are often better for
component level.
For Load testing one should use load testing tools such as JMeter
Continuous Delivery Process should execute tests through interface such as
maven or gradle so that it can be Test Tool agnostic.
57. Areas Not Covered
To do Continuous Delivery well we also have to...
Reproducible Test Environments
Production like Test Environments
“Non functional requirements” as functional requirements
A/B testing
Power of Metrics
and lots more...