Digital disruption is omnipresent. Organizations are asked (more than ever before) to constantly reinvent their businesses at the speed of ideas and to keep their customers at their epicenter at the same time. Venerable companies have become comfortable "breaking down silos", changing cultural norms and driving efficiency at all costs to always be one step ahead of their competition. It is not any longer the strongest that survives, nor the most intelligent that survives. It is the one that is most adaptable to change. Change seems to be the only thing constant, the new currency, the silver bullet to stay competitive. Big words, such as DevOps, emerged out of this fear to be second-class, and even managed to penetrate into our daily testing routines. The ability to change fast while delivering high quality has become imperative for all businesses (small, medium, large). This level of transformation requires testing to no longer be a bottleneck. It requires continuous testing to make sure that the quality of the software that is being developed is measured continuously. In this talk, Ingo Philipp outlines how to unfold continuous testing into reality. ​

1. @IngoPhilipp
Continuous Testing
Digital Transformation Requires Continuous Testing
Ingo Philipp
@IngoPhilipp

2. @IngoPhilipp
Increased
Efficiency
Reduced
Costs
Accelerated
Innovation
Enhanced Customer
Interaction
Higher Employee
Productivity
Enhanced Customer
Experience
‹› Digital transformation, a catch-all phrase.@IngoPhilipp

3. @IngoPhilipp
Businesses must
continuously exploit
digital technologies to
both create new sources
of customer value and
increase operational
agility in service of
customers.
∷ Forrester Research
‹› Digital transformation, a catch-all phrase.
Increased
Efficiency
Reduced
Costs
Accelerated
Innovation
Enhanced Customer
Interaction
Higher Employee
Productivity
Enhanced Customer
Experience
@IngoPhilipp

4. @IngoPhilipp
Software fuels digital transformation,
and so accelerates digital disruption
Ingo Philipp
‹› Digital transformation is a software-led transformation.@IngoPhilipp

5. @IngoPhilipp
Owns no vehicles Owns no real estate
Everyone can be a taxi driver Everyone can be a landlord
World's largest taxi company World's largest accommodation provider
Their value is all digital. This digitalization creates new
ways of business operations, and customer interactions

6. @IngoPhilipp
Plan Code Version Build Provision Test Deploy Monitor
∷ Melissa Perri
The product is a solution.
If the problem isn't solved,
the product doesn't work.
The goal is to
accelerate this
value exchange
Identify
Needs
Receive
Money
Build
Solution
Your Company Your Customer

7. @IngoPhilipp
Requirements Design Implementation Testing Acceptance Deployment
Inception Elaboration Construction Transition
Development Operations
DevOps
Agile Horizon
Digital Singularity
Incremental
Rational Unified Process
4
Agile
Scrum, Kanban
2
Continuous
DevOps
1
6+
Big Bang
Waterfall
Flexibility
Rigidity
Following a Plan
Contract Negotiation
Copious Documentation
Processes & Tools
Individuals & Interactions
Working Software
Customer Collaboration
Responding to Change

8. It’s not the strongest
that survive, nor the most
intelligent, but the one most
responsive to change
∷ Charles Darwin
‹› Software is not written, it is rewritten.@IngoPhilipp

9. @IngoPhilipp
∷ Dan Ashby ‹› We're testing everywhere!
Dev Ops
Now then, we
test here
We also
test here
And we test
here too Can we test here?
Of course we can!
And here of
course we test
Oh yeah, we
test here
And here
we test
We then
test here
‹› We all have the same finish line. Testing must not be an isolated, disconnected island in the company's landscape.

10. @IngoPhilipp
The bitterness of poor software quality remains long
after the sweetness of fast software delivery is forgotten
Ingo Philipp
‹› Poor software quality will erase your speed gains in the long run.
∷ Inspired by Benjamin Franklin
@IngoPhilipp

11. @IngoPhilipp

12. @IngoPhilipp
80%
Manual
20%
Automated
The reliance on
manual testing is the
top technical challenge
in app development
World Quality Report, 2015/16
Test case automation
requires developers
UI LoadAPI
Present
Traditional Testing

13. @IngoPhilipp
Future
Continuous Testing
+85%
Automated
Exploratory Tests
UI Checks
API Checks
Load Checks
80%
Manual
20%
Automated
Present
Traditional Testing
Service Virtualization

14. @IngoPhilipp
100%
0%
Sprint
Start
Sprint
End
API
UI
Start your automation
journey earliest possible
InterfaceCompleteness
Implementation Time
4x
Faster Creation
6x
Less Maintenance
20x
Faster Execution

15. @IngoPhilipp ‹› Enterprise environments are alike disease gene networks.
Welcome to the
Tester's Hell
@IngoPhilipp

16. Make your automation fast through APIs,
make it stable through service virtualization
Ingo Philipp
‹› Our two bosom automation buddies.@IngoPhilipp

17. @IngoPhilipp
Automation doesn't make testing
easy, it makes testing possible
Wolfgang Platz
‹› Automation is the dictate of today, but it's not a silver bullet to testing.@IngoPhilipp

18. @IngoPhilipp
A
B
100 Billion Billion Billion Billion Billion Billion Billion Combinations
‹› You can't test everything, but you can always test as much as possible.@IngoPhilipp

19. @IngoPhilipp
Testing harder isn’t the
answer, testing smarter is!
Wolfgang Platz
‹› Do the right things right at the same time.@IngoPhilipp

20. @IngoPhilipp
Testing is no more about test cases
than astronomy is about telescopes
Ingo Philipp
‹› A recipe (test case) is not cooking (testing).
∷ Inspired by Edsger Wybe Dijkstra & Michael Bolton
@IngoPhilipp

21. @IngoPhilipp
If we’re testing badly, then automation can
help us to test badly at an accelerated rate
Michael Bolton
‹› Automation accelerates, intensifies, amplifies, often simplifies, and sometimes even enables testing.
@vertaontest ‹› If you automate chaos, all you get is faster chaos.
@IngoPhilipp

22. @IngoPhilipp
∷ Inspired by Brendan Leonard
Testing? There's only automation!
We automate everything
OK, there's testing, and testing
is probably more than just automation.
We're just not convinced that we really need it
Oops
Damn!
‹› Confusing testing with automation is a dangerous thing.

23. If you don't pay attention to what
has your attention, it will take more of
your attention than it deserves
∷ David Allen ∷ Getting Things Done
‹› Sooner or later, testing grabs your attention. Pay attention to testing, not just to automation.@IngoPhilipp

24. Test Case
@IngoPhilipp

25. Passing
Test Case
@IngoPhilipp

26. Passing
Test Case
@iamdevloper
@IngoPhilipp

27. @IngoPhilipp
© BBC
@IngoPhilipp

28. @IngoPhilipp
Testing is exactly like
washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.
∷ Inspired by Luke Sullivan © BBC
@IngoPhilipp

29. @IngoPhilipp
Testing is exactly like
washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.
∷ Inspired by Luke Sullivan
‹› We actually wash a performing pig.@IngoPhilipp

30. @IngoPhilipp
Testing is exactly like
washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.
∷ Inspired by Luke Sullivan
That’s what we can know
‹› We actually wash a performing pig.@IngoPhilipp

31. @IngoPhilipp
Testing is exactly like
washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.
∷ Inspired by Luke Sullivan
That’s what we can know
That’s what we know
That’s
what we
check
Problem
Knowledge Gap
© Richard Bradshaw
@IngoPhilipp ‹› We actually wash a performing pig.

32. @IngoPhilipp
∷ Inspired by Luke Sullivan
The purpose of testing is to
close the knowledge gap
The goal is information,
not gratuitous automation
Testing is and always will be
a search for information
That’s what we can know
That’s what we know
That’s
what we
check
Problem
Knowledge Gap
© Richard Bradshaw
@IngoPhilipp

33. @IngoPhilipp
That’s what we can know
That’s what we know
That’s
what we
check
Problem
Knowledge Gap
© Richard Bradshaw
∷ Cem Kaner
We are an efficient
information service
We enable business people
to make business decisions
We are in situ investigators,
we are like Sherlock Holmes
@IngoPhilipp

34. @IngoPhilipp
That’s what we can know
That’s what we know
That’s
what we
check
Problem
Knowledge Gap
© Richard Bradshaw
@IngoPhilipp

35. @IngoPhilipp
That’s what we can know
That’s what we know
That’s
what we
check
Problem
Knowledge Gap
© Richard Bradshaw
We go to crime scenes
(software) and search for
evidence (risks) to enable the
police (developers) to arrest
(fix) the culprits (bugs)
@IngoPhilipp

36. @IngoPhilipp © www.wikipedia.org – How to use toilet paper?
OverUnder
Confirmation
« Demonstrate your depth of knowledge »
Exploration
« Demonstrate your breadth of knowledge »
@IngoPhilipp

37. Mechanical Testing
Process pre-defined data in pre-designed steps
Monitor Known Risks
Confirm what you already know
Low Information Value
Check what you have already learned
Change Detector
Confirmation
« Demonstrate your depth of knowledge »
Exploration
« Demonstrate your breadth of knowledge »
Creative Testing
Create new test ideas based on what you have learned
Analyze Potential Risks
Focus on the things you don’t know
High Information Value
Learn something new
Problem Detector
∷ Bolton ‹› Kaner ‹› Bach
@IngoPhilipp

38. @IngoPhilipp
Checking
« Requires Processing »
Evaluate a product by applying algorithmic
decision rules to specific observations of a product
Change Detector
∷ Bolton ‹› Kaner ‹› Bach
Exploration
« Demonstrate your breadth of knowledge »
Creative Testing
Create new test ideas based on what you have learned
Analyze Potential Risks
Focus on the things you don’t know
High Information Value
Learn something new
Problem Detector
@IngoPhilipp

39. @IngoPhilipp
Checking
« Requires Processing »
∷ Bolton ‹› Kaner ‹› Bach
Evaluate a product by applying algorithmic
decision rules to specific observations of a product
Change Detector
Exploring
« Requires Thinking »
Problem Detector
Evaluate a product by learning about it
through exploration and experimentation
@IngoPhilipp

40. @IngoPhilipp
Checking
« Requires Processing »
∷ Rich Rogers
Verify through
Instructions
Pay attention to
Deviations
Create
Test Cases
Follow
Procedure
Examine
Requirements
Factory
Process
Monitor Known Risks
Exploring
« Requires Thinking »
Problem Detector
Evaluate a product by learning about it
through exploration and experimentation
@IngoPhilipp

41. @IngoPhilipp
Checking
« Requires Processing »
∷ Rich Rogers
Verify through
Instructions
Pay attention to
Deviations
Create
Test Cases
Follow
Procedure
Examine
Requirements
Factory
Process
MechanicalProcess Monitor Known Risks
Investigate through
Experiments
Pay attention to
Oracles
Create
Test Ideas
Follow
Clues
Examine
Risks
Adaptive
Investigation
Analyze Potential Risks
CreativeProcess
Exploring
« Requires Thinking »
@IngoPhilipp

42. Checking
Efficient Confirmatory Testing
Exploring
Effective Exploratory Testing
Testing
Thorough Testing
+
=
Agile
Testing Equation
∷ Elisabeth Hendrickson
‹› The testing coin has two sides.@IngoPhilipp

43. @IngoPhilipp ‹› If you think that this will happen soon to you then you're probably doing testing in the wrong way.
∷ Picture by Shelly Palmer
@IngoPhilipp

44. Testing is not so much a thing you do,
it’s far more a way you think
Michael Bolton
‹› Change the way you think to change the way you test.@IngoPhilipp

45. Whatever your job title, you
are actually being paid to think
Seb Rose
‹› Testing happens in our heads, not in our tools.
Tester
QA Engineer
Test Engineer
DevOps Tester
Agile Tester
QA Tester
SDET
Test Automation Developer
Automation Tester
Test Analyst
QA Analyst
Quality Engineer
Manual Tester
Test Consultant
Test Strategist
…
@IngoPhilipp

46. @IngoPhilipp
If it doesn't make an impact, then
its absence doesn’t make a difference
Trey Smith
@IngoPhilipp

47. @IngoPhilipp
100%
Manual
Testing
0%
Automated
Testing
Testing
Present
There’s a better
way to do it. Find it!
Project Charter
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution

48. @IngoPhilipp
Risk Coverage Optimization Test Data Management UI/API Automation Continuous Integration & Distributed Execution
48
Hours
Execution
Single Agent
8
Hours
Execution
Multiple Agents
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution
89%
Business Risk
Coverage
1193
Manual
Test Cases
37%
Business Risk
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
53%
Business Risk
Coverage
Smoke Testing
34
Minutes
Execution
Smoke Testing
Test Case Count 7%
75% Redundancy
64% API Tests
50% Effort Test Data

49. @IngoPhilipp
8
Hours
Execution
Multiple Agents
72%
Reduced Critical Defects
Production Defects
89%
Business Risk
Coverage
1193
Manual
Test Cases
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
34
Minutes
Execution
Smoke Testing
53%
Business Risk
Coverage
Smoke Testing
Test Case Count 7%
64% API Tests
Testing happens in our
heads, not in our tools
Exploratory Testing
Risk Coverage Optimization Test Data Management UI/API Automation Continuous Integration & Distributed Execution
50% Effort Test Data

50. @IngoPhilipp
Your testing future doesn’t just happen,
it gets happened, so make it happen
Ingo Philipp
‹› Your testing future depends on your testing present.
∷ Inspired by Paul Tagliablue
@IngoPhilipp

51. @IngoPhilipp
Questions
The show is over. It’s your turn.
?
@IngoPhilipp