This document discusses brittle tests and strategies for addressing brittle tests. Brittle tests are tests that are fragile and produce inconsistent results. They require high maintenance costs to debug and fix. They can hide real bugs and provide useless feedback. This undermines trust in testing and team morale. The document recommends moving to a testing pyramid with more unit tests and fewer integration/end-to-end tests. It also suggests improving test design, reducing technical debt, using consistent environments, avoiding hard-coded data, implementing continuous testing, and adopting a team approach to quality.
2. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Moral of the Story
3. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
What exactly is a
brittle test?
4. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
What exactly is a brittle test?
Fragile, flaky, easy to break
Not deterministic result
Run 1
Input
Run 2
Run 3
Run 4
Not due to only the test code
5. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Raise your hand if
you ever had to deal
with brittle tests!
6. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
What are the
consequences
of brittle tests?
7. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
High Maintenance Cost
Keyword
test 1
test 2
test 3
test 4
75% Fail
Rollercoaster result trend
8. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
High Maintenance Cost
Debug and manual test
Fix broken tests
9. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Real Example
Test 1...50 Precondition Step Assert
Nightly regression tests (1000+)
Most of the time working
This time many tests failed
RELEASE DROP
What would you do?
10. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Hidden Fail
Test 1 Precondition Step Assert
Test 50 Precondition Step Assert
Cognitive bias: test fails only because of the precondition
What if a real fail has been hidden?
Test 1 Precondition Step Assert
Test 50 Precondition Step Assert BUG!
12. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Low Trust - Low Morale
Worthless feedback ignored
Maintenance avoided
Tests get even more broken
Automation is not trusted
V
i
c
i
o
u
s
L
o
o
p
Effort with no
results
No result ==
No commitment
13. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
What makes our
tests brittle?
14. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
How can we defeat
the Dragon?
15. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Ice Cream
Testing Pyramid Real world: Ice Cream
Practical Test Pyramid Ham Vocke - Martin Fowler’s blog
16. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Ice Cream
Move
Down to the Pyramid
Tests
Part of tests
Setup & Teardown
Diamo una chance alle codebase legacy Nicola Mincuzzi
18. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Multipurpose / Holistic
“There should never be more than
one reason for a class to change”
Robert C. Martin (Uncle Bob)
“There should never be more than
one reason for a test to fail”
Single Responsibility Principle
19. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Multipurpose / Holistic
Test 1 Registration Check User
Single assert
Single Responsibility
Setup 1 Checkout
Setup 2 Streaming
Test 1 Registration
Test 2 Login
Test one thing
20. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Poor Design
Sleep
Best effort automation
Quick and Dirty
Longer learning curve
21. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Poor Design
Invest in automation
Agree on development guidelines
(Pekka’s how to write good test cases)
How to write good test cases using Robot Framework Pekka Clark
22. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Poor Design
Reduce technical debt
Pair development
Boy Scout Rule:
“Always leave the code you are editing better
than you found it” Robert C. Martin (Uncle Bob)
23. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Good and Bad Technical Debt
“Anything about your code that slows you down over the long term.
Hard-to-read code, duplication, tangled dependencies, etc.”
Good And Bad Technical Debt Henrik Kniberg
24. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Unreliable and slow environment
Undersized
Environment
Front End Developer
Dev - Ops experiment
Business test
Backend Mad Developer
(internal server error hero)
Super complicated auto implosion mega query
Third part never-working services
25. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Unreliable and slow environment
Dedicated test environment
Pay for limitless resources
26. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Unreliable and slow environment
Mock third party systems
Strive for performances
27. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Hard coded data
Limited products
Test product: 3E24B278-D1EC-358C-ADE5-F51703BBB342
28. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Hard coded Data
Data Preparation
Data Targeting
Teardown Matters
29. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
No continuous Testing
Test suite needs to be executed
People is lazy forget to run
Not comparable results
30. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
No continuous Testing
Continuous testing to collect feedback
31. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Remove/reduce barriers to test execution
No continuous Testing
32. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Daily Standup
No continuous Testing
Include test feedback
in release process
Discuss reports together
35. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Quality assistance
Whole team in charge for quality
More TDD and pair programming
Quality Assistance over
Quality Assurance
BDD and TDD Outside in
36. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Robot Framework BDD
*** Settings ***
Library SeleniumLibrary
Resource settings.resource
*** Test Cases ***
App Game Available
Given The Browser
When Navigate To The App Game
Then Title Should Be Tic Tac Toe
*** Keywords ***
Given The Browser
Open Browser ${EMPTY} ${BROWSER}
When Navigate To The App Game
Go To ${APP_URL}
37. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
Quality Assistance Team
Angelo Caovilla
Roberta Massimino
Fabrizio Premoli
Emanuele Bonanno
Art Director
Matteo Villa
All CHILI ex-colleagues
Special Thanks to
