Nauka na błędach – root cause analysis w praktyce.

1. ROOT CAUSE
ANALYSIS
@kasia_balcerzak
LEARN BY FAILING

2. WHO AM I?
➤ Quality Engineer @ Spartez
➤ Testing traveler in Atlassian
products.
➤ Root Cause Analysis
Facilitator.
➤ Teaches developers how to
test.

4. What can we do
to find similar bugs?
What is the root of bugs?

8. OH S**T…

10. CREATE
USER
ASSIGN
USER
PROCESSING EVENTS
QUEUE

11. PROCESSING EVENTS
QUEUE
t1 t2
t2
t1

12. WHOSE FAULT WAS IT?

13. PLAY THE BALL,
NOT THE PLAYER!

14. create
timeline

15. GATHER
FACTS TO BE
OBJECTIVE IN
ANALYSIS.

16. DEVELOPER FINDS
DIFFERENCE IN
DEFAULT QUEUE
IMPLEMENTATION
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO
SUPPORT TICKET
FROM CANARY
CHANGE RELEASED
TO PRODUCTION
ENTERPRISE
CUSTOMERS REPORT
ISSUES, REOPENED
SUPPORT CASE
VERSION
ROLLED BACK

17. meet
the team
create
timeline

18. INVITE PEOPLE WHO…
introduced problem detected problem
communicated
problem
investigated problem other interested

19. EVERYONE IS
TRULY AN
EXPERT
THERE ARE
STORIES TO
UNCOVER
EACH STORY IS
ESSENTIAL

20. DEVELOPER FINDS
DIFFERENCE IN
DEFAULT QUEUE
IMPLEMENTATION
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO
SUPPORT TICKET
FROM CANARY
CHANGE RELEASED
TO PRODUCTION
ENTERPRISE
CUSTOMERS REPORT
ISSUES, REOPENED
SUPPORT CASE
VERSION
ROLLED BACK

21. DEVELOPER FINDS
DIFFERENCE IN
DEFAULT QUEUE
IMPLEMENTATION
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO
SUPPORT TICKET
FROM CANARY
CHANGE RELEASED
TO PRODUCTION
ENTERPRISE
CUSTOMERS REPORT
ISSUES, REOPENED
SUPPORT CASE
VERSION
ROLLED BACK

22. DEVELOPER FINDS
DIFFERENCE IN
DEFAULT QUEUE
IMPLEMENTATION
SUPPORT TICKET CAN’T
BE REPRODUCED;
MARKED AS NOT
RELATED TO CHANGE
TESTING REPLACED
WITH DOG FOODING
AND MONITORING
CHANGE RELEASED
TO CANARY
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO
TESTING IGNORED;
DIDN’T PERFORM
IMPACT ANALYSIS
SUPPORT TICKET
FROM CANARY
CHANGE RELEASED
TO PRODUCTION
ENTERPRISE
CUSTOMERS REPORT
ISSUES, REOPENED
SUPPORT CASE
VERSION
ROLLED BACK

23. BAD?BAD?LUCKY?GOOD?
GOOD? LUCKY?
GOOD?
BAD?
LUCKY?
Developer
quickly
identified
problem with
queue.
Audit log
helped rerun
failed
operations.
Didn’t show
error message
which helped
ignore initial
support ticket.

24. meet
the team
Find causal
factors
create
timeline

25. WHAT CONDITIONS
ENABLED THE
PROBLEM?

26. CAUSAL FACTOR
!=
ROOT CAUSE

27. INCIDENT:
CAR INCIDENT
CAUSAL FACTOR:
DRIVER DID NOT KEEP CAR ON
ROAD
ROOT CAUSE:
HIGH SPEED WHEN ICE ON THE
ROAD.

28. DEVELOPER FINDS
DIFFERENCE IN
DEFAULT QUEUE
IMPLEMENTATION
SUPPORT TICKET CAN’T
BE REPRODUCED;
MARKED AS NOT
RELATED TO CHANGE
TESTING REPLACED
WITH DOG FOODING
AND MONITORING
CHANGE RELEASED
TO CANARY
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO
TESTING IGNORED;
DIDN’T PERFORM
IMPACT ANALYSIS
SUPPORT TICKET
FROM CANARY
CHANGE RELEASED
TO PRODUCTION
ENTERPRISE
CUSTOMERS REPORT
ISSUES, REOPENED
SUPPORT CASE
VERSION
ROLLED BACK
EVENT QUEUE
LIBRARY UPGRADED
-LIFO OVER FIFO

29. meet
the team
Find causal
factors
Find roots
create
timeline

31. Event queue library updated. LIFO over FIFO.
Developers didn’t know how external library can
impact the application.
There wasn't any test for external libraries.
Libraries responsibilities were unclear.
Writing expectations of libraries in tests doesn’t
happen when libraries are introduced.
Adding contract tests for external libraries is not
a common thing and lacks procedure.
Why?
Why?
Why?
Why?
Why?

32. meet
the team
define causal
factors
define roots
create
timeline
create
actions

33. PREVENTIVE
ACTIONS
CORRECTIVE
ACTIONS
CONTRACT
TESTS FOR
NEW
LIBRARIES
CONTRACT
TESTS FOR
ALL EXTERNAL
LIBRARIES
RISK
ANALYSIS
PERFORMED
WHEN
UPGRADING
LIBRARY
DISPLAY
ERROR
MESSAGE TO
USER

34. Create realistic
actions
Establish mutual
agreement
Assign
owners

35. Does it end now?

36. ANALYSE ROOT
CAUSE
DEFINE
ACTIONS
APPROVE
ACTIONS
REVIEW
RESULTS

37. TRY AGAIN.
FAIL AGAIN.
TEST BETTER.
Kasia Samuel Beckett*

40. ROOT CAUSE ANALYSIS FOR DUMMIES
➤ Bad thing happens…
➤ Resolve it.
➤ Schedule time for analysis and book a whiteboard.
➤ Invite interested people.
➤ Work up the chain to find the root cause.
➤ List improvements to avoid future failures of the system.
➤ Identify owners. Assign tasks. Set time expectations. Get
agreement.
➤ Publish post-mortem.
➤ Follow up on actions and share your learnings.

41. THANKS!
@kasia_balcerzak