1. .
...... Defect Localization
Leo Arias
Canonical
September 24, 2013
Leo Arias (Canonical) Defect Localization September 24, 2013 1 / 26

2. Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion
Leo Arias (Canonical) Defect Localization September 24, 2013 2 / 26

3. Goals
Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion
Leo Arias (Canonical) Defect Localization September 24, 2013 3 / 26

4. Goals
Goals of Test Automation
Improve quality.
Understand the system under test.
Reduce and not introduce risk.
Leo Arias (Canonical) Defect Localization September 24, 2013 4 / 26

5. Goals
Improve Quality
Tests enable us to specify the behavior in various scenarios
captured in a form that we can then execute.
Prevent bugs from being introduced.
Defect localization.
Leo Arias (Canonical) Defect Localization September 24, 2013 5 / 26

6. Goals
Defect Localization
Customer tests tell us that some behavior isn’t working as expected,
the Unit tests tell us why.
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails but no unit tests fail, it indicates a missing unit
test.
Leo Arias (Canonical) Defect Localization September 24, 2013 6 / 26

7. Goals
Defect Localization
Customer tests tell us that some behavior isn’t working as expected,
the Unit tests tell us why.
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails but no unit tests fail, it indicates a missing unit
test.
Leo Arias (Canonical) Defect Localization September 24, 2013 6 / 26

8. Isolation
Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion
Leo Arias (Canonical) Defect Localization September 24, 2013 7 / 26

9. Isolation
Isolation
We should strive to isolate the system under test as much as possible
the parts we choose not to test.
Failure to isolate lead to context sensitivity because we end up testing
too much software all at once.
Leo Arias (Canonical) Defect Localization September 24, 2013 8 / 26

10. Isolation
Isolation
We should strive to isolate the system under test as much as possible
the parts we choose not to test.
Failure to isolate lead to context sensitivity because we end up testing
too much software all at once.
Leo Arias (Canonical) Defect Localization September 24, 2013 8 / 26

11. Isolation
Isolation, cont.
Leo Arias (Canonical) Defect Localization September 24, 2013 9 / 26

12. Isolation
Isolation, cont.
Leo Arias (Canonical) Defect Localization September 24, 2013 10 / 26

13. Isolation
Isolation, cont.
Leo Arias (Canonical) Defect Localization September 24, 2013 11 / 26

14. Defect Localization problems
Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion
Leo Arias (Canonical) Defect Localization September 24, 2013 12 / 26

15. Defect Localization problems
Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 13 / 26

16. Defect Localization problems
More Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 14 / 26

17. Defect Localization problems
Canonical Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 15 / 26

18. Defect Localization problems
Canonical Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 16 / 26

19. Defect Localization problems
Canonical Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 17 / 26

20. Defect Localization problems
Canonical Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 18 / 26

21. Defect Localization problems
Canonical Dependencies
Leo Arias (Canonical) Defect Localization September 24, 2013 19 / 26

22. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26

23. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26

24. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26

25. Proposal
Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion
Leo Arias (Canonical) Defect Localization September 24, 2013 21 / 26

26. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26

27. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26

28. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26

29. Proposal
Proposal
We should focus on defect localization and isolation while adding test
coverage.
Leo Arias (Canonical) Defect Localization September 24, 2013 23 / 26

30. Proposal
Short-term actions
When an automated test fails because of an error or a behavior change
on another project:
File a bug describing the problem and the projects affected.
<QA> Subscribe to the bug and follow the process to its
resolution.
Make sure all the people affected are notified.
<developers> Add automated customer tests with every fix to
prevent regressions.
<QA> Make sure no fix comes without a customer test. Check that
they are added on the right project.
Leo Arias (Canonical) Defect Localization September 24, 2013 24 / 26

31. Proposal
After the release...
<QA> Analyze the current API and GUI coverage.
<developers> Add isolated tests to fill the holes.
<developers> Make sure we only rely on APIs and behaviors with
automated customer tests.
<QA, developers> Write a test suite specific to test the
integration of all the projects on the different Ubuntu images and
devices.
<design, QA, developers> Start adding user tests since the
expected user experience is defined.
Establish an effective way for us to communicate between teams.
Leo Arias (Canonical) Defect Localization September 24, 2013 25 / 26

32. Open discussion
Open discussion
Thank you!
Creative Commons - atribución 3.0
Leo Arias (Canonical) Defect Localization September 24, 2013 26 / 26