Pivotal Labs Open View Presentation Quality Assurance And Developer Testing
1. Quality Assurance
and Developer Testing
Edward Hieatt, VP Engineering
Ian McFarland, VP Technology
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
2. Whatâs the goal?
⢠To develop software quickly, cheaply, with
the lowest defect rates possible
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
3. In practice, we want...
⢠Ability to deploy new releases frequently
o at a bare minimum, once per iteration/sprint
o ideally, as each new feature is finished
⢠Confidence that everything we build has a very low defect
rate
⢠Confidence that adding features/performing refactorings -
won't introduce defects
⢠Minimal firefighting/bug fixing - focus on new features
⢠Keep code flexible enough to keep feature cost constant
⢠Sleep well at night
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
4. The Old Way
⢠Write code
⢠Deploy to QA
⢠QA finds and reports bugs
⢠Development fixes bugs
⢠Deploy to QA
⢠QA finds bugs
⢠Development fixes bugs, repeat
⢠QA doesn't find anymore bugs
⢠Deploy to production
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
5. Whoâs responsible for
Quality?
â˘QAâers are responsible
â˘Quality tested after development is
complete
â˘Developer-QA team relationship is
loose & adversarial
â˘Automated tests are an afterthought
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
6. The Result
⢠Ever-increasing percentage of development time
spent on bug fixing and firefighting
⢠Ever-decreasing percentage available for feature
development
⢠Increasing fear that new features will inject bugs
⢠Architecture changes become prohibitively
expensive
⢠Performance concerns conflict with stability concerns
⢠Eventually, feature production grinds to a halt
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
7. Rethinking our attitudes
torward testing
⢠Everyone is responsible for quality
o Developers responsible for the quality of their code
o 'Customer' responsible for accepting work
o QA operates at a higher level of abstraction
⢠Reduce the cost of defects
o Move defect detection as early in lifecycle as
possible
⢠Make quality transparent
o Make bug detection easy
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
8. Quality as the focal point
⢠Tests are the core of the development process
⢠Test all the time, not just at the end
⢠Automate everything
⢠Invest in making it fast and easy to run testsÂ
⢠Write automated tests before we write the code
⢠Make build status visible to the entire team
⢠Run regression suites at every check-in
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
10. Kind of Tests
⢠Unit Tests
⢠Functional Tests
⢠Acceptance Tests
⢠Smoke Tests
⢠Integration Tests
⢠End-to-End tests
⢠Scenario Tests
⢠System Tests
⢠UI Tests
⢠Manual tests
⢠Exploratory Tests
⢠Regression Tests
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
11. Kind of Tests
⢠Unit Tests
⢠Functional Tests
⢠Acceptance Tests
⢠Smoke Tests
⢠Integration Tests
⢠End-to-End tests
⢠Scenario Tests
⢠System Tests
⢠UI Tests
⢠Manual tests
⢠Exploratory Tests
⢠Regression Tests
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
12. Testing Frameworks
â˘Domain Specific Languages (DSLs)
â˘FIT and FITnesse
â˘DSLs and FIT let PMs, analysts and QA
write more and better tests
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
16. Whereâs the Bug?
Test Fixture Client
View
Controller
Test Fixture Model
Database
17. Isolation
â˘Tests at different granularities help you
more quickly identify the source of the
problem
â˘Good test coverage tells you whatâs wrong
as quickly as possible
18. The Food Pyramid
Confidence in System as a Whole Increases
Test Isolation, Focus, Speed Increases
Scenario
Tests
Functional
Tests
Integration Tests
Unit Tests
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
19. Testing as part of
Development
â˘Test-Driven Development
â˘Red-Green-Refactor
â˘No code before a red test demands it
â˘Outside-in vs. Inside-out
â˘Fast test run frequently
â˘Continuous Integration
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
21. A Typical Day at Pivotal
⢠Standup - team communication
⢠Pair-up
⢠Pair pops next story from stack in Tracker
⢠Update from RCS
⢠Test-drive story [repeat]
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
22. Test-Driving a Story
⢠Write functional test (test is red)
⢠Write unit tests [repeat]:
⢠Write unit test (test is red)
⢠Fix unit test by writing code (test is green)
⢠Refactor (tests are green)
⢠Rerun functional test (test is green)
⢠Run all tests (tests are green)
⢠Update from RCS
⢠Run all tests (tests are green)
⢠Check in
⢠Mark story finished
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
23. How do we get there?
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
24. Action Plan
â˘Stop the bleeding
â˘Transition to testability
â˘Retrofit coverage where needed
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
25. Stop the bleeding
â˘Write a test every time a new defect is
detected, to isolate the test
â˘Test all new code
â˘Refactor touch points with old code for
testability
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
26. Transition to
Testability
â˘Characterization/black box tests first
â˘Refactor for testability
â˘Gradually work down to unit tests in
legacy code
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
27. Retrofit coverage
â˘Make retrofitting tests part of everyday
work
â˘Donât âstopâ and cover all your old code
at once
â˘Measure coverage as you go
â˘Have goals and measure progress
â˘Understand what coverage you have
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
28. Steady State
â˘Maintain Testability
â˘Continuously improve test suite
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
29. Maintaining Testability
â˘...requires vigilance
â˘Itâs easy to slip into old habits...
â˘when thereâs time pressure
â˘when youâre tired
â˘when youâre soloing
â˘When itâs hard to write tests, do it more
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
30. Maintaining Testability
â˘Keep code amenable to testing
â˘Keep coverage high, and focused
â˘Keep the test corpus small
â˘Less is more, assuming your coverage is
complete
â˘Donât punt on hard testing problems
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
31. Continuous
Improvement
â˘Test Speed
â˘Test Clarity
â˘Test Quality
â˘Test Readability
â˘Remove stale tests
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.
32. The role of QA
on an Agile team
⢠Different pair of eyes
⢠Can focus on exploratory testing and test
planning
⢠Catch gaps in requirements and coverage
⢠Should write additional coverage
⢠Less monkey work, more brain work
⢠Any bug discovered by QA must have an
automated regression test case written
Copyright Š 2008 Pivotal Labs, Inc. All rights reserved.