Pivotal Labs Open View Presentation Quality Assurance And Developer Testing

Quality Assurance
and Developer Testing
Edward Hieatt, VP Engineering
Ian McFarland, VP Technology

Copyright © 2008 Pivotal Labs, Inc. All rights reserved.

What’s the goal?

• To develop software quickly, cheaply, with
the lowest defect rates possible


In practice, we want...
• Ability to deploy new releases frequently
o at a bare minimum, once per iteration/sprint
o ideally, as each new feature is finished
• Confidence that everything we build has a very low defect
rate
• Confidence that adding features/performing refactorings -
won't introduce defects
• Minimal firefighting/bug fixing - focus on new features
• Keep code flexible enough to keep feature cost constant
• Sleep well at night


The Old Way
• Write code
• Deploy to QA
• QA finds and reports bugs
• Development fixes bugs
• Deploy to QA
• QA finds bugs
• Development fixes bugs, repeat
• QA doesn't find anymore bugs
• Deploy to production


Who’s responsible for
Quality?
•QA’ers are responsible
•Quality tested after development is
complete
•Developer-QA team relationship is
loose & adversarial
•Automated tests are an afterthought


The Result
• Ever-increasing percentage of development time
spent on bug fixing and firefighting
• Ever-decreasing percentage available for feature
development
• Increasing fear that new features will inject bugs
• Architecture changes become prohibitively
expensive
• Performance concerns conflict with stability concerns
• Eventually, feature production grinds to a halt


Rethinking our attitudes
torward testing
• Everyone is responsible for quality
o Developers responsible for the quality of their code
o 'Customer' responsible for accepting work
o QA operates at a higher level of abstraction
• Reduce the cost of defects
o Move defect detection as early in lifecycle as
possible
• Make quality transparent
o Make bug detection easy


Quality as the focal point

• Tests are the core of the development process
• Test all the time, not just at the end
• Automate everything
• Invest in making it fast and easy to run tests
• Write automated tests before we write the code
• Make build status visible to the entire team
• Run regression suites at every check-in


Kind of Tests


Kind of Tests
• Unit Tests
• Functional Tests
• Acceptance Tests
• Smoke Tests
• Integration Tests
• End-to-End tests
• Scenario Tests
• System Tests
• UI Tests
• Manual tests
• Exploratory Tests
• Regression Tests

Testing Frameworks

•Domain Specific Languages (DSLs)
•FIT and FITnesse
•DSLs and FIT let PMs, analysts and QA
write more and better tests


Where’s the Bug?
Test Fixture Client

View

Controller

Model

Database

Where’s the Bug?
Client

View

Controller

Test Fixture Model

Database

Where’s the Bug?
Test Fixture Client

View

Controller

Test Fixture Model

Database

Isolation

•Tests at different granularities help you
more quickly identify the source of the
problem
•Good test coverage tells you what’s wrong
as quickly as possible

The Food Pyramid

Confidence in System as a Whole Increases
Test Isolation, Focus, Speed Increases

Scenario
Tests

Functional
Tests

Integration Tests

Unit Tests


Testing as part of
Development
•Test-Driven Development
•Red-Green-Refactor
•No code before a red test demands it
•Outside-in vs. Inside-out
•Fast test run frequently
•Continuous Integration

TDD Example


A Typical Day at Pivotal

• Standup - team communication
• Pair-up
• Pair pops next story from stack in Tracker
• Update from RCS
• Test-drive story [repeat]


Test-Driving a Story
• Write functional test (test is red)
• Write unit tests [repeat]:
• Write unit test (test is red)
• Fix unit test by writing code (test is green)
• Refactor (tests are green)
• Rerun functional test (test is green)
• Run all tests (tests are green)
• Update from RCS
• Run all tests (tests are green)
• Check in
• Mark story finished

How do we get there?


Action Plan

•Stop the bleeding
•Transition to testability
•Retrofit coverage where needed


Stop the bleeding

•Write a test every time a new defect is
detected, to isolate the test
•Test all new code
•Refactor touch points with old code for
testability


Transition to
Testability

•Characterization/black box tests first
•Refactor for testability
•Gradually work down to unit tests in
legacy code


Retrofit coverage
•Make retrofitting tests part of everyday
work
•Don’t “stop” and cover all your old code
at once
•Measure coverage as you go
•Have goals and measure progress
•Understand what coverage you have

Steady State

•Maintain Testability
•Continuously improve test suite


Maintaining Testability
•...requires vigilance
•It’s easy to slip into old habits...
•when there’s time pressure
•when you’re tired
•when you’re soloing
•When it’s hard to write tests, do it more

Maintaining Testability

•Keep code amenable to testing
•Keep coverage high, and focused
•Keep the test corpus small
•Less is more, assuming your coverage is
complete
•Don’t punt on hard testing problems

Continuous
Improvement
•Test Speed
•Test Clarity
•Test Quality
•Test Readability
•Remove stale tests


The role of QA
on an Agile team
• Different pair of eyes
• Can focus on exploratory testing and test
planning
• Catch gaps in requirements and coverage
• Should write additional coverage
• Less monkey work, more brain work
• Any bug discovered by QA must have an
automated regression test case written


Pivotal Labs Open View Presentation Quality Assurance And Developer Testing

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Pivotal Labs Open View Presentation Quality Assurance And Developer Testing

Ähnlich wie Pivotal Labs Open View Presentation Quality Assurance And Developer Testing (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pivotal Labs Open View Presentation Quality Assurance And Developer Testing