The right quality lifecycle management tools can cut defects and time out of the software development lifecycle. The wrong tools – or lack of resources to support the tools—can cost you precious time and waste your money. Get Dr. Steven G. Anderson's take on how Microsoft's new tool set compares to Quality Center.

1. Tools for Managing Quality
How Microsoft s Emerging Tool Set
Stacks Up Against Quality Center
Software Quality Forum
Thursday, November 11, 2010
5:00 7:00 pm

2. BenchmarkQA helps project teams deliver higher
quality software through:
Quality Assurance Consulting
Contract & Permanent Staffing
QA/Test Training & Seminars
Local Outsourcing Test Lab
© 2010 BenchmarkQA

3. Agenda
Deciphering the MS Marketing
Comparisons
1. Project/requirements management
2. Test case management
3. Test execution
4. Defect Management
5. Reporting
6. Administration
7. Automated Testing
Conclusions
11/15/2010 © 2010 BenchmarkQA Slide 3

4. Overview of MS Tools
4 primary components
This is primarily a developer
view of the world
11/15/2010 © 2010 BenchmarkQA Slide 4
Reporting
Services

5. Deciphering the MS Marketing
TFS
Database (think Access application)
Visual Studio
Developer tool (unit testing)
Automated testing (Coded UI)
MTM
Microsoft Test Manager
Reporting
Reporting Services
11/15/2010 © 2010 BenchmarkQA Slide 5

6. Team Foundation Server (TFS)
Database repository
Views from Visual Studio, Reporting Services, IE and
MTM
Query tools
11/15/2010 © 2010 BenchmarkQA Slide 6

7. Visual Studio
Developer tool
Unit testing
Coded UI testing
Source code management
11/15/2010 © 2010 BenchmarkQA Slide 7

8. Microsoft Test Manager (MTM)
Test case management
Test execution
Integrated defect management
11/15/2010 © 2010 BenchmarkQA Slide 8

9. Reporting
Reporting Services
Primarily warehouse based
Excel
11/15/2010 © 2010 BenchmarkQA Slide 9

10. © 2010 BenchmarkQA
Comparison 1
PROJECT/ REQUIREMENTS MANAGEMENT

11. TFS Project/Requirements Management
User Stories (Validation (UAT) tests linked here)
Tasks (Verification tests linked here)
11/15/2010 © 2010 BenchmarkQA Slide 11

12. TFS Requirements
User Stories
High level
More detailed user stories linked to higher levels
States
Active/Resolved/Closed
Area Path
Modules or groupings of application functionality
Can be used to sort of create a tree structure of user stories in a report
Iteration
Can be used to manage releases
11/15/2010 © 2010 BenchmarkQA Slide 12

13. Task Form
11/15/2010 © 2010 BenchmarkQA Slide 13

14. Tasks
Tasks (linked to user stories)
Assigned to person
Estimate of effort
Actual hours
Activity
Design (Verification tests linked here)
Development
Testing
Task States
Active
Closed
11/15/2010 © 2010 BenchmarkQA Slide 14

15. Requirement-Design Links
Use task activities to deal with design elements
11/15/2010 © 2010 BenchmarkQA Slide 15

16. Scoring - Project Management
Scheduling TFS
Resource management TFS
QC
Can be added to requirements section
Workflow would be challenging
Additional reporting needed (burn down charts, etc.)
11/15/2010 © 2010 BenchmarkQA Slide 16

17. Scoring Requirement Management
Enter requirements Wash
Organize requirements QC
Link requirements to test cases Wash
Link requirements to defects Wash
11/15/2010 © 2010 BenchmarkQA Slide 17

18. Comparison 2
TEST CASE MANAGEMENT
© 2010 BenchmarkQA

19. TFS Manual Test Cases
Similar to QC
Action/step
Expected results
Parameterized test fields
Defined at test level rather than run-time
Can import from Excel
Test case importer (CodePlex)
Needs customization
Organization similar to Test Suite in QC
11/15/2010 © 2010 BenchmarkQA Slide 19

20. MTM Test Case Management
11/15/2010 © 2010 BenchmarkQA Slide 20

21. Test Case Entry
Test case entry very similar to QC
Action
Result
Parameters
Shared steps (call another test in QC)
11/15/2010 Slide 21© 2010 BenchmarkQA

22. Scoring - Test Case Management
Enter test cases QC
Organize test cases QC
Import from Excel QC
Link test cases to requirements Wash
Link test cases to defects Wash
Integration of automated tests QC
Unit tests - VS
11/15/2010 © 2010 BenchmarkQA Slide 22

23. Comparison 3
TEST EXECUTION
© 2010 BenchmarkQA

24. MTM Test Execution Management
Note configurations
Progress bar
11/15/2010 © 2010 BenchmarkQA Slide 24

25. TFS Test Execution
Very similar to QC
Filter by environment
Cannot stop and restart test
Recorded as fail.
Rearranges your desktop
History not linked to test case
List of tests to run
11/15/2010 © 2010 BenchmarkQA Slide 25

26. Scoring - Test Execution (Manual)
Organize test runs Wash
Execution speed Wash
Restart test QC
Environment support MTM
Execution history - QC
11/15/2010 © 2010 BenchmarkQA Slide 26

27. Comparison 4
DEFECT MANAGEMENT
© 2010 BenchmarkQA

28. TFS Defect Management
Remember this is a developer view of the world
Active/Resolved/Closed
Changing workflow is not as easy as QC
Option to verify defect
Major issue-Defect verification is recorded in test suite defect
was found rather than test suite it was fixed in.
Only reruns if defect created from test run
11/15/2010 © 2010 BenchmarkQA Slide 28

29. Scoring - Defect Management
Defect entry Wash
Creating defect while running test Wash
Managing defects QC
Verifying defects - Wash
11/15/2010 © 2010 BenchmarkQA Slide 29

30. Comparison 5
REPORTING
© 2010 BenchmarkQA

31. TFS Reporting
Uses a warehouse so data is old
Reporting services
Much higher skill level and permissions required to create
reports
Good for roll-up/summary reporting
Difficult to nearly impossible to generate accurate requirements
coverage report
11/15/2010 © 2010 BenchmarkQA Slide 31

32. Scoring - Reporting
No contest - QC
11/15/2010 © 2010 BenchmarkQA Slide 32

33. Comparison 6
ADMINISTRATION
© 2010 BenchmarkQA

34. Scoring - Administration
Project customization QC
User administration Wash
Defect workflow QC
Other workflow Not easy in either
11/15/2010 © 2010 BenchmarkQA Slide 34

35. Comparison 7
AUTOMATED TESTING
© 2010 BenchmarkQA

36. Unit Testing
Integrated part of VS
Easy for developer to create tests for methods that
return values
MOQ for visual unit test
Code coverage
11/15/2010 © 2010 BenchmarkQA Slide 36

37. Coded UI Testing
Capture/replay
Manually create object repository
Higher skill level than QTP to customize tests
Much more flexible than QTP
No waiting for support
11/15/2010 © 2010 BenchmarkQA Slide 37

38. Case Study 1
QTP 9 Crashed QTP on login screen
QTP 10 QTP hung on login screen
QTP 11 QTP recorded, could not replay login screen
(some customization would get past this issue)
VS 2010 Records and plays back for simple actions.
Customization for dealing with grid actions
11/15/2010 © 2010 BenchmarkQA Slide 38

39. Case Study 2
QTP 10
Cannot find any objects on screen, not even the menus
QTP 11
Can find most standard controls
Cannot see any 3D objects
VS2010
Sees all objects
View object properties (center, radius, etc.)
11/15/2010 © 2010 BenchmarkQA Slide 39

40. Scoring - Automated Testing
No contest Visual Studio
11/15/2010 © 2010 BenchmarkQA Slide 40

41. Summary
Project management TFS
Requirements QC (by a hair)
Manual test case management QC
Manual test execution QC (by a hair)
Defects Toss up
Reporting QC
Administration QC
Automated testing VS2010
11/15/2010 © 2010 BenchmarkQA Slide 41

42. Considerations
Cost
Initial (Project, QC, VS, QTP)
Maintenance
Conversion
Customization
Productivity improvements
Skill level of testing staff
Best of Breed vs. One Solution
If starting from scratch (i.e. want to get off Excel)
11/15/2010 © 2010 BenchmarkQA Slide 42

43. And the winner is?
Microsoft s Team Foundation Server/Visual Studio 2010
11/15/2010 © 2010 BenchmarkQA Slide 43

44. How can we help?
Project engagements
Tool selection and implementation
As needed support
Staffing
Senior automation resources
Quality Path Tools and Technology
Broad and deep assessment
Process improvement program
11/15/2010 Slide 44© 2010 BenchmarkQA

45. Thank You!
For additional information on how BenchmarkQA can be of
assistance, please contact:
Warren McLeod
952.392.2400
warren.mcleod@benchmarkqa.com
© 2010 BenchmarkQA