1. Most Common Validation ErrorsIdenti Fy Frequent Deficiencies To Accelerate Your Validation Projects (Based on Frank Houston research) QPack™ Application Lifecycle Management May 2010

2. About Orcanos Develop and implement QPack Medical ALM system Expert in Application lifecycle management Leading ALM 2.0 and CFR 21 Part 11 Forums Over 50 installations Industry based solutions: Software/Medical Device/Semiconductors Siemens Business and Technology partners Oracle partner (OEM) Offices: Israel, UK, Germany

3. Customers

5. Inconsistency

6. Lack of Needed Detail

7. Traceability

8. Vague Wording

9. Unverifiable Test Results

10. GDP

11. Incomplete Testing

12. AmbiguityTest Results

13. QPack Application Lifecycle Management Market Definition Develop Test Design Test Plan Definition Market Gathering Requirement Definition 2.0 Risk Mng. 1.0 Customer Facing Test Execution Delivery Defect Reporting

14. QPack ALM Solution Teamcenter PLM ALM ANALYTICS Product Requirements Tasks Marketing Requirements Test Plan Project Management QPack Test Execution Risk Management Defects Tracking SOA

15. Lessons Learned from 1,720 Observations The goal was to bring the company’s software validation evidence up to the level of the U.S. FDA expectation. Identified which documents most frequently contains errors The results are typical problems most companies face Applying Pareto Analysis Finds 9 Types of deficiencies representing about 41% of categories

16. Missing Information Documents or records omitted fundamental information or content that should have been included. Compliancy to specific standards Missing other engineering information HW & Mechanics

17. Inconsistency Documents contained statements inconsistent with other statements about the same topic in the same document or in the same validation package. Jargon Varying Terminology Contradiction in logic

18. Lack of Needed Detail This deficiency applied mostly to requirements documents. The requirements in the validation package did not adequately describe the software. Poor requirements documents Data User Interaction Key process

19. Traceability 3 frequent traceability problems: The traceability matrix did not account for a traceable specification or an observation step in a test script Broken trace (Barren or Orphans) Requirement details were not explicitly numbered and traced to associated test steps.

20. Vague Wording Documents used generalities “in accordance to an approved procedure”, or “applicable regulatory requirements”, or “all associated GxP and business processes” vague words such as “may”, “possibly”, “more or less”, and “approximately”

21. Unverifiable Test Results Expected results were not described sufficiently so that an independent reviewer could compare and verify actual results. The IEEE Standard for Software Test documentation, Std. 829.1988, Clause 6.2.4 says you should, “...provide the exact value (with tolerances where appropriate) for each required output or feature” For executed scripts, actual results were not recorded or captured in a way that allowed an independent reviewer to compare them to expected results. Actual-result column with no reference to a screen shot

22. GDP 3 frequent Good Documentation Practice problems: Hand-recorded data and testing evidence, such as test results, were presented in a way that could cause doubts about their authenticity (Ex. Cross-Out) Data that confirmed a specific requirement was hard to find in the evidence provided (for example, a busy screen shot crammed with data). Handwritten corrections were made that changed the sense of a requirement or an expected test result

23. Incomplete Testing Test scripts did not fully or adequately test the associated requirement.

24. Ambiguity Text could be interpreted more than one way The words “either” and “or” in a requirement are strong clues the text is ambiguous.

25. Identifying the Most Vulnerable Documents categorized the documents and records where most frequent deficiencies were found. They discovered that about 85% of findings were concentrated in six key documentation areas

26. User dashboard Personalized views Easily add charts, and filters

27. Full Coverage matrix

28. Risk Management (FMEA) Reduce project risk Relate risk objects to ALM artifacts Generate Risk Management Document Easily track high-risk features

29. Advanced ALM Analytics - BI Include external data sources Personalize your own dashboard drill down reports

30. Thank You www.orcanos.com

31. QPack Main Screen Manage Full Lifecycle Manage Your project Html/Word editor

32. Solution –Multiple Projects/Versions Solution

33. Requirements Management Central repository Automate process to improve communication Full Traceability of customer needs Freeze content to assure quality Requirements backlog for future design MS-Word Integration allow working with MS-Word

34. Requirements Management requirements tree (SRS) workflow Set priorities and effort estimation Customized templates

35. Market -> Software Traceability

36. Requirements Relations Manage logical relations Change in specific item might affect other places

37. MS-Word Integration (2003/2007) Manage multiple document types with full traceability (MRD, PRD, SRS, STP, STR…) Generate MS-Word documents and synch changes back Embed tractability reports in documents

38. MS-Word Integration

39. Requirements Backlog/Pool Manage future releases in backlog Evaluate change before implementation Customers Market Product Product Road Map - Pool Product Implementation and versions

40. QA – Test Plan Plan tests according to requirements to assure requirements are tested Manage test parameters for complex hardware configurations

41. Test Plan Test is part of release content Automatic cover on creation Test covers multiple requirements reduce test maintenance Requirement Requirement Test case

43. WinXP Pro

44. Windows Vista

45. Linux

46. Browser type

47. Explorer 6

48. Explorer 7Test parameters reduce test maintenance overhead Check scenarios: Run test on Linux and IE6 Run test on Linux and IE7 Run test on WinXP and IE6 Run test on WinXP and IE7 ….. …..

49. QA – Test Execution Easily build test run suites for QA teams Report defects from test run improves QA and development team communication Automatic traceability of requirements by defects and test results Track test results for better decision making and quality assurance

50. Run Tests – QPack Test Execution QA work plan for each release Distribute tests to run between testers

51. Test Run Screen Tester receives test to dashboard Run tests by steps Pass/Fail test on step level Parse test data by parameters

52. Automatic Defect Report On Test Fail Report defect from test on step failure Defect automatically connected to the tests Defect automatically routed to specific developer Defect contains found version, severity, priority, etc. Defect contains test data Steps from test inserted to defect

53. Test execution results Create and store filters by release, status, execution sets, etc.

54. Defect Tracking Create custom views, reports and charts to make sure no defect is overlooked Manage defects versions, branch defects to coop with multiple versions management Dynamic defects workflow for team collaboration Defects are traced back to requirements provides tracking in lower resolutions

55. Defect Tracking Report defects on specific requirement Use dynamic filters to generate reports Manage defects delta between versions Allow customer to report defects Defects list Dynamic filters

57. Common access to all ALM artifacts

58. Get Up-To-Date coverage of requirements by all ALM artifacts

60. User dashboard Personalized views Easily add charts, and filters

61. Internal/Customer Defects Create and store filters by release, status, execution sets, etc.

62. Defects Per Build Create and store filters by release, status, execution sets, etc.

63. Release manager Track changes on each release Monitor release quality Monitor test execution progress

64. Implementing QPack Organization Analysis Organization structure Products and solutions R&D teams Existing tools Existing methodologies Implementation Standards and documents management Market requirements Product requirements Coverage and traceability Test plan and execution Defects tracking Reporting Administration

65. Roadmap Integrate with Siemens Teamcenter PLM solution Provide regulations support kits (ISO, CMMI, FDA…) Hardware testing support (Complex configurations, integration with HW testing platforms) Agile support 3rd party plug-ins such as test automation tools, IDE, source control, CRM and helpdesk Document Management System BI next generation – ALM prediction Provide internet based solution

66. Delivering Real Value “QPack really gives us the possibility to control a project. We set up a workflow according to our company´s structure and size. This workflow ensures a standardized development process..." Quategra We have chosen the QPack system because we believe that it provides a comprehensive solution for managing our products development process in one integrated tool, with its intuitive user interface and reporting capabilities. Incredimail "Orcanos and QPack™ system are the ultimate solution for us; Orcanos managers helped us with professional experience on the methodology and with system customization to our needs...“ Alvarion

67. Summary

69. One Stop-shot for better collaboration

70. Full traceability from market to R&D

71. Regulatory compliance (ISO, CMMI, FDA…)

72. ALM analytics and BI executive dashboards

73. Easy implementation, small learning curve

74. Better visibility of change impact on resources and budget

75. Track customer needs to improve satisfaction

76. Make the right investments according to customer, risk, quality, and timeline