1. Using Defect Reports to Build Requirements Knowledge in Product Lines2nd International Workshop on Managing Requirements Knowledge (MaRK’09) Robyn Lutz Flight Software Systems Engineering & Architectures Jet Propulsion Lab/Caltech & Iowa State University Nicolas Rouquette Flight Software Systems Engineering & Architectures Jet Propulsion Lab/Caltech Pasadena, CA nicolas.rouquette@jpl.nasa.gov rlutz@cs.iastate.edu The research described in this paper was carried out in part at the Jet Propulsion Laboratory, California Institute of Technology ,and funded by NASA’s OSMA Software Assurance Research Program.

2. Problem Overview Goal: use defect analysis to incrementally improve the requirements knowledge for a product line Approach: use information in defect reports from previous product-line members to improve the quality of future members and reduce their requirements-related defects Challenge: can we improve propagation of the requirements knowledge exposed by past defects to future products in the product line? Courtesy NASA/JPL-Caltech Courtesy NASA/JPL-Caltech 2 Courtesy Classroom Clipart Courtesy NASA/JPL-Caltech Lutz_Rouquette_MaRK'09

3. Results Overview Our analysis of defect reports from testing of flight software product line implicated undocumented, tacit requirements information We describe 4 types of new requirements knowledge found in the defect reports We are adapting two mechanisms, new to product lines, to better preserve and convey defect-derived requirements knowledge to upcoming product-line members 3 Lutz_Rouquette_MaRK'09

4. Some Related Work Knowledge sharing: Savolainen and Sajaniemi; Maalej and Happel; Kruchten, Lago & van Vliet Tacit knowledge:Grunbacher & Briggs; Stone & Sawyer Rationale management:Dutoit and Paech; Lago & van Vliet Product Lines: Jirapanthong and Zisman; Trew; Cho, Lee & Kang; Pohl, Bockle & van der Linden; Weiss & Lai Defect analysis for improvement:Chillarege et al.; Dalal et al.; Ostrand & Weyuker; Mohagheghi, et al.; Lausen & Vinter; Fenton & Ohlsson; Leszak, Perry & Stoll; Lutz and Mikulski; Shull et al. Power of anecdotes: Petroski, Herschbach, Bayer, Rasmussen, Dvorakthe To the best of our knowledge, the work reported here is the first attempt to systematically use defect analysis to incrementally improve the requirements knowledge for a product line 4 Lutz_Rouquette_MaRK'09

5. Application Domain Product line of flight software for spacecraft, previously used on seven missions managed by JPL, with two more missions in development Example of a commonality: all products have software fault protection to respond to power loss Example of a variability: the choice of reaction wheels used to control spacecraft attitude (i.e., position) differs among the products 5 Lutz_Rouquette_MaRK'09

6. Approach Product Lines offer an opportunity for incremental requirements knowledge and quality improvement. 6 Lutz_Rouquette_MaRK'09 Defect analysis of earlier PL members, together with forward propagation of results, provides a way to achieve this.

7. Tasks How to capture relevant product-line requirements knowledge from the defect reports Used Orthogonal Defect Classification to identify defect patterns ODC previously used on spacecraft anomalies [RE’03, TSE’04] How to pro-actively communicate the new requirements-related information to developers of future product-line members Preserve: Specify the information so as to enable retrieval of the defects-learned requirements knowledge by future developers (a “pull” mechanism) Convey: Also need a “push” mechanism so that the requirements knowledge that could have prevented the defects in previous product-line applications will be remembered in future product-line applications 7 Lutz_Rouquette_MaRK'09

8. 1. Defect-derived Requirements Knowledge Newly discovered requirements, e.g., complicated interface information surfaced during testing Unexpected requirements dependencies, e.g., incorrect assumptions about relationships among variabilities Tacit* requirements rationales (insufficiently documented or incorrect), e.g., hardware idiosyncrasy known, but not by testers Misunderstood requirements, e.g., software behavior surprised testers, often due to ambiguous or partial documentation *Tacit knowledge: “knowledge that is essential but not documented” [Kruchten, Lago & van Vliet] Lutz_Rouquette_MaRK'09 8

9. 2a). Preserve Defect-derived Tacit Requirements Knowledge Extended Feature Model: FMs are extended with assumption specifications [Lago & van Vliet, ICSE’05] 9 Lutz_Rouquette_MaRK'09

10. 2b). Convey Defect-derived Tacit Requirements Knowledge PLA-DPs (Product Line Analysis-Defect Paradigms)* are stories of past PL anomalies, packaged in the product-line repository & told to future PL projects to “remember old lessons learned” “During testing on MRO the power software got confused [defect due to erroneous assumption] when the Reaction Wheel (to aim the spacecraft) was turned off by an uplinked software command & right back on by the fault-protection software. It turned out that there was a race condition which revealed a new software requirement. This is why [rationale] the software sometimes disables the schedule-verification function when the switch is being commanded.” Courtesy Classroom Clipart *Modeled on Petroski’s Design Paradigms 10 Lutz_Rouquette_MaRK'09

11. Conclusion Defect reports from past product-line members are a rich, and insufficiently tapped, source of tacit requirements knowledge for future product-line members Future work will expand scale (to include more members of the product line) and continue investigating how to better preserve and convey defect-derived requirements knowledge feature models extended with assumption specifications (formal) structured anecdotes of paradigmatic product-line defects (informal) 11 Lutz_Rouquette_MaRK'09