This project was to address the unstable and multiple builds/ multiple machines in the production environment for Dell Financial Services which had more than 30 apps running on Production.
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Yellow Belt DFS Austin Taxas
1. Analyze: Tools Used : Fishbone, Flow charts. Measure: Before Measures : 12 % build failure in 2004 Data from remedy Improve: The build Time has been reduced. Example DBC build and NRD Quarterly update have been automated. Instead of 14 Part Time resources , 2 Full Time resources are doing the Release work. As compared to 12 % failure in 2004 there is only 4 % failure in 2005.Continued improvements in 2006. Through Build Documentation review has led to lesser build failures and improved System Stability in 2005 continued in 2006. The three Year Benefit Losses has been reduced. Control & Report Standardization : Documents related to Release are kept in a centralized location with access to all those who need to know the process. ( URL : ausdsdfsdata01ataFS Release Management) Processes for Sustained Results This Process is being defined in other Groups like PG, HR and Corp. Future aim is to share the best practices throughout the entire AMCS organization Will be training the GSC Release folks in the DFS Release process Matrix will be maintained and no build failure and backout should occur. From lessons learned the new builds should be verified and then implemented Define: Problem Definition : There is no defined Release process for the releases in DFS IT Production Environment Yellow Belt 6-Up Presentation Title: DFS_IT_RELEASE_PROCESS_IMPROVEMENT Area: DFS It and Business PTT #: 23236 Time Period : Oct , 2005 to March 2006 Team :Meenakshi Ganeriwala Steven O’Hara Greg, Pantzer Why Selected : Lack of defined process for build deployment Lack of Release management group for deployment Root Cause Findings No dedicated resource for build implementation.. On call and Problem Solving people are pulled. Lack of proper build versions being deployed. Lack of Trial builds in the QA environment. Build issues not worked on to be eliminated in the future build Objective : Reduce Time To implement build( Automation) Reduce Resources doing the implementations Improve Accuracy of deployments to 98 % No new change should reduce the System stability Reduce the Three Year Benefit losses As-Is Process Improved Process Raw Data and Baseline Performance Key Tools
2. DEFINE (PROJECT INFORMATION, DEFINITION) PTT Number : 23236 Team : Meenakshi Ganeriwala O’Hara Steven Greg Pantzer Executive Sponsor : Diana Keating Chartering Manager : Sandeep Kulkarni Finance Analyst : SenGupta, Paul BPI Mentor : Pande, Kaunteya Project Type : Yellow Belt Project Lead : Meenakshi Ganeriwala, Start Date : Oct 4 , 2005 End Date : March, 2006 DFS_IT_RELEASE_PROCESS_IMPROVEMENT % of Build Failure in 2004 ---- 12% As Is Statement (2004) There is no defined Release process for the releases in DFS IT Production. Desired Statement (2005 and part of 2006) To define a process of implementing changes to the DFS Production Environment To reduce the % of Build Failure in 2004 ---- 3% to 5 % Have 98 % good build and 0% backed out builds. To deliver changes that will not break the stability of the environment To reduce the number of resources working on the build implementations To reduce the loss in the Three year Benefit.
4. DEFINE IS/ IS NOT ANALYSIS, Why Selected Why Selected : DFS wanted a defined process for Build implementation to the Production Environment Refer to the Flow Chart for the existing process of build deployment to the Production platform As-Is Process Development, Test Application services Not entire DELL IT DFS IT ( part of DELL IT ) Is Not Is Scope Statement Is/ Is Not Analysis
5. DEFINE (COPIS) Stakeholders Dell Financial Services Development and Test Team. Dell Financial Services Release Management Team Dell Financial Services IT and Business Suppliers Process Owners Customers Where does the process begin? Test Team delivers new Features to be deployed to Production What does the process exclude? Other data sharing, problem solving, support improvements What does the process include? Defined process of Build deployment to the Production environment Where does the process end? Build deployed to the Production environment and master CT closed. Documents successfully published on ausdsdfsdata01ataFS Release Management Boundaries Key Metrics Reduce the % failure Only two Full Time resource Reduce the Benefit loss % of build Failure Resources involved Three Year Benefit Measurable Outputs Process Measurable Inputs
6. MEASURE CTQ Tree Process Steps Metric Table : Customer Need Drivers (PFQT) CTQs (Metric) DFS IT Mgmt Reduce time and resource spent on Release management Less Build Failures Better stability DFS Business No of Successful Builds deployed to Production Environment stability Effective Build Quality Business Loss Builds completed on less hours Time % Failed reduced Resources $1,584,000 14 App Ser Part time 12 % Before $528,000 2 full time 3 to 5 % After Decrease the num Lagging Attribute productivity num num Resources involved Decrease the loss Lagging Variable Financial $ $ loss 3 year Benefit Decrease the mean Reduce variability Lagging Attribute Productivity/ Quality % Percentage % of the Build failure / backed out Desirability Leading/Lagging Attribute /Variable Family of Measure Unit of measure Metric Descriptor
11. IMPROVE ( These are the steps related to the Proposed improved Release Process for DFS
12. IMPROVE A notice for Build violation would look like this ( This section is referred from the Flow chart ) Change Ticket Number XXXXXXXXX Project/Express lane item XXXXXX Systems Impacted XXXXXXXXX Build Implementer Meenakshi Ganeriwala Subject of the email Build violation for CT XXXXXXXXX Matter sent in the email Your build request for change ticket CT XXXXXXXXXX has been reviewed by DFS-Release-Process Management Team. The following items have been identified as not being in compliance. _____ QA to Prod handoff meeting did not take place _X__ 5 day lead time was not provided for review of scripts and doc. __X_ CT-number folder missing in the QA_Prod_Builds folder under ausdsdfsdata01ata . _____ CT-number folder under ausdsdfsdata01ata is market “uncertified” __X__ UAT not signed off on this Item. _____ Implementation Document not in proper format/ contains insufficient information. _____ Build Document has insufficient Backout Instructions _____ Contact information not completed. _____ Server names missing. _____ Appropriate Login information not received in Build Doc. _____ Trial build requested was not scheduled and implemented with DFS Apps Member Please address these items prior to implementation. All required information must be completed and accurate in the implementation document. Failure to comply with DFS-Build-Process control procedures will result in not implementing your build. Necessary management will be sent a notification. Thanks, DFS-Release-Process Control Team Email : Meenakshi_ganeriwala@dell.com