Sometime in your career as a test manager, you’ll be assigned to lead the effort for a program so large that the CEO and board of directors monitor it. These are programs that bet the organization’s future and come with a high degree of risk, visibility, pressure, and fixed deadlines. Internal audit and external third-party reviews become de rigueur. Your upstream partners—analysis, design, development, and suppliers—all appear (at least to you) to miss their deadlines with no apparent consequences. Everyone looks to you to “make your dates” so the project delivers on time and keeps the organization in business. Bob Goetz shares key strategies, techniques, and processes to help you survive and even thrive in these mega programs. Learn ways to build teams, standardize processes, measure progress, and work transparently so your business will have full confidence in you and your team—and let you sleep at night!

1. W7
Concurrent Class
10/2/2013 1:45:00 PM
"Key Strategies to Survive the
Mega Test Program"
Presented by:
Robert Goetz
Kaiser Permanente
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Robert Goetz
Kaiser Permanente
With more than twenty years of leadership experience in all aspects of IT, Bob Goetz has
focused on leading QA organizations for the last several years. As part of this, Bob has led the
testing of several mega programs—more than $500 million. The first mega program turned
Bob’s hair grey as he gained the experience to survive and thrive. That multi-year effort
implemented new core business software while transforming business processes and
introducing new products to the market.

3. Key Strategies to Survive the
Mega Test Program
Bob Goetz, Executive Director, Test Services
Kaiser Permanente - Health Plan BIO
Topics
About Kaiser Permanente
What is a Mega Program?
Key Strategies, Methods & Processes
Final Takeaways
Q&A
2 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
1

4. Kaiser Permanente:
The Largest Not-for-Profit Health Plan
• A Durable Partnership
– Kaiser Foundation Health Plans and Hospitals
– Eight Permanente Medical Groups
•
•
•
•
•
•
•
•
•
More than 9 million members
Serves 9 states and District of Columbia
37 medical centers; 611 medical office buildings
175,000+ employees
17,000+ physicians
$50.6 billion operating revenue
Largest labor management partnership in history
Largest non-academic research organization in the U.S.
Largest private user of electronic medical records in the world
Data as of year-end 2012
3 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
The Kaiser Permanente Mission
Kaiser Permanente’s mission is to
provide high-quality, affordable healthcare services and to
improve the health of our members and the communities we
serve.
4 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
2

5. The Markets We Serve
Northwest Region


Portland, OR
Vancouver, WA
Colorado Region

Northern
California Region

Denver / Boulder, CO
Colorado Springs, CO
Ohio Region
•
•
Cleveland, OH
Akron, OH
Mid-Atlantic Region



Washington, DC
Maryland
Virginia
Southern
California Region
Georgia Region

Atlanta, GA
Hawaii Region
5 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Kp.org and My Health Manager –
World’s Leading Personal Health Record
YE 2012 –
– 116 million visits
– 32 million lab test results viewed
– 13 million secure messages
through Email My Doctor
– 12 million Rx refills
3

6. What is a Mega Program?
Or how did my hair turn grey?
7 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
What is a Mega Program?
8 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
4

7. What is a Mega Program?
9 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
What is a Mega Program?
• Huge Opportunity… Huge Risks
• Bets your company’s future
• Has Board of Director (BOD) Oversight
• External Review / Audit
• Your job is on the line
10 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
5

8. What is a Mega Program?
• Huge Opportunity… Huge Risks
• Bets your company’s future
• Has Board of Director (BOD) Oversight
• External Review / Audit
• Your job is on the line
How can you SURVIVE and even THRIVE a Mega Program?
11 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Set the Stage
12 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
6

9. Set the Stage
 Everything is a business problem
– What is the mega program trying to solve?
– Why is it important?
 Discover the situation you are facing
– Who are the stakeholders?
– Who holds the decision rights?
– Who decides “quality”? What is “highest level of quality”?
– What are the Risks? Issues?
13 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Standardize Processes
14 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
7

10. Standardize Processes
• Do a very quick analysis of process AND compliance
• Don’t assume your teams are following your “standards”
15 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Standardize Processes
 Test Strategy
 Naming Standards
 Test Plan
 Component Names
 Test Suites/Cases
 Automation
 Data Creation
 Test Closeout Report
 Data Usage
 Defect Standards
16 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
8

11. Standardize Processes
• Do a very quick analysis of process AND compliance
• Don’t assume your teams are following your “standards”
• Create New Tester Orientation Guide
• Test Suite/Test Case Standards
• Defect Standards
17 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Standardize Processes
Defect Severity
Severity Level Description
Complete system outage and/or is detrimental to the
1. Critical
majority of the development, user, and/or testing efforts
and there is no known workaround. This would also be
known as an emergency defect.
2. High
Serious adverse impact to the user and there is not an
effective work-around, however work can continue.
3. Medium
Moderate adverse impact to the user but there is an
effective workaround.
4. Low
No immediate adverse impact to the user.
Normally cosmetic.
18 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
9

12. Build Allies
19 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Build Allies
 Understand the history of your partners
– Do they trust you?
– Find out who they trust and why – learn from that
 Be transparent
– Admit what you know AND (more importantly!) what you don’t know
 Give them a seat at the table
– Treat them as they deserve: as professionals
– Loop them in on decisions
– Find out what they need and work their problems for them
20 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
10

13. Don’t Forget Your “Friends”
21 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Don’t Forget Your “Friends”
 Truism
– A divided house falls
– Never throw development (or UAT) under the bus – even as you
shake off their dust!
– To the business I.T. is one org – they don’t care who does what
 Establish early in the project your accountability for metrics
– Own it – if at all possible
– Be transparent – share all data possible – with everyone
 Get data to development early
– Give them a chance to consume it and be able to respond
22 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
11

14. Measure Progress
23 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Measure Progress
 Treat Defect Data as a Precious Commodity
– Be consistent in how you log defects (e.g. root cause)
 Make it public and protected
– Daily reports, open SharePoint Site
– Converted to PDF format
 Formats
– Static “point in time” reports
– Trending Charts
24 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
12

15. Burn Up Chart
25 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Burn Down Chart
26 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
13

16. Backlog Chart
27 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Defect Metrics
28 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
14

17. Root Cause
29 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Speak in the Language of Business
30 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
15

18. Speak in the Language of Business
 Remember
– This is never a technology project – it is a business project using
technology.
 Speak in their terms
– Call center?
Impact to Average Speed to Answer
– Ordering System?
Impact to Average Processing Time
– Manufacturing?
Impact to Work in Process (WIP)
31 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Speak in the Language of Business
 Know your audience
– Do they want a short “Yes” / “No” answer or long story?
 Don’t conflate your answer
 Answer the question behind the question
 Back it up with data – current data
 If you don’t know – that is OK
– Tell them you’ll find out and get back to them
 Remember to tell the story behind the data
32 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
16

19. “Even though the overall defect backlog increased during the last month,
we see the Critical defect backlog decreasing.
This is a good sign that Critical defects are getting the priority they need.
We still have many older then 90 day Critical defects, but this is a good
sign that work is being done to address this.”
33 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Create a safe space
34 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
17

20. Create a safe space
 Testing / QA is an easy target
– We are at the “end” of the project
– We are easily measured
 Find a weekly time for the team to vent and support each
other
– What is said in the room, stays in the room
 Double check the pulse of the team:
– What is working? What is not?
– What can we change? What should I (as test leader) change?
– What is blocking that I need to unblock?
35 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Final Take Aways
36 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
18

21. Final Take Aways
 Top Things to Remember
1. You are not alone
2. Standardize, Standardize, Standardize
3. Build Allies and Monitor your Friends
4. Measure your Progress and Treat Data as Gold
5. Tell the Story in Business Teams
6. Protect the Team
37 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Q&A
38 July 29, 2013
|
© 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
19