carrier grade software quality assurance system design, the overall view

1. Backward ThinkingBackward Thinking
Design QA System to Meet Quality GoalsDesign QA System to Meet Quality Goals
Liang Gao

2. Types of TestingTypes of Testing
• User testing
• Alpha testing
• Beta testing
• Function testing
• Integration testing
• Domain testing
• Boundary testing
• Best representative testing
• Logic testing
• State-based testing
• Path testing
• Performance testing
 2
• Specification-based testing
• Requirements-based testing
• Combination testing
• Regression testing
• Scripted testing
• Smoke testing
• Exploratory testing
• Guerilla testing
• Scenario testing
• Installation testing
• Load testing
• System testing
Don’t Panic

3. And Lots of Questions Need To Be AnsweredAnd Lots of Questions Need To Be Answered
• 产品的组织结构图 产品测试后， 测试人员如何流动
• 测试人员的职位和角色的划分 有没有独立的测试设计团队
• 测试组织内部的回报关系， 测试人员如何考核 开发和测试的比例
• 测试人员的要求，资历，经验等。 质量部对测试团队的监控作用。
• 开发流程和开发模式 开发人员给测试人员的质量的保证
• 测试工作量如何来评估，如何制定测试计划，依据什么来估计工作量
• 开发和测试人员在各个阶段的工作职责测试人员是否参与百合子和黑盒子的测试
• 版本发布谁来说了算，如何评估版本的质量。 版本能不能发布要考虑那方面的因素。
• 工具和自动化， 仔细介绍一下好的自动化工具。有那些工具。日常测试中用那些工具。是否
自己开发的，还是商用工具。
• 测试技术用例设计考虑的输入边界值 有否测试类型的划分
• 测试用例设计的步骤，需求分析，测试设计从那些角度来考虑，是从用户需求还是从哪些。
• 测试策略，一个版本的测试策略，有老特性和新特性。
• 如何来自动化，进入维护状态，测试如何进行。 什么组织来负责。是否是独立的组织，人员
的交替关系。
• 平台的测试和产品的测试的关系，是否有不同的测试团队来负责。测试内容和范围有没有差别
• 一个版本，测试是否基于风险，需要考虑那些因素主要问题是在什么阶段发现的。转测试之前
还是之后。
• 开发人员， SE ，市场人员和服务人员在测试过程中能发挥什么作用
 3

4. Nature of the SoftwareNature of the Software
• Continuous Release
 4

5. Product QualityProduct Quality
• “Rock Solid Product”, what does it mean?
• Your product is only as good as customer’s saying
• The ultimate goal is to increase customer
satisfaction.
• C company publish a customer satisfaction index
every day on its internal website.
• Customer are not satisfied when product is not
function as they expected to be.
 5

6. Why Not Satisfied?Why Not Satisfied?
• Lack of product knowledge – need a TAC, not
the topic today ， and not quality problem.
• Product has problem
o Manufacture problem – not the topic today
o Defects – yes, we need to catch those.
 6
Internal Found Defects
Customer
Found Defects

7. Product Quality MetricsProduct Quality Metrics• Product complexity
• Numbers of the customer found defects
• Customer found defects priority
• Number of the unit shipped
Can you come up with an equation to calculate
the “Product Quality Index” number?
You boss will love to say:” Lets reduce our quality
index from 4.7 to 3.5 this year”.
 7

8. QA’s goalQA’s goal• Reduce customer found defects!
• Reduce high profile customer found defects first.
(80 percent of the customer only use 20 percent
of the features, 20 percent of the customer make
80 percent of your revenue)
• So QA strategy is just anther ROI (return on
investment)
 8

9. 2 Kinds of Testing2 Kinds of Testing
OrganizationOrganization• By Feature
• By Testing roles
 9

10. 2 Types of Defects -2 Types of Defects -
EngineeringEngineering
• New feature defects – new code broken
• Regression defects – new code break the old
code
o Enable new code
o Not enable new code
 10

11. 1Type of Defects - Customer1Type of Defects - Customer
• Feature interaction defects
 11

12. Why Regression Bug?Why Regression Bug?
• Software Modules and Modules are coupled,
change one module’s code will affect others.
• Network equipments process same packet in
different place when a packet traverse the box –
sequential processing
• Protocols are layered (ISO layer)
• Protocols depends on one another (one protocol
use another protocol as input)
 12

13. Analysis Your CustomerAnalysis Your Customer
Found BugsFound Bugs
• New feature defects (single or multi-feature
interaction)
o More testing need to be done on new feature testing.
o Your testers need to be trained on how design effective
test cases to catch new bugs.
o Different testing methodologies.
o Increase your new feature testing coverage on your
test plan.
o End-to-end customer oriented solution testing
• Regression defects
o Will not decrease no matter how good your testers are
o You need a regression strategy
o All code need to be tested whenever release a new
version ( huge work)
 13

14. Combination is endlessCombination is endless
• Feature interaction is endless
• The way to use your product/system is endless
• Lucky you if you can cover them all
• If not, you need a strategy
• The best strategy is customer’s usage (20/80 rules)
 14

15. To Reduce Customer FoundTo Reduce Customer Found
DefectsDefects
• You need a process to test new
• You need a process to test old
• You need a process to test solution (whatever on
the customers site).
 15

16. Bugs in a Product’s lifeBugs in a Product’s life
 16
New Feature
Bug
Regression Bug Regression Bug
Active Development, Market
Share
Cash Cow End of Life
Product Revenue
Maintenance, feature enhancement
1.0
2.0
3.0
4.0 5.0
6.0

17. Nowadays in Company CNowadays in Company C
• Most of the new feature bugs can be found in
the early development – strong testers, end-to-
end system testing, early customer field trial.
• Many customer found defects are regression
bugs
o 10 million lines of code need to be tested whenever a new version is
released
o Release 2 new versions every week.
o Is that possible? Yes, but only via Automation
• Many companies use frequent regression to
stabilize a code branch.
 17

18. Strategy To Release aStrategy To Release a
Quality SoftwareQuality Software
• We are talking about carrier grade product.
• Test case/test plan design needs to start when
code writing start, and finish when code writing
finish.
• Be very strict on review process. (C company
spent multi-million $$ on this)
• Closely monitor your bug trend during testing
new.
• Repetitive regression to stabilize your code
branch
 18

19. Repetitive regression to stabilize your code branchRepetitive regression to stabilize your code branch
• In order to fix regression bugs, new code need to
be added  introduce another regression bugs?
• Multiple regression cycles are needed to carve
off regression bus.
• The more regression you are able to do before
FCS, the more stabilize your code branch.
 19

20. Testing Release TimingTesting Release Timing
 20
Dev Test Regression1 Regression2 Regression3
Solution EFT
Golden
Week

21. Development and Testing At a GlanceDevelopment and Testing At a Glance
 21
MRD/PRD
Functional
Specification
Code
Development Bug Fixes
Test Plan Test Case
Code
Release
to Test
Daily/Frequent Code Drops
Code to
Test
Alpha
Start
Beta
Start FCS
Various Testing Scripting
Dev
Test
Code
Control Opened Bug ID Code
Review
Form
Golden
Week

22. No Need to Be In SyncNo Need to Be In Sync
• Make regression a train
• Make System testing and solution a train as you
like.
• Only new feature testing need to be in sync with
development and release schedule.
• Scheduled regression testing make branch stable
• Scheduled system and solution testing make
optimize your code for performance
 22

23. Branch is QA’s BabyBranch is QA’s Baby
• Code branch should be owned by QA.
Developer’s checking in new code need QA
permission
• Design a nightly smoke test suite to monitor the
build quality.
• Test new, and exploratory test new to take the
new code bug out (daily build is necessary)
• Close the code branch before final regression.
• Only regression commit is permitted during
regression testing
• Multiple regression cycle to shrug off the
regression bug
 23

24. Automate the Process to Make Your Life EasyAutomate the Process to Make Your Life Easy
• When developer check in a code, he can
specify a bug id. Then this bug in the bug
tracking system’s state automatically change to
“R”
• A new code commit automatically trigger a
smoke testing, report automatically sent to QA
team. If there are large amount of failures, an
alarm can be automatically triggered (send a
SMS to your cell phone?)
• ………
 24

25. Not Enough TimeNot Enough Time
Testing?Testing?• More Attributes on the test cases can help
• Which functionality is most important to the project’s intended purpose?
- Which functionality is most visible to the user?
- Which functionality has the largest safety impact?
- Which functionality has the largest financial impact on users?
- Which aspects of the application are most important to the customer?
- Which aspects of the application can be tested early in the development cycle?
- Which parts of the code are most complex, and thus most subject to errors?
- Which parts of the application were developed in rush or panic mode?
- Which aspects of similar/related previous projects caused problems?
- Which aspects of similar/related previous projects had large maintenance
expenses?
- Which parts of the requirements and design are unclear or poorly thought out?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of problems would cause the worst publicity?
- What kinds of problems would cause the most customer service complaints?
- What kinds of tests could easily cover multiple functionalities?
- Which tests will have the best high-risk-coverage to time-required ratio?
 25

26. Ready to ReleaseReady to Release
• QA is always asked: can we release it?
o QA’s job is to deliver report (a risk assessment),
what works, what not. Testing report & Bug
Report
• QA can only answer this question by giving a full
version testing report
• QA use defects to talk to other department and
bosses.
 26

27. Big PictureBig Picture
 27

28. Testing TipsTesting Tips
• Plan ahead
o Perform good interview to create strategy and methodology
• Discussion is crucial:
o Interview & review with right set of people: developers, marketing, sales,
manufacturer
o Use white board as much as possible to discuss new test cases, resolving
technical questions
• Work with developers
o Find out the changes and possible breakages
o Resolve ambiguities in varies documentations
 28

29. Testing TipsTesting Tips
• Testing
o Test the code changes and its dependencies
o GUI testing != functionality testing. If possible, do not use
GUI testing to test core functionalities
o Use 80/20 rules to focus on the right area to test
o Not only ask what to test, but ask what not to test
• Remove unnecessary duplicate efforts
o Types of release can determine the testing, see next slide
o Weapon: Schedule. Use it carefully
o Code scale linearly but the complexity grows exponentially
 29

30. Version controlVersion control• Version 1.0, Oct 10th, 2007, Liang Gao
• Version 1.1, Oct 29th, 2007, Liang Gao
• Version 1.3, Jan 10th, 2008, Liang Gao
• Version 1.4, March 13th, 2008, Liang Gao
• Version 1.5 June, 2008, Liang Gao
 30