1. How to Become A Testing
Expert
Liang Gao

2. Typical Interview Question
• Do you know programming
• Answer: Yes, a little
• Which language do you know the best
• Answer: I know C a little
• How many lines of code have you written
• Answer : Umm.., about 300 lines
• Do you know Java or .NET,
• Answer : No
• Do you know Linux
• Answer : No
• Do you know networking
• Answer : I installed Windows XP on my PC….
• Ok, you can work for us as testing engineer

3. Tester’s Dilemma
• Testing is not cool
• Tester is second class when compared with
developer
• Developer using me as servant
• I don’t see I can make difference in our company
• Testers are keep coming, they are young and
better than me.
• 我在吃青春饭， I don’t know where my career
path is
• OK ， I need to be a developer and write code!

4. Top Management's Dilemma
也知道， 在招到好的 工程 是有多你 现 测试 师 难
- 北京研究所副所华为 长 2008
It took Juniper 6 month plus to find a new QA
director

5. Career Story of Ting
• Graduate at 1990, only job found is the tester at Sun
• Join Cisco as tester in 1993, then test automation
engineer (design Cisco’s automation framework)
• Cisco grows from 2000 to 40000 from 1993 to 1998
• Join Netscreen as the 1st testing engineer at 1998
• NetScreen went to Nasdaq on 2001
• Grow with NetScreen as testing lead, testing manager,
testing senior manager and testing director
• Found Sigma at 2004, served as CEO as today

6. Lesson
• You may have to stay long enough to be
competitive.

7. What is a testing expert?
• Ability to find critical bugs in given time frame
(hot gun)
• Ability to build a comprehensive testing
strategy in given time frame (expert)
• Ability to manage the release process (top
expert)

8. What is a testing expert Looks like?
• Play the video of James Bach

9. Find Bug that could make News

10. Bugs in the News
• A Cisco Secure Access Control Server (ACS) that is
configured to use Extensible Authentication
Protocol-Transport Layer Security (EAP-TLS) to
authenticate users to the network will allow
access to any user that uses a cryptographically
correct certificate which can be expired, or come
from an untrusted Certificate Authority (CA) and
still be cryptographically correct.
• CSCse58195. The WLC contains a bug when
processing WLAN ACLs that causes the
WLANvACL configuration to be saved with an
invalid checksum. When the configuration is
subsequently reloaded at boot time, the
checksum fails and the WLAN ACLs are not
installed.

11. Bugs in the News
• CSCdv24925 It is possible to read stored
configuration file from the Storage Router
without any authorization.
• CSCdu45417 It is possible to halt the Storage
Router by sending a fragmented packet over
the Gigabit interface.
• CSCdv24925 An unauthorized person may
read the configuration of the Storage Router.
That may lead to unauthorized access of a
storage space.

12. Bugs in the News
• Versions of the Cisco ACE 4710 Application
Control Engine appliance prior to software
version A1(8a) use default administrator, web
management, and device management
account credentials. The appliance and
module do not prompt users to modify system
account passwords during the initial
configuration
process.
• Crafted SSH Packet Vulnerability
• Crafted SNMPv2c Packet Vulnerability

13. 2 Factors of a hot gun
• Technical Expertise
• Thinking methodology

14. Hot Gun’s Bug Percentage
P4
P2
P1
P3
Cosmetic
Functionality
Major Functionality
Critical Functionality, Crash, Hang
10%
30%
40%
20%

15. Thinking Methodology

16. Thinking Methodology
• Customers
• Information
• Developer
revelations
• Team
• Equipment &
Tools
• Schedule
• Test Items
• Deliverables
 Structures
Functions
 Data
 Platform
 Operation
 Time
 Capability
 Reliability
 Usability
 Security
 Scalability
 Performance
 Installability
 Compatiabilty
 Supportability
 Testabilty
 Measurability
 Portability
 Localizations
 Function
testing
 Domain
testing
 Stress testing
 Flow testing
 Scenario
testing
 Claims testing
 User testing
 Automatic
testing

17. Thinking Methodology
• Customers
• 客服
• 售前
• 售后
• 市场

18. Thinking Methodology
发布
升级 / 安装
配置
入网
运营
维护
发布流程测试
Installation/Upgrade 测试
HA
你知道客户典型配置么？
你知道客户跑什么业务？你知道客户跑什么业务？
你知道客户典型应用场景么？
客户 1-2 年运营的
可能遇到情况
可维护性测试
兼容性？

19. Thinking Methodology
Negative Stress Boundary Features
Interaction
Security
Feature
Point 1
Feature
Point 2
Feature
Point 3
Feature
Point 4
Feature
Point 5

20. Develop Testing Strategy
• Be the review person
• 知其然也知其所以然

21. Develop Testing Strategy
• What is your goal first?
– Find more bugs?
– Find more critical bugs?
– Ensure product or feature has no critical defects
– Ensure customers will be ok after the release?
• The strategy
– Bug oriented?
– Coverage oriented?
– Customer oriented?

22. What is Coverage Strategy?
• How to thoroughly test OSPF Hello protocol?
– Function points? (tester)
– User Scenarios? (test expert)
– Scalability? (test expert)
– Performance?
– Security? (test expert)
– ………

23. What is Coverage Strategy?
• 7 platforms
• 6 different line cards
• 2 modes (main/aggressive)
• AH/ESP
• CA/No CA
• HA/No HA
• Hub Spoke/Partial Mesh/Full Mesh
7x6x2x2x2x3 = 1088

24. Types of Testing Covered on Different Release
Major
Release
Minor
Release
Patch
Release
Platform
Release
SFR CSP
New Feature Test Full Full TBD Full
Regression Test Full Partial Partial Partial Partial
System Test Full Full Full Full
Interoperability Test Full TBD TBD TBD
Performance Test Full TBD Full TBD
Capacity Test Full Full TBD
Security Test Full Full Full Full Full
Automation Test
(partial regression)
Full Full Full Full Full Full
SFR – Special Feature Release CSP – Customer Specific
Patch

25. Advise to the New Expert
• Practice, Practice, Practice
• Don’t confuse experience with Expertise
• Don’t trust the folklore – but learn it anyway
• Take nothing on faith, own your methodology
• Drive your own education, no one else will
• Reputation = Money: Build and protect your
reputation
• Relentlessly gather resources, materials and tools.
• Associate with demanding colleagues.
• Write, speak

26. An Expert’s Vision
• An Expert’s Vision
– I can test anything
– Under any condition
– On any given time frame.

27. An Expert’s Vision

28. Top Expert
• Release process
• Testing organization management

29. Release process

30. Release Process
• Start from the MRD
• End with the FCS (first customer ship)

31. 敏捷开发与持续集成 – 迭代中的持续集成

32. Test Organization management

33. Thank you!