lessoned learned from the field.

1. Lessons Learned on
Automation
Liang Gao (liangg@gmail.com)

2. Lesson -1 Automation Cost Money

You need people to write scripts

You need testbed to run those scripts (lots of
equipments)

You need people to monitor those test bed
runs, and debug running failures.

You need people to maintain those scripts to
reflect product change

Can you justify all those to your upper
management?

3. You might if you

Show some proof in the front.

Can go to your boss and say: I can reduce
major release cycle from 8 month to 5
month if I do this.

Can go to your boss and say: I can reduce
customer found bugs down by 20% if I do
this.

4. Lesson 2: Automated Testing is Dangerous

Once it is automated, chances are, it will not
be manually executed anymore.

No exploratory testing can be done

If your script has problem, and can not catch
bugs (output is pass even it should be fail), it
will be going into darkness for ever

You may lose your chance to catch bugs for
ever and you don’t know.

5. Lesson 3: No Framework, No Automation

If you have one couple of hundred scripts in
hand, you might be fine without a framework.

You need one if you have thousands of scripts
to manage.

How to know which test case passed/failed?

Where to get a decent running report.

How to group test cases easily.

How to debug

You need one if there are many people develop
scripts in parallel.

6. Lesson 4: Use Standard Script Language

VB/TCL/Perl/Python/Ruby

Customized scripts need learning period

Customized script language – Who maintain
it? No community support

Hard to communicate with others –
developers, other test outside the company

7. Lesson 5: Separate writer and runner

Engineer should not develop script and then
execute script. Script execution is a dedicated
job.

Debug takes time

Test bed problem?

Script problem?

Image regression bug?

Script integration takes time.

Script execution should be a 24/7 factory,
should be a machine. Script is just a by-
product, it is the full version report that you
want

8. Lesson 6: Test Bed Independent is very Important

Separate the writer and runner requires the script
should be testbed independent.

Script should be able to run from testbed to testbed
with minimum change to the test framework
configuration, not to script itself.

Hard coded router/switch names, IP addresses,
interface names are not good when switch testbeds

A handover process is needed between the writer
and runner.

Develop a “script checker” tool to check the hard
coded values in the script as an acceptance criteria.

9. Lesson 7:
Manage Your Scripts the Same as Your Bugs

Script need to have states like bugs

(S) – Submitted: Script is submitted to the
regression team

(A) – Assigned: Script is assigned to a regression
engineer

(I) – Integration: regression engineer is putting it on
the regression test bed.

(P) – Production: Script is in the regression testbed
and will be run on regression testing against release

(F) - Feedback needed: Script has errors, more
feedback from writer needed.

10. Lesson 8: Design You Scripts As Data Driven.

Script need to be data driven

Different data means different test cases.
Test_case.tcl {router1 eth0} {router2 eth0}
{router3 eth0 eth1} {router4 eth0 eth1 eth2}
{mode 1} {phase 1} {traffic 1 speed}
Can test all combinations of Mode, Phase and
Traffic with one single script.

Data generation can be automated too.

Can catch more bugs when vary the data

11. Lesson 9: Log is More Important Than
Script

When fail, most of the time we only look at
the logs, not scripts for debugging.

Read log like read a book

More debugging info dumped when fail.

12. Lesson 10:
test case designer and automator separate?

Don’t use automator who doesn’t respect
testing

Don’t use automator who doesn’t
understand testing

C company use same tester to design,
manual execute and automate the test
cases. And so is J company

13. Lesson 11: User Standard Testbed
DUT2 DUT3 DUT4
VLAN1
VLAN2
VLAN3
VLAN4
Four Ethernet on each of the Device Under Test
1 2 3 4 1 2 3 4 1 2 3 4
被测设备 2
DUT1
1 2 3 4

14. Lesson 11: User Standard Testbed
DUT1 DUT2 DUT3 DUT4
Ethernet1 Open Open Shutdown Shutdown
Ethernet 2 Open Shutdown Open Shutdown
Ethernet 3 Shutdown Open Open Shutdown
Ethernet 4 Shutdown Shutdown Shutdown Shutdown
Logical Topology:
DUT2
VLAN1
VLAN2
VLAN3
DUT1
DUT3

15. Lesson 12:
Dynamic script allocation

Different release has different feature set

Each regression run need to pick different
scripts to run

Tricks for dynamic script allocation

Static file for version matching

Database

16. Lesson 13:
Use Database to keep automation result history

Database for full version report

Database for regression bug debugging

17. Lesson 14:
Script maintenance – version per branch

Certify scripts for each new release

Keep them in a separate code branch

18. Version control

Version 1.0, Dec 24th, 2007, Liang Gao

Version 1.1 March 2008, Liang Gao