A Test Automation Framework

A Test Automation
Framework
Gregory Solovey
Alcatel–Lucent
www.qualitymanagementconference.com
1

2
Test related
Processes
Provide
decomposition/
abstraction
Design Tests
Build Harness
Automate
Testing
Include in CI
Framework
There is nothing new under the Sun

3
• 10 minutes to automate a test case
• 10 minutes to identify an error
• 10 minutes to update the testware
10/10/10 Goals for Test
Automation Projects

4
To paraphrase a quote from Tolstoy’s Anna Karenina:
All successful test automation projects
are alike; each unsuccessful automation
project fails in its own way.
Test Automation

5
Test Tools Implementation
Programming language Test language
Data
structures
variables, arrays, hashes TS,UC,TC,TA, buffer
Control
statements
if, case compare
for, while, until loop for TS,UC,TC,TA
Functions subroutine, procedure TC {set, test, compare, tear
down}
Control
flow
algorithm unconditional sequence
Users Developers prefer to use
open source software
which can be adapted
Testers prefer to use
commercial tools, with
technical support

6
The Three Pillars of a Framework
• testware hierarchy
• testware organization
• test language

7
Application
Layer
Abstraction
Layer
Middleware
Layer
Driver
Layer
Behavior Layer
User documents
Structural Layer
Architectural documents
Functional Layer
Requirements documents
Interface Layer
Design documents
System Hierarchies

8
Test Design Techniques:
• Boundary analysis
• Path sensitization
Derived Test Design Methods:
• Arithmetic expressions
• Relational expressions
• Logical expressions
• Algorithm
• State machine
• Instruction set
• Syntax
Test Design Techniques
From test techniques to test methods,
Professional Tester, issue 29, October 2014

9
• Test Set (TS) is a part of the testware that verifies a
particular architectural unit of the system.
• Use Case (UC) is a collection of end-to-end scenarios to
cover the functionality of a subsystem.
• Test Case (TC) is a single verification act that moves the
object-to-test from an initial state, executes a test, compares
the actual and expected results, and returns it back to its
initial state; a UC consists of TCs.
• Test Action (TA) is the simplest act of communication with
the object-to-test. A TA supports the interfaces of the object-
to-test (CLI, GUI, SNTP, HTTP).
Testware Hierarchy

Testware Hierarchy
10
TS1
UC1
TC1
TA1
TA2
TC2
TA3
TA4
TA5
UC2
TC3
TA6
TA7
TC4
TA1
TA3
TA4
TS2
UC3
TC7
TA30
TA32
TA36
TC8
TA41
TA42
UC4
TC12
TA50
TA55
TC13
TA90
TA92
UC
5
TC21
TA77
TA78

Test
Script 2
Testware File Organization
11
Language Commands
interface action words
Test Case calls
business action words
Test
Script 1
Test
Script 3
Config file
TS1
UC1
TC1
TA1
TA2
TC2
TA3
TA4
TA5
UC2
TC3
TA6
TA7
TC4
TA1
TA3
TA4
TS2
UC3
TC7
TA30
TA32
TA36
TC8
TA41
TA42
UC4
TC12
TA50
TA55
TC13
TA90
TA92
UC5
TC21
TA77
TA78
Driver
Libraries
Test case
Libraries

Testware File Organization
12
Language Commands
interface action words
Test Case calls
business action words
Config file
Driver
Libraries
Test case
Libraries
TS: ‘verifyMidleware’, ‘verifyAudioDrivers’,
‘verifyMapApplication’
UC: ‘vrfBufferManagement’,
‘vrfProgramMonitor’, ‘vrfFaultNotifications’,
‘vrfNetworkManager’
TC: ‘createBuffer’, ‘FillBuffer’, ‘SendMessage’,
‘SendAlarm’’
TA: ‘set’, ‘send’, ‘push button’, ‘compare’,
‘repeat’, ‘connect’

Testware Scale
13
Number
per unit
Total Unique
TS 2-50 25 25
scripts 1-4 60 60
UC 2-10 250 250
TC 2-30 5,000 50
TA 2-10 25,000 25

Test Language
• A test is presented as an unconditional
sequence of TS/UC/TC/TA executions
• The instruction set includes the following
command groups:
– run TS/UC/TC commands (one or multiple
times)
– connection commands
– compare commands
– variable manipulation commands
– limited control commands-pause/stop/exit
14

Reliability
• returning the object to its initial state
• handling multiple echoes
• support for stimulus-response
synchronization
• multiple connection attempts
15

Output Buffer Structure
16
Text
Text
Text
Text
Text
Match Text

Test Case Structure
17
Test Case
Tear down phase
Test phase Action
Action
Action
Compare
Action
Action

Verdicts
18
Test
object
Verdict Conditions
TA OK
TIMEOUT
ERROR
‘MATCH’ detected in Output Buffer
‘MATCH’ not found after predefined elapsed time
Predefined error detected in Output Buffer
TC PASS
FAIL
All TAs are ‘OK’ & expected result is found in
Output Buffer
At least one TA is not ‘OK’ or a compare failed
UC PASS
FAIL
All TCs ‘PASS’ed
At least one TC ‘FAIL’ed
TS PASS
FAIL
All UCs ‘PASS’ed
At least one UC ‘FAIL’ed

userFuncLib.pm
Framework Structure
19
tesToy.pl
ssh/ telnet/ ksh
Failed Cases
Result Log
Debug Log
Capture Log
Config file
connectLib.pm
testCommLib.pm
fullPathib.pm
Test Script
Test Script
Test Script
Test Script
Test Script
Test Script
Test Script
TC Library
Object to
test

The test framework is a data driven
application that reads test data,
provides syntax checking and compiles
the data into internal test structures,
creates the necessary connections to
the object to test, executes the tests
and finally creates the log and result
files.
20
Framework Functionality

Debug Log Structure
• Supports TS/UC/TC/TA hierarchy and IDs
• TA: output buffer, rules to capture it, and
how verdict was determined, based on the
buffer content.
• TC: expected and real results and how
verdict was determined, based on the
comparison constraints.
• UC/ TS: how verdict determined, based on
the children verdicts
21

10/10/10 Goals
• Test design in 10 minutes
A framework shifts the testers’ focus from writing new
test scripts to reusing existing ones
• Error detection in 10 minutes
The debug file structure allows for easy error
identification and creation of “defect tracker”
• Testware maintenance in 10 minutes
Testware is organized in a fashion that allows to make
unique changes for any single changes in the object-to-
test.
22

Example: Triangle Algorithm
23
a< b+c and
b<a+c and
c<a+b
a = b a = c
b = c b =c
Not a
triangle (NT)
Equilateral
(ET)
Isosceles
(IT)
Regular
(RT)
0
2
1
1
1
1
0
0
00
Enter a, b, c
1
53
4
1

24
TC # description a b c result
1-14 less than 3 parameters , symbols, negative and 0 values for
each parameter
NT
15-17 a<b+c 6
6
11
2
3
3
3
3
9
NT
NT
RT
18-20 b<a+c 2
3
3
6
6
11
3
3
9
NT
NT
RT
21-23 c<a+b 2
3
3
3
3
9
6
6
11
NT
NT
RT
24-26 2: a=b 11
11
3
3
11
11
9
9
9
RT
IT
RT
27-29 3: b=c (a=b) 3
11
11
3
11
11
2
11
3
IT
ET
IT
30-32 4: a=c (a<>b) 3
11
11
11
9
9
12
11
3
RT
IT
RT
33-35 5: b=c (a<>b and a<>c) 11
9
9
3
11
11
2
11
3
RT
IT
RT
Triangle Test Set

tesToy Configuration File
25
[tt_defaults]
testwareDir = "/home/user/testware"
swDesc="Triangle Test"
prompt_base = "myServer!user> "
time_out = 5
logDir = '/home/user/logs'
keep_going = 1
expectEchoes = 1
debug_level = 3
[test_files]
TS "verify triangle application" "telnet" "myServer"
run triangle.uc
[libraries]
/home/user/libs/general.lib
[user_vars]
myServer = "135.135.135.135"

tesToy Test Script
26
UC set test environment
run general::setDir ("/home/user/bin")
UC verify out-boundary data
run general::whichTriangle ( "", "", "", "error")
run general::whichTriangle ("5", "", "", "error")
run general::whichTriangle ("5", "5", "", "error")
run general::whichTriangle ( "", "5", "5", "error")
run general::whichTriangle ("5", "5", "a", "error")
UC verify that length of first side is less than other two
run general::whichTriangle ( "6", "2", "3", "NT")
UC verify 'a=b' condition
run general::whichTriangle ("11", "3", "9", "RT")
run general::whichTriangle ("11","11", "9", "IT")

tesToy Library
27
# execute any command and compare results
TC setDir
send ("cd $P[0]")
send ("pwd")
compare ("$P[0]")
reset_phase
end_of_case
# execute the triangle application and verify output
TC whichTriangle
send ("bin/triangle.pl $P[0] $P[1] $P[2]")
compare ($P[4])
reset_phase
end_of_case

Further Reading
• Embedding testability, Professional Tester, issue 27, August 2014,
8– 15. – how to build testability for an embedded system
• From test techniques to test methods, Professional Tester, issue
29, October 2014, 4 – 14. – how to build the minimum number of
tests that covers all implementation errors
• Automating Inside the Lines, Professional Tester, 2015, issue 30.
– how to build a test automation framework
• A Continuous Integration Test Framework, Testing Experience,
issue 28, December 2014, 30 – 34. – how to build a continuous
integration test framework
• QA of Testing, Professional Tester, issue 28, October 2014, 9 – 12.
- how to automatically monitor the test completeness
28

Thank you for attending this session
Send your comments or questions to
gregory.solovey@alcatel-lucent.com
29

A Test Automation Framework

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to A Test Automation Framework

Similar to A Test Automation Framework (20)

Recently uploaded

Recently uploaded (20)

A Test Automation Framework