Mater Test Specification for SImple Rail road command protocol

1. Master Test Specification
Simple Rail Road Command Protocol (SRCP)
Submitted by:
Rishu Seth
High Integrity Systems
Fachhochschule Frankfurt am Main
Matriculation number-936161
1

2. Table of Contents
1.Introduction ...................................................................................................... 4
1.1 Overview ............................................................................................... 4
1.2 Test Object ............................................................................................ 4
1.3 Test Environment .................................................................................. 5
1.4 Test Plan................................................................................................ 6
1.5 Test Organisation .................................................................................. 7
1.6 Test Resources ...................................................................................... 7
2. Test Planning ................................................................................................... 8
2.1 Test Areas ............................................................................................. 8
2.2 Test Metrics .......................................................................................... 9
2.2.1 Defect Severity ...................................................................... 10
2.2.2 Defect Priority ....................................................................... 10
2.3 Test Requirement ................................................................................ 12
2.3.1 Network Connection ............................................................. 12
2.3.2 Lexical Units ......................................................................... 12
2.3.3 Commands ............................................................................. 13
2.3.4 Replies ................................................................................... 13
2.3.5 State of Server ....................................................................... 13
2.3.6 Hand Shake Mode ................................................................. 14
2.3.7 Command Mode .................................................................... 15
3. Test Specification .......................................................................................... 16
3.1 Equivalence Classes ........................................................................... 16
3.1.1 Feedback Sensors .................................................................. 16
3.1.2 Generic Accessories .............................................................. 17
3.1.3 Generic Loco ......................................................................... 17
2

3. 3.2 Black Box Test Cases ......................................................................... 17
3.2.1 Hand Shake Area ................................................................... 18
3.2.2 Command Area ..................................................................... 19
3.2.3 Info Area................................................................................ 20
3.3 White Box Testing .............................................................................. 20
3.3.1 SRCPD Source ...................................................................... 20
3.3.2 Masking of Condition............................................................ 26
3.3.3 Cause Effect Graph ............................................................... 26
4. Findings .......................................................................................................... 30
5. Conclusion...................................................................................................... 30
References .......................................................................................................... 30
3

4. 1. Introduction
1.1 Overview
This document is Master Test Specification for Simple Railroad Command
Protocol (SRCP: Version 0.8.3). The initial focus of this document is to analyze
SRCP, it's specification and provide test cases & scenarios, that will
eventually lead to errors in the system and then after that also to look inside the
code and find anomalies and errors. Although to provide the full test coverage
is not exactly possible. The main objectives of this document are:
1. It provides a status report of the test object in comparison to the
requirements. Is the product fit for release? Adding value to the product by
finding defects.
2. To Test the most prioritized elements of the test-object. The realistic use of
the software should be considered.
3. Testing should be unbiased and from different point of views considering the
needs of different users.
4. Test process is well structured. Is it repeatable and predictable?
1.2 Test Object
SRCP is an internet protocol for controlling and programming (digital) model
railway systems. It produces client server communication. The server is the
component that is connected with the plant.
The client can be run on the same computer as the server. The server has no
clue about how the plant looks like. It is the server’s task to translate the
commands of SRCP into action.
A SRCP server is unaware of the exact concrete railway system. And the
topology and even it is not being informed about the installed equipment. Its
4

5. task is to collect all available information as precisely as possible handling
knowledge about the possibilities and limits of the controlled plant. Its data
have always to be understood as the best knowledge about the state of the
railway system.
Limitation of SRCP
SRCP does not fulfill any formal real time demands.
Merits of SRCP
1. A SRCP server abstracts the control of the construction.
2. Paying attention to different safety strategies.
3. Scaling Capability.
An SRCP server represents one central unit (CU). A CU provides at least one
bus over which information is read or commands are executed. A client contacts
the server in order to send commands to the plant and to receive information
from there. So there are SRCP channels between client and server, where the
client sends commands to the server and SRCP channel between server and the
plant, where the server has to execute the command on the plant. SRCP
connection is based on TCP (Transmission Control Protocol).
The Test object will be the protocol version 0.8.3 of SRCP and the SRCP
Daemon 2.0.12, which will emulate the SCRP environment.
1.3 Test Environment
The SRCP Test Environment consists of:
1. SRCPD - will emulate the SRCP server and a digital plant.
2. Server - A machine with OS Linux. SRCP Daemon will run on the server.
5

6. 3. Client - The Client represents a working station with a python written
script, which will connect to the server. OS can be either Linux or
Windows.
4. Java - will provide a client program that sends commands to the SRCP
server.
5. Documentation – MS (Microsoft) Word, Excel, Latex.
1.4 Test Plan
The overall test strategy consists of 6 main phases or levels .The levels are
given below
Test-Planning
• To Analyze the SCRP specification.
• It provides test metrics to measure the quality of the test object.
• It formulates the requirements of the test object.
Test-Case-Generation
• Use the test metrics & requirements to generate test cases for black box
testing. Also divide the test cases in different test areas.
• Provide a python written client program which will help us to execute the
test cases on the test object.
• Inspect the source code of SRCP Daemon and specify white box test
cases.
Test-Case-Execution
• Execute test cases & log results.
Analysis Test Result
6

7. • To analyze test result and compare it with desired result.
Bugs Analysis
• To find the bugs that means deviation from our previous stag.
Test-Documentation
• Visualize test results.
• Defect Reporting.
Final-Test-Result
• Finalized the test result after Bugs Analysis.
1.5 Test Organization
• Developers: Dr. Schönfelder.
• Testers: Students.
• Execution of Tests: Workstations at the university.
1.6 Test Resources
• SCRPD 2.0.12 – source code.
• Specification of SRCP 0.8.3
• Configurations file "srcpd.conf".
• A client program written in Java will be used to connect to the server and
execute test cases.
7

8. 2. Test Planning
2.1 Test Areas
There are mainly three main test areas:
• Hand Shake Area: On initial connection hand Shake Mode is activated.
Then after this connection, its according to the demand of the client
whether to activate Command Mode or Info Mode.
• Command Area
• Info Area
There are some different Sub Test Areas also which are used in combination
with these three Test Areas:
1. Server: A SRCP server represents exactly one central unit (CU) in the
usual sense. Every CU disposes of at least one bus for addressing on
which information is read and/or commands are executed. If a SRCP
server can manage and provide several CU it is valid.
8

9. 2. Feed Back (FB): Feedback devices are encoders that signal an occurrence
on construction
3. Generic Accessoire: A generic Accessoire generally indicates a decoder
that can serve one or more ports under one address. Often these are
switch decoders or signal decoders working as impulse decoders.
• Generic Loco (GL): It indicates engine decoders in the engine
address room of the hardware.
• Locks: Devices locking other devices. The LOCK devices are
devices providing a lock over another device. They are optional.
2.2 Test Metrices
It is the measure of software quality and confidence of our test object. Test
metrices have various uses:
1. It can reduce working time - once we have a matrix for a specific issue, it
will take our less time to test it or to think how to test it.
2. It is logical and testing challenging - It is a challenge to make our own
matrix and to find common issues that are repeatable from project to project.
For Testing of this project, used metrics will look at:
• Functional areas
• Requirements Check
For Test coverage and consistency of Test Effort
During Test Preparation this metric uses:
• Number of Test requirements compared to Functional Area/
Requirements
9

10. • Number of Test Cases planned compared to Test Cases ready for
execution
During Test Execution this metric uses :
• Number of Test Cases Passed, Failed or Blocked.
• Number of Test Cases Executed compared to Test Cases Planned
Severity: Grouping by potential danger caused.
2.2.1 Defect Severity
It is the measure of how lethal a defect can be and how strong is the effect of
failure on software user.
• Critical: It is the highest order of severity and results in the failure of
complete software system of a subsystem or of a software unit within the
system.
• Major: It is almost relevant to “Critical” but there are still some chances
of processing some acceptable alternatives to get desired result.
• Minor: This is referred to as some common normal defects which are not
as lethal as other two but somehow effect the normal working of a
software.
DEFECT SEVERITY:
TEST CLASS DEFINITION STANDARD % PAS
CRITERIA RATE
CRITICAL ESSENTIAL 100 %
MAJOR NECESSARY 90%
MINOR NOT IMMEDIATE 70%
2.2.2 Defect Priority
10

11. It is the measure of priority that is set in regard to addressing defects and
according to its severity, to set minimum time within which it should be fixed.
• High: This is the most prioritized area and defect falling in this category
severely affect the system’s working and should be fixed as soon as
possible.
• Medium: This is having a bit less priority and the defects can be fixed in
normal course of development activities.
• Low: This is having the lowest priority and the defects can be fixed after
the more severe n priortised defects are take care of.
For example when a client connects to the SRCP server, the server generates a
welcome message containing information about the server. The defect of not
generating the welcome would have minor severity, but the process is repeated
every time we connect to the server and that is the first thing we see. So the
priority for this defect would be high and this would be test cases we should
definitely consider.
Another example would be that all commands are case sensitive and have to be
written in uppercase letters. In case some commands would be accepted with
lower case letters that would result in a defect, which doesn't cause a failure and
doesn't really impair usability. So the defect would have a minor severity and
minor priority. So a test case for this defect would not be relevant, considering
our matrices.
DEFECT PRIORITY:
SEVERITY LEVEL DEFINITION
High High system crash data loss
Medium Wrong result operation error
Low Low minor problems
11

12. The table tells us that we will only consider relevant combinations for our test
cases. We cannot provide full test coverage of the Test-Object, because the
number of all possible test cases could be from high to astronomically high.
Implementing all possible test cases would consume too much time, so that by
the time we are finished will all possible test cases, new versions of our Test-
Object would be available, which also need to be tested. So for the test cases we
only focus only on the important ones with the help of our matrices
A test-area from the main test areas (Chapter 2.1) has passed the test process if
it does not contain any defects of the severity level "Critical“. It is essential to
the product. Here need 100% pass rate.
1. A test-area has passed the test process if it does not contain less than 90%
pass rate. It is called important.
2. A test-area has passed the test process if must not be less than 70% pass rate
and it is called desirable test class.
2.3 Test Requirements
Test-Requirements tell us what requirements the Test-Object has to full fill,
by analyzing the specification of the Test-Object. They will help us to derive
test cases.
2.3.1 Network Connection
The SRPC is based on the data transfer technologies of TCP (Transmission
control Protocol). One port is occupied. At present the port 12345 SHALL
be used. Registering with IANA can change this for the future and give the
port a name, too. The temporary name is srcp 12345/tcp
2.3.2 Lexical Units
1. In this case, all character codes are described in their decimal form as #
followed by the numeric value of the character.
12

13. • The entire communication consists of the characters of the 7bit ASCII
character set in the range from #32 to #127 (including the borders).
• Again the characters #9 (TAB) is used for space, #10 and #13 (CR, LF)
for end of line are valid.
• All further characters (esp. with a code value >127) are ignored in
incoming data and removed.
• In outgoing data they can be included, but is not used within SRCP.
2. The characters #9and #13 are seen as white space.
2.3.3 Commands
Commands are send in the “Command Mode” and in “Hand Shake” from client
to server. The SRCP server processes the command and generates a response
that is to be sent to the client. Command consists of:
1 Commands consist of a command word, followed by a command
parameter list separated by white space.
2 The end of a command is the end of line.
3 The end of line always consists of the character #10(n, LF).
4 A prefixed #13(r, CR) is valid. The character #13 is ignored.
2.3.4 Replies
A server HAS TO reply to every command of a client. That is to be sent to the
client on the same connection.
A reply SHALL be single-line. With the sign (Backslash, #92) immediately
before the LF(resp CRLF) it is marked that the reply contains another line.
A single line including the end of the line MUST NOT exceed the length of
1000 characters.
If the result of a command can be ascertained due to communication with
connected model railway devices this opportunity SHALL be used. In case there
are insufficient results for a task the answer is “416 ERROR no data”.
13

14. 2.3.5 State Of Server
SRCP server is always in one of three possible states towards the client: hand
shake, command mode or info mode.
Hand Shake Mode
In this Mode, after establishing a connection and sending the welcome message
the server waits for a command from the client. The server executes it and sends
a single line reply to the client. Then the server waits for the next command.
That way information and commands are being exchanged. Some example
commands used in this mode are:
1 SET CONNECTIONMODE SRCP <MODE> The type of connection is
agreed upon. Valid are INFO for the unidirectional information mode and
COMMAND for the bidirectional command mode. The server sends either 202
OK or the error message 401 ERROR unsupported connection mode to the
client.
2 SET PROTOCOL SRCP <VERSION> The desired protocol that is to be used
is arranged. If this command is missing the SRCP version given during
14

15. welcome is accepted. Every released version number from and including 0.8 is
valid as <VERSION>.
2.3.6 Command Mode
During command mode the server waits for commands from the client and
executes these. As the result of the execution the server always creates a reply
due to be sent to the client in the same TCP session. Subsequently the next
command is executed.
The server‘s replies to the client in the command mode always consist of a time
stamp at the beginning, a numerical answer code and further data. The
numerical answer code is structured in groups. 100-199 marks information and
results, 200-299 includes receipts confirming the processing according to the
rules, 400-499 marks error while executing commands, 500-599 errors by the
server itself, 600-699 specific error codes in implementation.
1xx INFO: Informations, results
2xx OK: Command is accepted and brought to execution. Attention: This is no
confirmation for actually executing the command.
4xx/5xx ERROR: An error condition has occurred. The command is ignored
and not executed.
6xx ERROR: A server specific error has occurred. The command in question is
not executed.
The significance of the time stamp arises from the kind of reply: For quittances
(OK, ERROR) it specifies the point in time when the command is processed and
the receipt OK or ERROR is generated.
Some conditions in command mode are :
1. Replies of the commands by the server are to be sent to the client in the
same TCP session.
2. GET, SET, CHECK, WAIT, INIT, TERM, RESET, VERIFY are
commands that are defined in command mode.
15

16. 3. Valid commands must always be executed.
4. It is not possible to go to info mode out of command mode.
2.4 Result of Planning Phase
PROJECT PLAN:
APRIL MAY JUNE
Test
planning
Test case
generation
Test case
execution
Analysis test result
documentation
Bugs analysis
documentation
(Repeat until desire
level)
Final result of test
cases
documentation
3. Test Specification
3.1 Equivalence Class
Equivalence class method will help us to develop test cases in a structured way
and Assists finding test data to provoke a special expected behaviour.
3.1.1 Feedback Sensors
1. Class of addresses allowed: 1 ≤ x ≤ 10.
Representative element = 1.
2. Class of addresses not allowed: x < 1 && x > 10.
Representative element = 11
16

17. 3.1.2 Generic Accessoires
1. Class of all possible addresses allowed: 0 ≤ x ≤ 324.
Representative element = 1.
2. Class of addresses not allowed: x < 0 and x > 324.
Representative element = 325
3.1.3 Generic Loco
1. Class of all possible addresses allowed: 1 ≤ x ≤ 81.
Representative element = 1.
2. Class of addresses not allowed: x < 1 and x > 81.
Representative element = 82
3.2 Black Box Test Cases
A python-written client program will run input/output driven tests on the SRCP
server. The client program will communicate with the server through a TCP
based connection. Input data will be provided through the specification of
SRCP.
Because of the configuration of SRCPD we will be able to use bus 1 for most of
the test cases and bus 0 for "SERVER", "SESSION", "LOCK",
"DESCRIPTION" or "TIME".
17

18. FIGURE: Server Client Communication
18

19. ID Command Description Output Severity/Priority
1 Set Connection Enter command 202 OK Critical / High
Mode SRCP
Command mode in session Connection
1 Mode OK
2 SET Set protocol 201 OK Critical / High
Protocol
PROTOCOL version for SRCP
SRCP 0.8.3 server
3 Set Connection Try enter 401 Error Critical / High
Unsupported
Mode SRCP invalid mode
Connection
Mode
4 Set Connection Enter info mode 202 OK Critical / Major
Mode SRCP
INFO Connection
Mode OK
5 GO Complete Hand 200 OK GO Critical / High
Shake
ID Command Description Output Severity/Priority
1 RESET 0 SERVER Reinitiate 200 OK Critical /High
server
2 INIT 1 POWER Initiate power 200 OK Critical /High
supply on bus
1
3 SET 1 POWER Set power 200 OK Critical / High
ON supply on
4 GET 1 FB 1 Get info from INFO 1 FB Major /Medium
updated FB 2 123
on bus 1 with
address 1
5 SET 0 POWER Get server out 415 ERROR Major/ High
19

20. OFF of power forbidden
supply (not
allowed)
ID Command Description Output Severity/Priority
1 TERM 0 Close 100 INFO 0 Critical / High
SERVER connection to SERVER
server
2 INIT 1 Initiate power 101 INFO 1 Critical/ Medium
POWER supply on bus 1 POWER
3 INIT 1 FB S88 Initialize a FB 101 INFO 1 Major /Medium
888 1 on bus 1 with FB
address 1
4 INIT 1 GA 0 S Initialize a GA 101 INFO 1 Critical / Medium
11 on bus 1 with GA 0 1 1
address 0 over
protocol S with
port & value 1
5 TERM 1 GL 1 Set engine to 102 INFO 1 Major / Medium
default state & GL 1
take it out of
communication
50
45
40
35
30
25
Series1
20
15
10
5
0
Total Test Cases Pass Fail
Figure Test Case pass and fail ratio Total 45 Pass 18 Fail 27
20

21. 3.3 White Box Testing
3.3.1 SRCPD Source
White Box Testing is a method of testing software that tests internal structures
or workings of an application, as opposed to its functionality. In white-box
testing an internal perspective of the system, as well as programming skills, are
required and used to design test cases. The tester chooses inputs to exercise
paths through the code and determine the appropriate outputs.
Main Criteria in White Box Testing is code coverage, which means all the
branches in the control flow graph are going to be covered. The White box
Testing is going to be performed on the Specific modules of the SRCP server to
ensure safety of the tested models.
First we need to look at the most important source files for out test process. For
each module of SRCPD there are 2 different files. The *.h files contain the
defined functions and constants. The *.c files contain the exact definition of the
functions which were defined in the corresponding *.h files.
I have performed white box testing on the following module of the SRCP:
Followings are the bugs findings performed by doing statement and logic
coverage white – box testing.
MODULE: srcp-fb.c
1. Function: initfb
Bug: Here bus number is not checked and ‘fb’ is invalid under under bus
number zero.
21

22. Suggestion: Bus number should be checked before initializing the ‘fb’.
2. Function: queue_reset_fb
Bug: While on sleep it does not show any error message.
Suggestion: It should return an error message or result in case an error occurs.
3. Function: enqueueInfoFB
Bug: Here declared message length is constant i.e. 1000 and if another
parameter is passed , it may crash.
Suggestion: It should be made Dynamic.
Problem FB: Take FB on bus 1 with address 1 out of communication.
It is not running on bus 1.
Suggestion: It should be running on bus 1.
Problem: SET 2 FB 1 SET FB with too short parameter list. Address is not
mentioned.
Suggestion: Parameter list is checked and also address of bus should be
checked
Flow graph:
It is a representation of all paths that have been traversed by a program during
its execution, using graph notation. Using this flow chart, we can compute the
number of independent paths through the code. In a control flow graph each
node represents a basic block. Direct edges are used to represent jumps in
control flow. There are two specially designated blocks:
Entry block: Through which control enters into flow graph.
Exit block: Through which all control flow leaves the graph.
1. Function: initfb
Its code part and corresponding flow graph is given below:
CODE:
22

23. FLOW GRAPH:
2. Function: queue_reset_fb
Its code part and corresponding flow graph is given below:
23

24. CODE:
FLOW GRAPH:
24

25. 3. Function: enqueueInfoFB
Its code part and corresponding flow graph is given below:
25

26. CODE:
FLOW GRAPH:
3.3.2 Masking of Conditions:
It is necessary for efficiently analyzing the test paths. In this case multi-
condition decision is made separately to analyse the test paths efficiently.
In my white box testing I have made a masking of condition graph for ‘init_fb’
function in the module ‘srcp-fb.c’.
Masking of condition:
26

27. 3.3.3 Cause Effect Graph
This is basically a hardware testing technique adapted to software testing. It
considers only the desired external behavior of a system. This is a testing
technique that aids in selecting test cases that logically relate Causes (inputs)
to Effects (outputs) to produce test cases.
A “Cause” represents a distinct input condition that brings about an
internal change in the system. An “Effect” represents an output
Steps to design the cause effect Graph
Step – 1: For a module, identify the input conditions (causes) and actions
(effect).
27

28. Step – 2: Develop a cause-effect graph.
Step – 3: Transform cause-effect graph into a decision table.
Step – 4: Convert decision table rules to test cases. Each column of the
decision table represents a test case.
ESTABLISHING A CONNECTION:
CAUSE EFFECT
(1) Client connects to the server by (E1) Welcome string sent by the server
TCP/IP connection
(2) Client desired operation mode (E2) Server reply according to
operation mode demand by client
(3) Server – Client sent GO command (E3) Server confirmed GO command
CEG 1:
CEG 2:
28

29. CEG 3:
29

30. DECISION TABLE:
TEST CASE 1 2 3 4
(1) Client connects 1 0 1 0
to the server by
TCP/IP connection
(2) Client desired 1 1 1 0
operation mode
(3) Client sends GO 0 1 1 0
command
4. Findings
• Delayed Response from server
• Not proficient details in Specification
• Time Out – Server gets stuck after certain period of time
5. Conclusion
In our master test case specification, the Black box test cases are selected
according to the SRCP protocol and our assumptions stated earlier. White Box
test case findings were done while analyzing code and it is very important to do
both Black Box and White Box testing together so as to find more bugs and be
more sure about working of the software.
6. References
[1] Class notes and slides of Dr. Schoenfelder
[2] The Art of Software Testing , Second Edition, Glenford J. Myers
30