2. Outline
• Bugzilla
• Testopia
• Jenkins
• Automated Testing Environment (Hypervisor
Testing as an example)
– Automated Testing Process
– Example
• References
2
3. Bugzilla Introduction
• Bugzilla is a Defect Tracking System and it is
server software designed to help you manage
software development
3
4. Bugzilla Features
• Excellent security to protect confidentiality
• Optimized database structure for increased
performance and scalability
• Integrated email capabilities
• Comprehensive permissions system
• Editable user profiles and comprehensive email
preferences
• Proven under fire as Mozilla's bug tracking
system
4
5. Bugzilla Requirements
• Hardware
– CPU : 3 GHz
– RAM : 4 GB or more recommended
– Hard Disk : 50 GB of free space is large enough
• Operating System : Including Windows, Linux, and
Mac OS X
• Software
– Perl : 5.12
– Database Server : Bugzilla supports MySQL 5.5,
PostgreSQL 8.3, Oracle and SQLite
– Web Server : Apache 2.2
5
19. Testopia Introduction
• Testopia is a test case management extension for
Bugzilla
• Testopia Requirements
– Bugzilla
– MySQL or PostgreSQL
– Mozilla compatible browser
– Additional Perl Modules
• Testopia Architecture
– Test Plans
– Test Cases
– Test Runs
19
20. Relationship between Bugzilla and Testopia
20
• Since Testopia is an extension to Bugzilla, it relies on many of the same
objects that Bugzilla uses to track bugs
• Testopia is comprised of several objects that are interdependent, so they
make managing the testing procedure possible
21. Relationship between Bugzilla and Testopia (cont.)
21
• Component
– A Bugzilla component
– An attribute of a product
23. 23
• Milestone
– A Bugzilla object
– Testopia builds are associated with milestones
Relationship between Bugzilla and Testopia (cont.)
24. 24
• Test Plan
– At the top of the Testopia hierarchy are Test Plans. Before you can do anything else in
Testopia, you need a Test Plan
– Test plan will serve as the storage point for all related Test Cases and Test Runs and it will
act as the Dashboard for your testing
Relationship between Bugzilla and Testopia (cont.)
25. Relationship between Bugzilla and Testopia (cont.)
25
• Test Case
– Test Cases are the heart of all testing
– Test Cases spell out what steps to take while running a test and what outcomes to expect
– Test Cases are associated with one or more Test Plans and with zero or more Test Runs
26. Relationship between Bugzilla and Testopia (cont.)
26
• Test Run
– Once you have defined a set of Test Cases, you are ready to run through those tests in a
Test Run
– Each run is associated with a single Test Plan and can consist of any number of Test
Cases from that plan
27. Relationship between Bugzilla and Testopia (cont.)
27
• Build
– Software development is usually an iterative affair. In Testopia, each iteration is called a build
– In software testing, a string denoting the compiled results of a period of development
– Builds are often associated with milestones of the project. This is reflected in the build's
relationship to Bugzilla's target milestones object
28. Relationship between Bugzilla and Testopia (cont.)
28
• Case Run
– A Test Case-Run is the record of how a particular Test Case fared in a particular run for a given
Build in a given Environment
29. Relationship between Bugzilla and Testopia (cont.)
29
• Category
– A property of a product that is used to classify Test Cases
30. Relationship between Bugzilla and Testopia (cont.)
30
• Environment
– A list of the surrounding conditions that a test run is performed in
31. Testing Process of Testopia
31
The testing process in
Testopia is as follows :
Step 1. Create a Product and
multiple Components
Step 2. Create a Test Plan
Step 3. Create Test Cases
Step 4. Create a Build
32. Testing Process of Testopia (cont.)
32
The testing process in
Testopia is as follows :
Step 5. Create Environment
Step 6. Create a Test Run
Step 7. Execute the Test Run
Step 8. Produce a test report
on Dashboard
33. Testopia Installation
Step 1. Download Testopia
– http://www.mozilla.org/projects/testopia/
33
Step 2. Decompress the file to Bugzilla folder
Step 3. Run checksetup.pl and install module
39. Integrate Jenkins with Testopia
39
• This plug-in integrates Jenkins with Testopia
and generates reports on automated test
execution
• With this plug-in you can manage your tests in
Testopia, schedule and control in Jenkins
• It reads TAP test report formats, used to
update Testopia test case's executions
41. Auto Test Process
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 1. Retrieve Test Case from Testopia
41
1.
Step 2. Invoke corresponding script
to initialize the test environment
2. Auto testing environment for ITRI
ARM hypervisor :
42. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 3. Initialize the test environment
Step 4. Launch necessary Guest VMs
42
3.
4.
43. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 5. Execute testing tasks
43
5.
44. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 6. Generate and return test
result (by TAP, Test Anything Protocol)
44
6.
45. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 7. Return test result to Jenkins
45
7.
46. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 8. Write corresponding result to
Test Case in Testopia based on TAP
46
8.
48. The process to create a Test Plan
48
• To create a new Test Plan in Testopia
– Step 1. Click the "New Plan" link in the Bugzilla footer
– Step 2. Enter a name for this Test Plan
– Step 3. Select a product from the product list
– Step 4. Select a type for this plan
– Step 5. Select a product version
– Step 6. Type or paste your plan document into the Plan
Document Editor
– Step 7. Click the "Submit" button
49. Step 2. Create a new Test Plan
49
• Plan Name
– Test Plan name
50. Step 2. Create a new Test Plan (cont.)
50
• Product
– Bugzilla product name
51. Step 2. Create a new Test Plan (cont.)
51
• Product Type
– The testing type for this Test Plan
52. Step 2. Create a new Test Plan (cont.)
52
• Product Version
– The Bugzilla product version
53. Step 2. Create a new Test Plan (cont.)
53
• Plan Document
– This is document that spells out the type of testing and testing methods
used for this plan
54. The process to create a Test Run
54
• To create a new Test Run in Testopia
– Step 1. Click "Create a New Test Run" link on your Test Plan
or the New Run link in the footer
– Step 2. Select which Test Cases to include
– Step 3. Enter a Summary for this Test Run
– Step 4. Select a build from the Build list or type the name
for a new one
– Step 5. Select an environment from the Environment list
– Step 6. Click the "Submit" button
55. Step 3. Create a new Test Run
55
• Product Version
– The Bugzilla product version
56. Step 3. Create a new Test Run (cont.)
56
• Run Manager
– The user in charge of this Test Run
57. Step 3. Create a new Test Run (cont.)
57
• Target Completion Rate
– The expect completion rate of target
58. Step 3. Create a new Test Run (cont.)
58
• Build
– The default product build
59. Step 3. Create a new Test Run (cont.)
59
• Environment
– The environment in use for this Test Run
60. Step 3. Create a new Test Run (cont.)
60
• Target Pass Rate
– The expect pass rate of target
61. Step 3. Create a new Test Run (cont.)
61
• Summary
– A short description of this Test Run
62. Step 3. Create a new Test Run (cont.)
62
• Notes
– This field is a place to make notes and observations about this run
63. The process to create a Test Case
63
• To create a new Test Case in Testopia
– Step 1. Click "Create a New Test Case" on the plan page or
New Case from the footer
– Step 2. Enter a short description of your Test Case in the
Summary field
– Step 3. Select a Category
– Step 4. Add a default tester
– Step 5. List the steps for testing in the Action field
– Step 6. Provide the expected outcomes in the Expected
Results field
– Step 7. Click the "Submit" button
64. Step 4. Create a new Test Case
64
• Summary
– A short description of the Test Case
65. Step 4. Create a new Test Case (cont.)
65
• Default Tester
– The person is assigned to this Test Case
66. Step 4. Create a new Test Case (cont.)
66
• Alias
– A unique string that uses to identify test result
67. Step 4. Create a new Test Case (cont.)
67
• Priority
– The level of testing
68. Step 4. Create a new Test Case (cont.)
68
• Category
– The product category
69. Step 4. Create a new Test Case (cont.)
69
• Estimated Time
– The estimated time that this Test Case should take to complete
70. Step 4. Create a new Test Case (cont.)
70
• Bugs
– You can attach bugs to your Test Cases
71. Step 4. Create a new Test Case (cont.)
71
• Blocks
– If this Test Case prevents others from being run their ID numbers will appear
here
72. Step 4. Create a new Test Case (cont.)
72
• Status : Test Case status
– PROPOSED : This test case is a work in progress and has not been reviewed for
accuracy. It is therefore not ready to be included in test runs
– CONFIRMED : This test case has passed review and is ready to be included in future test
runs. Only test cases with this status can be included in new test runs
– DISABLED : This test case is no longer applicable to current testing
73. Step 4. Create a new Test Case (cont.)
73
• Add Tags
– A user defined string used to classify Test Cases
74. Step 4. Create a new Test Case (cont.)
74
• Requirements
– The requirement number or URL to a document containing the requirement
this Test Case is designed to test
75. Step 4. Create a new Test Case (cont.)
75
• Automated
– Automatic Test Cases are run by a script
76. Step 4. Create a new Test Case (cont.)
76
• Scripts
– The name of the script that runs this Test Case
77. Step 4. Create a new Test Case (cont.)
77
• Arguments
– The specific arguments sent to the script
78. Step 4. Create a new Test Case (cont.)
78
• Add to Run
– Entering a run ID here will include this Test Case in a Test Run if it is not
already included
79. Step 4. Create a new Test Case (cont.)
79
• Depends On
– If this Test Case requires other Test Cases be run before this one, their case
numbers should appear here
80. Step 4. Create a new Test Case (cont.)
80
• Action
– This field lists the steps of testing
81. Step 4. Create a new Test Case (cont.)
81
• Expected Results
– This details the expected outcomes of a test
82. Process to Launch Test Script
82
Configure
1. Clean Environment
2. Check Image
3. Launch Test Script
Start Test Script
1. 3.
2.
1. Login to Host
2. Start VM
3. Ping Host
4. Turn off VM
5. Return Result
Testopia
Contents :
1. Parameters
2. Functions
4.
1 . Retrieve Test Script
83. Set up a new project
83
Step 5. Create a project on Jenkins to invoke Test
Script
84. Step 6. Set Testopia configuration section
84
• Testopia Version
– The version name of Testopia
85. Step 6. Set Testopia configuration section (cont.)
85
• Test Run ID
– The Test Run ID of Testopia
86. Step 6. Set Testopia configuration section (cont.)
86
• Single Test Build Steps
– The command to execute by single step
87. Step 6. Set Testopia configuration section (cont.)
87
• Iterative Test Build Steps
– The parameters of Testopia are used to execute by test cases
88. Step 7. Set Result configuration section
88
• Test Anything Protocol (TAP)
– TAP is a simple text-based interface between testing
modules in a test harness
– It allows individual tests to communicate test results
to the testing harness