1. Understand Release Engineering
Quality Software Release
Liang Gao

2. We’ll cover
• What is release engineering.
• Why do we need release engineering
• Release version planning
• Branch management
• Build process
• Release process
• Source code control
• Release engineering best practice

3. What is Release Engineering
Delive
r
Delive
r

4. Why do We need Release Engineering
• Do you know how many releases Cisco IOS
has?

5. Why do We need Release
Engineering
• Do you know how many platforms Cisco IOS runs?
1000 Series 1600 Series 1700 Series 2500 Series MC 3810 4500 Series
7000 Series 7400 Series 7500 & 7000 RSP Series 7600 OSR Series 10000
Series
12000 GSR Series
Catalyst 4000
Catalyst 5000
Catalyst 6000 & 6500

6. What is Release Engineering
• build a good quality binary
• Source + tools = product

7. What is Release Engineering
• Compile
• Verifying that the software on the
(uncharacterized) end user system continues
to execute properly
• Source code quality control
• branching and merging separate but parallel
codelines of development
• Make it a predictable, quality release

8. Overall Picture
Customer
Marketing/Product
Release Planning
Release
Engineering
QC
DE
Customer

9. Types of Releases
• Major: Introduce major functionalities, major infrastructure changes
unaware by users
• Minor: Introduce minor enhancements and bug fixes
• Maintenance/service: Bug fixes, mostly critical bugs found by customers
• Patch: Bug fixes targeted toward on customer
• Platform: Release only one selected platforms, mostly introduce new
hardware platforms (ASIC, new hardware revision, component changes

10. Sample Release Process IntroductionQC lead identifies an official build as
release candidate
Build master rebuilds the candidate with a release name
using build tool
Build master puts the signed image to release
directory
Auto Regression
QC Lead to announce Exit QC
Technical Writer and QC provide Release Note
Store image, Go Live, Sales notify customer

11. Release Manager's question
• Which release contains this defect repair?
• Which releases have this defect fixed?
• Which codelines contain this defect repair?
• What files have changed between these
two releases?
• What defects have been added/dropped
between this release and that release?
• What changes have been added/dropped
between this release and that release?

12. Introducing Branch and its
Management

13. Branch Management

14. Release Management
• Customers may use different versions
• How many branch you need to support
• XP and Vista will be co-exist.

15. Release Management1.1 1.2 1.3 1.4 2.0 2.1
1.1r1 1.1r2 1.1r3 1.1r4 1.1r5
1.1Throttle
1.2FCS 1.2r1 1.2r2 1.2r3
2.0
3.0 3.0FCS
1.2Beta
Bug Fix
Bug Fix
Bug Fix
2 month

16. Maintain Multiple Releases
1.1
1.2
1.1r1 1.1r2
1.2r1 1.2r2
One Testing Group

17. Maintain Multiple Releases
P4 – Beta3
1 week 2008-09-17 2008-09-23
P4 - Golden Week 10 days 2008-09-24 2008-10-9
Extended time period 4 days 2008-10-10 2008-10-15
P4 - FCS - Internal 5 days 2008-10-16 2008-10-22
Sanity on CTU0z0r1, all New Features need to be
run.
10/16-10/17
FCS regression Suites :
Security Scan
flow
Sanity
Release job
All Featuress sanity (Besides private script,
please also do manual sanity on al lFeaturess)
System level testing automation jobs
Upgrade testing
Checksum testing
10/17-10/23
Official
images
Confirm files are on staging srv 10/23 FCS

18. Merge Process

19. Merge Process
• Get the Diff
• Use CVS or Subversion to find code conflict
• Get developers to result
• Compile binary
• Auto regression
• Code commits

20. Branch Process
• Copy over

21. Bug Fix Checkin Process
• A field in bug report: To be fixed in
• It is filled by DE manager case by case
• Forward Fix
– older version found bug, that will need to be
fixed in the later version
• Backward fix
– Current version’s old code found bug, need to
be fixed in the previous version as well
• Both forward and backward fix
• A tool can be used to auto propagate the fix

22. CM Tools For Source Code Control
• CVS
• Subversion
• Rational Clearcase

23. Stable Branch
• After a full regression, no P1 bugs, high
percentage of the regression test cases
passed. No many regression bugs.
• Stable branch take a great effort to get.
– Regression and bug filling
– Bugs fix
– More regression
– ….

24. Stable Branch Strategy
• Main branch is stable
– Release branch is for new code development
– Strictly code merge criteria
• Release branch is stable
– New branch contain most of the new code
(every one can check in
– Extensive testing on the release branch

25. Branch Management in Real Life

26. Some of The Best Practice in
Process
• New feature Code Checkin Process
• Bug Fix code checkin process
• Branch open and close time process
• QC on Branch Control
• Code Freeze
• Nightly build and smoke test
• Customer Patch
• Testing Coverage on different release
• Release Criteria Sample

27. New code check in process
• Developer checkout the private branch and develop the code.
• Developer drop the private branch build image for tester to test
• Test file bugs and developer fix bugs on the private branch
• Code check in criteria
– No P1 bugs
– P2 bugs no more than 2
– Code coverage 75%
– Automation script is developed.
– Tester has authority to veto the commit
– `

28. Bug fix process.
• Developer checkout a private branch for
bug fix
• Developer done code review
• Developer provide code diff in the bug
report
• Developer provide the simple test to show
bug is fixed
• Developer run small scale regression on the
core feature that might be affected by this
bug fix

29. Branch Open and Close Windows
Timing
• For new features, open the branch (that
code can be checked in) on a specific time.
Accept commit only it meets the commit
criteria.
• If the feature missed the branch open time,
it has to wait for the next round
(performance impact on the developer)
• Customer found bug, regression bug code
fix can be checked in through out the cycle.
Use nightly build and daily smoke test

30. Branch Open and Close Windows
Timing
Close
CloseCloseOpen Open

31. QC on Branch Control
• The firmware is always built by QC to
ensure the source code is in repository and
the quality of the firmware. Every firmware
built including the source code is kept and
backup for reference.
• QC will not accept firmware from developer
for testing. If developer request to test
his/her firmware for bug fixes, QC will re-
test again using QC built firmware.
• The firmware build can be done by demand

32. Code Freeze
• Feature Freeze
• Code Freeze
– When the code is ready for beta and freezed,
all source code checked in require bug id, no
new feature check in will be allowed and source
tree is locked. The code will be reviewed by
code reviewer. Once it is reviewed, the QC
project lead will open the permission for the
developer to check in.
– Any last minutes bug fixes require reviewed by
two chief architects. QC Project lead will

33. Development and Testing At a Glance
MRD/PRD
Functional
Specification
Code
Development
Test Plan Test Case
Code
Release
to Test
Code to
Test
Alpha
Start
Scripting
Dev
Test
Code
Control Opened Bug ID
Feature Freeze
Code Freeze
Beta
Start FCS
Various Testing
Code
Review
Bug Fixes
Form
Golden
Week
Daily/Frequent Code Drops

34. Nightly build and smoke test
• The Nightly Build and Smoke Test is a process in which a software product
is completely built every night and then put through a series of tests to
verify its basic operations. This process is a coding-stage process, and it
gets initiated at the start of the coding phase of the project. The process
produces its savings by reducing the likelihood of several common, time-
consuming risks – unsuccessful integration, low quality, and poor progress
visibility.
• The Nightly Build and Smoke Test can be used effectively on projects of
virtually any size and complexity.
• It is the responsibility of the Build Group to create the required
environment for running the nightly builds and monitoring it on a daily
basis.

35. Customer Patch
• For customer patch release, a bug id is
required.
• Test Engineer will try to reproduce the same
problem in test environment.
• Developer will determine the cause and fix
accordingly.
• Since this is patch release, we will normally
make a copy of release source directory,
changed the version to identify the
customer, check in only the fixes, and build.

36. Types of Testing
Test Type Description
New Feature Test Perform new feature tests according to MRD, RFE and functional specification. The
new feature test cases will become part of regression test once the product is released.
Regression Test Perform feature tests that is already available in the past releases, this is to ensure the
existing working features do not break in the new release.
System Test Ensure device/software works as a whole by exercising all major components in
hardware and software.
Interoperability Test Software interoperability ensures the implementation works with our products and other
security products. Hardware interoperability ensures our hardware devices work with
other hardware vendors to test such as interface link, NSRP, etc.
Performance Test For NetScreen devices, measure the throughput of firewall and IPSec and ramp up
rate. For Global PRO, we measure the number of devices Global PRO can support.
Capacity Test Measure the maximum supported capacity listed in product specifications such as
MRD/PRD/RFE or datasheet.
Security Test Run through list of well-known security scanners and attack software to assure our
devices are not vulnerable to these well-known attack software.
Automation Test Automation does not provide new test cases, we develop automation software to
facilitate manual regression test, we have more than 30% of ScreenOS regression test
cases run in automation environment.

37. Types of Testing Performed
Major
Release
Minor
Release
Patch
Release
Platform
Release
SFR CSP
New Feature Test Full Full TBD Full
Regression Test Full Partial Partial Partial Partial
System Test Full Full Full Full
Interoperability Test Full TBD TBD TBD
Performance Test Full TBD Full TBD
Capacity Test Full Full TBD
Security Test Full Full Full Full Full
Automation Test
(partial regression)
Full Full Full Full Full Full
SFR – Special Feature Release CSP – Customer Specific
Patch

38. Sample Release Criteria
• Long run tests should be executed for 5
days without QC showstopper being found.
• There are no open critical bugs
characterized by data corruption or card or
machine freeze.
• There are no open ship stopper bugs
characterized by bad customer experience
such as failure to install and configure.
• Less than 1 incoming critical bugs per week
Incoming bug trend should be checked for
decline over time.