1.
MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26
Abstract
Users can leverage the attributes of Perforce to create
a mechanism that allows seamless file transfer among
the multiple OSes that Perforce supports, without the
overhead of setting up different permissions and
protocols that may be proprietary to each OS. Such a
solution can be written in a single codebase, effectively
reducing maintenance overhead. The file-transfer
mechanism can then be extended to support auto-
updating client software, so users always will be
guaranteed that the locally installed applications will be
the latest version executed at run time.
Novel Use of Perforce for Software
Auto-Updates and File Transfer in
a Multi-OS Environment
Xavier Galvez, AMD

2. 2 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment
This is text for annotations in footer. Similar to footnotes treatment.
Background and Motivation
The tools team in the graphics division at AMD created an in-house, end-to-end solution for
pre-submit developer builds (PSDB). This PSDB mechanism:
1) Takes a code change before it is submitted into version control,
2) Overlays the changes on the last-known-good version of a successful build,
3) Compiles these using an accelerated build farm to ensure compilation does not break, and
4) Deploys tests on the resulting build to ensure the changes are safe.
Once the changes pass the tests, the developer can submit the changes into version control.
However, two problems arise from Step 1:
Problem #1: How will the files be uploaded to the servers that run the mechanism?
Developers at AMD work on Windows®
and Linux operating systems. Different permissions and
privileges have to be set up for file transfer between these systems. Ideally, if a software client
performing the file transfer is installed locally on the user’s computer, then the source code for
this client should be compilable for all OSes (i.e., a single codebase should run on multiple
OSes) for ease of maintenance.
Also, development teams work with different version-control systems; some teams submit
code into Perforce while others use Subversion (SVN). The client should work with any
version-control system.
Problem #2: Given that a client will be installed on the developer’s computer to upload
the files, how can we ensure that the developer is running the latest version of the
client?
Software is a living document that gets updated frequently, especially when bugs are found
and features are added. Client updates can be deployed proactively (i.e., pushed) to ensure
developers use the latest version.
File Transfer
A crucial component of PSDB is a way to upload the code changes to the solution without
submitting them into revision control. This component must be both OS- and version-control
system-agnostic. AMD developers work in Windows and Linux OSes, and with Perforce depots
and SVN repositories for revision control.
First Attempt
To address these challenges, the first generation of the file-transfer mechanism had two
separate codebases (one codebase determined modified files in Perforce, and another in
SVN). The first attempt at writing this component used a remote file server with a shared
directory. However, file transfer was done differently for each OS.
In Windows, file transfer was done using Robocopy, a third-party tool used for synchronizing
files in two different locations. After the modified files were listed, this tool took the modified
files from the user’s local computer and copied them to a shared directory on the remote file
server. With the file server hosted on a Windows backend, the permission setup was
straightforward.
This was not the case in Linux. The Robocopy tool was not available for this OS, prompting a
different process. A Linux user would go through the additional preliminary steps of mounting

3. 3 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS
Environment
This is text for annotations in footer. Similar to footnotes treatment.
the remote directory as a superuser and ensuring the mount happened on login. Additional
packages needed to be installed (e.g., application wmctrl was needed for manipulating
windows management). Permissions also had to be set up correctly. To simplify the setup, a
script and installation recipe manual was written for Linux users. However, support calls
increased due to the complexity of the instructions, affecting quality of service.
The first generation of the file-transfer mechanism exposed the need for a simple workflow to
provide a better user experience: for instance, simple installation and a one-click upload. A
status bar could provide visual validation to display progress. Also, to reduce development
time, maintenance, and support, the succeeding version could be written in a single codebase
and employ the same mechanics for performing file transfer on different OSes.
Enter Perforce.
Current Version
The idea behind using Perforce as a back-end solution is to set up an intermediary “pre-
submit” Perforce depot that would house the files that users have modified. Once the files have
been “uploaded” to this Perforce depot, the PSDB solution can then “download” the files from
the depot and overlay them on the last-known-good version of a successful build. In Perforce
parlance, the modified files will be submitted to the pre-submit Perforce depot, and
synchronized by the PSDB for retrieval.
No changes are submitted to the main Perforce depot or SVN repository—all writeable actions
are done on the intermediary pre-submit Perforce depot. The intermediary depot exists only to
perform the file transfer. Use of this intermediary depot must be transparent to the user, as if
the depot didn’t exist.
Using Perforce
The following steps describe how AMD used Perforce to create the second generation of the
file-transfer mechanism.
1) Create a new workspace (clientspec) on the pre-submit depot.
A developer using Perforce as a version-control system already has an active
workspace for viewing and editing files. A new workspace is created on the pre-submit
depot with a View that is an exact duplicate of this active workspace’s View on the main
depot. Mimicking the active workspace guarantees that the files will be checked into the
correct relative locations in the pre-submit depot, no matter how convoluted the View is.
SVN does not adhere to the notion of a workspace, so files in a directory appear “as is”
in the repository (i.e., the new workspace View is wide, as in “//depot/…”). The danger
of a convoluted workspace is non-existent, and modified files in SVN can be uploaded
to the pre-submit Perforce depot as-is.
When creating the workspace on the pre-submit depot, ensure that the allwrite option is
enabled (i.e., the noallwrite option is disabled). Because this workspace is used to
submit the modified files into the pre-submit depot, these files should remain untouched
on the local computer from the main depot’s perspective. Enabling the allwrite option is
crucial because this ensures that the files are not switched to read-only after being
submitted to the pre-submit depot.

4. 4 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment
This is text for annotations in footer. Similar to footnotes treatment.
2) Create a pending changelist on the pre-submit depot using the new workspace;
populate this with the modified files.
This pending changelist will contain the modified files. The output of a p4 describe on
the description of the changelist on the main depot can be parsed to enumerate all files
that have been modified. These modified files on the main depot are then added to this
pending changelist, which will then be submitted to the pre-submit depot.
Modified files in SVN can be parsed from svn info. Other version-control systems would
have a similar command for listing modified files.
Files that have been deleted or removed (due to a rename or integrate/move) cannot be
added to this pending changelist. Instead, the description of the pending changelist will
enumerate these files. The PSDB mechanism will then remove these files from its copy
of the last-known-good in the next step, thereby accurately mimicking the modified state
on the user’s local computer.
3) Submit the pending changelist and trigger the pre-submit build.
Once the pending changelist has been submitted, the PSDB mechanism takes over.
PSDB takes a copy of the last-known-good version of the latest successful build,
overlays the added/edited files, removes the deleted/moved files as described in the
previous step, and initiates the accelerated build.
Benefits
By using Perforce as the main mechanism for transferring files from the user’s computer to the
PSDB server, a single codebase can be written that treats the Perforce command-line calls as
an API. With the codebase written in Perl, a write-once/run-on-all-OSes model is achieved.
The only per-OS task involved is packaging the Perl script into an executable for each OS.
Having Perforce as the back end means that the changelist submitted into the pre-submit
depot and its accompanying changelist description can be used as a rudimentary database for
keeping track of metadata and storage of modified files. Having a file server and separate SQL
database is not necessary.
AMD leveraged a Perforce installation that already existed at the company by simply creating a
new depot. We take advantage of the robustness inherent with Perforce, and merely extend its
application by piggybacking on an existing framework.
Another benefit is eliminating race conditions; when accessing the back end for uploading files,
each transaction—the Perforce submit—is atomic. Unlike FTP and rsync, this also makes
handling error conditions in the script simpler and more reliable than using an alternate data
store.
Shortcomings
By design, uploading modified files into the PSDB system can be done only through a
numbered pending changelist. This can be modified to use a default pending changelist (i.e.,
unnumbered), but numbered changelists force users to adhere to best practices.
The pre-submit depot can be filled quickly with PSDB requests. On a normal file server,
cleanups can be done easily by deleting older files. When using Perforce as an intermediary
depot, a p4 obliterate would be necessary to conserve space on the server. Maintenance
overhead can be reduced by having the obliterate command called when the server is not busy

5. 5 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS
Environment
This is text for annotations in footer. Similar to footnotes treatment.
and automated on a schedule.
Doing a PSDB request on a virtual integration is not possible because the modified files will
exist only on the main server, and not locally on the user’s computer. If a PSDB request is
necessary in this scenario, users are asked to perform the integration locally. (The actual
submission, if the PSDB tests pass, can be performed virtually.)
Given this, using a version control system as a file storage solution is overkill because
Perforce is used as an intelligent file server. However, the need to maintain a separate
database and dumb file server is removed, making this solution practical and efficient.
To implement this solution, an executable client that initiates this mechanism (i.e., the file
upload action) must be installed on the user’s local computer. Ensuring that the user has the
latest version of this client may be an issue. This is addressed in the next section.
Self Auto-Update
Software is a living document that gets updated frequently as bugs are fixed and features
added. Ideally, all users would be running the same latest version to have the best experience;
users benefit by having access to the newest features and fixes, and this also reduces
maintenance work because developers do not spend time debugging legacy versions.
However, deployment may be a necessary overhead because this requires the developer to
properly package the software (e.g., create the installer), make this available to users (i.e.,
announce and publish), and enforce (i.e., nag users) to ensure that users install the latest
version.
Web apps are not susceptible to this shortcoming. However, web apps have limitations if a
desired function can be executed only as a binary running on a local user’s computer (such as
uploading multiple files in the background — modern web browsers do not allow this because
it presents a security hole). In the case of the file-upload client, creating this client as a web
app was not an option.
The rationale behind a self auto-updating mechanism is to provide a seamless experience in
which the user is not required to perform any manual actions and is assured that the latest
version of the binary always is executed at run time.
Components
The self auto-updating mechanism consists of three entities: (1) a centralized version control
system, (2) the “caller” program installed on the user’s local computer, and (3) the “client” files
that are updated, also installed on the user’s local computer.
The centralized version control system is a file repository that tracks users’ files and their
revisions. Typically, the file repository is used for revision control: users check out files for
editing and check in files with the desired modifications. The file repository keeps track of
which files the users have on their computers, and at which revision. The file repository runs
on a server accessible to users running different OSes across the network. In our case, we
leveraged an existing Perforce implementation.
The caller is one part of the software application installed by the user manually. The caller is
not automatically updated. The caller consists of a Perforce command-line client and an
executable that calls the Perforce client with the necessary commands to perform the auto-

6. 6 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment
This is text for annotations in footer. Similar to footnotes treatment.
update.
The client files make up the rest of the software application installed by the user. These files
are the core of the installed software and handle the file upload functionality. This portion of the
software application can be updated on the user’s local computer when necessary.
Workflow
This section describes the workflow that Figure 1 illustrates.
1) Initially, the user installs the deployed software package from the author. On the local
user’s computer, the caller and client files are installed. When the user executes the
software application, the caller executable is first run in the background.
2) In the background, unknown to the user, the caller executable connects to Perforce
(using the Perforce command-line client) and creates a workspace for the user.
3) With this workspace, the caller synchronizes the client files from the Perforce server.
This action effectively overwrites the user’s local copy, thus assuring that the user’s
local copy is the latest version. When the sync is complete, the caller runs the client.
The client is essentially the core of the software, and is the portion that is updated and
executed.
Every time the user runs the software, the caller synchronizes with the Perforce server.
If all files are up to date, no action is necessary and the installed application (i.e., the
client) runs as usual.
4) When a new revision of the software is ready, the author checks in the modified client
files to the file repository. No further action is required from the author.
5) If the user runs the installed software after the update, the caller connects to Perforce
and detects that the client files on the user’s local computer are not up to date through
p4 sync.
6) The same p4 sync call grabs the latest version of the client files. After the sync, the
updated version of the client files—now the latest—is executed.

7. 7 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS
Environment
This is text for annotations in footer. Similar to footnotes treatment.
Figure 1: The workflow for the self auto-updating mechanism
Benefits
This method leverages the existing Perforce set up. A separate database or file server is not
necessary.
Whenever users initiate the file upload from their local computers (regardless of using Perforce
or SVN as their main repository), the caller ensures that the latest client is always
synchronized locally and executed. This happens entirely in the background, silently, and
users are not required to perform any explicit actions to update their local copy of the client.
The version number is logged on the submit changelist in the intermediary Perforce depot
each time the file upload is performed. Auditing these version numbers can verify that the auto-
update mechanism is working and that the user is running the latest version.
To verify initially that the auto-update mechanism is working, the package used for the first
installation contains a client that is one version behind. When the user runs the installer for the
first time, the auto-update mechanism engages and retrieves the latest version from the

8. 8 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment
This is text for annotations in footer. Similar to footnotes treatment.
Perforce server. The version number is then logged and inspected. If the version number is not
the latest, contact is initiated with the user to debug the issue. This proactive measure
improves the quality of service provided to users of the application.
Shortcomings
The full package initially installed by the user is not updated automatically; only the client files
are updated silently. This is sufficient for deploying features and bug fixes that affect file-
upload functionality.
If a full application upgrade is required (i.e., the caller needs to be updated), then the original
manner of deployment (i.e., packaging/publishing/enforcing) is pursued. However, this is
performed rarely because the caller portion of the software application has matured.
The caller currently consists of a separate Perforce command-line client and the executable
that calls the Perforce client. The executable calls the Perforce commands through the shell.
The executable can be rewritten to take advantage of existing Perforce APIs instead
Walk-through
The caller and client scripts are written in Perl and converted into executable binaries using
ActivePerl for Windows (32-bit) and Linux (Ubuntu 32-bit and 64-bit). The Windows 32-bit
executable can be used on 64-bit versions of Windows. These binaries are then packaged
using InstallShield (for Windows) or tarballed (Linux) and uploaded to a server, ready for
download.
The user downloads the package and runs the installer. The installer takes care of adding the
caller to the list of Custom Tools (see Figure 2). An install script written for Linux users
performs this task.
Figure 2: The caller added to the list of Custom Tools in P4V
The file-upload mechanism can be initiated by right-clicking a numbered pending changelist in
P4V for Perforce users (see Figure 3). In SVN, right-clicking a folder in Windows Explorer
presents this option. The file-upload mechanism also can be initiated from the command line.

9. 9 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS
Environment
This is text for annotations in footer. Similar to footnotes treatment.
Figure 3: Calling the file uploader mechanism from P4V
The caller validates all parameters and synchronizes the client. Figure 4 shows the client as up
to date, ensuring that the user is running the latest version. The caller then runs the client to
perform the file upload.
Figure 4: The log shows the caller running p4 sync; the client is at the latest version
The client parses the pending changelist using p4 describe and lists the files. A workspace
mimicking the user’s local workspace is created on the intermediary Perforce depot, and a
pending changelist is created. Added/modified files are included in this pending changelist.
Deleted/renamed/moved files are noted in the changelist description and will be removed from
the PSDB copy. Figure 5 shows these steps.

10. 1
0
Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment
This is text for annotations in footer. Similar to footnotes treatment.
Figure 5: The log shows the client preparing the files for upload
The pending changelist is then submitted to the intermediary Perforce depot. Once this has
been verified, a SOAP call is made to the PSDB mechanism to initiate the build. A web
browser is launched to display the progress of the build and the eventual test results. Figure 6
presents these steps.
Figure 6: the log shows the client uploading the files and making the SOAP call to initiate the build

11. 1
1
Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS
Environment
This is text for annotations in footer. Similar to footnotes treatment.
For simplicity, the user sees only a progress bar and status to demonstrate the stages of the
file upload during this time (see Figure 7). The user has the option to view the log if desired;
the log also appears when an error occurs. The user can then send the log to the PSDB team
for debugging.
Figure 7: Progress bars provide visual validation
Conclusion
This white paper describes an unintended use of Perforce to perform file transfers by
leveraging Perforce as an intelligent file storage server. Although this may be overkill for a
version-control system and other file transfer protocols such as FTP or SCP could have been
used, this novel method removes the overhead of setting up and maintaining a separate
database to manage the files that have to be transferred.
By taking advantage of Perforce’s robustness and its consistency of command calls being
supported on different operating systems, the described method can be applied without having
to maintain different codebases and set-up procedures for each OS.
The file transfer method is then extended into a self auto-updating mechanism to ensure that
users have the latest version of software installed on their local computers.