White Paper for NYSE Euronext's presentation by Adam Breashears and Valerie Hendrickson

1. Scalable
lifecycle
management
via
Perforce
Adam Breashears - Release Automation Tools & Services
Valerie Hendrickson - Release Management
Abstract
Managing complex, dynamic production environments where speed & accuracy of deployment intersect
with a requirement of deep information about the state of each environment (current and historical) and
a guarantee of application integrity is a hallmark of high-transaction low-latency environments. The same
qualities that make Perforce a fantastic source code repository, the attention to protecting and tracking a
company’s source and intellectual property, overlap to a large degree with the requirements of
maintaining a critical production environment.
Introduction1
NYSE Euronext faces a wide range of challenges stemming from its leading market position globally, its
recent mergers which created a global exchange and financial service provider, with offices and data
centers across two continents bound together and connected to its customer base by a global private
network the Secure Financial Transactions Infrastructure (SFTI) created and maintained by NYSE
Euronext. In addition to synergistic challenges created from the various mergers, the server infrastructure
maintained by the Release Management team at NYSE Euronext has expanded from 300 (as the start-up
Archipelago) to 9,000 servers (at peak during a data center migration), with a target of 6,000 globally by
EOY 2011, in a corporate culture which prizes increases in scalability and efficiency (alongside an
indifference to extremes in variance), not increases in headcount.
Continuous corporate growth that is fueled by both organic and M&A activity has necessitated a process
where variant processes can be made common at certain systemic points. Hardware and software
migrations take time and involve organizational risks, but 'synergies' require that they be managed as a
single 'environment' on both side of the life-cycle, whether it be radically different styles of SDLC entering
into the production environment or interactions with said production environment. There's no room for
1:1 solutions to development styles and environments. Therefore, virtual consistency must be high, and
certain functions must look the same for example software installs/rollbacks.
In addition to virtual consistency, a common prerequisite to our environment is the ability to have
absolute traceability path into the production environment, binary to source, configs/scripts to baselines,
and historical definitions of all changes. Every script run in a production environment must be available
for review at a later time and of course, all of this has to happen with no human driven interaction or
time delay to the ability to execute.
Perforce at NYSE Euronext is much more than a source code repository2), it is the foundation of
information transfer and validation between any software artifact that goes between production and non-
production. As such, it is under high load 24/7 and is considered a mission critical resource for IT.
What is the full scope of Life Cycle management?
There are two related concepts at NYSE Euronext in terms of Life Cycle management - Software Life
1
This should be read in conjunction with the presentation presented at the 2011 Perforce conference.
2
Perforce fulfills that function , along with many others to various groups within the company, however
here we'll be limiting the scope of discussion to its role in Production focused Life Cycle management

2. Cycle Management (controlling how software is used in production throughout its usable life cycle) and
the Software Development Life Cycle3 (SDLC) and both utilize Perforce as a critical component.
Specifically within the scope of this white paper we'll be focusing on the final cycles of the SDLC and the
production component of software utilization.
Some specific challenges that needed to be addressed (in no order of importance):
• Detecting unauthorized use/deployment
• Detecting unauthorized change (config/script/binary)
• Making sure software still exists on appropriate systems
• Creating common patterns of usage for unique systems
• Having full traceability from binary on machine X in production to source files in development
• Be able to deploy dozens of releases to thousands of servers in under an hour
• Verify explicit patterns of tested software configuration
• Rapidly adapt (intraday) to shifting asset states by the hundreds (new/deprecate old/changes)
• Be able to absolutely verify the software level of all assets real-time
• Remove any possible source of inaccuracy (humans) from the release/install process
How does Perforce help us accomplish this task?
Perforce specific features
The key features that Perforce offers its development community also coincide perfectly with the needs
of a secure production environment, scalability, fast data operations, designed to accommodate global
user communities/solutions, Easy administration/support, flexible/fast/Intuitive branching, an affordable
pricing model and world-class technical support4.
In addition to the 'high-level' value adds of the Perforce design, some more specific points include:
• Maintaining integrity of data objects along pipelines (by atomic change lists)
• Easy & secure integrations between secure dev/qa/production sections of the repository (as a
result of the branching system)
• Detailed trace of data objects as they route along pipelines for audit requirements (as a result of
the embedded versioning mechanism)
• Detailed trace of Perforce's own data objects, such as the client specs that represent meta-data
around production systems (spec depot)
• Protection of production meta-data objects from tampering (triggers)
• Secure sections within each repository for different levels of access (dev/qa/prod) combined with
triggers to explicitly limit access to appropriate users & domains
• With all software objects defined by change list and pulled from a central source, Quality
validation against known configurations can be explicitly defined and baselined for risk against
component specific (as opposed to release) level rollbacks.
• The production repository is a meta-map of all production machines due to the easy yet robust
flexibility of client specs (with spec components pulling from different repositories for 3rd party
software packages, Release objects, scripts, configs, etc)
3
SDLC - creating or altering systems, and the models and methodologies used to develop these systems
4
Reference Perforce website (www.perforce.com)

3. • Production machines are compared to baseline each night for possible change and additional of
extra components (p4 diff/p4 add)
• Production rollbacks occur within seconds across entire machine farms and are limited only by
dependencies on other applications (atomic changelists with release naming conventions in the
description guarantee precise conditions can be easily duplicated during a rollback).
As an example of scalability of this method, while keeping the same general level of release management
resources (who have to manage not only Perforce, but our release/issue tracking system, our build tool,
our custom automation and provide user & branching/depot support), we've gone from approximately
100 releases per human (over a server population of 300) in 2007 to 3000 releases per human (over a
server population of 6000 with a spike at the end of 2010 of 8,500 due to data center migrations) in
2011.
Implementation of Perforce as a Life Cycle management tool
Release Installation Process
All production machines are set up in Perforce with a 'meta-map' of what their software configuration
should be in production. The meta-map is a locked client spec, drawing from multiple sections of the
repository; with a particular syntax in its description that we can use to map the client to its production
host and purpose (for automation purposes it helps to have more data than just the 'host'). Where
multiple user accounts are required for applications, multiple client specs are created and locked to their
owner only (by depot permission settings). While the locked down client specs allow us to prevent
changes to the meta-map in production, new spec creation mapping to a production section is restricted
by triggers, only Release Management can create new meta-maps for production.
Each line of software has its own sections for development, quality assurance and production, with each
being integrated to the next through the p4 integrate command set. The descriptions of each have syntax
that define what level of software they represent in our release tracking tool, which also stores the
change list number corresponding to that release. The dual storage of release/changelist in each system
allows for easy access, depending on 'which way' the user is trying to get to a particular piece of data -
with the servers as the starting point (you'd look at the p4 have with the changelist description) or the
end point (where you'd look at the release tracking tool, identify the change list number and then ask to
install that number on a particular asset). Combined with a source of reference data (our Environment
Mapping Database) we can automatically create client-specs (and meta-maps) as needed depending on
business, software, and machine/OS type requested.
The end result of this allows for very specific & rapid environment configurations in multiple
environments and across multiple branches (in QA) as client-spec behavior has an attribute that when
you change the meta-map of the spec, it removes all of the old synced files and replaces with the new
ones, allowing groups to swap between completely different development efforts in minutes without
having to work about binary overlap of differing functional levels.
Automated Production Deployment
For deployment outside of local environments, multiple proxies are setup to reduce latency of files to the
local environment, but by far the most resource intensive component of the installation is the update to
the 'Have' table in Perforce (currently at 30 gig and rising), one of the most important components of a
front end Perforce server in this configuration is RAM (and local disk for the database - running release
installations in parallel generates high levels of database activity that can exceed SAN capacity).
Since Perforce requires a 'pull' methodology, there's a built in level of 'assurance' that we can't

4. 'accidentally' trigger an install at an inappropriate time. The 'install event' is a series of cascading scripts
(automatically generated each night before the installs) that specify what each client should be from the
client and under the watchful eye of our operations division.
Once a sync has been completed and the Have table updated (Perforce doesn't update the Have table
unless the file transfer is successful), release rollbacks are as simple as using the same scripted
methodology to sync to an earlier changelist, which removes all file versions submitted after that version,
removing the fear of having an untested release configuration in production (for example, version 7 of X
tested with version 5 of Y, but the requested rollback scenario only wants a rollback of X). Rollbacks to N-
1 releases are auto-generated and included with the release package, but rollbacks to earlier releases are
easily accommodated and take minutes to set-up and deploy. Should multiple systems require rollbacks
and there be confusion over the desired release levels, rollbacks using a variant of p4 sync @05/01/2011
are always possible as well (although fortunately we've never had to do this).
Once the software objects have been integrated into the production staging area, release times are
literally the time for any pre-install scripts to run5, a few milliseconds of database time, the raw speed of
an ftp transfer to the target system plus the time of any post-install scripts (in Perforce and a point of
variance depending on system design).
At the end of the install, all information results from scripted activity is stored in a text file and submitted
to Perforce for later use.
Post-Install
Once objects have been installed into the production environment, Perforce becomes the 'root' of our
application environment that we use to watch for change. Unlike applications that watch for file system
change, Perforce allows us to differentiate between known changes and unknown changes by doing a P4
diff against the 'have' version currently on the client. Run after shutdown and before we start for the day,
the diffs give us an excellent insight into application and configuration integrity before the beginning of a
trading day, and quickly highlight any changes made during the day based on decisions by operational
staff so those changes can be incorporated back into the baseline upon confirmation.
After each diff is run an email that groups all of the results by project area is sent out (sorted by those
projects with failures at the top) which links to an interface allowing current and historical perusal of diff
data, with links to breakdowns of the server farms and the actual diffs per machine. This allows us a
quick method of seeing the impacted business areas, the role of the servers effected and the timeliness
desired for follow-up and resolution.
With complete historical knowledge of software configurations, troubleshooting production issues
becomes much more focused, as every piece of information around how the system was setup at that
point in time is stored within Perforce and can be easily replicated in a lab environment or reviewed in
detail.
Operationally, the meta-map allows us to check the state of all software and configurations on a
production machine without actually going to the machine, which creates not only a very convenient and
easy to use method of checking any machine maintained by Perforce but also a much safer method of
giving visibility into machine specific setups without giving everyone access to secure areas.
5
Stored in Perforce and designed to validate various criteria per project, generically refer to as 'release
readiness checks' s consisting of steps ranging from checking disk space to validating user accounts and
directory permissions.

5. The information stored within the meta-map6 is used throughout the company in anything involving the
production environment, the visualization tools using Perforce data to view machines related to software
projects and roles, and the historical information about software levels on that box for use in scheduling
application deployments for phased roll-outs of new software. Furthermore, it will be used on ongoing
projects building management consoles that will dynamically manage and monitor resources from a point
of view related to the business unit and software project and not from the machine perspective.
Conclusion
Perforce adds value far beyond is initial role as a source code repository for NYSE Euronext, it is a fast
flexible information rich robust 'root' of our software production environment. As we increasingly look to
expand globally and our global resource pools find themselves working in far flung environments,
Perforce will continually provide a key common structural component that allows us to continue to scale
by growing our automation platforms and not our resource pools.
6
The combination of data from the diffs, baseline, client spec, change and have tables.