The Great Migration: from TeamWare to Mercurial

The Great
Migration:
from TeamWare to
Mercurial

James C. McPherson
Senior Kernel Engineer
Sun Microsystems
The Great Migration: from TeamWare to Mercurial, what we learned along the way
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Introduction
Nomenclature and History
TeamWare
Replacement SCM Candidates
Mercurial
What we have learned
References

2

Introduction
January 2005
Sun releases DTrace under CDDL
14 June 2005
Sun releases the bulk of the core
Solaris kernel and basic userland
under CDDL
30000+ files, several million LoC
Source Code Management system?
SCCS wrapped in TeamWare
It was Time For A Change(tm)
3

Some nomenclature......
“Solaris” and “Solaris Express” are
constructed from several Consolidations:

OS and Networking, aka OS/Net, aka “ON”
Admin and Install, aka “AI”
Java
JDS (Desktop) and X
SunCluster
NetWorkStorage (now part of ON), and
xVM

4

A bit more nomenclature
Under TeamWare, each Consolidation is
housed in The Gate, which is updated to
The Clone every night (11pm US/Pacific is
the convention)
You putback to The Gate
You bringover from The Clone

The Gate and Clone filesystems are
exported read-only, but TeamWare's
metadata directories are exported read-
write
5

A picture, please?

The Gate
exported r/o
TeamWare metadata putback
exported r/w

Updated at 11pm
US/Pacific every night
Development
process

The Clone
exported r/o
TeamWare metadata bringover
exported r/w

6

Why not stick with TeamWare?
TeamWare's code was not Open
TeamWare had been EOL'd (End Of Life'd)
Getting patches for bugfixes was hard
Getting patches for new features was
really hard
TeamWare was ill suited to globally
distributed development, and
The operational model did not allow
moving Gates outside Sun

7

Development with TeamWare
TeamWare is not just a collection of
wrappers around SCCS, it also provides
application/scripting triggers, tracks name
changes, aggregates operations and even
provides checkpointing.
Heavy use of NFS – fine when all the
developers are in one building
Over time, engineers wrote their own
wrappers around TeamWare, to
workaround its shortcomings

8

Other development infrastructure
Developer scripts include
nightly – to build your workspace
wx (Workspace Extensions) – to
manage your workspace
bfu (Bonwick-Faulkner Upgrade)
webrev – generates webpages for
code review

9

What do these scripts do? (1)
nightly – general purpose gate and
workspace build script. So-called because
it's run every night on the gate source,
and used to take all night to run.
nightly is generally maintained by ON
engineers, and used by several other
consolidations as well
nightly has a number of options, including
package generation, lint checking, C and
Java style checking

10

wx sits atop TeamWare
wx helps track checked out files, bug and
ARC comments and sanity checks your
workspace on request
wx shields the user from most raw SCCS
operations

11

bfu, the Bonwick-Faulkner Upgrade, takes
cpio archives produced by 'nightly' and
blats them onto your system under test
Currently the fastest way of getting a new
kernel and consistent minimal userland
(commands + libraries) on your system.
Gate machines are bfu'd every morning –
we “eat our own dogfood”

12

Webrev generates web pages of changes
for easy code review

13

14

Gate infrastructure
Scripts written by the gate staff (yes,
called gatekeepers... no, there is no
keymaster)
nightly
buglist
backout – used for reverting bad
changes
lock-gate / unlock-gate – used when
large changes are integrated
update-flagdays
15

Time For A Change
Both Sun and the OpenSolaris Community
agreed that updating the SCM system was
required
OpenSolaris project created to oversee
this change: scm-migration
Requirements list generated by the
Community and tuned by the project
Candidate SCM systems evaluated against
requirements, evaluation reports
published on opensolaris.org

16

Major Requirements
Must be Open Source
Unbiased and disconnected distribution
Networked operation
Interface stability and completeness
Standard operations and transactions
Per-changeset metadata

For the full list, visit
http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/

17

Candidate SCM systems
SubVersion
Mercurial
Bazaar / Bazaar-NG
Git
Monotone
SVK
TeamWare

18

Evaluations
SubVersion, SVK, Monotone and
TeamWare were winnowed out via mailing
list discussion (tools-discuss) in the
preliminary phase

Bazaar, Git and Mercurial all continued to
the next phase
Evaluation reports are available at
http://www.opensolaris.org

19

Successful SCM candidate

Mercurial

20

Once Mercurial was chosen, the real work
could get started

.... and boy, was there a lot of it!

21

Migration tasks
Work divided into four major areas:

Developer-focused tools
Changing source in the gate
Gate infrastructure tools
Education

22

Developer-focused tools
Three major scripts needed updating:
nightly
wx
webrev
Many wx features were ported to
Mercurial extensions (thus written in
Python) and called “Cadmium”
nightly and webrev were enhanced to
understand Mercurial as well as
TeamWare
23

Why call it Cadmium?

http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm
24

Gate source changes
While managed by TeamWare, all files
had embedded SCCS ident strings:

%Z%%M% %I% %E% SMI

Since these are an anachronism with
Mercurial, they were removed in files that
were under active development at the
migration point

25

Gate infrastructure changes
The infrastructure saw a lot of the heavy
lifting for the migration, leveraging the
changes for the wx to Cdm work

Scripts (ksh88) were re-written in Python
as Mercurial hooks

These are still being tweaked, but the
majority of changes are bedded down

26

Gate back-end model

Images by David Marker, used with permission
27

New model: general

The Gate
hg push

Development
process
The Clone

Updated after every
successful hg push
(almost immediately) hg pull

28

New model: Open and Closed

Gate (Closed)
hg push
Gate (Open)
Separate hg push and hg pull/clone
required for Closed and Open parts

Development
process
Clone (Open)

Clone (Closed)
hg pull
Updated after every successful hg
push (ie, almost immediately)
29

ON Go-live
ON went live with Mercurial on 6 August
2008, when build snv_97 opened:

6733918 Teamware has retired, please
welcome your new manager, Mercurial

30

What did we learn?

31

What did we learn? (1)
The OpenSolaris community now knows a
lot more about Sun's internal processes
Switching to a new SCM is like a sausage
(it's really ugly while it's happening)
Corporate firewalls are necessary, but get
in the way of migrating
The longer you've been using the old
system, the harder it is to change
The longer you've been using the old
system, the more obscure its foibles
32

Your time estimates are too optimistic
Push up to date tools to the community as
soon as possible
We had (and have!) essential developer
tools that very few people understood
We have developer tools so old that the
last people to understand them properly
have retired
The better you know SCMs in general, the
easier your migration will be
33

The existing SCM dictates workflow
implementation, which influences what
developers see as gaps needing plugs
Engineers don't like change. At least, not
too much change
A consistent developer tool experience
requires SCM-agnosticism
Migrating to a modern, sophisticated SCM
is a great opportunity to change lots of
niggles that require coordination

34

The tools in use must support both the old
and new SCMs
developers need to migrate existing
workspaces
Non-integrated projects need to
change and sync up
Sustaining organisations need to
work with both SCMs
Productivity will dip while people work
their way up the learning curve
35

Finally....

SCM migration is an
all-or-nothing change

Your bags must be packed
for the journey

36

References

http://www.opensolaris.org/os/community/tools/scm
SCM Migration project page
http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/

SCM Migration requirements list
http://www.opensolaris.org/os/community/tools/scm/scm_milestone

SCM Migration milestones
http://www.opensolaris.org/os/community/tools/scm/on-scm-tools

ON consolidation tools

37

Further reading

http://opensolaris.org/os/community/on/os_dev_process/
OpenSolaris development process

http://opensolaris.org/os/community/on/devref_toc
OpenSolaris ON developer's reference
guide

An extended version of this presentation
will appear at http://blogs.sun.com/jmcp

38

What Questions Do You Have?
What
Questions
What
What
Questions
Questions
Do
Do
You
You
Have?
Have?
Do You Have?
39

The Great
Migration:
from TeamWare to
Mercurial
James C. McPherson
Senior Kernel Engineer
Sun Microsystems
jmcp@Sun.COM

The Great Migration: from TeamWare to Mercurial

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie The Great Migration: from TeamWare to Mercurial

Ähnlich wie The Great Migration: from TeamWare to Mercurial (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The Great Migration: from TeamWare to Mercurial