Weitere ähnliche Inhalte
Ähnlich wie The Great Migration: from TeamWare to Mercurial (20)
Kürzlich hochgeladen (20)
The Great Migration: from TeamWare to Mercurial
- 1. The Great
Migration:
from TeamWare to
Mercurial
James C. McPherson
Senior Kernel Engineer
Sun Microsystems
The Great Migration: from TeamWare to Mercurial, what we learned along the way
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 3. Introduction
January 2005
Sun releases DTrace under CDDL
14 June 2005
Sun releases the bulk of the core
Solaris kernel and basic userland
under CDDL
30000+ files, several million LoC
Source Code Management system?
SCCS wrapped in TeamWare
It was Time For A Change(tm)
The Great Migration: from TeamWare to Mercurial, what we learned along the way
3
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 4. Some nomenclature......
“Solaris” and “Solaris Express” are
constructed from several Consolidations:
OS and Networking, aka OS/Net, aka “ON”
Admin and Install, aka “AI”
Java
JDS (Desktop) and X
SunCluster
NetWorkStorage (now part of ON), and
xVM
The Great Migration: from TeamWare to Mercurial, what we learned along the way
4
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 5. A bit more nomenclature
Under TeamWare, each Consolidation is
housed in The Gate, which is updated to
The Clone every night (11pm US/Pacific is
the convention)
You putback to The Gate
You bringover from The Clone
The Gate and Clone filesystems are
exported read-only, but TeamWare's
metadata directories are exported read-
write
The Great Migration: from TeamWare to Mercurial, what we learned along the way
5
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 6. A picture, please?
The Gate
exported r/o
TeamWare metadata putback
exported r/w
Updated at 11pm
US/Pacific every night
Development
process
The Clone
exported r/o
TeamWare metadata bringover
exported r/w
The Great Migration: from TeamWare to Mercurial, what we learned along the way
6
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 7. Why not stick with TeamWare?
TeamWare's code was not Open
TeamWare had been EOL'd (End Of Life'd)
Getting patches for bugfixes was hard
Getting patches for new features was
really hard
TeamWare was ill suited to globally
distributed development, and
The operational model did not allow
moving Gates outside Sun
The Great Migration: from TeamWare to Mercurial, what we learned along the way
7
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 8. Development with TeamWare
TeamWare is not just a collection of
wrappers around SCCS, it also provides
application/scripting triggers, tracks name
changes, aggregates operations and even
provides checkpointing.
Heavy use of NFS – fine when all the
developers are in one building
Over time, engineers wrote their own
wrappers around TeamWare, to
workaround its shortcomings
The Great Migration: from TeamWare to Mercurial, what we learned along the way
8
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 9. Other development infrastructure
Developer scripts include
nightly – to build your workspace
wx (Workspace Extensions) – to
manage your workspace
bfu (Bonwick-Faulkner Upgrade)
webrev – generates webpages for
code review
The Great Migration: from TeamWare to Mercurial, what we learned along the way
9
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 10. What do these scripts do? (1)
nightly – general purpose gate and
workspace build script. So-called because
it's run every night on the gate source,
and used to take all night to run.
nightly is generally maintained by ON
engineers, and used by several other
consolidations as well
nightly has a number of options, including
package generation, lint checking, C and
Java style checking
The Great Migration: from TeamWare to Mercurial, what we learned along the way
10
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 11. What do these scripts do? (2)
wx sits atop TeamWare
wx helps track checked out files, bug and
ARC comments and sanity checks your
workspace on request
wx shields the user from most raw SCCS
operations
The Great Migration: from TeamWare to Mercurial, what we learned along the way
11
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 12. What do these scripts do? (3)
bfu, the Bonwick-Faulkner Upgrade, takes
cpio archives produced by 'nightly' and
blats them onto your system under test
Currently the fastest way of getting a new
kernel and consistent minimal userland
(commands + libraries) on your system.
Gate machines are bfu'd every morning –
we “eat our own dogfood”
The Great Migration: from TeamWare to Mercurial, what we learned along the way
12
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 13. What do these scripts do? (4)
Webrev generates web pages of changes
for easy code review
The Great Migration: from TeamWare to Mercurial, what we learned along the way
13
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 14. The Great Migration: from TeamWare to Mercurial, what we learned along the way
14
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 15. Gate infrastructure
Scripts written by the gate staff (yes,
called gatekeepers... no, there is no
keymaster)
nightly
buglist
backout – used for reverting bad
changes
lock-gate / unlock-gate – used when
large changes are integrated
update-flagdays
The Great Migration: from TeamWare to Mercurial, what we learned along the way
15
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 16. Time For A Change
Both Sun and the OpenSolaris Community
agreed that updating the SCM system was
required
OpenSolaris project created to oversee
this change: scm-migration
Requirements list generated by the
Community and tuned by the project
Candidate SCM systems evaluated against
requirements, evaluation reports
published on opensolaris.org
The Great Migration: from TeamWare to Mercurial, what we learned along the way
16
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 17. Major Requirements
Must be Open Source
Unbiased and disconnected distribution
Networked operation
Interface stability and completeness
Standard operations and transactions
Per-changeset metadata
For the full list, visit
http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/
The Great Migration: from TeamWare to Mercurial, what we learned along the way
17
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 18. Candidate SCM systems
SubVersion
Mercurial
Bazaar / Bazaar-NG
Git
Monotone
SVK
TeamWare
The Great Migration: from TeamWare to Mercurial, what we learned along the way
18
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 19. Evaluations
SubVersion, SVK, Monotone and
TeamWare were winnowed out via mailing
list discussion (tools-discuss) in the
preliminary phase
Bazaar, Git and Mercurial all continued to
the next phase
Evaluation reports are available at
http://www.opensolaris.org
The Great Migration: from TeamWare to Mercurial, what we learned along the way
19
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 20. Successful SCM candidate
Mercurial
The Great Migration: from TeamWare to Mercurial, what we learned along the way
20
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 21. Once Mercurial was chosen, the real work
could get started
.... and boy, was there a lot of it!
The Great Migration: from TeamWare to Mercurial, what we learned along the way
21
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 22. Migration tasks
Work divided into four major areas:
Developer-focused tools
Changing source in the gate
Gate infrastructure tools
Education
The Great Migration: from TeamWare to Mercurial, what we learned along the way
22
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 23. Developer-focused tools
Three major scripts needed updating:
nightly
wx
webrev
Many wx features were ported to
Mercurial extensions (thus written in
Python) and called “Cadmium”
nightly and webrev were enhanced to
understand Mercurial as well as
TeamWare
The Great Migration: from TeamWare to Mercurial, what we learned along the way
23
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 24. Why call it Cadmium?
http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm
The Great Migration: from TeamWare to Mercurial, what we learned along the way
24
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 25. Gate source changes
While managed by TeamWare, all files
had embedded SCCS ident strings:
%Z%%M% %I% %E% SMI
Since these are an anachronism with
Mercurial, they were removed in files that
were under active development at the
migration point
The Great Migration: from TeamWare to Mercurial, what we learned along the way
25
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 26. Gate infrastructure changes
The infrastructure saw a lot of the heavy
lifting for the migration, leveraging the
changes for the wx to Cdm work
Scripts (ksh88) were re-written in Python
as Mercurial hooks
These are still being tweaked, but the
majority of changes are bedded down
The Great Migration: from TeamWare to Mercurial, what we learned along the way
26
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 27. Gate back-end model
Images by David Marker, used with permission
The Great Migration: from TeamWare to Mercurial, what we learned along the way
27
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 28. New model: general
The Gate
hg push
Development
process
The Clone
Updated after every
successful hg push
(almost immediately) hg pull
The Great Migration: from TeamWare to Mercurial, what we learned along the way
28
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 29. New model: Open and Closed
Gate (Closed)
hg push
Gate (Open)
Separate hg push and hg pull/clone
required for Closed and Open parts
Development
process
Clone (Open)
Clone (Closed)
hg pull
Updated after every successful hg
push (ie, almost immediately)
The Great Migration: from TeamWare to Mercurial, what we learned along the way
29
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 30. ON Go-live
ON went live with Mercurial on 6 August
2008, when build snv_97 opened:
6733918 Teamware has retired, please
welcome your new manager, Mercurial
The Great Migration: from TeamWare to Mercurial, what we learned along the way
30
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 31. What did we learn?
The Great Migration: from TeamWare to Mercurial, what we learned along the way
31
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 32. What did we learn? (1)
The OpenSolaris community now knows a
lot more about Sun's internal processes
Switching to a new SCM is like a sausage
(it's really ugly while it's happening)
Corporate firewalls are necessary, but get
in the way of migrating
The longer you've been using the old
system, the harder it is to change
The longer you've been using the old
system, the more obscure its foibles
The Great Migration: from TeamWare to Mercurial, what we learned along the way
32
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 33. What did we learn? (2)
Your time estimates are too optimistic
Push up to date tools to the community as
soon as possible
We had (and have!) essential developer
tools that very few people understood
We have developer tools so old that the
last people to understand them properly
have retired
The better you know SCMs in general, the
easier your migration will be
The Great Migration: from TeamWare to Mercurial, what we learned along the way
33
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 34. What did we learn? (3)
The existing SCM dictates workflow
implementation, which influences what
developers see as gaps needing plugs
Engineers don't like change. At least, not
too much change
A consistent developer tool experience
requires SCM-agnosticism
Migrating to a modern, sophisticated SCM
is a great opportunity to change lots of
niggles that require coordination
The Great Migration: from TeamWare to Mercurial, what we learned along the way
34
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 35. What did we learn? (4)
The tools in use must support both the old
and new SCMs
developers need to migrate existing
workspaces
Non-integrated projects need to
change and sync up
Sustaining organisations need to
work with both SCMs
Productivity will dip while people work
their way up the learning curve
The Great Migration: from TeamWare to Mercurial, what we learned along the way
35
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 36. Finally....
SCM migration is an
all-or-nothing change
Your bags must be packed
for the journey
The Great Migration: from TeamWare to Mercurial, what we learned along the way
36
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 37. References
http://www.opensolaris.org/os/community/tools/scm
SCM Migration project page
http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/
SCM Migration requirements list
http://www.opensolaris.org/os/community/tools/scm/scm_milestone
SCM Migration milestones
http://www.opensolaris.org/os/community/tools/scm/on-scm-tools
ON consolidation tools
The Great Migration: from TeamWare to Mercurial, what we learned along the way
37
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 38. Further reading
http://opensolaris.org/os/community/on/os_dev_process/
OpenSolaris development process
http://opensolaris.org/os/community/on/devref_toc
OpenSolaris ON developer's reference
guide
An extended version of this presentation
will appear at http://blogs.sun.com/jmcp
The Great Migration: from TeamWare to Mercurial, what we learned along the way
38
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 39. What Questions Do You Have?
What Questions Do You Have?
What Questions Do You Have?
What Questions Do You Have?
What Questions Do You Have?
What
What Questions Do You Have?
Questions
What
What
Questions
Questions
Do
Do
You
You
Have?
Have?
Do You Have?
What Questions Do You Have?
What Questions Do You Have?
What Questions Do You Have?
The Great Migration: from TeamWare to Mercurial, what we learned along the way
39
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
- 40. The Great
Migration:
from TeamWare to
Mercurial
James C. McPherson
Senior Kernel Engineer
Sun Microsystems
jmcp@Sun.COM
The Great Migration: from TeamWare to Mercurial, what we learned along the way
James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved