The Xen Project has been producing open source virtualization technologies for the past 10 years. It currently enables some of the largest clouds in the industry. However, decisions made on both the community and business fronts a few years back nearly caused the Xen Project to collapse on itself. Through a series of corrective actions, Xen is once again moving forward with major advances and a growing community. We will discuss some of the lessons learned from this experience, including lessons from both community and business vantage points. We will also focus on maintaining a healthy relationship between a community and a business entity as a means of keeping a project vibrant.

1. How to (Almost) Kill a Successful Project and Bring
It Back to Life Again
Russell Pavlicek
Xen Project Evangelist
Citrix Systems
Russell.Pavlicek@XenProject.org
Lessons Learned from the Xen Project

2. A guy with a lot of experience and a
really big mouth
So Who is the Old, Fat Geek Up Front?

3. • Linux user since 1995; Linux desktop since 1997
• Linux advocate before I ever saw the software
• Early advocate in DEC, Compaq
• Former columnist for Infoworld, Processor
• Former panelist on The Linux Show
• Wrote book, Embracing Insanity: Open Source Software
Development
• Speaker at 40+ conferences
• Currently Xen Project Evangelist employed by Citrix
About the Speaker...

4. • There is a Xen Project overview and status talk
later in the conference
• “Xen Project: A 10th Anniversary Status Report”
Sunday @2:45 PM
• If you want to learn about Xen Hypervisor
Architecture, details, and plans, please plan to
attend
About The Upcoming Talk

5. • We will spend a couple minutes reviewing the
project
• We will spend a few minutes considering its
history
• But we will spend the bulk of the time
considering lessons to take away
• We are not here for the project's history; we are
here for your future
About This Talk

6. • The premier Open Source Hypervisor
• Powering some of the biggest Clouds in the
industry (Amazon, Rackspace, Terremark)
• Celebrating its 10th Anniversary
• Now a Linux Foundation Collaborative Project
– Sponsoring organizations include: Google, AMD,
Intel, Samsung, Cisco, Oracle, Amazon, Verizon and
more
What is the Xen Project?

7. • The Xen Hypervisor, including ARM servers
– Type 1 (Bare Metal)
• Xen Cloud Platform & XAPI
– Cloud readiness out of the box
• Other Projects
– Mirage
– ARM Hypervisor for mobile devices
What Does the Xen Project Produce?

8. • It was the first industrial-strength Open Source
Hypervisor
• It enjoyed a very high rate of adoption
• It employed excellent technology
• It had a FOSS-friendly corporation behind it
• And, yet, 2 years ago, it ran the risk of being
abandoned by the FOSS community before its
10th birthday
The Xen Project Story (30 second
version)

9. • The project, though viable, developed an inward
focus
– Reach out to the rest of the Open Source community
was limited
– Reach out to its users was minimal
– Code development continued, but the community
became insulated and stagnated
– No one stepped up to contradict the rumor that Xen
was dying technology, overcome by competitors
How Did This Happen?

10. • The project forgot the importance of working
with its ecosystem
– Upstream projects (Linux Kernel, QEMU) were
branched rather than engaged with patches
– The project decided that others in the ecosystem
(i.e., the distributions) would have to carry the
burden of maintaining and supporting those
differences
– This went on too long, and the ecosystem got fed up
How Did This Happen? (continued)

11. • The corporation backing it (XenSource) was sold
to a company with a long closed source
software history (Citrix)
• The new corporation was interested in the
technology, but had no particular interest in the
project itself
How Did This Happen? (continued)

12. • It was not about malice
• It was not about fear
• It was about disconnection
– The project became disconnected from the FOSS
community
– The project became disconnected from the users
– The new company became disconnected from the
needs of the project, because, in part, the project
never really explained what it needed from the
company
Why Did This Happen?

13. • The prognosis was not good
• Xen Hypervisor had been overtaken by a
commercial offering in IT mindshare
• Xen Project had been overshadowed by another
Open Source Hypervisor in the community
• Distributions stopped facilitating Xen
• The FOSS Community began to forget Xen
About Two Years Ago: Battling for a
Future

14. • About 2 years ago, a new direction was plotted
• Citrix decided it wanted to understand FOSS and
reinvigorate the Xen community
• The company began to hire FOSS-savvy people
to reconnect with the community and with
users
• Brought about efforts to birth Apache
CloudStack, OpenDaylight, and to move the Xen
Project under the Linux Foundation
A Conscious Reversal in Direction

15. • Common themes heard at FOSS events:
– What is Xen?
– Xen is dead, right?
– Isn't Xen closed source?
– No one uses Xen anymore
Reality Two Years Ago: Xen Who?

16. • Linux kernel 3.0 contains all that Xen needs to
exist by default
• Most Linux distributions are Xen-enabled
– CentOS has a project to give RHEL6 users a Xen
option
• Xen Project now part of Linux Foundation
• Launch of a new user-friendlier website
(XenProject.org)
Reality Today: Xen is Back!

17. So What Did We Learn?

18. • It is possible to die while you are winning
– Being first is not enough
– Great technology is not enough
– Having a FOSS-friendly corporation behind you is not
enough
• A project must stay vibrant as an Open Source
organism or it will perish
Lesson 1

19. • Disconnection from users can make you a “Dead
Project Walking”
– You can be adding functionality, issuing new
releases, and still be dying
– The connection between project and users is
essential
– Focusing on software alone is not enough
• If you are not interacting with your users, you
are at serious risk
Lesson 2

20. • Connecting with your developers != connecting
with your users
– You need information sources for both users and
developers
– If users have to dig through technical websites,
wikis, etc. to answer simple questions, you are in
trouble
– Even Linux kernel development – arguably one of
the most insular projects – cannot thrive in a
vacuum
Lesson 2 (continued)

21. • Never ignore your project's Open Source root
structure
– Cut flowers are beautiful – until they die
– Living plants need their roots
– FOSS community is the root structure, and it must
spread wide
– The project team cannot stand alone
• Pay attention to your partners in FOSS: libraries,
kernel, packaging
Lesson 3

22. • Never ignore your support structure
– Xen needed cooperation from Distributions to be
properly supported
– The relationship with the distributions was allowed
to stagnate; it was not continuously cultivated
– When one distribution invested in another Open
Source virtualization solution, other distributions
were swayed
• Your distribution route can be critical to success
Lesson 4

23. • Having corporate backing is not enough
– The corporation has its own set of goals, and they
rarely align exactly with the project's goals
– When the project and the company don't mesh,
friction can occur
– This isn’t about good versus evil; business and
projects are just two separate animals with different
needs
Lesson 5

24. • Having a FOSS company backing you is no
guarantee
– Even FOSS-centric companies can be sold
– Sometimes they are sold to other FOSS companies
(e.g., JBoss, Gluster)
– Sometimes they are sold to closed source
companies (e.g., MySQL, Xen, Cassatt)
• If a project won’t survive without FOSS company
backing, consider options (e.g., Linux
Foundation)
Lesson 6

25. • In FOSS, there is no such thing as autopilot
– Intent is critical
– If you are not planning to succeed, you are planning
to fail
– Great software is not enough; you can have the best
technical solution, but if a “big dog” starts throwing
its weight around, you need to be able to respond
– If you're not looking at your whole ecosystem, you
are inviting failure
Lesson 7

26. • If it ain't growing, it's dying
– If your project team is seeing no new blood over
time, be concerned
– Open Source organisms must move and grow
– New folks are needed from time to time to add new
ideas and keep focus on what users need
Lesson 8

27. • Know where your project could fit in the world
and make a plan to get there
– Competition means you probably won't be best fit
for every situation
– It may not be possible to have every feature your
competitors have (especially if they have much
corporate backing)
– Figure out who your users are, what they need, and
what you need for them to use your code
Lesson 9

28. • Competition Increases Innovation
– Lack of competition can cause stagnation (consider
Unix CDE)
– Competing technologies keep the ball moving
forward continuously
– Xen's competition with KVM and VMware insured
that new virtualization capabilities would keep
flowing
– A competing project has to stay on top of its game
or it won't make it
Lesson 10

29. • Major new features can keep your mindshare
alive in the community
– Large advances (e.g., ARM and Mirage) generate
attention from the FOSS ecosystem and the
userbase
– If you aren’t making headway, your root and support
structures may stop working to give you what you
need
– Periodic advances keeps the project growing
Lesson 11

30. • Sometimes, perception really is reality
– You can have the best code in the world, but if no
one cares, it’s useless
– If the rumor arises that you are dying, outmoded, or
outdone by some other project, you must fight that
perception
– The unchallenged lie will become fact for many
people
Lesson 12

31. • In contrast, KVM managed perceptions well
– It could have looked like a purely Red Hat/IBM
business play when Red Hat purchased Qumranet
– The relationship between Red Hat and the KVM
project was well-defined & appropriate; no
disconnect occurred
– FOSS community embraced KVM project
– Clearly, Red Hat and IBM are still focusing major
business initiatives on KVM, but the community
accepts that because it was done correctly
Lesson 12 (continued)

32. • Go to local FOSS events
– Submit talks
• Get used to rejection and learn from it
• Talk to the track chairman
– “I don’t like speaking” – get over it; calculus was
way harder than this
– Get an ORG booth for cheap
• Stock it with flyers, CDs, business cards,
stickers
• Shoot your mouth off
– Blogs
– A usable website
– Podcasts (TLLTS)
– Social Media
• Twitter, Facebook, Google+, LinkedIn
• More mouthing off
– YouTube demos and tutorials
– Write for Linux.com LXer, LWN.net
• Get a “kick-*aas” mascot!
– But our buddy the Xen panda is taken!
• Shout out and live, or shut up and die!
– Passion is your ally
– Let it leak over everyone
– Don’t imitate the suits; do what fits you
Crash Course in Perception
Management

33. • There's a new reality for FOSS projects: the
corporate connection
– Projects used to be primarily volunteers working
nights and weekends
– Today, corporations play a big role in development
• You need to have a good grip on what your
corporate sponsors want from you, and what
you need from them; disconnection can be fatal
Lesson 13

34. • Manage the relationship between business and
project
– Prevent the loss of the project’s identity
– If the project appears “owned” by a business, the
FOSS community might become suspicious and back
away
– In this case, perception is as dangerous as reality
– If you forget what the project is, every else will too
– Project ecosystem will wither away; only the
business remains
Lesson 13 (continued)

35. • Establish a symbiotic relationship
– Business provides user feedback, resources
– Project provides overall focus, goal, direction, labor
– Both sides need to color in the lines
– Otherwise, you get “fake Open Source:” the code is
open, but there is no community, no support, no
ecosystem
Lesson 13 (continued)

36. • Make sure your project addresses its entire
ecosystem:
– Is the code good?
– Are you reaching out to your users?
– Is your development community active, engaged,
and growing?
– Are reaching out to the FOSS community?
– Have you insured you have proper support (libraries,
distros, kernels, etc.)?
Final Thoughts

37. • If you are in a relationship with a corporation, is
that relationship healthy?
– Do you have freedom to do what you need to do?
– Are you getting user feedback to seed new growth in
the project?
– Is your project's identity getting lost in the corporate
identity? (if so, consider a foundation route)
• Whatever else, don't give up!
Final Thoughts (continued)

38. Russell.Pavlicek@XenProject.org
XenProject.org
Questions?