Project-Builder.org is a tool to help upstream projects build their own packages for various Operating Systems directly from their VCS/CMS on a regular basis. It provides a continuous packaging approch

1. Project Builder:
A GPL continuous packaging solution
Bruno Cornec HP EMEA Open Source Profession Lead
V2.0 April 2013

2. Introducing Myself
●
Software engineering and Unices since 1988
●
Mostly Configuration Management Systems (CMS), Build systems, quality tools, on
multiple commercial Unix systems
●
Discover Open Source & Linux (OSL) & first contributions in 1993
●
Full time on OSL since 1995, first as HP reseller then @HP
●
Currently:
●
Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble
●
OSL HP Advocate
●
EMEA OSL HP Profession Lead
●
Solutions Linux Conference and OWF board member
●
MondoRescue, Dploy.org, Project-Builder.org project lead
●
LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor
●
FOSSBazaar and OSL Governance enthusiast
●
Mandriva, Mageia, Fedora packager
2 HP / OSSI / Bruno Cornec

3. What is the problem ?
Users / Sysadmins environment:
>= 90%
< 10%
Users: easyness (GUI, CLI)
Admins: distribution compliance, < 1%
smooth integration – reduce admin load
3 HP / OSSI / Bruno Cornec

4. What is the problem ?
Developers environment:
Sourceforge
Flat Files ftp, web
Project
CMS/VCS autotools ?
CVS Install proc ?
Soft. Eng. ?
SVN Fear of distro ?
GIT, ...
4 HP / OSSI / Bruno Cornec

5. What is the problem ?
Packagers environment:
Distro
ftp, web
Flat Files
Package
CMS/VCS
Does s/he only exists ? CVS
On time ? Lag behind ? rpmlint ?
Distro coverage (ver, #) SVN
lintian ?
Beta avail ? GIT, ...
5 HP / OSSI / Bruno Cornec

6. My answer with my various hats
I'm a sysadmin so I want to deal with packages only (deployment
server integration)
I'm a developer so I want to ease my tests without screwing up my
system and testing as a std user/admin
I'm a software engineer and want no duplication of code nor
information (metadata)
I'm a a packager so I want to streamline that work for new projects I'm
interested in packaging
I'm the marketing department of my own project and use packages as
a “competitive advantage”
Founding no tool existing to help me doing all that (2005), so I started
writing one to scratch my own itch
6 HP / OSSI / Bruno Cornec

7. What is the solution ?
Packaging should be a project concern as well as coding, testing,
installing, .... especially for smaller projects
New mantra: “Package early, package always”
Use packaging as your only way of delivery (not a dream)
Overhead minimal (not much longer than providing a tgz), maximum
benefit (consistancy, reproduceability, team work improvement,
distribution integration, improved deployment)
Packaging is also a marketing activity based on a technical content. It's
an easy way to extend your user base, and improve your community.
Makes tests easy.
THE SOLUTION IS CONTINUOUS PACKAGING
7 HP / OSSI / Bruno Cornec

8. Cloud based Continuous Packaging
Local Build Server (may host VMs and/or VEs)
Packagers VM or VE Build
Repository
Build + Server
metadata
Local build
Project
RM Build
Farm
(may host VMs) Remote build
Developers
9 HP / OSSI / Bruno Cornec

9. Provisioning the Continuous Packaging Cloud
Local Build Server (may host VMs and/or VEs)
VM or VE Build
Build +
metadata Local build
Repository
Server
Project
RM Build Farm (may
host VMs)
Remote build
10 HP / OSSI / Bruno Cornec

10. Goals
Project-builder main goal is to help you package continuously being
agnostic:
CMS agnostic: no CMS but guys it's 21st century now, SVN, CVS,
Mercurial, GIT, SVK....
OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 120+ distro
tuples at the moment – repositories for yum, urpmi, apt. Solaris pkg.
HP-UX sd in progress
Build environment agnostic: local, VM (QEMU, KVM...), VE
(rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm)
No project impact: preserves the md5sum of the delivered upstream
sources. Can be completely external to the upstream project.
11 HP / OSSI / Bruno Cornec

11. Goodies
Project-builder provides additional goodies:
Easy VMs/VEs/RMs management
Macro system with perl variables to avoid duplication
Skeleton generation to help starting
Manages delivery up to your repository (ssh based)
Manages announces on mailing lists
Integration of tests in the process
Manages patches/additional sources when not upstream
Checks validity of packages built (lintian, rpmlint)
Easy creation of new versions for upstream management
13 HP / OSSI / Bruno Cornec

12. VM, VE & RM Management
VE Creation:
debootstrap for .deb based ones.
rpmbootstrap (modeled after debootstrap): create chroot for
multiple RPM based distributions (Mageia, Fedora, Mandriva,
CentOS, OpenSuSE)
VM|RM Creation:
KVM or QEMU iso install possibilities.
LinuxCOE/OpenStack as provisioning tool for your VM|RM
VM|VE|RM setup: adds pb tools and build account
VM|VE Snapshot feature
Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg).
ssh communication for VM/RM
14 HP / OSSI / Bruno Cornec

13. Metadata Management
Packaging process independent of the project and external
Metadata isolated - No duplication
15 HP / OSSI / Bruno Cornec

14. Configuration file usage
Macro example:
./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar
to debootstrap for RPM based distributions
./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY
./rpmbootstrap/deb/control:Description: PBSUMMARY
./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY"
Instantiation possible from distro-ver-arch, distro-ver, distro, distro-
family, distro-type, os
Similar approach for all other parameters (Cf: man pb.conf)
16 HP / OSSI / Bruno Cornec

15. Commands
[cms|sbx]2build: Create tar files for the project under your CMS. Parameters are
packages to build if not using default list
build2pkg: Create packages for your running distribution
build2ssh: Send the tar files to a SSH host
pkg2ssh: Send the packages built to a SSH host
build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and
send those packages to a SSH host once built
launchvm: Launch one virtual machine
script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it
test2vm|ve|rm: Test inside a VM/VE/RM
newvm|ve: Create a new VM/VE/RM
setupvm|ve|rm: Setup a VM/VE/RM for pb usage
newver: Create a new version of the project derived from the current one
newproj: Create a new project and a template set of configuration files under
pbconf
17 HP / OSSI / Bruno Cornec

16. The Big picture
Packagers
newver / newproj
newvm|ve / setupvm|ve|rm
Packaging
build2vm|ve|rm
cms|sbx2build
Project Building Packages
build2pkg
pkg2ssh
build2ssh
Developers
18 HP / OSSI / Bruno Cornec

17. Multi platform life cycle
User Requirements Validation
Functional Integration tests
Specification
Design
Unit tests
Specification
19 Coding
HP / OSSI / Bruno Cornec

18. HP+Intel Solution Brief
21 HP / OSSI / Bruno Cornec

19. Multi platform packaging cycle
VMs/VEs/RMs
Developers
UUWL Packages
VM/VE/RM
23 Packagers HP / OSSI / Bruno Cornec

20. Q & A up to now ?
24 HP / OSSI / Bruno Cornec

21. DEMONSTRATION
25 HP / OSSI / Bruno Cornec

22. To be done
LSB chroot support. (or not)
LXC support
Support for libvirt, virsh, openstack API
Other CMS (Bazaar, …) only when/if needed
Other VMs (VMWare, Xen, ...) only when/if needed
Multiple delivery means
Look at interactions with Buildbot
Add non-interactive signature support for .deb
Config-Model for configuration file management
REST API and Web Interface
26 HP / OSSI / Bruno Cornec

23. Learn Project-Builder.org
Start with the Lab (63 pages)
Use man (pb, pb.conf + 8
ProjectBuilder::* man pages)
Use the mailing-list pb-announce
and pb-devel at
http://www.mondorescue.org/sympa
Use examples from
http://trac.project-builder.org/browser/projects/
27 HP / OSSI / Bruno Cornec

24. Web Resources
Project-Builder Web site / Trac / Wiki:
●
http://www.project-builder.org
●
http://trac.project-builder.org
Projects using project-builder.org:
●
http://www.project-builder.org (of course :-)
●
http://www.mondorescue.org
●
Http://www.linuxcoe.org
●
http://www.fossology.org
●
http://kde-apps.org/content/show.php/Meganizer?content=129907
28 HP / OSSI / Bruno Cornec

25. Related tools
Project-Builder is mostly suited for upstream projects wanting to package their
applications
Distributions provide each their build tools
● SuSE: Open Build Service (Multi distro, Web based, BaaS)
● Fedora (Koji)
● Mandriva/Mageia (Youri, mdvsys, mgarepo...)
Other complementary tools:
● Buildbot
● KVM/QEMU
● rpmbootstrap, rinse, mock
● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail
Project-Builder.org is one of the tool of the vcs-pkg.org initiative.
29 HP / OSSI / Bruno Cornec

26. Bruno_Cornec@hp.com
Contact
(Linux Solution Consultant in the HP/Intel
Solution Center)
http://www.hp.com/linux
”Changes are never easy to make.
There is comfort and safety in
tradition, but change must come,
no matter how painful or expensive
it may be.”
Bill Hewlett Linus Torvalds, Richard Stallman, Eric
Raymond, Nat Makarevitch, René
Cougnenc, Eric Dumas, Rémy Card, Bdale
Thanks Garbee, Bryan Gartner, Gallig Renaud,
Joachim Langenbach among others, for
their work and devotion to the Open
Source Software cause... and my family
Open Source at HP for his patience :-)
http://opensource.hp.com
30 HP / OSSI / Bruno Cornec