2. Me
Assistant professor (Maître de conférences) at Univ. de Lorraine
Teach Free Software in a system administration curriculum
(Licence Professionnelle Administration Système, Réseaux et
Applications à base de Logiciel Libre)
Debian involvement:
Collaboration between Debian and Ubuntu
Ruby packaging
Archive rebuilds
Ultimate Debian Database
lucas@debian.org Debian Quality Assurance 2 / 31
3. Debian
Well known and respected for:
lucas@debian.org Debian Quality Assurance 3 / 31
4. Debian
Well known and respected for:
Its long and bumpy release cycles
lucas@debian.org Debian Quality Assurance 3 / 31
5. Debian
Well known and respected for:
Its long and bumpy release cycles
Its quality
But it was not a given:
> 1000 Debian Developers
Mostly volunteers, with their own agendas
> 2000 Package maintainers
18117 software packages
Some very popular packages
A lot of niche packages
lucas@debian.org Debian Quality Assurance 3 / 31
7. Quality factors
Culture: "We release when it’s ready "
Many DDs won’t compromise on this
lucas@debian.org Debian Quality Assurance 4 / 31
8. Quality factors
Culture: "We release when it’s ready "
Many DDs won’t compromise on this
Packages ownership: individuals or small teams
Many of the maintainers are experts of their packages
lucas@debian.org Debian Quality Assurance 4 / 31
9. Quality factors
Culture: "We release when it’s ready "
Many DDs won’t compromise on this
Packages ownership: individuals or small teams
Many of the maintainers are experts of their packages
Release Critical (RC) bugs
Prevent packages from being part of a release
Everybody is welcomed to fix them
Non-Maintainer Uploads
lucas@debian.org Debian Quality Assurance 4 / 31
10. Quality factors
Culture: "We release when it’s ready "
Many DDs won’t compromise on this
Packages ownership: individuals or small teams
Many of the maintainers are experts of their packages
Release Critical (RC) bugs
Prevent packages from being part of a release
Everybody is welcomed to fix them
Non-Maintainer Uploads
Debian Quality Assurance team
lucas@debian.org Debian Quality Assurance 4 / 31
11. Debian QA team
Goal: Improve the quality of Debian as a whole
Not really a team (no strict membership), more like a central place
to discuss and work on QA
IRC: #debian-qa@irc.debian.org
Mailing list: debian-qa@lists.debian.org
lucas@debian.org Debian Quality Assurance 5 / 31
12. Debian QA team: Tasks
Run archive-wide checks and mass bug filings
Maintain QA infrastructure
Take care of the dirty areas of Debian
lucas@debian.org Debian Quality Assurance 6 / 31
14. Archive-wide checks and MBF
Developers are volunteers:
Focus on interesting things
Manual testing is boring → not done
Need automated tests
Other advantage: treats all packages equally
Niche packages are very important to some users
lucas@debian.org Debian Quality Assurance 8 / 31
15. Archive rebuilds
Rebuild all packages from source
7 hours on 40 build nodes (using Amazon EC2)
Detect packages that Fail To Build From Source (FTBFS)
Bugs in packages
Toolchain bugs
Failures caused by changes in other packages
Ran every 2-3 weeks
Scripts to analyze failures and file bugs semi-automatically
(about 200 bugs filed per hour)
Also used for test rebuilds with new compilers, linkers, . . .
→ Valuable information for upstream
lucas@debian.org Debian Quality Assurance 9 / 31
16. Debcheck
Check (statically) that dependencies can be satisfied
Not trivial: versioned dependencies, alternate depends, conflicts
2 versions:
"Simple one", in PHP
http://qa.debian.org/debcheck.php
Better one: edos-debcheck (EDOS project)
Can also work with RPMs
http://edos.debian.net/
lucas@debian.org Debian Quality Assurance 10 / 31
17. Piuparts
Testing package installation, upgrade and removal
Find bugs in maintainer scripts
(Shell) scripts executed during package installation/removal
Developers don’t run it on their systems
→ Archive-wide setup
http://piuparts.debian.org
lucas@debian.org Debian Quality Assurance 11 / 31
18. Lintian
Static analysis of packages
Usually run locally by the developer, before uploading
Archive-wide setup for continuous testing
Packages now rejected at upload time for some errors
Some example findings:
Use of deprecated packaging features
Common packaging mistakes
Embedded copies of libraries
Spelling errors in binaries
usefull visiters treshold targetted superseeded dependancies
splitted maintainance explicitely equiped informations mispelled
http://lintian.debian.org
lucas@debian.org Debian Quality Assurance 12 / 31
19. DEHS - Debian External Health Status
Monitor upstream version of packages
Detect when the Debian package is outdated
HTML scraping + regular expressions
Configured in debian/watch
lucas@debian.org Debian Quality Assurance 13 / 31
20. And more . . .
Scan build logs for dangerous warnings
Track security issues
Track popularity of packages
...
lucas@debian.org Debian Quality Assurance 14 / 31
22. Infrastructure: mitigating the data hell
Debian: aggregation of loosely-connected services
No Launchpad!
Everyone can start their own service
Data everywhere
A dozen of places to get interesting data
Tools required to gather data in a central place
and expose it to the maintainers
lucas@debian.org Debian Quality Assurance 16 / 31
25. Ultimate Debian Database (UDD)
Gather everything into a single SQL DB
Perfect tool for data-mining Debian
Currently imported in UDD:
Debian Sources and Packages, bugs, popcon, testing migrations, upload history,
orphaned packages, carnivore, lintian, debtags, translations, NEW queue,
screenshots, DEHS, ldap, wanna-build, removals, Ubuntu Sources and Packages,
Ubuntu bugs, Ubuntu popcon
Possible questions you can answer with UDD:
RC buggy packages in testing, sorted by popcon?
Packages for which the last 4 uploads were NMUs?
Packages not maintained by official DDs?
http://udd.debian.org/
lucas@debian.org Debian Quality Assurance 19 / 31
26. Some UDD queries
Packages with the highest number of different lintian errors or
warnings
select package, count(distinct tag) as cnt
from lintian
where tag_type in(’error’,’warning’)
group by package
order by cnt desc limit 15;
lucas@debian.org Debian Quality Assurance 20 / 31
27. Some UDD queries
Packages with the highest number of different lintian errors or
warnings
select package, count(distinct tag) as cnt
from lintian
where tag_type in(’error’,’warning’)
group by package
order by cnt desc limit 15;
package count
tsocks 18
pygopherd 18
cyclades-serial-client 17
muddleftpd 17
elib 17
lucas@debian.org Debian Quality Assurance 20 / 31
28. Some UDD queries
Who uploaded wheezy’s packages?
select changed_by, count(*) from sources s, upload_history uh
where s.source = uh.source and s.version = uh.version
and s.distribution=’debian’ and s.release = ’wheezy’
group by changed_by order by count desc limit 8;
lucas@debian.org Debian Quality Assurance 21 / 31
29. Some UDD queries
Who uploaded wheezy’s packages?
select changed_by, count(*) from sources s, upload_history uh
where s.source = uh.source and s.version = uh.version
and s.distribution=’debian’ and s.release = ’wheezy’
group by changed_by order by count desc limit 8;
changed_by count
gregor herrmann <gregoa@debian.org> 790
Joachim Breitner <nomeata@debian.org> 237
Christian Perrier <bubulle@debian.org> 229
Alessio Treglia <alessio@debian.org> 222
Salvatore Bonaccorso <carnil@debian.org> 207
Clint Adams <clint@debian.org> 206
Jonas Smedegaard <dr@jones.dk> 195
lucas@debian.org Debian Quality Assurance 21 / 31
30. Taking care of the dirty areas of Debian
lucas@debian.org Debian Quality Assurance 22 / 31
31. Taking care of the dirty areas of Debian
Two complementary approaches:
Focus on packages neglected by their maintainers
Focus on maintainers neglecting their packages
lucas@debian.org Debian Quality Assurance 23 / 31
32. Neglected packages
Already orphaned:
Do minimalistic maintenance
Problem: > 700 orphaned packages
Not (yet) orphaned:
Find them, orphan or remove them
Goal: Put maintenance in the hands of people with time
lucas@debian.org Debian Quality Assurance 24 / 31
33. Bapase: finding neglected packages
Multi-criteria search for neglected packages
Based on Ultimate Debian Database
http://udd.debian.org/bapase.cgi
lucas@debian.org Debian Quality Assurance 25 / 31
34. Inactive maintainers
Missing In Action (MIA) team
Find them, track them, orphan their packages
Mostly based on reporting by users or fellow developers
Echelon: records activity of DDs ; Carnivore: tracks identities
Collaboration with the Account Managers:
possible removal of their account
(Unused accounts → possible security problems)
lucas@debian.org Debian Quality Assurance 26 / 31
37. Use of VCSes
lucas@debian.org Debian Quality Assurance 29 / 31
38. Following Debian evolutions
Need to improve QA infrastructure
Team-aware
VCS-aware
Abstraction layer?
If I’ve learned anything, it’s that OSS projects can’t make a choice. And
when you can’t make a choice, what do you do? You support all options!
Yay! How do we support all options? An abstraction layer! Yay! Abstraction
layers are fun to write! Yay! – – Linux Hater’s Blog
Standard workflows?
Keep track of sponsorship requests
http://mentors.debian.org + sponsorship-requests
Track state of packages in VCS
Partial solution: PET (http://pet.debian.net/)
lucas@debian.org Debian Quality Assurance 30 / 31
39. Conclusions
Debian:
A large-scale volunteer-based distribution
A continuous effort to organize development and improve quality
lucas@debian.org Debian Quality Assurance 31 / 31