apt-get install debian-wizard

Insider infos, master your Debian/Ubuntu distribution

  • About
    • About this blog
    • About me
    • My free software history
  • Support my work
  • Get the newsletter
  • More stuff
    • Support Debian Contributors
    • Other sites
      • My company
      • French Blog about Free Software
      • Personal Website (French)
  • Mastering Debian
  • Contributing 101
  • Packaging Tutorials
You are here: Home / Archives for News / Ubuntu News

The secret plan behind the “3.0 (quilt)” Debian source package format

October 21, 2010 by Raphaël Hertzog

New source package formats do wondersWhile I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0” format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0”). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0” was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

  • Supports compression formats other than gzip: bzip2, lzma, xz.
  • Can use multiple upstream tarballs.
  • Can include binary files in the debian packaging.
  • Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
  • Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

  • it supports compression formats other than gzip: bzip2, lzma, xz.
  • it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

  • January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
  • March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
  • April 2008: I ask ftpmasters to support the new source packages in #457345.
  • June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
  • February 2009: Lenny release.
  • March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
  • November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
  • March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
  • October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
  • June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

Understanding Membership Structures in Debian and Ubuntu

August 30, 2010 by Raphaël Hertzog

Debian and Ubuntu have a set of official membership roles that can be granted to regular contributors. Those roles come with rights that enable the contributors to do their work and to participate in the project governance (elections and other official decision-making processes). It’s also a way for the distributions to acknowledge the work done: most contributors are proud of the status they reached.

The membership structure plays an important role in the development of a distribution: it defines the kind of contributors that are welcome in the project, it sets expectations of the project towards its contributors and defines their rights. In the end, this shapes the project’s ability to recruit new contributors to keep the project alive and kicking. This article introduces the existing statuses in Debian and Ubuntu, and defines the — sometimes confusing — jargon associated with them.

The Debian Case

Debian only has two types of official members: Debian Developers (DD) and Debian Maintainers (DM). The rights of the developers are codified in the Debian Constitution while those of the maintainers have been defined in a general resolution of 2007. The Debian Maintainer status is still mostly documented in a wiki page. The integration of this new status in Debian’s official processes has been slow to come largely because it was introduced — at that time — without enough negotiation with the involved parties. Nowadays, it is preferred that people get the DM status before applying for DD.

DM is a very limited role: maintainers can only upload packages that already have their name on them (either in the Maintainer or Uploaders field) and a specific flag (DM-Upload-Allowed: yes) that only Debian Developers can add. They have no other rights and limited access to Debian’s resources.

Besides those official roles, there are also maintainers of packages that have no official status within Debian except that they are listed in the “Maintainer” field of the package. They are doing the maintenance work but all uploads are done by a Debian Developer after verification of the work done (this is called “sponsorship” and is the only way to start with official packaging work). Once the DD trusts the maintainer, the developer will typically ask the maintainer to apply for DM status in order to be relieved from the sponsorship work.

In the end, that makes three different kind of package maintainers and a lot of confusion when you discuss membership issues… in particular when the New Maintainer process is the path that you follow to become a Debian Developer. Don’t be fooled by the names when reading Debian’s documentation!

The Ubuntu Case

Ubuntu had, from the start, an official Ubuntu Member status that includes all contributors: developers of course, but also documentation writers, artists, translators, etc. This status notably grants the right to vote in elections of the Community Council, the right to participate on Planet Ubuntu, and the @ubuntu.com email alias.

For developers, the situation is more complicated: the wiki page lists no less than five different statuses. Initially, developers were split between Ubuntu Core Developers and the MOTU (Masters Of The Universe). The latter were responsible of the universe/multiverse sections of the archive while the former also had upload rights for the main/restricted sections. But, inspired by the Debian Maintainers concept and facing concrete problems in terms of archive management, they changed their infrastructure to offer more fine-grained control on package uploads.

Ubuntu can now grant upload permissions on a package-per-package basis, but it can also delegate the right to grant upload permissions with the same granularity. This lead to the new Per-Package Uploader status which is simply an Ubuntu Member with upload rights on a limited set of packages where they have a specific expertise. The more generic Ubuntu Developers status now encompasses members of various development teams that have been delegated the right to manage upload permissions on a (usually large) package set (the current teams are Ubuntu Desktop, Mythubuntu, Kubuntu, and Edubuntu). Those teams can define their own policy to add new members provided they follow the basic rules defined by the Developer Membership Board (see this wiki page).

Ubuntu Contributing Developer is an intermediate status for someone who is not yet ready for one of the other developer statuses but who has still shown enough commitment to be an Ubuntu Member.

All those statuses can be obtained in a similar way: you prepare a wiki page listing your past contributions, you collect testimonials from existing members that you have worked with, you add yourself in the agenda of the next meeting of the board (or council) that grants the status that you seek, and you attend the meeting. The members of the board will decide whether you are ready for the status (or not) based on what you provided in the wiki, based on your answers during the meeting (and on a mailing-list for developers), and based on what others have to say about you.

The most important boards are usually elected by the community while others are commonly appointed by the community council. Those governance bodies include Canonical employees but not as many as one would expect: two out of eight in the Developer Membership Board, two out of eight in the Community Council, but all six members of the Technical Board. The last figure, while not intended, is not surprising given the high expectations set on potential members of the technical board. Mark, as the founder, is the only person to have a permanent seat on both the Community Council and the Technical Board.

Comparison of the Statuses Between Debian and Ubuntu

The following table summarizes the rights given to each developer role in the two projects (Put the mouse over the abbreviations to know what they are referring to).

Rights Debian Ubuntu
DM DD UM PPU/UD MOTU UCD
Package maintenance via sponsorshipYN/AYYYN/A
Official email alias–YYYYY
Participate in votes for members–YYYYY
Participate in votes for developers–Y–YYY
Upload rights restricted to pre-approved packagesY––Y––
Upload rights restricted to a section of the archive––––Y–
Unlimited upload rights–Y–––Y
Number of contributors (as of 2010-07-27)117904462278563

Please note that the number of contributors are not 100% accurate for Ubuntu. A contributor can have multiple statuses (direct membership to a launchpad group) granted over time (while gaining experience). The problem has been mostly avoided by calculating differences between number of members of the various groups but it’s not perfect and it can’t be: some MOTU are also PPU for packages in main and it’s legitimate (but I only counted them as MOTU and not as PPU). Another limitation is that members of some administrative teams are included indirectly in many teams and thus appear in the count while they should not.

Anyway, this simple table makes it obvious that Ubuntu’s structure offers a broader choice of statuses. They acknowledge the work of all contributors from the start while still giving the most critical rights only to those who have proven that they deserve them. Despite this difference, Debian still has a significant advantage in terms of number of developers. That number does not tell the whole story though: the Ubuntu contributors include many Canonical employees (e.g. 36 out of 63 core developers have a @canonical.com email registered on their launchpad account) that are likely to spend more time working on the distribution than the average Debian member. But even if comparing person-hours would be a challenging thought experiment, in practice it’s of not much interest if both projects continue to cooperate and if more and more of the contributions flow in both directions.

Debian is aware of the shortcomings of its structure. Changes to better accommodate non-packagers have been discussed several times already. The last efforts in that direction were unfortunately perceived as a solution ready to go rather than a proposal to be discussed, and the project got quickly buried by a general resolution (GR). Even if that resolution invited for further discussion and a new proposal, the truth is that when someone’s initiative is “corrected” by way of GR, it usually kills any motivation to go forward.

Possible Evolution?

On the Ubuntu side, the infrastructural changes were completed recently and they don’t expect any further change in the near future. They do plan, however, to expand usage of those new features so that more teams benefit from the possibility to control upload rights on packages that are relevant to them, and so that more individuals developers apply to become Per-Package Uploaders on packages that they know very well.

On the Debian side, a recent discussion on the debian-project list brought back the topic of the bad terminology and it was agreed that the “New Maintainer process” should be renamed into something else (“New Developer process” has been suggested). But Christoph Berg — Debian Account Manager and hence heavily involved in the New Maintainer Team — suggested that Debian would be better off implementing the long-awaited membership changes before trying to update all the documentation. It would certainly imply some more vocabulary updates. Later in the discussion, he confirmed that membership reform is on the top of the TODO list of the new maintainer team (just after the rewrite of the nm.debian.org website).

What can be expected from this reform? The following answers are my own guesses based on my experience of Debian, but the project hasn’t decided anything yet.

  • First of all: a new status for contributors that are not packagers. The tricky part will be defining the process to follow and the rights granted.

  • Changes to the technical implementation of the DM status. The current implementation does not allow to give upload rights to a single DM if two are listed in the Uploaders field of a package (and both might not have the same experience for that package). Furthermore, it suffers from annoying restrictions like the inability to upload new binary packages.

  • A change of the Debian constitution to integrate those new statuses is almost unavoidable.

  • Other more invasive changes have been proposed like replacing the NM process by a simple designation by other DD, but it’s unlikely to happen. The NM process can already be greatly simplified by the application manager if the applicant can show good testimonials from other developers and if he has a track record of real contributions (e.g. as witnessed by changelog entries in Debian packages).

Almost two years have elapsed since the previous efforts in that direction, the new maintainer team has recruited new members and is in a general better shape. Hopefully, the next episode of this saga will have a better outcome.

This article was first published in Linux Weekly News. In a comment, Mark Shuttleworth tried to explain how the Ubuntu community is being setup.
  • « Previous Page
  • 1
  • 2
  • 3
  • 4

Get the Debian Handbook

Available as paperback and as ebook.
Book cover

Email newsletter

Get updates and exclusive content by email, join the Debian Supporters Guild:

Follow me

  • Email
  • Facebook
  • GitHub
  • RSS
  • Twitter

Discover my French books

Planets

  • Planet Debian

Archives

I write software, books and documentation. I'm a Debian developer since 1998 and run my own company. I want to share my passion and knowledge of the Debian ecosystem. Read More…

Tags

3.0 (quilt) Activity summary APT aptitude Blog Book Cleanup conffile Contributing CUT d-i Debconf Debian Debian France Debian Handbook Debian Live Distro Tracker dpkg dpkg-source Flattr Flattr FOSS Freexian Funding Git GNOME GSOC HOWTO Interview LTS Me Multiarch nautilus-dropbox News Packaging pkg-security Programming PTS publican python-django Reference release rolling synaptic Ubuntu WordPress

Recent Posts

  • Freexian is looking to expand its team with more Debian contributors
  • Freexian’s report about Debian Long Term Support, July 2022
  • Freexian’s report about Debian Long Term Support, June 2022
  • Freexian’s report about Debian Long Term Support, May 2022
  • Freexian’s report about Debian Long Term Support, April 2022

Copyright © 2005-2021 Raphaël Hertzog