The secret plan behind the “3.0 (quilt)” Debian source package format

New source package formats do wondersWhile I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0″ format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0″). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0″ was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

  • Supports compression formats other than gzip: bzip2, lzma, xz.
  • Can use multiple upstream tarballs.
  • Can include binary files in the debian packaging.
  • Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
  • Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

  • it supports compression formats other than gzip: bzip2, lzma, xz.
  • it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

  • January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
  • March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
  • April 2008: I ask ftpmasters to support the new source packages in #457345.
  • June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
  • February 2009: Lenny release.
  • March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
  • November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
  • March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
  • October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
  • June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

Additional Resources

Get the Debian Administrator's Handbook

After a successful liberation campaign, the Debian Administrator's Handbook is now freely available. If you appreciate my articles and what I do for Debian, check out the book and grab a copy.

Comments

  1. What happened to 2.0?

    • 2.0 has been implemented at the same time than “3.0 (quilt)”, it was specified a long time ago under the name “wig&pen”. Before then, the initial support of that format by dpkg-source was partial: it was only able to unpack those package.

      In the end, this format will never be used in Debian, it’s similar to 3.0 (quilt) but the design is IMO inferior.

  2. Manuel A. Fernandez Montecelo says:

    Great work! I already converted the packages that I control (except one where co-maintainers are still not sure) as soon as almost 1 year ago — I didn’t know that it was not very mature by then!

  3. I appreciate the effort you put in putting together this post.

  4. Would a new source format be a good approach to solving the data package issue (where the packaging work is virtually entirely install(1)/dh_install, and the data is very large, so duplicating it between orig.tar.gz and _all.deb on the archives is a pain)

    • Jon, I don’t see how a new format could help here.

      A way to avoid the duplication would be for the .deb to be empty, a sort of data-installer. The postinst would fetch the .orig.tar.gz and install the required data. It would be a rather ugly hack.

      Or it could be the opposite, the source package could fetch the data at build time but official debian packages are not allowed to fetch stuff over the network. Maybe this requirement could be relaxed for the upcoming data.debian.org meant for this type of packages.

  5. Would’ve been nice if the next source format was just a git archive with a debian branch

    • Sven Joachim says:

      That’s already available as format 3.0 (git), currently marked as experimental. However, I don’t think such a format will ever be accepted by the FTP masters, because checking the whole git history for undistributable or non-DFSG-free files is much more work than with the current formats.

  6. I’m currently in the process of building a packaging system for a cloud framework that I have underdevelopment. I’ve essentially ignored compression protocols other than LZx and I was just wondering… what other compressions do you support, and … why?

Trackbacks

  1. [...] While I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this. More here [...]