dpkg – Page 9

The secret plan behind the “3.0 (quilt)” Debian source package format

October 21, 2010 by Raphaël Hertzog

New source package formats do wonders While I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0” format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0”). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0” was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

Supports compression formats other than gzip: bzip2, lzma, xz.
Can use multiple upstream tarballs.
Can include binary files in the debian packaging.
Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

it supports compression formats other than gzip: bzip2, lzma, xz.
it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
April 2008: I ask ftpmasters to support the new source packages in #457345.
June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
February 2009: Lenny release.
March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

Everything you need to know about conffiles: configuration files managed by dpkg

September 21, 2010 by Raphaël Hertzog

The Debian policy dictates that package upgrades must take care of preserving user changes to configuration files. This article will explain you how most packages ensure this. This is important knowledge for anyone who has to manage upgrades: knowing how it works lets you easily automate most of it and deal correctly with the fallout.

How dpkg manages configuration files

Most packages rely on dpkg to properly install configuration files. Dpkg keeps a checksum of the last installed version of configuration file. When it must install a new version, it calculates the checksum of the currently installed file and if it doesn’t match anymore, it knows that the user has edited the file. In that case, instead of overwriting the configuration file, it asks the user what to do. You probably already have seen those questions, they look like this:

Configuration file `/etc/bash.bashrc'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** bash.bashrc (Y/I/N/O/D/Z) [default=N] ?

In this specific example, if you answer “Y” or “I” (for “yes” or “install”), dpkg will install the new version of /etc/bash.bashrc but it will also backup the current version in /etc/bash.bashrc.dpkg-old. If you answer “N” or “O” (for “no” or “old”), dpkg will install the new version in /etc/bash.bashrc.dpkg-dist and /etc/bash.bashrc is left untouched. The two other answers allow you to examine the differences before taking a decision. Note that if you choose to start a shell, the new version is currently available as /etc/bash.bashrc.dpkg-new (and since Squeeze there are convenient environment variables $DPKG_CONFFILE_OLD and $DPKG_CONFFILE_NEW in case you want to create a custom review script).

All configurations files managed by dpkg are called “conffiles” because that’s the name of the field where they are recorded in the dpkg database. You can display the list of conffiles for any package:

$ dpkg --status bash
[...]
Conffiles:
 /etc/skel/.profile ecb6d3479ac3823f1da7f314d871989b
 /etc/skel/.bashrc 2afdd6c53990f2387a7ef9989af0bc07
 /etc/skel/.bash_logout 22bfb8c1dd94b5f3813a2b25da67463f
 /etc/bash.bashrc 5b3c3bc73d236e4e1b6f9b6c1ed5964e
[...]

The command “dpkg-query --showformat='${Conffiles}\n' --show bash” can give you the same information if you need to retrieve only that field. The 32 characters after the filename are the MD5 checksum of the original configuration file provided by the package.

Avoiding the conffile prompt

Every time that dpkg must install a new conffile that you have modified (and a removed file is only a particular case of a modified file in dpkg’s eyes), it will stop the upgrade and wait your answer. This can be particularly annoying for major upgrades. That’s why you can give predefined answers to dpkg with the help of multiple --force-conf* options:

--force-confold: do not modify the current configuration file, the new version is installed with a .dpkg-dist suffix. With this option alone, even configuration files that you have not modified are left untouched. You need to combine it with --force-confdef to let dpkg overwrite configuration files that you have not modified.
--force-confnew: always install the new version of the configuration file, the current version is kept in a file with the .dpkg-old suffix.
--force-confdef: ask dpkg to decide alone when it can and prompt otherwise. This is the default behavior of dpkg and this option is mainly useful in combination with --force-confold.
--force-confmiss: ask dpkg to install the configuration file if it’s currently missing (for example because you have removed the file by mistake).

If you use Apt, you can pass options to dpkg with a command-line like this:

$ apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

You can also make those options permanent by creating /etc/apt/apt.conf.d/local:

Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

Bringing up the conffile prompt at any time

The conffile prompt is only displayed when dpkg detects that the package provides an new version of the conffile. Thus reinstalling the same package will not bring up the prompt. But you can instruct dpkg to ask nevertheless with the --force-confask option. This is a new feature in Debian Squeeze. It will only ask for files that are locally modified.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

How to customize dpkg-source’s behaviour in your Debian source package

September 14, 2010 by Raphaël Hertzog

dpkg-source is the program that generates the Debian source package when a new package version is built. It offers many interesting command-line options but they are often not used because people don’t know how to ensure that they are used every time the package is built. Let’s fill that gap!

It is possible to forward some options to dpkg-source by typing them on the dpkg-buildpackage command line but you’d have to remember to type them every time. You could create a shell alias to avoid typing them but then you can’t have different options for different packages. Not very practical.

The proper solution has been implemented last year (in dpkg 1.15.5). It is now possible to put options in debian/source/options. Any long option (those starting with “--“) can be put in that file, one option per line with the leading “--” stripped.

Here’s an example:

# Bzip2 compression for debian.tar
compression = "bzip2"
compression-level = 7
# Do not generate diff for changes in config.(sub|guess)
extend-diff-ignore = "(^|/)config.(sub|guess)$"

Notice that spaces around the equal sign are possible contrary on the command line. You can use quotes around the value but it’s not required.

The debian/source/options file is part of the source package so if someone else grabs the resulting source package and rebuilds everything, they will use the options that you defined in that file.

You can also use debian/source/local-options but this time the file will not be included in the resulting source package. This is interesting for options that you want to use when you build from the VCS (Version Control Repository, aka git/svn/bzr/etc.) but that people downloading the resulting source package should not have. Some options (like --unapply-patches) are only allowed in that file to ensure a consistent experience for users of source packages.

You can learn more about the existing options in the dpkg-source manual page. Read it, I’m sure you’ll learn something. Did you know that you can tell dpkg-source to abort if you have upstream changes not managed by an existing patch in debian/patches? It’s --abort-on-upstream-changes and it’s only allowed in debian/source/local-options.

Be sure to subscribe to the RSS feed or to the email newsletter to not miss useful documentation for Debian contributors!

How to make 110.28 EUR in one month with free software and Flattr

August 12, 2010 by Raphaël Hertzog

Create an account on Flattr.com (mine is here);
Submit the URL of your free software project to the Flattr directory with a good description and the proper tags (here’s my entry for dpkg);
Add a Flattr button on your website (I put one on wiki.debian.org/Teams/Dpkg);
Inform your users and reach out to free software users using Flattr by registering your project in Flattr FOSS.

And I’m not alone, Joey Hess made 25 EUR in one week only.

And right now only 30K people are using Flattr. I expect this number to increase significantly now that invites are no longer required to open a Flattr account (see the announce on Flattr’s blog).

It’s time to spread the word about Flattr FOSS!