Contributor Documentation

Howto to rebuild Debian packages

December 15, 2010 by Raphaël Hertzog

Being able to rebuild an existing Debian package is a very useful skill. It’s a prerequisite for many tasks that an admin might want to perform at some point: enable a feature that is disabled in the official Debian package, rebuild a source package for another suite (for example build a Debian Testing package for use on Debian Stable, we call that backporting), include a bug fix that upstream developers prepared, etc. Discover the 4 steps to rebuild a Debian package.

1. Download the source package

The preferred way to download source packages is to use APT. It can download them from the source repositories that you have configured in /etc/apt/sources.list, for example:

deb-src http://ftp.debian.org/debian unstable main contrib non-free
deb-src http://ftp.debian.org/debian testing main contrib non-free
deb-src http://ftp.debian.org/debian stable main contrib non-free

Note that the lines start with “deb-src” instead of the usual “deb”. This tells APT that we are interested in the source packages and not in the binary packages.

After an apt-get update you can use apt-get source publican to retrieve the latest version of the source package “publican”. You can also indicate the distribution where the source package must be fetched with the syntax “package/distribution“. apt-get source publican/testing will grab the source package publican in the testing distribution and extract it in the current directory (with dpkg-source -x, thus you need to have installed the dpkg-dev package).

$ apt-get source publican/testing
Reading package lists... Done
Building dependency tree       
Reading state information... Done
NOTICE: 'publican' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/collab-maint/publican.git
Need to get 727 kB of source archives.
Get:1 http://nas/debian/ squeeze/main publican 2.1-2 (dsc) [2253 B]
Get:2 http://nas/debian/ squeeze/main publican 2.1-2 (tar) [720 kB]
Get:3 http://nas/debian/ squeeze/main publican 2.1-2 (diff) [4728 B]
Fetched 727 kB in 0s (2970 kB/s)  
dpkg-source: info: extracting publican in publican-2.1
dpkg-source: info: unpacking publican_2.1.orig.tar.gz
dpkg-source: info: unpacking publican_2.1-2.debian.tar.gz
$ ls -dF publican*
publican-2.1/                 publican_2.1-2.dsc
publican_2.1-2.debian.tar.gz  publican_2.1.orig.tar.gz

If you don’t want to use APT, or if the source package is not hosted in an APT source repository, you can download a complete source package with dget -u dsc-url where dsc-url is the URL of the .dsc file representing the source package. dget is provided by the devscripts package. Note that the -u option means that the origin of the source package is not verified before extraction.

2. Install the build-dependencies

Again APT can do the grunt work for you, you just have to use apt-get build-dep foo to install the build-dependencies for the last version of the source package foo. It supports the same syntactic sugar than apt-get source so that you can run apt-get build-dep publican/testing to install the build-dependencies required to build the testing version of the publican source package.

If you can’t use APT for this, enter the directory where the source package has been unpacked and run dpkg-checkbuilddeps. It will spit out a list of unmet build dependencies (if there are any, otherwise it will print nothing and you can go ahead safely). With a bit of copy and paste and a “apt-get install” invocation, you’ll install the required packages in a few seconds.

3. Do whatever changes you need

I won’t detail this step since it depends on your specific goal with the rebuild. You might have to edit debian/rules, or to apply a patch.

But one thing is sure, if you have made any change or have recompiled the package in a different environment, you should really change its version number. You can do this with “dch --local foo” (again from the devscripts package), replace “foo” by a short name identifying you as the supplier of the updated version. It will update debian/changelog and invite you to write a small entry documenting your change.

4. Build the package

The last step is also the simplest one now that everything is in place. You must be in the directory of the unpacked source package.
Now run either “debuild -us -uc” (recommended, requires the devscripts package) or directly “dpkg-buildpackage -us -uc”. The “-us -uc” options avoid the signature step in the build process that would generate a (harmless) failure at the end if you have no GPG key matching the name entered in the top entry of the Debian changelog.

$ cd publican-2.1
$ debuild -us -uc
 dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): 
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): 
dpkg-buildpackage: source package publican
dpkg-buildpackage: source version 2.1-2rh1
dpkg-buildpackage: source changed by Raphaël Hertzog 
 dpkg-source --before-build publican-2.1
dpkg-buildpackage: host architecture i386
[...]
dpkg-deb: building package `publican' in `../publican_2.1-2rh1_all.deb'.
 dpkg-genchanges  >../publican_2.1-2rh1_i386.changes
dpkg-genchanges: not including original source code in upload
 dpkg-source --after-build publican-2.1
dpkg-buildpackage: binary and diff upload (original source NOT included)
Now running lintian...
Finished running lintian.

The build is over, the updated source and binary packages have been generated in the parent directory.

$ cd ..
$ ls -dF publican*
publican-2.1/                    publican_2.1-2rh1.dsc
publican_2.1-2.debian.tar.gz     publican_2.1-2rh1_i386.changes
publican_2.1-2.dsc               publican_2.1-2rh1_source.changes
publican_2.1-2rh1_all.deb        publican_2.1.orig.tar.gz
publican_2.1-2rh1.debian.tar.gz

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

4 tips to maintain a “3.0 (quilt)” Debian source package in a VCS

November 18, 2010 by Raphaël Hertzog

Most Debian packages are managed with a version control system (VCS) like git, subversion, bazaar or mercurial. The particularities of the 3.0 (quilt) source format are not without consequences in terms of integration with the VCS. I’ll give you some tips to have a smoother experience.

All the samples given in the article assume that you use git as version control system.

1. Add .pc to the VCS ignore list

.pc is the directory used by quilt to store its internal data (list of applied patches, backup of modified files). It’s also created by dpkg-source so that quilt knows that the patches are in debian/patches (and not in patches which is the default directory used by quilt). For that reason, the directory is kept even if you unapply all the patches.

However you don’t want to store this directory in your repository, so it’s best to put it in the VCS ignore list. With git you simply do:

$ echo ".pc" >>.gitignore
$ git add .gitignore
$ git commit -m "Ignore quilt dir"

The .gitignore file is ignored by dpkg-source, so you’re not adding any noise to the generated source package.

2. Unapply patches after the build

If you store upstream sources with non-applied patches (most people do), and if you don’t build packages in a temporary build directory, then you probably want to unapply the patches after the build so that your repository is again in a clean status.

This is now the default since dpkg-source will unapply any patch that it had to apply by itself. Thus if you start the build with a clean tree, you’ll end up with a clean tree.

But you can still force dpkg-source to unapply patches by adding “unapply-patches” to debian/source/local-options:

$ echo "unapply-patches" >>debian/source/local-options
$ git add debian/source/local-options
$ git commit -m "Unapply patches after build"

svn-buildpackage always builds in a temporary directory so the repository is left exactly like it was before the build, this option is thus useless. git-buildpackage can also be told to build in a temporary directory with --git-export-dir=../build-area/ (the directory ../build-area/ is the one used by svn-buildpackage, so this option makes git-buildpackage behave like svn-buildpackage in that respect).

3. Manage your quilt patches as a git branch

Instead of using quilt to manage the Debian-specific patches, it’s possible to use git itself. git-buildpackage comes with gbp-pq (“Git-BuildPackage Patch Queue”): it can export the quilt serie in a git branch that you can manipulate like you want. Each commit represents a patch, so you want to rebase that branch to edit intermediary commits. Check out the upstream documentation of this tool to learn how to work with it.

There’s an alternative tool as well: git-dpm. Its website explains the principle very well. It’s a more complicated than gbp-pq but it has the advantage of keeping the history of all branches used to generate the quilt series of all Debian releases. You might want to read a review made by Sam Hartman, it explains the limits of this tool.

4. Document how to review the changes

One of the main benefit of this new source format is that it’s easy to review changes because upstream changes are kept as separate patches properly documented (ideally using the DEP-3 format). With the tools above, the commit message becomes the patch header. Thus it’s important to write meaningful commit messages.

This works well as long as your workflow considers the Debian patches as a branch that you rebase on top of the upstream sources at each release. Some maintainers don’t like this workflow and prefer to have the Debian changes applied directly in the packaging branch. They switch to a new upstream version by merging it in their packaging branch. In that case, it’s difficult to generate a quilt serie out of the VCS. Instead, you should instruct dpkg-source to store all the changes in a single patch (which is then similar to the good old .diff.gz) and document in the header of that patch how the changes can be better reviewed, for example in the VCS web interface. You do the former with the --single-debian-patch option and the latter by writing the header in debian/source/patch-header:

$ echo "single-debian-patch" >> debian/source/local-options
$ cat >debian/source/patch-header <<END
This patch contains all the Debian-specific
changes mixed together. To review them
separately, please inspect the VCS history
at http://git.debian.org/?=collab-maint/foo.git

END

Subscribe to this blog by RSS, by email or on Facebook.

Managing distribution-specific patches with a common source package

November 5, 2010 by Raphaël Hertzog

In the comments of the article explaining how to generate different dependencies on Debian and Ubuntu with a common source package, I got asked if it was possible to apply a patch only in some distribution. And indeed it is.

The source package format 3.0 (quilt) has a neat feature for this. Instead of unconditionally using debian/patches/series to look up patches, dpkg-source first tries to use debian/patches/vendor.series (where vendor is ubuntu, debian, etc.). Note that dpkg-source does not stack patches from multiple series file, it uses a single series file, the first that exists.

So what’s the best way to use this? Debian should always provide debian/patches/series, they are supposed to provide the default set of patches to use. Any derivative cooperating with Debian can maintain their own series files within the common VCS repository used for package maintenance. They can drop Debian-specific patches (say branding patches for example), and they can add their own on top of the remaining Debian patches.

It’s worth noting that it’s the job of the maintainers to keep both series files in sync when needed. dpkg-source offers no way to have stacked series files (or dependencies between them).

If you want to use quilt to edit an alternate series file, you can temporarily set the QUILT_SERIES environment variable to “vendor.series”. Just make sure to start from a clean state, i.e. no patches applied. Otherwise quilt will be confused by the sudden mismatch between the series file and its internal data (stored in the .pc directory).

Found it useful? Click here to see how you can encourage me to provide more articles like this one.

Correctly renaming a conffile in Debian package maintainer scripts

October 14, 2010 by Raphaël Hertzog

After having dealt with the removal of obsolete conffiles, I’ll now explain what you should do when a configuration file managed by dpkg must be renamed.

The problem

Let’s suppose that version 1.2 of the software stopped providing /etc/foo.conf. Instead it provides /etc/bar.conf because the configuration file got renamed. If you do nothing special, the new conffile will be installed with the default configuration, and the old one will stay around. Any customization made by the administrator are lost in the process (in fact they are not lost, they are still in foo.conf but they are unused).

Of course, you could do mv /etc/foo.conf /etc/bar.conf in the pre-installation script. But that’s not satisfactory: it will generate a spurious conffile prompt that the end-user will not understand.

The solution

In the preinst script, you have to verify if the old conffile has been modified by the administrator. If yes, you want to keep the file around. Otherwise you know you will be able to ditch it once the upgrade is over, and you rename it to /etc/foo.conf.dpkg-remove to remember this fact.

In the postinst script, you remove /etc/foo.conf.dpkg-remove. If the old conffile (/etc/foo.conf) still exists, it’s because it was modified by the administrator. You make a backup of the new conffile in /etc/bar.conf.dpkg-dist and rename the old one into /etc/bar.conf.

In the postrm, when called to abort an upgrade, you move /etc/foo.conf.dpkg-remove back to its original name.

In practice, use dpkg-maintscript-helper

dpkg-maintscript-helper can automate all those tasks. You just have to put the following snippet in the maintainer scripts (postinst, postrm, preinst):

if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
    dpkg-maintscript-helper mv_conffile /etc/foo.conf /etc/bar.conf 1.1-3 -- "$@"
fi

In this example, I assumed that version 1.1-3 was the last version of the package that contained /etc/foo.conf (i.e. the last version released before 1.2-1 was packaged).

You can avoid the preliminary test if you pre-depend on “dpkg (>= 1.15.7.2)” or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper’s manual page.

Debhelper support

Debhelper makes it easy to inject those commands in the maintainer scripts. You just have to provide debian/*.maintscript files. See dh_installdeb’s manual page for details.

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email (just check “Send me blog updates”).