How to use quilt to manage patches in Debian packages

Most Debian source packages are using the “3.0 (quilt)” format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don’t worry, this tutorial will teach you how to use quilt in the context of Debian packaging.

Pre-requisites

Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What’s quilt?

Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop).

Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved

Stack of files PictureTo better understand the various quilt commands, you should have a basic idea of how the tool works. The “stack of patches” is maintained in a dedicated directory (“patches” by default, but in Debian packages we override this value to “debian/patches”). This directory contains the patch files and a “series” file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it’s named “.pc”). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches.

Configuring quilt

Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e ${where}debian/rules -a -d ${where}debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use “debian/patches” instead of “patches” when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple “export QUILT_PATCHES=debian/patches” in that file.

As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too.

Applying and unapplying patches, navigating in the stack of patches

When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt.

You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top.

Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py

Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1

Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1

Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py

No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py

Applying patch 03_manpage.diff
patching file docs/man/django-admin.1

Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch

If there’s no quilt series yet, you want to create the “debian/patches” directory first.

If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run “quilt push -a” first.

Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!).

Now to add changes in this patch, you’re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you’re done with your changes. It’s quite common to forget this step and to be unable to generate the patch afterward. That’s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor.

If you want, you can review your work in progress with quilt diff.

When you’re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you’re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header.

Importing an external patch

If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option “-P better-patch-name”. Like quilt new, it inserts the patch after the topmost patch.

Updating patches for a new upstream version

With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn’t. Also, you’ll have to update those patches because dpkg-source doesn’t accept any fuzz.

If you’re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let’s deal with the case where some of the patches no longer apply. There’s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed.

But there’s a less desirable case where the patch is still relevant but it doesn’t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt’s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands

You should probably read quilt’s manual page too to learn about the various other commands and options that exist.

There’s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use “-P patch-to-rename” to rename a patch which is not currently at the top).

Feedback

Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article.

Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: http://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

My Debian Activities in February 2012

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (384.14 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg and multiarch

The month started with a decision of the technical committee which allowed me to proceed with an upload of a multiarch dpkg even if Guillem had not yet finished his review (and related changes). Given this decision, Guillem made the experimental upload himself.

I announced the availability of this test version and invited people to test it. This lead to new discussions on debian-devel.

We learned in those discussions that Guillem changed his mind about the possibility of sharing (identical) files between multiple Multi-Arch: same packages, and that he dropped that feature. But if this point of the multiarch design had been reverted, it would mean that we had to update again all library packages which had already been updated for multi-arch. The discussions mostly stalled at this point with a final note of Guillem explaining that there was a tension between convenience and doing the right things every time that we discuss far-reaching changes.

After a few weeks (and a helpful summary from Russ Allbery), Guillem said that he remained unconvinced but that he put back the feature. He also announced that he’s close to having completed the work and that he would push the remaining parts of the multiarch branch to master this week (with the 1.16.2 upload planned next week).

That’s it for the summary. Obviously I participated in the discussions but I didn’t do much besides this… I have a “mandate” to upload a multiarch dpkg to sid but I did not want to make use of it while those discussions remained pretty unconclusive. Also Guillem made it pretty clear that the multiarch implementation was “buggy”, “not right” and “not finished” and that he had reworked code fixing at least some of the issues… since he never shared that work in progress, I also had no way to help even just by reviewing what he’s doing.

We also got a few multiarch bug reports, but I couldn’t care to get them fixed since Guillem clearly held a lock on the codebase having done many private changes… it’s not quite like this that I expect to collaborate on a free software project but life is full of surprises!

I’ll be relieved once this story is over. In the mean time, I have added one new thing on my TODO list since I made a proposal to handle bin-nmu changelogs and it’s something that could also fix #440094.

Misc dpkg stuff

After a discussion with Guillem, we agreed that copyright notices should only appear in the sources and not in manual pages or --version output, both of which are translated and cause useless work to translators when updated. Guillem already had some code to do it for --version strings, and I took care of the changes for the manual pages.

I merged some minor documentation updates, fixed a bug with a missing manpage. Later I discovered that some recent changes lead to the loss of all the translated manual pages. I suggested an improvement to dh_installman to fix this (and even prepared a patch). In the end, Guillem opted for another way of installing translated manual pages.

Triggered by a discussion on debian-devel, I added a new entry to my TODO list: implementing dpkg-maintscript-helper rm_conffile_if_owner to deal with the case where a conffile is taken over by another package which might (or might not) be installed.

Misc packaging

At the start of the month, I packaged quilt 0.51. The number of Debian specific patches is slowly getting down. With version 0.51, we dropped 5 patches and introduced a new one. Later in the month I submitted 4 supplementary patches upstream which have been accepted for version 0.60.

This new version (just released, I will package it soon) is an important milestone since it’s the first version without any C code (Debian had this for a long time but we were carrying an intrusive patch for this). Upstream developer Jean Delvare worked on this and based his work on our patch, but he went further to make it much more efficient.

Besides quilt, I also uploaded dh-linktree 0.2 (minor doc update), sql-ledger 2.8.36 (new upstream version), logidee-tools 1.2.12 (minor fixes) and publican 2.8-2 (to fix release critical bug #660795).

Debian Consultants

The Debian Project Leader is working on federating Debian Companies. As the owner of Freexian SARL, I was highly interested in it since Freexian “contributes to Debian, offers support for Debian and has a strategic interest in Debian”. There’s only one problem, you need to have at least 2 Debian developers on staff but I have no employees (it’s me only). I tried to argue that I have already worked with multiple Debian developers (as contractors) when projects were too big for me alone (or when I did not have enough time). Alas this argument was not accepted.

Instead, and since our fearless leader is never afraid to propose compromises, he suggested me (and MJ Ray who argued something similar than me) to try to bring life to the Debian Consultants list which (in his mind) would be more appropriate for one-man companies like mine. I accepted to help “animate” the list, and on his side, he’s going to promote both the “Debian Companies” and the “Debian Consultants” lists.

In any case, the list has seen some traffic lately and you’re encouraged to join if you’re a freelancer offering services around Debian. The most promising thing is that James Bromberger offered to implement a real database of consultants instead of the current static page.

Book update

We made quite some progress this month. There’s only one chapter left to translate. I thus decided to start with proofreading. I made a call for volunteers and I submitted one (different) chapter to 5 proofreaders.

The liberation campaign made a nice leap forwards thanks to good coverage on barrapunto.com. We have reached 80% while we were only at 72% at the start of the month (thanks to the 113 new supporters!). There’s thus less than 5000 EUR to raise before the book gets published under a free license.

Looking at the progression in the past months, this is unlikely to be completed on time for the release of the book in April. It would be nice though… so please share the news around you.

Speaking of the book’s release, I’m slowly preparing it. Translating docbook files is not enough, I must be able to generate HTML, ePub and PDF versions of the book. I’m using Publican for most formats, but for the PDF version Publican is moving away of fop and the replacement (webkit-based) is far from being satisfactory to generate a book ready for print. So I plan to use dblatex and get Publican to support dblatex as a backend.

I have hired Benoît Guillon, the upstream author of dblatex, to fix some annoying bugs and to improve it to suit my needs for the book (some results are already in the upstream CVS repository). I’m also working with a professional book designer to get a nice design.

I have also started to look for a Python Django developer to build the website that I will use to commercialize the book. The website will have a larger goal than just this though (“helping to fund free software developers”) but in free software it’s always good to start with your own case. :-)

Hopefully everything will be ready in April. I’m working hard to meet that deadline (you might have noticed that my blog has been relatively quiet in the last month…).

Thanks

See you next month for a new summary of my activities.

My Debian Activities in December 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (364.18 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg and Multiarch

I had some hope to have a multiarch-enabled dpkg in sid for Christmas as Guillem told me that it was realistic. Alas Guillem got sick. We’re in January and we’re still not there.

While some of Guillem’s commits in December were related to multi-arch, the size of his pu/multiarch/master branch did not really shrink. We still have 36 commits to merge… most of the work he did was refactoring some parts of the code that were already merged. And he initiated some discussion on interface changes. I participated to those discussions hoping to bring them to a quick resolution.

I’m still maintaining my own pu/multiarch/full branch, it is based on Guillem’s branch but with further fixes that I developed and that he has not yet merged and with a change reverted (Guillem’s branch allows crossgrading packages between different architectures while dpkg does not manage this correctly yet).

I can only hope that January will be the last month of this never-ending saga. It’s been one year since I started working on this project. :-|

Misc dpkg work

I reviewed (and later merged) a patch of Kees Cook to enhance dpkg-buildflags so that it can report which hardening features are enabled. This feature might then be used by tools like lintian to detect missing hardening features.

I mentored Gianluca Ciccarelli who is trying to enhance dpkg-maintscript-helper to take care of replacing a directory by a symlink and vice-versa.

I took care of #651993 so that dpkg-mergechangelogs doesn’t fail when it encounters an invalid version in the changelog, and of #652414 so that dpkg-source --commit accepts a relative filename when a patch file is explicitly given.

Guillem also merged a fix I developed for LP#369898.

Packaging work

WordPress 3.3 came out so I immediately packaged it. Despite my upstream bug report, they did not update their GPL compliance page which offers the corresponding sources for what’s bundled in the tarball. So I hunted for the required sources myself, and bundled them in the debian.tar.xz of the Debian source package. It’s a rather crude solution but this allowed me to close the release critical bug #646729 and to reintroduce the Flash files that were dropped in the past… which is great since the Flash-based file uploader is nicer than the one using the browser’s file field.

Quilt 0.50 came out after 2 years of (slow) development. The Debian package has many patches and several of them had to be updated to cope with the new upstream release. Fortunately some of them were also merged upstream. It still took an entire morning to complete this update. I also converted the packaging from CDBS to dh with a short rules file.

Zim 0.54 came out and I immediately updated the package since it fixed a bug that was annoying me.

Review of the ledgersmb packaging

As the sql-ledger maintainer (and a user of this software for my accounting), I have been hoping to get ledgersmb packaged as a possible replacement for it. I have been following the various efforts initiated over time but none of them resulted in a real package in Debian.

This is a real pity so I tried to fix this by offering to sponsor package uploads. That’s why I did a first review of the packaging. It took several hours because you have to explain everything that’s not good enough.

I also filed a wishlist bug against lintian (#652963) to suggest that lintian should detect improper usage of dpkg-statoverride (this is a mistake that was present in the package that I reviewed).

nautilus-dropbox work

I wanted to polish the package in time for the Ubuntu LTS release and since Debian Import Freeze is in January, I implemented some of the important fixes that I wanted.

The Debian package diverges from upstream in that the non-free binaries are installed in /var/lib/dropbox/ instead of $HOME.
Due to a bug, the files were not properly root-owned so I first fixed this (unpacking the tarball as root lead to reuse of the embedded user & group information, and those information changed recently on the Dropbox side apparently).

Then we recently identified other problems related to proxy handling (see #651065). I fixed this too because it’s relatively frequent that the initial download triggered during the package configuration fails… and in that case it’s the user that will re-trigger a package download after having given the appropriate credentials through PackageKit. Without my fix, usage of pkexec would imply the loss of the http_proxy environment variable and thus it would not be possible for a user to download through a proxy.

Last but not least I reorganized the Debian specific patches to better separate what can and should be merged upstream, from the changes that upstream doesn’t want. Unfortunately Dropbox insists on being able to auto-update their non-free binaries, they are, thus, against the installation under /var/lib/dropbox and the corresponding changes.

Book update

We’re making decent progress in the translation of the Debian Administrator’s Handbook, about 6 chapters are already translated (not yet reviewed though).

The liberation campaign is also (slowly) going forward. We’re at 67% now (thanks to 90 new supporters!) while we were only at 60% at the start of December.

Thanks

See you next month for a new summary of my activities.

Quilt patch management with debhelper 7

Since quilt 0.46-7 (see #527255) you can automatically apply/unapply a quilt patch serie with debhelper’s dh command. It’s very practical in tiny rules files:

%:
	dh --with quilt $@

There’s also dh_quilt_patch and dh_quilt_unpatch for those who don’t use tiny rules files.

Enjoy !