Contributor Documentation

How to use quilt to manage patches in Debian packages

August 8, 2012 by Raphaël Hertzog

Most Debian source packages are using the “3.0 (quilt)” format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don’t worry, this tutorial will teach you how to use quilt in the context of Debian packaging.

Pre-requisites

Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What’s quilt?

Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop).

Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved

Stack of files Picture To better understand the various quilt commands, you should have a basic idea of how the tool works. The “stack of patches” is maintained in a dedicated directory (“patches” by default, but in Debian packages we override this value to “debian/patches”). This directory contains the patch files and a “series” file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it’s named “.pc”). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches.

Configuring quilt

Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e ${where}debian/rules -a -d ${where}debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use “debian/patches” instead of “patches” when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple “export QUILT_PATCHES=debian/patches” in that file.

As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too.

Applying and unapplying patches, navigating in the stack of patches

When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt.

You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top.

Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py

Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1

Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1

Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py

No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py

Applying patch 03_manpage.diff
patching file docs/man/django-admin.1

Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch

If there’s no quilt series yet, you want to create the “debian/patches” directory first.

If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run “quilt push -a” first.

Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!).

Now to add changes in this patch, you’re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you’re done with your changes. It’s quite common to forget this step and to be unable to generate the patch afterward. That’s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor.

If you want, you can review your work in progress with quilt diff.

When you’re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you’re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header.

Importing an external patch

If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option “-P better-patch-name”. Like quilt new, it inserts the patch after the topmost patch.

Updating patches for a new upstream version

With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn’t. Also, you’ll have to update those patches because dpkg-source doesn’t accept any fuzz.

If you’re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let’s deal with the case where some of the patches no longer apply. There’s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed.

But there’s a less desirable case where the patch is still relevant but it doesn’t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt’s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands

You should probably read quilt’s manual page too to learn about the various other commands and options that exist.

There’s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use “-P patch-to-rename” to rename a patch which is not currently at the top).

Feedback

Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article.

Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: https://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

Contributing to the translation of Debian

January 31, 2012 by Raphaël Hertzog

If you’re not into packaging and if you asked how you could help Debian, someone probably suggested that you help to translate it.

It’s true that translating Debian is essential if we want to make Debian available to everybody on the world. There are many persons who are stuck as soon as they get a message in English, so it’s important to aim for 100% coverage in terms of localization.

Some vocabulary: localization vs internationalization

Internationalization (i18n) is the work that makes it possible to translate messages in a given application.

Localization (l10n) is the work of translating messages of said application. So as a translator, you’ll be doing “localization” but some knowledge of “internationalization” is still useful… because it will define how you’re supposed to provide the translations. We’ll come back to that later.

Join your localization team

Usually the translation work is shared among multiple translators within a localization team. Check out the Debian International page on www.debian.org to find out instructions for translators for each language.

Many teams have a debian-l10n-*@lists.debian.org mailing list used for coordination, feel free to ask questions on those lists when you start (but make sure that you have read the relevant documentation before).

Each team has its own workflow, so observe for a while to get used to what’s happening before asking your first questions.

What is there to translate?

The translation of most of the software provided by Debian is not handled by Debian. The Debian translation teams “only” handle the translation of:

the software that are specific to Debian (debian-installer, dpkg, APT, etc.) (*);
the Debconf prompts in all Debian packages (*);
the Debian documentation (*);
the Debian website;
the Debian wiki;
the descriptions of packages.

Now before contributing to your first translation, I have to come back to internationalization to teach you a few things. In the above list, the projects marked with “(*)” do use PO files for their translation and the next sections will explain you how to work with those files.

Introduction to Gettext

The free software community has mostly standardized on a single internationalization infrastructure known as Gettext. With this tool, you’re provided a “POT file” which contains all the translatable strings. It looks like this:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR Software in the Public Interest, Inc.
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR , YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: dpkg 1.16.1\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME \n"
"Language-Team: LANGUAGE \n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"

#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr ""

#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr ""

[…]

The lines starting with “#:” are comments that indicate the source files where the (English) string is used. This can be useful if you want check the source to have more information about how the string is used.

The lines starting with “#,” contain flags that can be important. If the “fuzzy” flag is set, the translated string is not used because it must be updated (or at least verified) since the original string evolved. The “c-format” flags indicates that the string must be a C format string, this has some implications in what’s allowed in the string (in particular when it embeds conversion specifier for arguments submitted to printf-like functions).

Another thing to note is that the translation of the empty string is used to store some meta-information about the translation itself.

Contributing a translation as a PO file

When you start a new translation, you copy that POT file to create a “PO file” for your own language (eg. fr.po for the French language). You replace some template values (identified with the upper case words in the POT file) and you replace all the empty strings on “msgstr” lines with the translation of the string that appears in the previous “msgid” line.

The result could be something like this:

# translation of fr.po to French
# Messages français pour dpkg (Linux-GNU Debian).
msgid ""
msgstr ""
"Project-Id-Version: fr\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: 2012-01-16 07:57+0100\n"
"Last-Translator: Christian Perrier \n"
"Language-Team: French \n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: Plural-Forms: nplurals=2; plural=n>1;\n"
"X-Generator: Lokalize 1.2\n"

#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr "caractère invalide « %1$c » dans la taille du membre « %3$.16s » de l'archive « %2$.250s »"

#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr "impossible d'écrire le fichier « %s »"

[…]

If there’s already a “PO file” for your language, there might still work to do: there might be strings that have not yet been translated and there might be “fuzzy” strings which have to be updated — strings which were already translated but where the original string has been modified.

There are software that can assist you to edit PO files: poedit, virtaal, lokalize, gtranslator. There are also special extensions for vim (packaged in vim-scripts) and for Emacs.

Submit the translation for inclusion

Once you have a complete PO file, you should submit it for inclusion. Sometimes you will have been granted commit rights to the source code repository so that you can include your translation by yourself. In the other cases, you should submit your translation with a bug report tagged “l10n” and someone else will include your work in the next release.

Depending on the team, the workflow might require a review before the submission. In that case, you usually have to send a call for review on the coordination mailing list.

Go ahead!

Hopefully those explanations will be enough to get you started. There are many other things to learn¹ but it’s good to learn while practicing…

¹ For example, can you find out why the French translation above changed “%c” in “%1$c”?

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Answering questions of Debian users on various support channels

January 24, 2012 by Raphaël Hertzog

When you start your journey with Debian, you tend to have lots of questions. You’ll find some answers in various documentations but there always are remaining questions. Those can be asked on various support channels:

on mailing lists (debian-user@lists.debian.org for the English-speaking one);
on IRC (channel #debian on irc.debian.org);
on questions & answers websites like ask.debian.net;
on web forums like forums.debian.net;
on every other places where Debian users hang out together (on a booth during a conference, during an install party of a Linux User Group, etc.).

Those are the places where you can also start your journey as a Debian contributor… instead of asking questions, you just have to answer questions of other users! Let me share some advice if you want to do some user support.

User support is difficult…

It’s not always an easy task. Some users are more skilled than others and there might be difficulties related to the language, English is not always the native language of a user who asks a question in English.

Be respectful and courteous when you answer user questions, even if they made mistakes. You’re effectively representing Debian and you should give out a good image of the project. If you don’t have the patience or the time needed to do a good answer, don’t reply and let someone else take care of this user. I invite you to read (and follow!) the Debian Community Guidelines.

Avoid RTFM answers, instead you should show the users how they could have found (alone) the solution to their problem. We don’t want to scare people away, we want to grow our community.

But it’s also rewarding

In some cases, the problem reported by the user will be a real problem and you’ll have an opportunity to file a good bug report, thus helping to improve Debian for everybody.

Often, you don’t even have the answer to the user’s question. But you’re more skilled than him/her to do researches on the web, or you know of a good documentation that might contain the relevant bits of information, in any case you’re doing further research to help this user. In this process, you also grow your own skills since you’re learning stuff that you didn’t know yet.

At least that’s how I learned many things during my first year in the Debian community… there’s no reason why you couldn’t learn lots of stuff that way, in particular if you also read the answers of other skilled people on those channels (it takes a bit of training to learn who are the skilled people though).

I still believe that doing user support is one of the best ways to join the Debian community and to start contributing. It helps you to grow your skills, and to slowly progress from “average user” to “advanced user”.

If you want to start contributing to Debian, click here to subscribe to my newsletter and get future updates for new contributors. You can also follow me on Identi.ca, Google+, Twitter and Facebook.

How to triage bugs in the Debian bug tracking system

September 16, 2011 by Raphaël Hertzog

Triaging bugs is one of the easiest way to start contributing to Debian. I’ll teach you the basics in this article.

1. Prerequisites

All interactions with the Debian Bug Tracking System (BTS) happen through email so you need to have an email account with an address that you’re willing to make public.

All the mail that you send to the BTS will be archived and publicly available through its web-interface. This also means that you should have some spam filters in place because it will inevitably be harvested by spammers. 🙁

To ensure that this email address is consistently used by the various tools that we’re going to use, it’s a good idea to put this email address in the DEBEMAIL environment variable. You can also specify your full name in DEBFULLNAME (in case you don’t want to use the name associated with your Unix account). You usually do this by modifying ~/.bashrc (if you use bash as login shell):

export DEBEMAIL="hertzog@debian.org"
export DEBFULLNAME="Raphaël Hertzog"

You should also install the devscripts package, it provides the bts command that we’re going to use.

2. Find a package or a team with too many bugs

You can literally pick any popular software that’s in Debian, they almost always get more bug reports than the maintainers can handle. Instead of picking a package, you can also select a packaging team and concentrate your efforts on the set of packages managed by the team.

In any case, it’s important to receive the bug traffic for the packages that you’re going to work on. If you went for a specific package, you should subscribe to the package via the Package Tracking System (there’s a subscribe box on the bottom left corner once you selected the source package of interest). If you decided to help a team, there’s usually a dedicated mailing list receiving all bug traffic.

You can browse a list of packages with the most bugs if you have troubles finding a package to work on.

A stack of bug reports to triage

3. Triage bugs!

Bug triaging is all about making sure that bugs are correctly classified so that when a developer looks at the bug list, he can quickly find bugs with all the information required to be able to fix them!

3.1 Adding information to bug

Adding supplementary information is easily done just by sending a mail to XXXX@bugs.debian.org (replace XXXX with the bug number).

But often you want to reply to a message in the bug history, in that case “bts --mbox show XXXX” is for you. It will grab the corresponding mailbox and open a mailer (mutt by default) on it. Now you can directly reply in your favorite mailer.

3.2 Classifying bugs

The Debian BTS uses tags (click the link and read the doc!) to classify bugs. “bts tag XXXX + foo” will add the foo tag (replace the + with a – to remove a tag). If you want to explain why you’re adding a tag, you should instead reply in the bug log as explained above, put control@bugs.debian.org in Bcc (Blind Carbon Copy) and start the body of your message with your tag command:

tag XXXX + foo
thanks

But what tag should you add? When a bug is submitted, you should try to reproduce the bug. If you can reproduce it, then tag the bug “confirmed” (example in #641710). If you can’t, you should request more information (ex: a sample document triggering the bug, a configuration file, the output of some relevant command, etc.) until you can reproduce it or conclude that it was a user mistake. When you request supplementary information due to this, you should tag the bug “unreproducible moreinfo” (example in #526774). “moreinfo” should be later dropped when the requested information are provided, and “unreproducible” should be dropped if those information were enough to actually help reproduce the bug (example in #526774).

During that initial evaluation, it’s also worth differentiating packaging bugs (which are specific to Debian) from upstream bugs (which are relevant also for non-Debian users). The latter should be tagged “upstream” (and forwarded upstream if the bug is reproducible or contains enough information for the upstream developers, example in #635112).

If you saw a (viable) patch in the bug log, the bug should be tagged “patch”. This is usually done by the patch submitter but sometimes it’s forgotten (example in #632460). Take care though to not reinstate the patch tag if it was initially set but then dropped by the package maintainer after a review of the patch.

If the title of the bug report is not descriptive enough, you can change it with a “retitle XXXX new-title” command (example in #170850).

You can also change the severity of the bug report depending on the impact of the problem (with a command “severity XXXX new-severity”, what a surprise!). Request for new features are “wishlist”, most documentation problems are “minor”. On the other side of the scale, you can use “important” for bugs that are very annoying but that should not block a release. “serious”, “grave” and “critical” are used for release critical bugs, check the official definitions of the severities (examples in 502738 or #506498).

3.3 Closing non-bugs and bugs that are already fixed

If your analysis of the bug report is that it’s not really a bug but a user mistake, then you should close it by sending a mail to XXXX-done@bugs.debian.org with some explanations of the user’s mistake so that he can get past his problem (example in #592853).

If the problem was a real bug, but one that is apparently already fixed, you should try to quickly find the version that fixed the bug. If you can’t find it in the changelog (there’s a link to it in the PTS, or you can use /usr/share/doc/package/changelog.Debian.gz), you’ll make the safe assumption that the upstream version you’re currently using is the first one where this is fixed. Then you send a mail to XXXX-done@bugs.debian.org but you start your mail with “Version: version-that-fixed-the-bug” and continue with a small explanation of why you believe the bug to be fixed by this version (example in #122948).

3.4 Reassigning misfiled bug reports

Bug reports are not always filed against the proper package. Users file bugs against applications where they experience the bugs, but the real bug might be in an underlying library or application.

When that happens, you should use the “reassign XXXX correct-package version” command to get it filed against the correct package. The version parameter is optional but should be provided if possible, it should be the oldest version that we know to have the problem (example in #626232).

3.5 Forwarding bugs

Forwarding bugs means opening bug reports in the upstream bug tracker for issues that have been reported in Debian but that applies to the upstream (unmodified) source code. Be sure to include all the relevant information and a link to the corresponding Debian bug.

Depending on the upstream bug tracker, you might have to open an account to be able to file new bug reports.

On the Debian side, you must record that a bug has been forwarded with “bts forwarded XXXX upstream-bug-url”. upstream-bug-url is the URL corresponding to the upstream bug report you created (ex: http://projects.ciarang.com/p/feed2omb/issues/21/ recorded in #609345″).

If the upstream authors fix the bug you reported, you can tag the Debian bug with “fixed-upstream” so that it’s easier to find bugs to close when the next upstream release comes out (example in #637275).

3.6 Updating version information

The Debian BTS uses “version tracking” to know which package versions are affected by a given bug. It’s particularly important to have correct version information for release critical bugs since it might affect the migration of packages to testing.

You can learn more on this topic here: http://wiki.debian.org/HowtoUseBTS.

4. More advice

Colin Watson wrote a constructive rant explaining some mistakes that bug triagers are often doing. While it refers mainly to Ubuntu’s launchpad, the advice apply equally as well to Debian. Check it out to become a better bug triager!

Note that you can refer to this article with this shorter URL: https://raphaelhertzog.com/go/bugtriaging/