apt-get install debian-wizard

Deciphering one of dpkg’s weirdest errors: unable to open ‘/path/to/foo.dpkg-new’

July 18, 2011 by Raphaël Hertzog

We already studied one weird error of dpkg, let’s do another one:

Unpacking replacement libexo-common ...
dpkg: error processing /var/cache/apt/archives/libexo-common_0.6.1-1_all.deb (--unpack):
 unable to open '/usr/share/doc/exo/html/C/images/exo-preferred-applications-internet.png.dpkg-new': No such file or directory

Rather difficult to understand on the first look right? Let’s see in detail what’s usually happening when you get this error.

The first hint comes from the file extension “.dpkg-new”. This extension is used by dpkg to unpack the updated files near the old files. When everything has been unpacked, they are renamed over the old files.

The failure happens precisely when dpkg tries to rename the file (in fact when it tries to fsync() it before the rename)… but why does it fail?

Usually because there are unexpected symlinks (or bind mounts) involved that resulted in two different files being installed in the same directory. For example consider package that provides /dir1/a-file and /dir2/a-file. Now imagine that on the target system, /dir1 is a real directory but /dir2 is a symbolic link that points to /dir1.

When dpkg processes /dir1/a-file.dpkg-new everything is fine, but when it tries to process /dir2/a-file.dpkg-new it will fail because that file is the same than /dir1/a-file.dpkg-new which has already been renamed.

Diagnosing further the problem requires to understand why there’s a symlink instead of a real directory. It might be two packages that were badly coordinated, or a problem in the package itself because it lacks some code that drops the symlink in the preinst (so that dpkg installs the real directory instead of keeping the symlink).

There might be variations in the way two files end up sharing the same directory, but this simple example should have clarified the nature of the underlying problem.

Subscribe to this blog by RSS, by email or on Facebook.

How to squash Debian release critical bugs

July 14, 2011 by Raphaël Hertzog

Squashing release critical (RC) bugs is one of the most helpful way to contribute to Debian. Because those are the bugs that we need to get rid of in order to release the next stable version. Let’s see how you can help.

The process is really simple:

Find a RC bug that you can fix
Prepare a fixed package
Send the patch to the BTS
Get your fixed package uploaded

I have already covered points 2 and 3 in a former article: How to prepare patches for Debian packages. So I’m not going to repeat myself in this article, instead I’ll cover how to find a release critical bug to fix and how to get your package uploaded.

Find a RC bug with UDD

The Ultimate Debian Database (UDD) provides a neat interface to query the bug tracking system: http://udd.debian.org/bugs.cgi. The form contains many options to customize the query, we’ll try explain some.

First, at the top, you can decide to list bugs based on whether or not they affect testing/stable/unstable. Indeed, thanks to version tracking the BTS knows if a bug applies to the version of the package present in a given distribution. I suggest you pick a choice that includes “sid” because that’s where you’re going to push your fixed package. If sid is not affected by the bug, you can’t do much about it (except if you want to join the release team and help ensure updates are flowing from unstable to testing).

I usually go for “wheezy and sid” because that filters out packages which are not in testing (and are thus not blocking the next release).

Then you have lots of filters to further reduce the list of bugs. It’s important to reduce the noise as you typically have several hundreds of RC bugs at a given time. I suggest you the following settings for a start, but you’re free to try other combinations depending on what you want to achieve (I’ll give some examples later):

ignore bugs tagged patch (a fix is already available);
ignore bugs tagged pending (a fix has already been committed to the VCS repository);
ignore bugs newer than 7 days (that’s the time we should leave for the maintainer to take care of the bug by himself);

I usually sort the results by source package so that I immediately see if several RC bugs are affecting the source package. And I might add “popularity contest” information to get a feel of whether the package is widely used or not.

You can see the results of the suggested query here.

Other interesting queries:

List of RC bugs blocking the migration of their package (i.e. affecting unstable but not testing, with different version in both distributions).
List of RC bugs on packages that you can adopt (i.e. bugs affecting any distribution but on orphaned packages, sorted by decreasing popularity).

Find a RC bug with rc-alert

The downside of the above method is that you get bugs on packages that you don’t know at all and that you might not care about. Some of them might even be better removed instead of fixed.

With the rc-alert command (provided by the devscripts package), you will get a list of RC bugs on packages that are installed on your computer. It should be much more restrictive. And you have command line options to further filter the list just like with the UDD form. Bonus feature: you can also filter by debtags allowing you to restrict the bugs to packages which are implemented in programming languages that you know.

$ rc-alert --include-dists TU --exclude-tags P+ --debtags implemented-in::perl,implemented-in::python
Package: gwibber
Bug:     608724
Title:   gwibber bypasses certificate checking when providing the login/password for OAuth
Flags:   [     S  ] (security)
Dists:   [STU] (stable, testing, unstable)
Debtags: implemented-in::python, interface::x11, role::program, uitoolkit::gtk, works-with::im, x11::application
[...]

Get your fixed package uploaded

Usually sending the patch to the BTS is more than enough. The maintainer or another Debian developer will pick it up at some point. But if you’re interested in becoming a Debian maintainer/developer, it might be worth to push your changes more quickly and get credited by a proper upload with your name.

Instead of just preparing a patch, you should also prepare a fixed package following the rules of a Non-Maintainer Upload. Put this package somewhere online (for example on mentors.debian.net).

When you send your patch to the BTS, include the link to the updated package and express your wish to get this update sponsored. If you haven’t heard anything from the maintainer after 2-3 days, try to find a sponsor on debian-mentors@lists.debian.org. NMUs are easier to review than completely new packages and many Debian developers like to kill RC bugs, so there’s a good chance that you’ll find someone.

Now happy RC bug-squashing! And if you really enjoy it, you can subscribe to debian-bugs-rc (~3000 mails per month)… 🙂

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

7 tips to file useful Debian bug reports and get your problem solved

July 11, 2011 by Raphaël Hertzog

Filing bug reports is the most common way for users to contribute. Even if it’s not too difficult, I’ll give you some advice to improve the quality of your reports. After all, when you go out of your way to report a bug, it is in the hope to see it fixed… so let’s see how we can make this more likely.

1. Try to reproduce the bug

If you can’t reproduce the bug, it’s next to impossible to find the root cause and thus to fix it. In that case, I would suggest you to wait until you experienced the bug multiple times. Maybe you’ll be able to find something that triggers it (or that makes it more likely to encounter it). If the application has a debug/verbose mode, it might be a good idea to enable it until you experience the bug a second time. The generated log might be helpful for the developer to understand what happens exactly.

So don’t file bug reports straight away unless you can reproduce it. The exception to the rule is when the application gives some useful information like a core-dump, a back-trace or an error message.

Obviously if the bug happens during an upgrade, it’s difficult to reproduce it (unless you have multiple computers) but you should still report it. Be sure to include all the relevant information (versions of packages before and after the upgrade, logs of the upgrade, etc.).

2. Do your best to identify the faulty package

When you report a bug to Debian, you must assign it to a package. While there are pseudo-packages useful for problems which are not directly attributable to a real package, in most of the cases you should report a bug against the specific package that seems to be the cause of the problem you encountered.

In turn this often requires you to attribute the problem to a file (for example the executable of the application that triggers the bug). Once you have a filename you can use dpkg -S to identify the corresponding package.

$ dpkg -S /usr/bin/hamster-time-tracker
hamster-applet: /usr/bin/hamster-time-tracker

Note that reportbug accepts a filename as parameter and will do the above conversion for you.

If you only know the name of the application (but not the filename of the associated executable), you can use dpkg -S with a pattern to let it return all possible matches:

$ dpkg -S hamster
hamster-applet: /usr/share/applications/hamster-applet.desktop
hamster-applet: /usr/share/gnome/help/hamster-applet/es/statistics.page
[…]
hamster-applet: /usr/bin/hamster-time-tracker
[…]

Or you can also verify in the list of installed packages:

$ dpkg -l "*hamster*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version         Description
+++-===============-===============-==============================================
ii  hamster-applet  2.32.1-1        time tracking applet for GNOME

3. Verify that the bug is not already reported and/or fixed

If there’s a newer version of the software available, it’s a good idea to try to reproduce the problem with this version too. Because the developers tend to care only about the latest version, they will want to reproduce it with this version, and they will be annoyed if the problem that you reported is already fixed. That’s why bug reports of users of testing/unstable tend to be more useful than bug reports of stable users.

In any case, you want to verify that the bug has not yet been reported: filing a duplicate bug is useless and only generates more work for the developers to merge both bugs together. On the opposite, it’s highly appreciated to add supplementary information to an existing bug report, even a simple confirmation that the bug still exists on a newer version is useful.

Note that reportbug will automatically show you the list of open bugs before allowing you to submit a new one.

4. Use reportbug

While the Debian bug tracking system allows anyone to submit a new bug with a simple mail, you should really use a dedicated program like reportbug (or reportbug-ng) because it will automatically include lots of useful information in the generated report (version of dependencies, current architecture, etc.) and will assist you in all the steps.

5. Describe the problem so that the developer can reproduce it

Ideally your report should include everything required so that the developer can reproduce the problem on his system. If a given document triggers the bug, attach it.

Describe the steps required to reproduce the bug in great details just like you would explain it to your grand-ma. Explain how you expected the program to react and what happened instead.

You can learn much more on how to draft a good bug report in this article: How to report bugs effectively. It’s a bit long but well worth it if you intend to report bugs and thus interact with developers.

6. Be kind and willing to help

When you draft a bug report, keep in mind that you’re writing to a volunteer free software developer and not to a customer service. You should be respectful and follow the usual rules of courtesy. Developers’ attention is scarce and should not be wasted.

Be willing to help, if the developer starts investigating your problem, he might need your help to get supplementary information (in particular if he can’t reproduce it) and you should be ready to provide it. Thus it’s important to keep whatever you need to reproduce the problem.

In some cases, the Debian maintainer might be overworked and you can offer your help to forward the bug to the upstream bug tracker, it’s always appreciated. If you’re reasonably confident that the problem is not Debian-specific, you can do it straight away and set the forwarded field to the URL of the upstream bug report (for example with bts forwarded <bug> <url>).

7. Use the correct severity

The Debian bug tracking system lets you set the initial severity of the bug report (in decreasing severity: critical, grave, serious, important, normal, minor, wishlist). Pick the correct severity according to the official definitions but don’t misread them.

In particular, don’t over-inflate the severity: for instance if you lost some data due to a misuse of the software, it’s not “critical” (i.e. “rm -rf *” doesn’t warrant a critical bug against rm). If you use only a tiny part of a software, and that part doesn’t work, the package might be unusable for you but it’s not unusable for everybody, so it doesn’t warrant the “grave” severity. The “important” severity is often a good choice in those cases.

Do not under-estimate the severity either, if a problem is important enough that it must be fixed before the next stable release (for example a regression compared to the previous release), pick a release-critical severity (i.e. at least “serious”). The maintainer and the release manager can always lower the severity if they do not agree with your initial judgment.

And now, happy bug-reporting! You can refer to this article with this shorter URL: https://raphaelhertzog.com/go/bugreporting/

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

How to prepare patches for Debian packages

July 4, 2011 by Raphaël Hertzog

You want to start contributing to Debian and/or Ubuntu, you decided to help a package maintainer and you’re now looking for how to change a source package and how to submit your changes.

1. Retrieve the source package and install build-dependencies

The first step is to retrieve the latest version of the source package and to install the required build-dependencies. I already covered how to do this with apt-get in the article explaining how to rebuild a source package.

If you prefer, you can use dget (from the devscripts package) to directly grab the source package. You can find the URL of the .dsc files in the Package Tracking System for example.

Sometimes apt-get will warn you that the source package is maintained in a VCS repository, like this:

$ apt-get source wordpress
[...]
NOTICE: 'wordpress' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/git/collab-maint/wordpress.git
[...]

In that case, you can use debcheckout to retrieve the VCS repository instead (provided that you have the corresponding VCS installed):

$ debcheckout wordpress
declared git repository at git://git.debian.org/git/collab-maint/wordpress.git
git clone git://git.debian.org/git/collab-maint/wordpress.git wordpress ...
Cloning into wordpress...

Note however that some maintainers use their VCS in a way that’s not really compatible with the explanations that I will give below.

It’s also a good idea to install the package “packaging-dev”. It’s a meta-package depending on the most common tools that are used for Debian packaging work.

2. Do the changes

Execute dch --nmu to record the fact that you’re working on an update prepared by someone who is not the maintainer (NMU means Non Maintainer Upload). This also ensures that if we build the package, we won’t overwrite the original source package that we downloaded, thus making it possible to generate a “diff” between both versions.

2.1. Modify Debian packaging files

Now fire your text editor and do the required changes in the “debian” sub-directory. You will probably run dch -a multiple times to document each subsequent change.

2.2. Modify upstream files

If you have to modify upstream files, the proper way to do it depends on the source package format (“1.0” vs “3.0 (quilt)” vs “3.0 (native)”, see the debian/source/format file) and on the presence or not of a patch system (the what-patch can help you identify it). In this explanation, I’ll assume that the package is using the recommended format: “3.0 (quilt)”. (It also works for “1.0” if quilt is used and if you configured ~/.quiltrc as recommended by /usr/share/doc/quilt/README.source).

First you should ensure that all patches have been applied with quilt push -a. If there’s no patch yet, you want to create the debian/patches directory (with mkdir debian/patches). Note that you should better invoke quilt from the root of the source package (and the examples below assume this).

2.2.1. Import a patch

If the upstream changes are already in a patch file (say /tmp/patch that you downloaded from the upstream VCS repository) you can import that patch like this:

$ quilt import -P fix-foobar.patch /tmp/patch
Importing patch /tmp/patch (stored as fix-foobar.patch)
$ quilt push
Applying patch fix-foobar.patch
[...]
Now at patch fix-foobar.patch

The -P option allows to select the name of the patch file created in debian/patches/. As you see, the new patch file is recorded in debian/patches/series but not applied by default, we’re thus doing it with quilt push.

2.2.1. Create a new patch

If the upstream changes that you want to make are not in a patch yet, you should tell quilt that you’re going to create one:

$ quilt new fix-foobar.patch
Patch fix-foobar.patch is now on top

Then you must record every file that you’re going to modify with a quilt add invocation. quilt then makes a backup of those files so that it can generate the patch later on. If you’re going to modify the files with your text editor you can just do quilt edit file-to-modify, it’s the same than quilt add file-to-modify followed by sensible-editor file-to-modify.

$ quilt edit foobar.c
File foobar.c added to patch fix-foobar.patch

The last step is tell quilt to generate the patch:

$ quilt refresh
Refreshed patch fix-foobar.patch

3. Test your changes

You should build your modified package with “debuild -us -uc”. You can easily install the resulting package with “debi”. Verify that everything works as expected. If not, continue your modifications until you’re satisfied with the result.

4. Generate a patch and mail it

If you followed the instructions, you should have two .dsc files in the parent directory, like this:

$ cd ..
$ ls wordpress_*.dsc
../wordpress_3.0.5+dfsg-1.1.dsc
../wordpress_3.0.5+dfsg-1.dsc

Generating the patch to send to the maintainer is then just a matter of running debdiff:

$ debdiff wordpress_3.0.5+dfsg-1.dsc wordpress_3.0.5+dfsg-1.1.dsc >/tmp/wp-debdiff

You can send the /tmp/wp-debdiff file to the wordpress maintainer. Usually you send it via the bugreport that your update is fixing and you add the “patch” tag to the report.

This can be automated with the nmudiff utility. By default it assumes that you’re using mutt but it can also directly feed the resulting mail to sendmail. The default text that nmudiff proposes assumes that you’re actually performing an NMU and that the result has been uploaded. If that’s not the case, you should edit the text and make it clear that you’re just sending a patch.

If you have been working in a VCS repository, instead of using debdiff you can simply use the diff feature integrated in your VCS (git diff, svn diff, etc.). But note that with a distributed VCS (like git/bzr/mercurial, unlike svn) you should probably have committed all individual changes in separate changesets. And instead of sending a single patch, you’re probably going to send a series of patches (though it might be easier to just upload your branch in a public repository and give the corresponding URL to the maintainer).

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.