You are here: Home / Archives for dpkg

My Debian activities in July 2011

August 5, 2011 By

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (170 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf.

Libre Software Meeting / RMLL

I attended “only” 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month.

During those 3 days I helped with the Debian booth that was already well taken care of by Frédéric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time.

One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world).

Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out… it’s far from being an international event and the number of talks doesn’t make for a better quality.

BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux?

dpkg-source improvements

During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to “3.0 (quilt)” source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there’s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option.

Build flags changes

Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly.

In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags.

Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly.

Makefile snippets provided by dpkg

At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere.

More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related).

dpkg-buildflags improvements

Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got.

I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements.

In the end, here are the main changes implemented:

  • new “prepend” directive to inject flags at the start (see commit);
  • new “strip” directive to strip flags from the result returned by dpkg-buildflags (see commit);
  • new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
  • new --export=configure command to inject build flags on the ./configure command line (see commit);
  • new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs.

DebConf: rolling and hardening build flags

I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it’s been a pleasure to meet again all my Debian friends. It’s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I’m pretty happy with the result.

I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688).

We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line.

Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It’s not ready to be merged but it’s working already (see my last update in the bug log).

A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel.

It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes.

More dpkg work

Lots of small things done:

  • I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we’re down to 154 bugs (from more than 300 a month ago!).
  • I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
  • I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS

Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian.

I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we’re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great!

Misc packaging work

I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process.

I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there’s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)!

Thanks

See you next month for a new summary of my activities.

Filed Under: Debian News, Meta, News Tagged With: , , , , , , , , ,

Understanding dpkg’s file overwrite error

August 1, 2011 By

This is probably one of the most common errors. You’re very likely to encounter it, in particular if you tend to mix packages from various origins/distributions, or if you’re using unstable. It looks like this:

Unpacking gbonds-data (from .../gbonds-data_2.0.3-2_all.deb) ...
dpkg: error processing /var/cache/apt/archives/gbonds-data_2.0.3-2_all.deb (--unpack):
 trying to overwrite '/usr/share/omf/gbonds/gbonds-C.omf', which is also in package gbonds 2.0.2-9
dpkg-deb: subprocess paste killed by signal (Broken pipe)

A given file can only be provided by a single package. So if you try to install a package that provides a file that is already part of another installed package, it will fail with a message similar to the above one.

Sometimes this failure will be meaningful because dpkg prevented you to install two unrelated packages that happen to have a real file conflict. In other cases, like in the example above, this failure is just the result of a mistake.

Folder with gears

The version 2.0.3-1 of gbonds split the architecture independent files in a separate package called gbonds-data but the maintainer forgot to add the required control field in gbonds-data (Replaces: gbonds (<< 2.0.3-1)). That field allows dpkg to take over files from the listed packages.

If you want to ignore the file conflict and let dpkg take over the file (even without the Replaces), you can pass the --force-overwrite command-line option.

But you’re not using dpkg directly, you’re probably using an APT frontend (like apt-get or aptitude). Don’t worry, there’s a simple way to define custom dpkg options to use:

# apt-get -o Dpkg::Options::="--force-overwrite" install gbonds-data

The syntax is a bit weird, but the “::” after “Options” is important, it’s the syntax that defines a list item value instead of a single value. And you can effectively pass multiple options to dpkg by putting multiple -o Dpkg::Options::="…".

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Filed Under: Documentation, User Documentation Tagged With: , , , , , , ,

Deciphering one of dpkg’s weirdest errors: unable to open ‘/path/to/foo.dpkg-new’

July 18, 2011 By

We already studied one weird error of dpkg, let’s do another one:

Unpacking replacement libexo-common ...
dpkg: error processing /var/cache/apt/archives/libexo-common_0.6.1-1_all.deb (--unpack):
 unable to open '/usr/share/doc/exo/html/C/images/exo-preferred-applications-internet.png.dpkg-new': No such file or directory

Rather difficult to understand on the first look right? Let’s see in detail what’s usually happening when you get this error.

The first hint comes from the file extension “.dpkg-new”. This extension is used by dpkg to unpack the updated files near the old files. When everything has been unpacked, they are renamed over the old files.

The failure happens precisely when dpkg tries to rename the file (in fact when it tries to fsync() it before the rename)… but why does it fail?

Usually because there are unexpected symlinks (or bind mounts) involved that resulted in two different files being installed in the same directory. For example consider package that provides /dir1/a-file and /dir2/a-file. Now imagine that on the target system, /dir1 is a real directory but /dir2 is a symbolic link that points to /dir1.

When dpkg processes /dir1/a-file.dpkg-new everything is fine, but when it tries to process /dir2/a-file.dpkg-new it will fail because that file is the same than /dir1/a-file.dpkg-new which has already been renamed.

Diagnosing further the problem requires to understand why there’s a symlink instead of a real directory. It might be two packages that were badly coordinated, or a problem in the package itself because it lacks some code that drops the symlink in the preinst (so that dpkg installs the real directory instead of keeping the symlink).

There might be variations in the way two files end up sharing the same directory, but this simple example should have clarified the nature of the underlying problem.

Subscribe to this blog by RSS, by email or on Facebook.

Filed Under: Documentation, User Documentation Tagged With: , , , , ,

My Debian activities in June 2011

July 3, 2011 By

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (195 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dropbox for Debian

This is not free software but Dropbox is very popular and they did only provide an Ubuntu package that did not work on Debian. So I created an official package.

I have been in touch with Dropbox developers and they have been very helpful. They’ll shortly release a signature mechanism (with GPG) so that we can further improve the package by verifying the origin of the downloaded binaries.

SAT Britney

At the start of the month, I continued my work on the britney reimplementation (the software that creates testing out of unstable) but I quickly stalled it because the release managers asked the feedback of Stefano Zacchiroli and Ralf Treinen (who have extensive knowledge on the topic with their research work on Mancoosi) and I did not want to invest further work in case they would identify a major flow… the feedback came only very late this month and while it was somewhat negative, I still think it’s worth pursuing the effort for a bit longer.

Converted ftplib to multiarch

While dpkg still doesn’t support multiarch (no news from Guillem and no visible sign of progress :-( ), unstable got all the remaining bits allowing us to convert libraries to multiarch (see the announce). As soon as the required libc6 landed in unstable, I looked into converting the only library package that I maintain. I had no major problem but I still identified 2 issues in Lintian (filed as #630164 and quickly fixed by Niels Thykier).

build-arch / build-indep support

For the 42th time in the last 10 years, the idea of using build-arch/build-indep targets in the rules file has surfaced again. I had already decided some time ago that I would accept a patch implementing a new field Build-Features to enable dpkg-buildpackage to use those targets and this time Bill Allombert completed such a patch so I merged it.

The technical committee also decided that it would take a final decision on this topic (see #629385). Roger Leigh provided useful input by doing an archive-wide rebuild with the various solutions suggested. Given that the majority would like to make the target mandatory at some point in the future, I provided the dpkg patch for my preferred solution. We would use “auto-detection” as a temporary measure until all packages have been converted to have the targets.

The technical committee has not yet taken any decision even though the discussion stalled since the 12th of June. But that’s usual with that body. I’m sure it will be solved during Debconf. ;-)

Misc dpkg work

  • Modified dpkg-source --after-build to automatically unapply patches if they have been applied by dpkg-source --before-build.
  • Lots of small bug fixes (#628726, #629582, #630996, #631435, #631439, #631547, #632168) and that’s just to keep with the flow of incoming bug reports!
  • Added 2 supplementary Perl modules to the supported API for the benefit of Lintian.
  • Spent an evening to track down the possible causes of an long-standing and annoying assertion failure related to triggers.
  • Updated my branch with improved triggers directives to take into account the feedback of Guillem, and merged it.
  • While doing this I discovered a design flaw with the usage of “prerm failed-upgrade” and merged a fix.
  • Discussed integration of dpkg-buildflags with debhelper in #544844 and decided of further improvements for dpkg-buildflags as a result.

Hamster applet update

Hamster-applet is a GNOME application which did not have a 3.0 release, but it had a development release (2.91.x). I checked out whether it was possible to package this version for experimental and have the applet work with the GNOME fallback mode. Apparently not, the code was not yet updated to be compatible with the newer panel.

Instead I uploaded the latest stable version (2.32.1) to unstable. It has some nice improvements in the standalone version (and the name of the executable changed). For usage with GNOME 3, I have created a custom shortcut to start it quickly (with gconf-editor set /apps/metacity/global_keybindings/run_command_1 to “<Mod4>t” and /apps/metacity/keybinding_commands/command_1 to “hamster-time-tracker” because the GNOME 3 control panel does not seem to work to set custom keybindings currently).

Translated my professional website into English

While I’m grateful for all the people who are supporting my work, I’m still far from my goal to have one third of my time funded through donations and sales of products on this blog.

So I decided to also bring more visibility to my company and in particular to its Debian-related service offering. It was only available in French up to now so I translated it and expanded it a bit. My “support page” on this blog now also links to my company’s website.

If your company needs help to create Debian packages, or needs Debian technical support by email, you just found the right partner. :-)

BTW, I have discounted prices for individuals and non-profits who would like to benefit from my help to create Debian packages.

The Debian Administrator’s Handbook

This is the title of the upcoming translation of my book. The project now has a dedicated website: debian-handbook.info.

You can subscribe to its RSS feed to keep up with the latest news. The full table of contents is online along with a FAQ.

I’m actively looking for partners to help me promote the fundraising once it goes live. If you can reach a large set of readers interested by a good Debian book, get in touch with me to join the affiliate program.

Thanks

See you next month for a new summary of my activities.

Filed Under: News Tagged With: , , , , , , , ,

Deciphering one of dpkg’s weirdest errors: short read on buffer copy

June 27, 2011 By

As a Debian/Ubuntu user, you’re likely to be exposed at some point to an error reported by dpkg. In a series of articles, I’ll explain some of the errors that you might encounter.

Some error messages can be confusing at times. Most of the error strings do not appear very often and developers thus tend to use very terse description of the underlying problem. In other cases the architecture of the software makes it difficult to pin-point the real problem because the part that displays the error is several layers above the one that generated the initial error.

This is for example the case with this error of dpkg:

Unpacking replacement xulrunner-1.9.2 ...
dpkg-deb (subprocess): data: internal gzip read error: '<fd:0>: too many length or distance symbols'
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing /var/cache/apt/archives/xulrunner-1.9.2_1.9.2.17+build3+nobinonly-0ubuntu1_amd64.deb (--unpack):
 short read on buffer copy for backend dpkg-deb during `./usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so'

First, the decompression layer discovers something unexpected in the data read in the .deb file and dpkg-deb outputs the error message coming from zlib (“too many length or distance symbols”). This causes the premature end of dpkg-deb --fsys-tarfile that dpkg had executed to extract the .data.tar archive from the deb file. In turn, dpkg informs us that dpkg-deb did not send all the data that were announced (and hence the “short read” in the error message) and that were meant to be part of the file ‘/usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so’.

That’s all nice but it doesn’t help you much in general. What you must understand from the above is that the .deb file is corrupted (sometimes just truncated). In theory it should not happen since APT verifies the checksums of files when they are downloaded. But computers are not infallible and even if the downloaded data was good, it can have been corrupted when stored on disk (for example cheap SSD disks are known to not last very well).

Try removing the file (usually with apt-get clean since it’s stored in APT’s cache) and let APT download it again. Chances are that it will work on the second try. Otherwise consider doing a memory and HDD check as something is probably broken in your computer.

Join my free newsletter and learn more tips for users. Or click here to support my work on dpkg with Flattr, consider subscribing for a few months.

Filed Under: Documentation, User Documentation Tagged With: , , , , , ,

apt-get, aptitude, … pick the right Debian package manager for you

June 20, 2011 By

This is a frequently asked question: “What package manager shall I use?”. And my answer is “the one that suits your needs”. In my case, I even use different package managers depending on what I’m trying to do.

APT vs dpkg, which one is the package manager?

In the Debian world, we’re usually thinking of APT-based software when we’re referring to a “package manager”. But in truth, the real package manager is dpkg. It’s the low-level tool that takes a .deb file and extracts its content on the disk, or that takes the name of a package to remove the associated files, etc.

APT is better known because it’s the part of the packaging infrastructure that matters to the user. APT makes collection of software available to the user and does the dirty work of downloading all the required packages and installing them by calling dpkg in the correct order to respect the dependencies.

But APT is not a simple program, it’s a library and several different APT frontends have been developed on top of that library. The most widely known is apt-get since it’s the oldest one, and it’s provided by APT itself.

Graphical APT front-ends

update-manager is a simple frontend useful to install security updates and other trivial daily upgrades (if you’re using testing or sid). It’s the one that you get when you click in the desktop notification that tells you that updates are available. In cases, where the upgrade is too complicated for update-manager, it will suggest to run synaptic which is full featured package manager. You can browse the list on installed/available packages in numerous ways, you can mark packages for installation/upgrade/removal/purge and then run in one go all the recorded actions.

software-center aims to be an easy to use application installer, it will hide most of the packaging details and will only present installed/available applications (as defined by a .desktop file). It’s very user friendly and has been developed by Ubuntu.

Of the graphical front-ends, I use mainly synaptic and only when I’m reviewing what I have installed to trim the system down.

Console-based GUI APT front-ends

In this category, I’ll cite only aptitude. Run without parameter, it will start a powerful console-based GUI. Much like synaptic, you can have multiple views of the installed/available packages and mark packages for installation/upgrade/removal/purge before executing everything at once.

Command-line based package managers and APT front-ends

This is where the well known apt-get fits, but there are several other alternatives: aptitude, cupt, wajig. Wajig and cupt are special cases as they don’t use libapt: the former wraps several tools including apt-get, and the latter is a (partial) APT reimplementation (versions 1.x were in Perl, 2.x are now is C++).

You’re welcome to try them out and find out which one you prefer, but I have never felt the need to use something else than apt-get and aptitude.

apt-get or aptitude?

First I want to make it clear that you can use both and mix them without problems. It used to be annoying when apt-get did not track which packages were automatically installed while aptitude did, but now that both packages share this list, there’s no reason to avoid switching back and forth.

I would recommend apt-get for the big upgrades (i.e. dist-upgrade from one stable to the next) because it will always find quickly a relatively good solution while aptitude can find several convoluted solutions (or none) and it’s difficult to decide which one should be used.

On the opposite for regular upgrades in unstable (or testing), I would recommend “aptitude safe-upgrade“. It does a better job than apt-get at keeping on hold packages which are temporarily broken due to some not yet finished changes while still installing new packages when required. With aptitude it’s also possible to tweak dynamically the suggested operations while apt-get doesn’t allow this. And aptitude’s command line is probably more consistent: with apt-get you have to switch between apt-get and apt-cache depending on the operation that you want to do, aptitude on the other hand does everything by itself.

Take some time to read their respective documentation and to try them.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

Filed Under: Documentation, User Documentation Tagged With: , , , , , , , , ,

My Debian activities in May 2011

June 3, 2011 By

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work, then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

I have been…

Doing some work towards Debian Rolling

At the start of the month, the discussions about Debian rolling were still very active on debian-devel. Declaring that testing would be rolling did not make it (as I hoped), the argument that some RC bugs last for far too long in that distribution carried the discussion and thus the most consensual proposition ended up being the one of Josselin Mouette were rolling would be testing plus a few selected cherry-picked packages from unstable.

I believe it’s a workable solution if we only care about a subset of architectures. Otherwise the same reasons that keep the fixed packages out of testing would probably also apply for rolling.

Given this, I did setup britney (the software that controls testing) on my laptop to investigate how we can create rolling. It turns out britney is a very specialized software with very few configuration knobs.

At the same time Joachim Breitner made a proposition that immediately grabbed my attention. He suggests to use SAT solvers to find out the set of packages that should migrate from unstable to testing. I thought that rolling would be a good testbed for this new implementation of britney (which he calls SAT-britney) so I jumped right in this project.

I was not at all familiar with this science field, so I looked up quite some documentation: I learned that all SAT solvers expect the problem to be presented in CNF form, and that DIMACS was the file format of choice to represent those boolean constraints. Several SAT solvers are available in Debian and picosat appears to be one of the best.

Then I started some early coding/prototyping to play with the concept. You can find the result in this git repository, you can grab a copy with git clone git://git.debian.org/~hertzog/sat-britney.git.

There’s not much yet, except some Python code to generate a SAT problem that can be fed to a SAT solver. But I really look forward to this project.

Representing Debian during Solutions Linux

During the second week, I spent 3 days in Paris to help manage the Debian booth at Solutions Linux.

We have responded to lots of queries but most visitors already knew Debian, and many of them use it at work and/or at home. We tried to recruit those people as new members for Debian France, the local association. We also sold all our remaining goodies.

The Ubuntu people were interviewed by France 3 (an important TV channel) and we took this opportunity (with the consent of the Ubuntu guys) to show our Debian t-shirts in the background: you can watch the video here (in French), you can see me with Carl Chenet at 1:21.

We have also been interviewed by Intelli’n TV: here and here (both in French). I’m not very good at this exercise. :-)

Improving dpkg triggers

The third week was a vacation week, in theory I should have stayed away from my computer but I really wanted to take this opportunity to improve the state of dpkg triggers in Debian.

I already covered my work in another article: Trying to make dpkg triggers more useful and less painful.

The result is not merged yet, I just asked a question to all package maintainers who are using triggers to be able to decide whether I’ll merge it as is, or if I can make the new behavior the default one.

Supporting users after Alioth’s migration

When I came back from my vacation, many services provided by Alioth.debian.org were non-functional after a migration to a new setup that involves two machines instead of one. Given that I used to be an Alioth admin, I know that in those periods you tend to be get bogged down on many user support requests. So I re-joined #alioth on IRC and tried to help a bit.

I did investigate some of the reported problems and prepared fixes (updated scripts, configuration files, etc.) for some of the issues. I also created a list of remaining issues that should have lasted only a few days but that’s still active because there are still regressions left.

The most important things still missing are:

  • proper support for delegation of rights. We used ACL setup by the admins in the past. With the new FusionForge, each project admin should be able to delegate rights to external “roles”. There’s a Debian Developer role already but trying to grant him right fails…
  • access to the Ultimate Debian Database. Many tools rely on this database to work.
  • anonymous FTP access to download project files.
  • clear guidelines on how we’re supposed to deal with websites that are updated by VCS hooks.
  • clear guidelines on how we’re supposed to deal with personal git repositories

Improving the “3.0 (quilt)” source format

I have made some proposals to change the way the new source format would work. The goals are to be less painful for packagers who are using a VCS, and to avoid unexpected changes slipping through a new patch generated by dpkg-source.

It seems that the proposals are relatively consensual so I’ll implement them at some point.

Missing in action on my blog

I did a lots of stuff for Debian between travel and vacation, and in the remaining time, I did not manage to write many articles for my blog.

In fact, besides the article on my triggers work mentioned above I only published one interview: People behind Debian: Steve Langasek, release wizard.

I’ll try to do better this month!

Thanks

Many thanks to the people who gave me 151.61 € in May.

See you next month for a new summary of my activities.

Filed Under: Meta Tagged With: , , , , , , , , , , ,

Trying to make dpkg triggers more useful and less painful

May 30, 2011 By

Lately I have been working on the triggers feature of dpkg. I would like to share my plan and what I have done so far. I’ll first explain what triggers are, the current problems, and the work I did to try to improve the situation.

Introduction

Dpkg triggers are a neat feature of dpkg that package can use to send/receive notifications to/from other installed packages. Those notifications take the form a simple string.

This feature is heavily used to track changes of packaged files in a list of predefined directories, and to update other files based on this. For instance, man-db is watching the directories containing manual pages so that it can update its cache (in /var/cache/man/). install-info is updating the index of info pages when there have been changes in /usr/share/info. gnome-menus is updating its own copy of the menu hierarchy (with entries from /etc/gnome/menus.blacklist blacklisted) every time that a .desktop file is installed/updated/removed.

From a user’s perspective

You see triggers in action very often during upgrades (in fact too often as we’ll see it later):

Preparing to replace zim 0.52-1 (using .../archives/zim_0.52-1_all.deb) ...
Unpacking replacement zim ...
Processing triggers for shared-mime-info ...
Processing triggers for menu ...
Processing triggers for desktop-file-utils ...
Processing triggers for man-db ...
Processing triggers for hicolor-icon-theme ...
Processing triggers for python-support ...
Processing triggers for gnome-menus ...
Setting up zim (0.52-1) ...
Processing triggers for python-support ...
Processing triggers for menu ...

As you guessed it, those “Processing triggers” lines correspond to the packages which received (one or more) trigger notifications and which are doing the corresponding task.

By default the triggers are processed at the end of the dpkg --unpack invocation which is often too soon because APT will often call dpkg --unpack repeatedly during important upgrades. There are some options to ask APT to use dpkg’s --no-triggers option in order to defer the trigger processing at the end of the APT run. You can put this in /etc/apt/apt.conf.d/triggers:

// Trigger deferred
DPkg::NoTriggers "true";
PackageManager::Configure "smart";
DPkg::ConfigurePending "true";
DPkg::TriggersPending "true";

I have now asked APT maintainers to use those options by default, I filed bug #626599 to track this. At the same time I fixed bug #526774 reported by APT maintainers. This bug forced them to put a work-around in APT which resulted in running triggers sooner than expected.

(And while writing this article I filed bug #628564 and #628574 because it was clearly not normal that the menu triggers was executed twice for the installation of a single package)

From a packager’s perspective

The implementation of triggers has several consequences on the status that packages can have.

Let’s assume that the package A installs a file in a directory that is watched by package B (and that B is currently in the “installed” state). When A is unpacked, dpkg adds B to its “Triggers-Awaited” field and lists the activated trigger in B’s “Triggers-Pending” field. Package A is in “unpacked” state, but B has been changed to “triggers-pending”.

When A is configured, instead of going to the “installed” state, it will go to the “triggers-awaited” state. In that state the package is assumed to NOT fulfill dependencies. However, B—which is still in “triggers-pending” state—does fulfill dependencies.

A and B will switch to “installed” at the same time when the trigger has been processed.

The fact that the triggers-awaited status does not fulfill dependencies means that some common triggers like man-db have to be processed regularly just to be able to ensure dependencies are satisfied before running the postinst of other installed packages.

But a package which ships a manual page can certainly be considered as configured when its postinst has been run even if man-db has not yet updated its cache to know about the new/updated manual page.

When you activate a trigger with the dpkg-trigger command you have an option --no-await to avoid awaiting the trigger processing (and thus to go directly to installed state after the postinst has been run). But with file triggers or activate trigger directives, you do not have this option.

My proposal to improve the situation

This is the problem that I tried to solve during my last vacation. But before changing the inner working of triggers, I wrote a non-regression test suite for that feature (commit here) so I could hack with some confidence that I did not break everything.

The result has been presented on the debian-dpkg mailling list: see the discussion here. I added new directives that can be used in triggers files that work exactly like the current triggers except that they do not put triggering packages in trigger-awaited status.

I believe the code to be mostly ready, but in its current form the patch brings zero benefits until all packages have been converted to use the trigger variants that do not require awaiting trigger processing (and the change requires a pre-dependency on dpkg to ensure we have the required dpkg that understands the new kind of trigger directives).

Remaining question

Thus I wonder if I should not change the default semantic of triggers. The packages which really provide crucial functionality to awaiting packages through triggers would then have to be updated to switch to the new directives.

If you’re a packager using triggers, you can thus help me by answering this question: do you know some triggers where it’s important that the awaiting packages are not considered as configured before the trigger processing? In most of the cases I checked, it’s important for the triggered package rather than for the triggering package.

In truth, a package in triggers-awaited status is usually in a good enough shape to be able to satisfy dependencies (i.e. requirements that other packages can have), but it would still be worth to record the fact that it’s not entirely configured yet because it might be true from the user’s point of view: for example if the menu trigger has not yet been processed, the software might not yet be visible in the application menu.

If you appreciate this kind of groundwork that benefits to the whole Debian ecosystem, please consider supporting my work. Click here and give it a look, there are many ways to contribute and to make a difference for me.

Filed Under: Debian News, News, Ubuntu News Tagged With: , , , , , ,

March 2011 wrap up

April 3, 2011 By

Since I’m soliciting donations to support my Debian work, the least I can do is explain what I do. You can thus expect to see an article like this one every month.

Multi-Arch work

I updated the code to use another layout for the control files stored in /var/lib/dpkg/info/. Instead of using a sub-directory per architecture (arch/package.type), we decided to use package:arch.type but only for packages which are Multi-Arch: same. dpkg is taking care to rename the files the first time it is executed with write rights and then updates /var/lib/dpkg/info/format to remember that the upgrade has been done and that we can rely on the new structure.

I filed a few bugs on packages that are improperly accessing those internal files instead of using the appropriate dpkg-query interface. I sent a heads-up mail on -devel to make other people aware of those problems in the hope to discover most of them as early as possible.

After that, the work stalled because Guillem went away for 2 weeks and thus stopped his review of my work. I hope he will quickly resume the review and that we will get something final this month.

With the arrival of dpkg 1.16.0, it’s now possible to start converting libraries to multi-arch even if full multi-arch support has not yet landed in dpkg proper. See http://wiki.debian.org/Multiarch/Bootstrapping for the detailed plan.

If you’re curious about Multi-Arch, you might want to read this article of Steve Langasek as well.

Bug triage for dpkg in launchpad

At the start of the month, there was close to 500 bugs reported against the dpkg package in Launchpad. Unfortunately most of it is noise… many of the reported bugs are misfiled, they show an upgrade problem of a random package and that upgrade problem confuses update-manager which tries to configure an already configured package. This generates a second error that apport attributes to dpkg and the resulting bug report is thus filed on dpkg. There are literally hundreds of those that have to be reclassified.

Michael Vogt and Brian Murray did some triaging, and I also spend quite some hours on this task. It’s a bit frustrating as I tend to mark many reports “Incomplete” because there’s no way they can be acted upon and many of them are so old that the reporter is unlikely to be able to provide supplementary information.

But in the middle of this noise, there are some useful bug reports, like LP#739179 which enabled me to fix a regression even before it reached Debian Unstable (because Ubuntu runs a snapshot of dpkg with multiarch support).

I subscribed to the Launchpad bugs for dpkg via the Debian Package Tracking System (thanks to the derivatives-bugs keyword) and will try to keep up with the incoming reports.

Misc dpkg work

The ftpmasters came up with a request for a new field (see 619131) in source packages. After a quick discussion and a round of review on debian-policy@l.d.o, I implemented the new Package-List field. This should allow the ftpmasters to save some time in NEW processing, but we deferred the change for the next dpkg version (1.16.1) to ponder a bit more on the design of the field.

I also fixed a bunch of bugs (#619541, #605719, #598922, #616096) and merged a patch of Mark Hymers to recognize the new Built-Using field.

Developers-reference work

The review process for changes to the developers-reference is not working as it should. And I suffered from it while trying to integrate the patch I wrote for the “Developer duties” chapter (see #548867).

We purposely changed the maintainer field from debian-doc to debian-policy in the hope to have more reviews of suggested changes and to seek some sort of consensus before committing anything. But we don’t get more reviews… and deciding to commit a patch is now even harder than it was (except for trivial stuff where personal opinions can’t interfere).

In my case, I only got the feedback of Charles Plessy which was very mixed to say the least. I tried to improve my patch based on what he expressed but I also clearly disagreed with some of his assertions and was convinced that my wording was in line with the dominant point of view within Debian.

We tried to involve the release team in the discussion because most of what I documented was about helping making stable release happen, but nobody of the team answered.

Instead of letting the situation (and my patch) rot, I solicited feedback from the DPL and from another developers-reference editor to see whether my patch was an improvement or not. After some more time, I went ahead and committed it.

It was not pleasant for anyone.

I don’t know how we can improve this. Contrary to the policy, the developers-reference is a document that is not normative, I believe the result is better when we put some “soul” into it. But it’s a real challenge when you seek a consensus and that the interest in reviewing changes is so low.

DVD shop listed on debian.org

In February, I launched a DVD shop whose benefits are used to fund my Debian work. Shortly after the launch I used the official form to be added to the official listing of Debian CD vendors and offered a few suggestions to deal with vendors who are selling unofficial images (with firmware in my case).

A few weeks later, I got no answers: neither for my request nor for my suggestions, I mailed the cdvendors@debian.org team directly asking for a status update and quickly got an answer suggesting that Simon Paillard usually does the work and can’t process the backlog due to some injury. At this point no concerns had been raised about adding me to the list. To save some time and some work for the team, I added myself to the list since I had commit rights and I informed them that I did it, so that they can review it.

Shortly after I did that, Martin Zobel Helas objected to my addition. I cleared some misunderstandings but the discussion also lead to some changes to please everybody: the listing now indicates that some images are unofficial and I have prepared a special landing page for people coming from the Debian website through this listing.

Debian column on OMG! Ubuntu

I have always been a firm believer that it’s important for Debian to reach out to the widest public with its message of freedom. Thus when Benjamin Humphrey contacted the debian-publicity team to find volunteers to write a Debian column on OMG! Ubuntu, I immediately jumped in.

I wrote 4 articles over there. The tone is very different from my articles on my blog and I like that duality. Check out Debian is dying! Oh my word!, Debian or Ubuntu, which is the best place to contribute?, Are you contributing your share? and Ubuntu’s CTO reveals DEX: an effort to close the gap with Debian.

It’s a great win-win situation, OMG! Ubuntu benefits from my articles, Debian’s values are relayed further, and OMG! Ubuntu’s large audience also helps me develop my own blog.

Work on my book

I had lots of paperwork to do this month (annual accounting stuff for my company) and I did not have as much time as I hoped for my book. Still I have a updated a few more chapters of my French book and I certainly hope to complete the update during April.

This means that the work on the English translation could start in may.

Work on my blog

Just like for my book, it has been relatively difficult for me to cope with my policy of two articles every week. But I still managed to get quite some good stuff out.

I interviewed Christian Perrier (Debian’s translation coordinator) and also Bdale Garbee (chair of Debian’s technical committee).

I finished my series of “Debian Cleanup Tips” with 2 supplementary articles:

The removal of firmware is causing troubles to quite some users so I wrote an article explaining how to deal with the problem. A regular reader also asked me to write an article about Jigdo, I executed myself because it was a good idea and that he has been very nice with me: Download ISO images of Debian CD/DVD at light speed with Jigdo.

Last but not least, I shared my package maintainer pledge which inspired my developers-reference patch (see discussion above).

Thanks

Many thanks to all the people who showed their appreciation of my work. The 324.37 EUR that you gave me in February represented 2 days and a half of my time that I have spent working on the above projects.

See you next month for a new summary of my activities.

Filed Under: Meta Tagged With: , , , , , ,

February 2011 wrap up

February 26, 2011 By

February has been again a busy month for me. Here’s a quick summary of what I did:

Multi-Arch work

I have spent many days implementing and refining dpkg’s Multi-Arch support with Guillem Jover (dpkg co-maintainer) and Steve Langasek (beta-tester of my code ;-) ). Early testers can try what’s in my latest pu/multiarch/snapshot/* branch in my personal git repository.

A Debian DVD shop

I’m always exploring new options to fund my Debian work (besides direct donations) and this month—with the Debian Squeeze release—I saw an opportunity in selling Debian DVD. Nobody provides DVD with included firmwares and quite a few people would like to avoid the SpaceFun theme. So I built unofficial Debian DVDs that integrate firmware and that install a system with the old theme (MoreBlue Orbit). Click here to learn more about my unofficial DVDs.

On my blog

In my “People behind Debian” series, I interviewed Mike Hommey (Iceweasel maintainer) and Maximiliam Attems (member of the kernel team).

I started a “Debian Cleanup Tip” series and already published 4 installments:

For contributors, I wrote two articles: the first gives a set of (suggested) best practices for sponsoring Debian packages and adapted my article as a patch for the Developers Reference. In the second article, I shared some personal advice for people who are considering participating on Debian mailing list: 7 mistakes to avoid when participating to Debian mailing lists.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

Filed Under: Meta Tagged With: , , , , ,
« Older Posts
Newer Posts »