Archives for 2010

Secret figures of a Debian/Ubuntu blogger: what you liked most on raphaelhertzog.com

October 25, 2010 by Raphaël Hertzog

Chart goes up on screen I launched raphaelhertzog.com this summer (taking over the English content of my former multi-lingual blog), when I decided that I would be more serious about blogging on Debian/Ubuntu related topics. On September, I decided to write 2 articles per week and up to now I managed to keep the schedule.

Two of my articles were published by Linux Weekly News, those are much more researched than the average blog article (they are tagged with [LWN] in the list below).

The most popular articles

Most people read my blog through the RSS feed which happens to be syndicated on Planet Debian and Planet Ubuntu. According to the feedburner’s statistics, the top-5 articles are:

5 reasons why I still contribute to Debian after 12 years (32700 views)
[LWN] Understanding Membership Structures in Debian and Ubuntu (31700 views)
Social Micropayment Can Foster Free Software, Discover Flattr (30100 views)
Everything you need to know about conffiles: configuration files managed by dpkg (29900 views)
How to make 110.28 EUR in one month with free software and Flattr (29400 views)

But I also have occasional readers visiting my blog because my articles are announced on Identi.ca, Twitter and Facebook (and they circulate on social networks, thanks to those who are sharing them!). The top-5 articles according to the statistics of my website are:

5 reasons why I still contribute to Debian after 12 years (6000 views)
[LWN] Can Debian offer a Constantly Usable Testing distribution? (5000 views)
Understanding Debian’s release process (1500 views)
Flattr FOSS (1400 views, not an article but I regularly blog about this project)
Can Debian achieve world domination without being on Facebook? (1100 views)

The most flattered

Since I am using Flattr on my blog, it can be interesting to see the articles which generated lots of flattr micro-donations. The top-3 articles are my articles about Flattr (1, 2, 3). Excluding articles related to Flattr, the top-5 is:

5 reasons why I still contribute to Debian after 12 years (12 flattr)
The secret plan behind the “3.0 (quilt)” Debian source package format (10 flattr)
How to use multiple upstream tarballs in Debian source packages? (5 flattr)
[LWN] Understanding Membership Structures in Debian and Ubuntu (4 flattr)
Do You Want a Free Debian Book? Read on. (4 flattr)

Most articles get 2 to 3 flattr clicks.

The most commented

I usually get 4-5 comments on most articles but some generate much more feedback:

[LWN] Can Debian offer a Constantly Usable Testing distribution? (40 comments)
5 reasons why I still contribute to Debian after 12 years (22 comments)
Can Debian achieve world domination without being on Facebook? (15 comments)
How to generate different dependencies on Debian and Ubuntu with a common source package (14 comments)
[LWN] Understanding Membership Structures in Debian and Ubuntu (12 comments)

Factoids

Here are my conclusions based on the above figures:

Writing about your Debian/Ubuntu work and your long term involvement makes for highly popular content that spreads well.
In-depth and well researched articles (like those written for LWN) do not generate more flattr revenues than the average article even if they take 4 to 8 times as long to write.
People are more likely to flattr you for your free software contribution than for the value they get out of your article.
People care a lot about the Debian release process, and like to discuss the topic.

If you also appreciate the above-linked articles, you should click here to subscribe to my email newsletter.

The secret plan behind the “3.0 (quilt)” Debian source package format

October 21, 2010 by Raphaël Hertzog

New source package formats do wonders While I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0” format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0”). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0” was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

Supports compression formats other than gzip: bzip2, lzma, xz.
Can use multiple upstream tarballs.
Can include binary files in the debian packaging.
Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

it supports compression formats other than gzip: bzip2, lzma, xz.
it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
April 2008: I ask ftpmasters to support the new source packages in #457345.
June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
February 2009: Lenny release.
March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

Understanding Debian’s release process

October 18, 2010 by Raphaël Hertzog

Currently, the main product of the Debian project is its stable release^[1]. Those release come out approximately every 18-24 months. This article gives a short overview of the process leading to the next stable release.

Creating a new distribution

Immediately after a stable release, a new distribution is created in the Debian archive. Its initial content is a copy of the (just released) stable distribution. Its codename is decided by the release managers and there’s a tradition of picking a character’s name from the Toy Story movie.

As an example, the “wheezy” distribution will be created once “squeeze” (aka Debian 6.0) is out.

For simplicity there’s a generic name to refer to the distribution used to prepare the next stable release: it’s testing. In the Debian archive, testing is just a symbolic link pointing to the right directory (squeeze currently).

Updating packages, working on release goals

During most of the cycle, developers work on packaging new upstream versions and implementing release goals. They upload their packages in the unstable distribution.

From there packages trickle to the testing distribution once they satisfy some quality checks: they must not have new release-critical bugs, they must have been built on all architectures that were previously supported, they must not break any dependency in testing, and they must have spent at least 10 days in unstable.

This minimal period ensures the package has been tested and gives enough time to users to file bugs if the package is suffering from problems. If the problems are deemed release-critical, they will block the migration of the package to testing.

During this part of the release cycle, the main work of the release team is ensuring that updated packages flow from unstable to testing. It can be a tricky task: package dependencies frequently tie packages together so that they can only migrate to testing together. If only one of the tied packages is not ready (for example if a new revision has been uploaded and has not spent 10 days in unstable yet), then none of them can migrate.

Stabilizing, polishing, fixing release-critical issues

The constant churn of new packages makes it very difficult to build a very polished release. That’s why, at some point, release managers freeze the testing distribution: automatic updates are stopped and they vet every single update made to testing. They have strong requirements, the goal is to only allow updates fixing release critical bugs, or those which are low-risk and bring significant value to the user experience (like new translations, updated documentation, etc.).

During freeze, some packages are also removed because the current upstream version can’t be supported for the lifetime of the stable release.

The freeze tends to slow down the pace of changes in unstable. Many maintainers opt to push new upstream versions in experimental instead so that if they need to update their packages in testing, they can still do it through unstable. This procedure is recommended by the release managers because it means that updates that they unblock have been tested as usual. It’s not the case for updates uploaded directly to testing (through testing-proposed-updates).

This behavior is rather annoying for the bleeding-edge users that use testing or unstable like a rolling release.

Release time

Once release managers are satisfied of the quality of the new distribution, some last minute work is needed, like generating the CD images. In the Debian archive, the release is made official by pointing the “stable” symbolic link to the new distribution (and the “oldstable” one to the previous distribution).

Now it’s party time, the cycle is over, and a new one can start. 🙂

^[1] The Constantly Usable Testing project aims to make testing a first-class product like stable—but with a very different update policy.

Follow me on Identi.ca, Twitter and Facebook.

Correctly renaming a conffile in Debian package maintainer scripts

October 14, 2010 by Raphaël Hertzog

After having dealt with the removal of obsolete conffiles, I’ll now explain what you should do when a configuration file managed by dpkg must be renamed.

The problem

Let’s suppose that version 1.2 of the software stopped providing /etc/foo.conf. Instead it provides /etc/bar.conf because the configuration file got renamed. If you do nothing special, the new conffile will be installed with the default configuration, and the old one will stay around. Any customization made by the administrator are lost in the process (in fact they are not lost, they are still in foo.conf but they are unused).

Of course, you could do mv /etc/foo.conf /etc/bar.conf in the pre-installation script. But that’s not satisfactory: it will generate a spurious conffile prompt that the end-user will not understand.

The solution

In the preinst script, you have to verify if the old conffile has been modified by the administrator. If yes, you want to keep the file around. Otherwise you know you will be able to ditch it once the upgrade is over, and you rename it to /etc/foo.conf.dpkg-remove to remember this fact.

In the postinst script, you remove /etc/foo.conf.dpkg-remove. If the old conffile (/etc/foo.conf) still exists, it’s because it was modified by the administrator. You make a backup of the new conffile in /etc/bar.conf.dpkg-dist and rename the old one into /etc/bar.conf.

In the postrm, when called to abort an upgrade, you move /etc/foo.conf.dpkg-remove back to its original name.

In practice, use dpkg-maintscript-helper

dpkg-maintscript-helper can automate all those tasks. You just have to put the following snippet in the maintainer scripts (postinst, postrm, preinst):

if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
    dpkg-maintscript-helper mv_conffile /etc/foo.conf /etc/bar.conf 1.1-3 -- "$@"
fi

In this example, I assumed that version 1.1-3 was the last version of the package that contained /etc/foo.conf (i.e. the last version released before 1.2-1 was packaged).

You can avoid the preliminary test if you pre-depend on “dpkg (>= 1.15.7.2)” or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper’s manual page.

Debhelper support

Debhelper makes it easy to inject those commands in the maintainer scripts. You just have to provide debian/*.maintscript files. See dh_installdeb’s manual page for details.

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email (just check “Send me blog updates”).

« Previous Page
1
…
5
6
7
8
9
…
14
Next Page »