How to generate different dependencies on Debian and Ubuntu with a common source package

There are situations in which a given package needs to have different dependencies in Debian and in Ubuntu. Despite this difference it’s possible to keep a single source package that will build both variants of the package. Continue reading to discover a step-by-step explanation.

1. When is that needed?

While it is possible to have different dependencies depending on the distribution on which the package is built, it should be usually avoided when possible. This infrastructure should only be used as a last resort when there are no better alternatives.

Here are some examples of when it might be needed:

  • Ubuntu has some packages that Debian does not have (or vice-versa), and the resulting package would benefit from having them installed.
  • The package names differ between Ubuntu and Debian and it’s on purpose (i.e. the difference is a justified choice and not a mistake because both distributions failed to coordinate).
  • The packages are built differently in both distributions, and the run time dependencies are not the same due to this. Maybe some associated patches are only applied in Ubuntu.

2. Using substitution variables in debian/control

The dependency that varies between both distributions can’t be hardcoded in debian/control, instead you should put a substitution variable (substvar) that will be replaced at build time by dpkg-gencontrol. You can name it ${dist:Depends} for example:

[...]
Depends: bzip2, ${shlibs:Depends}, ${misc:Depends}, ${dist:Depends}
[...]

Note that you typically already have other substitution variables (${shlibs:Depends} comes from dpkg-shlibdeps and ${misc:Depends} from debhelper and its dh_* scripts).

3. dpkg-gencontrol needs a -V option

The value used to replace this new variable needs to be communicated to dpkg-gencontrol. You can use the -V option for this, the syntax would be something like this:

dpkg-gencontrol [...] -Vdist:Depends="foo (>= 2), bar"

If you use debhelper, you have to pass the option to dh_gencontrol after two dashes (--):

dh_gencontrol -- -Vdist:Depends="foo (>= 2), bar"

If you use CDBS, you can set the DEB_DH_GENCONTROL_ARGS_ALL make variable:

include /usr/share/cdbs/1/rules/debhelper.mk
DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="foo (>= 2), bar"

The value given to dpkg-gencontrol is static in all those examples, now let’s see how we can use give a different value depending on the distribution that we’re targetting.

4. Using dpkg-vendor in debian/rules

dpkg-vendor is a small tool (provided by the dpkg-dev package) that parses the /etc/dpkg/origins/default file (provided by the base-files package) to know the current distribution and its ancestry. It can be used in debian/rules to adjust the behavior depending on the current distribution. You can check its man-page to learn about the various options supported but we’re only going to use --derives-from <vendor> in this sample. With this option the script exits with zero if the current distribution is or derives from the indicated distribution, or with 1 otherwise.

Now combining all together, we can use dpkg-vendor to dynamically define the content of the substitution variable in debian/rules. Let’s suppose that you want ${dist:Depends} to be “foo (>= 2)” on Ubuntu (and its derivatives) and “bar” everywhere else. Using debhelper’s 7 tiny rules file, this example could be:

ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	SUBSTVARS = -Vdist:Depends="foo (>= 2)"
else
	SUBSTVARS = -Vdist:Depends="bar"
endif

%:
	dh $@

override_dh_gencontrol:
	dh_gencontrol -- $(SUBSTVARS)

If you use CDBS, it could be this:

include /usr/share/cdbs/1/rules/debhelper.mk
ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="foo (>= 2)"
else
	DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="bar"
endif

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Everything you need to know about conffiles: configuration files managed by dpkg

The Debian policy dictates that package upgrades must take care of preserving user changes to configuration files. This article will explain you how most packages ensure this. This is important knowledge for anyone who has to manage upgrades: knowing how it works lets you easily automate most of it and deal correctly with the fallout.

How dpkg manages configuration files

Most packages rely on dpkg to properly install configuration files. Dpkg keeps a checksum of the last installed version of configuration file. When it must install a new version, it calculates the checksum of the currently installed file and if it doesn’t match anymore, it knows that the user has edited the file. In that case, instead of overwriting the configuration file, it asks the user what to do. You probably already have seen those questions, they look like this:

Configuration file `/etc/bash.bashrc'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** bash.bashrc (Y/I/N/O/D/Z) [default=N] ? 

In this specific example, if you answer “Y” or “I” (for “yes” or “install”), dpkg will install the new version of /etc/bash.bashrc but it will also backup the current version in /etc/bash.bashrc.dpkg-old. If you answer “N” or “O” (for “no” or “old”), dpkg will install the new version in /etc/bash.bashrc.dpkg-dist and /etc/bash.bashrc is left untouched. The two other answers allow you to examine the differences before taking a decision. Note that if you choose to start a shell, the new version is currently available as /etc/bash.bashrc.dpkg-new (and since Squeeze there are convenient environment variables $DPKG_CONFFILE_OLD and $DPKG_CONFFILE_NEW in case you want to create a custom review script).

All configurations files managed by dpkg are called “conffiles” because that’s the name of the field where they are recorded in the dpkg database. You can display the list of conffiles for any package:

$ dpkg --status bash
[...]
Conffiles:
 /etc/skel/.profile ecb6d3479ac3823f1da7f314d871989b
 /etc/skel/.bashrc 2afdd6c53990f2387a7ef9989af0bc07
 /etc/skel/.bash_logout 22bfb8c1dd94b5f3813a2b25da67463f
 /etc/bash.bashrc 5b3c3bc73d236e4e1b6f9b6c1ed5964e
[...]

The command “dpkg-query --showformat='${Conffiles}\n' --show bash” can give you the same information if you need to retrieve only that field. The 32 characters after the filename are the MD5 checksum of the original configuration file provided by the package.

Avoiding the conffile prompt

Every time that dpkg must install a new conffile that you have modified (and a removed file is only a particular case of a modified file in dpkg’s eyes), it will stop the upgrade and wait your answer. This can be particularly annoying for major upgrades. That’s why you can give predefined answers to dpkg with the help of multiple --force-conf* options:

  • --force-confold: do not modify the current configuration file, the new version is installed with a .dpkg-dist suffix. With this option alone, even configuration files that you have not modified are left untouched. You need to combine it with --force-confdef to let dpkg overwrite configuration files that you have not modified.
  • --force-confnew: always install the new version of the configuration file, the current version is kept in a file with the .dpkg-old suffix.
  • --force-confdef: ask dpkg to decide alone when it can and prompt otherwise. This is the default behavior of dpkg and this option is mainly useful in combination with --force-confold.
  • --force-confmiss: ask dpkg to install the configuration file if it’s currently missing (for example because you have removed the file by mistake).

If you use Apt, you can pass options to dpkg with a command-line like this:

$ apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

You can also make those options permanent by creating /etc/apt/apt.conf.d/local:

Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

Bringing up the conffile prompt at any time

The conffile prompt is only displayed when dpkg detects that the package provides an new version of the conffile. Thus reinstalling the same package will not bring up the prompt. But you can instruct dpkg to ask nevertheless with the --force-confask option. This is a new feature in Debian Squeeze. It will only ask for files that are locally modified.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

How to create Debian packages with alternative compression methods

While gzip is the standard Unix tool when it comes to compression, there are other tools available and some of them are performing better than gzip in terms of compression ratio. This article will explain where you can make use of them in your Debian packaging work.

In the source package

A source package is composed of multiple files. The .dsc file is always uncompressed and it’s fine since it’s a small textual file. The upstream tarballs can be compressed with gzip (orig.tar.gz), bzip2 (orig.tar.bz2), lzma (orig.tar.lzma) or xz (orig.tar.xz), so choose the one that you want if upstream provides the tarball compressed with multiple tools. Put it at the right place and dpkg-source will automatically use it. Note however that packages using source format “1.0” are restricted to gzip, and the main Debian archive currently only allows gzip and bzip2 (xz might be allowed later) even if the source format “3.0 (quilt)” supports all of them.

The debian packaging files are provided either in a .diff.gz file for source format “1.0” (again only gzip is supported) or in a .debian.tar file for source format “3.0 (quilt)”. The latter tarball can be compressed with the tool of your choice, you just have to tell dpkg-source which one to use (see below, note that gzip is the default).

In a native package, dpkg-source must generate the main tarball and you can instruct it to use another tool than gzip with the --compression option. That option is usually put in debian/source/options:

# Use bzip2 instead of gzip
compression = "bzip2"
compression-level = 9

For “3.0 (quilt)” source packages, this option is not very useful as the debian tarball that gets compressed is usually not very large. But some maintainers like to use the same compression tool for the upstream tarball and the debian tarball, so you can use this option to harmonize both.

In native packages, it’s much more interesting: for instance the size of dpkg’s source package has been reduced of 30% by switching to bzip2, saving 2Mb of disk.

In the binary packages

.deb files also contain compressed tar archives and by default they use gzip as well:

$ ar t dpkg_1.15.9_i386.deb 
debian-binary
control.tar.gz
data.tar.gz

data.tar.gz is the archive that contains all the files to be installed and it’s the one that you can compress with another tool if you want. Again this is mostly interesting for (very) large packages where the size difference clearly justifies deviating from the default compression tool. Try it out and see how many megabytes you can shove. Another aspect that you must keep in mind is that those alternative tools might use important amount of memory to do their job, both for compression and decompression. So if your package is meant to be installed on embedded platforms, or if you want to build your package on low-end hardware with few memory, you might want to stick with gzip.

Now how do you change the compression tool? Easy, dpkg-deb supports a -Z option, so you just have to pass “-Zbzip2″ for example. You can also pass “-z6″ for example to change the compression level to 6 (it’s interesting because a lower compression level might require less memory depending on the tool used). The dpkg-deb invocation is typically hidden behind the call to dh_builddeb in your debian/rules so you have to replace that invocation with “dh_builddeb -- -Zbzip2“.

If you are using a debhelper 7 tiny rules files, you have to add an override like in this example:

%:
	dh $@

override_dh_builddeb:
	dh_builddeb -- -Zbzip2

If you are using CDBS, you have to set the variable DEB_DH_BUILDDEB_ARGS:

include /usr/share/cdbs/1/rules/debhelper.mk
[...]
DEB_DH_BUILDDEB_ARGS = -- -Zbzip2

I hope you found this article helpful. Follow me on identi.ca or on twitter.

How to customize dpkg-source’s behaviour in your Debian source package

dpkg-source is the program that generates the Debian source package when a new package version is built. It offers many interesting command-line options but they are often not used because people don’t know how to ensure that they are used every time the package is built. Let’s fill that gap!

It is possible to forward some options to dpkg-source by typing them on the dpkg-buildpackage command line but you’d have to remember to type them every time. You could create a shell alias to avoid typing them but then you can’t have different options for different packages. Not very practical.

The proper solution has been implemented last year (in dpkg 1.15.5). It is now possible to put options in debian/source/options. Any long option (those starting with “--“) can be put in that file, one option per line with the leading “--” stripped.

Here’s an example:

# Bzip2 compression for debian.tar
compression = "bzip2"
compression-level = 7
# Do not generate diff for changes in config.(sub|guess)
extend-diff-ignore = "(^|/)config.(sub|guess)$"

Notice that spaces around the equal sign are possible contrary on the command line. You can use quotes around the value but it’s not required.

The debian/source/options file is part of the source package so if someone else grabs the resulting source package and rebuilds everything, they will use the options that you defined in that file.

You can also use debian/source/local-options but this time the file will not be included in the resulting source package. This is interesting for options that you want to use when you build from the VCS (Version Control Repository, aka git/svn/bzr/etc.) but that people downloading the resulting source package should not have. Some options (like --unapply-patches) are only allowed in that file to ensure a consistent experience for users of source packages.

You can learn more about the existing options in the dpkg-source manual page. Read it, I’m sure you’ll learn something. Did you know that you can tell dpkg-source to abort if you have upstream changes not managed by an existing patch in debian/patches? It’s --abort-on-upstream-changes and it’s only allowed in debian/source/local-options.

Be sure to subscribe to the RSS feed or to the email newsletter to not miss useful documentation for Debian contributors!

Can Debian achieve world domination without being on Facebook?

Debian Developers who like Debian on FacebookFacebook is not very popular among free software hackers. When I announced my Facebook page on identi.ca (see here) I got a few replies suggesting it was odd for me to use Facebook.

Indeed there are many good reasons why Facebook should be avoided: it is a centralized and proprietary service that is not very privacy-friendly. But the truth is that lots of people are using it (even Debian developers, can you recognize them on the picture?) and some are using their Facebook news feed as their main source of news.

You might wonder how many persons that represents, so here are the figures: the Debian Facebook page has 48,361 fans and the Ubuntu one has 247,932 fans. That’s right, an announce put on the Debian facebook page would reach more persons than the most popular announce list that Debian is currently running (debian-announce@lists.debian.org has about 29,000 subscribers). Unfortunately that Debian Facebook page is empty and I don’t even know who the administrator is. The Ubuntu page on the contrary is properly configured to relay news from The Fridge and Jono Bacon is posting some custom updates from time to time.

Debian contributors regularly mention “world domination” as the ultimate goal of the universal operating system (I even feature this in my blog header banner! :-)). But working towards world domination means — in my opinion — that we should communicate our ideals of freedom to as many people as possible, even if they are using a service that we don’t want to promote.

The Debian social contract acknowledges that some users have to use non-free software and we provide the non-free section for them. In the same spirit, I believe we must have a presence on Facebook. That does not mean that we endorse Facebook, and we should surely promote Joindiaspora once it’s usable. But in the mean time we should reach out to Facebook users and allow them to follow us with the platform of their choice.

That’s the choice I recently made when I decided to setup a public facebook page featuring my free software work, my blog and my book. So if you are a Facebook user, click here to visit my page and click on “Like” if you want to follow this blog in your Facebook news feed. I also share interesting Debian or Ubuntu related articles that I discover while reading my RSS subscriptions.

Below are the usual facebook widgets for the 3 pages quoted in this article (they might not show up if you read this article through an RSS feed):


Understanding Membership Structures in Debian and Ubuntu

Debian and Ubuntu have a set of official membership roles that can be granted to regular contributors. Those roles come with rights that enable the contributors to do their work and to participate in the project governance (elections and other official decision-making processes). It’s also a way for the distributions to acknowledge the work done: most contributors are proud of the status they reached.

The membership structure plays an important role in the development of a distribution: it defines the kind of contributors that are welcome in the project, it sets expectations of the project towards its contributors and defines their rights. In the end, this shapes the project’s ability to recruit new contributors to keep the project alive and kicking. This article introduces the existing statuses in Debian and Ubuntu, and defines the — sometimes confusing — jargon associated with them.

The Debian Case

Debian only has two types of official members: Debian Developers (DD) and Debian Maintainers (DM). The rights of the developers are codified in the Debian Constitution while those of the maintainers have been defined in a general resolution of 2007. The Debian Maintainer status is still mostly documented in a wiki page. The integration of this new status in Debian’s official processes has been slow to come largely because it was introduced — at that time — without enough negotiation with the involved parties. Nowadays, it is preferred that people get the DM status before applying for DD.

DM is a very limited role: maintainers can only upload packages that already have their name on them (either in the Maintainer or Uploaders field) and a specific flag (DM-Upload-Allowed: yes) that only Debian Developers can add. They have no other rights and limited access to Debian’s resources.

Besides those official roles, there are also maintainers of packages that have no official status within Debian except that they are listed in the “Maintainer” field of the package. They are doing the maintenance work but all uploads are done by a Debian Developer after verification of the work done (this is called “sponsorship” and is the only way to start with official packaging work). Once the DD trusts the maintainer, the developer will typically ask the maintainer to apply for DM status in order to be relieved from the sponsorship work.

In the end, that makes three different kind of package maintainers and a lot of confusion when you discuss membership issues… in particular when the New Maintainer process is the path that you follow to become a Debian Developer. Don’t be fooled by the names when reading Debian’s documentation!

The Ubuntu Case

Ubuntu had, from the start, an official Ubuntu Member status that includes all contributors: developers of course, but also documentation writers, artists, translators, etc. This status notably grants the right to vote in elections of the Community Council, the right to participate on Planet Ubuntu, and the @ubuntu.com email alias.

For developers, the situation is more complicated: the wiki page lists no less than five different statuses. Initially, developers were split between Ubuntu Core Developers and the MOTU (Masters Of The Universe). The latter were responsible of the universe/multiverse sections of the archive while the former also had upload rights for the main/restricted sections. But, inspired by the Debian Maintainers concept and facing concrete problems in terms of archive management, they changed their infrastructure to offer more fine-grained control on package uploads.

Ubuntu can now grant upload permissions on a package-per-package basis, but it can also delegate the right to grant upload permissions with the same granularity. This lead to the new Per-Package Uploader status which is simply an Ubuntu Member with upload rights on a limited set of packages where they have a specific expertise. The more generic Ubuntu Developers status now encompasses members of various development teams that have been delegated the right to manage upload permissions on a (usually large) package set (the current teams are Ubuntu Desktop, Mythubuntu, Kubuntu, and Edubuntu). Those teams can define their own policy to add new members provided they follow the basic rules defined by the Developer Membership Board (see this wiki page).

Ubuntu Contributing Developer is an intermediate status for someone who is not yet ready for one of the other developer statuses but who has still shown enough commitment to be an Ubuntu Member.

All those statuses can be obtained in a similar way: you prepare a wiki page listing your past contributions, you collect testimonials from existing members that you have worked with, you add yourself in the agenda of the next meeting of the board (or council) that grants the status that you seek, and you attend the meeting. The members of the board will decide whether you are ready for the status (or not) based on what you provided in the wiki, based on your answers during the meeting (and on a mailing-list for developers), and based on what others have to say about you.

The most important boards are usually elected by the community while others are commonly appointed by the community council. Those governance bodies include Canonical employees but not as many as one would expect: two out of eight in the Developer Membership Board, two out of eight in the Community Council, but all six members of the Technical Board. The last figure, while not intended, is not surprising given the high expectations set on potential members of the technical board. Mark, as the founder, is the only person to have a permanent seat on both the Community Council and the Technical Board.

Comparison of the Statuses Between Debian and Ubuntu

The following table summarizes the rights given to each developer role in the two projects (Put the mouse over the abbreviations to know what they are referring to).

Rights Debian Ubuntu
DM DD UM PPU/UD MOTU UCD
Package maintenance via sponsorshipYN/AYYYN/A
Official email alias-YYYYY
Participate in votes for members-YYYYY
Participate in votes for developers-Y-YYY
Upload rights restricted to pre-approved packagesY--Y--
Upload rights restricted to a section of the archive----Y-
Unlimited upload rights-Y---Y
Number of contributors (as of 2010-07-27)117904462278563

Please note that the number of contributors are not 100% accurate for Ubuntu. A contributor can have multiple statuses (direct membership to a launchpad group) granted over time (while gaining experience). The problem has been mostly avoided by calculating differences between number of members of the various groups but it’s not perfect and it can’t be: some MOTU are also PPU for packages in main and it’s legitimate (but I only counted them as MOTU and not as PPU). Another limitation is that members of some administrative teams are included indirectly in many teams and thus appear in the count while they should not.

Anyway, this simple table makes it obvious that Ubuntu’s structure offers a broader choice of statuses. They acknowledge the work of all contributors from the start while still giving the most critical rights only to those who have proven that they deserve them. Despite this difference, Debian still has a significant advantage in terms of number of developers. That number does not tell the whole story though: the Ubuntu contributors include many Canonical employees (e.g. 36 out of 63 core developers have a @canonical.com email registered on their launchpad account) that are likely to spend more time working on the distribution than the average Debian member. But even if comparing person-hours would be a challenging thought experiment, in practice it’s of not much interest if both projects continue to cooperate and if more and more of the contributions flow in both directions.

Debian is aware of the shortcomings of its structure. Changes to better accommodate non-packagers have been discussed several times already. The last efforts in that direction were unfortunately perceived as a solution ready to go rather than a proposal to be discussed, and the project got quickly buried by a general resolution (GR). Even if that resolution invited for further discussion and a new proposal, the truth is that when someone’s initiative is “corrected” by way of GR, it usually kills any motivation to go forward.

Possible Evolution?

On the Ubuntu side, the infrastructural changes were completed recently and they don’t expect any further change in the near future. They do plan, however, to expand usage of those new features so that more teams benefit from the possibility to control upload rights on packages that are relevant to them, and so that more individuals developers apply to become Per-Package Uploaders on packages that they know very well.

On the Debian side, a recent discussion on the debian-project list brought back the topic of the bad terminology and it was agreed that the “New Maintainer process” should be renamed into something else (“New Developer process” has been suggested). But Christoph Berg — Debian Account Manager and hence heavily involved in the New Maintainer Team — suggested that Debian would be better off implementing the long-awaited membership changes before trying to update all the documentation. It would certainly imply some more vocabulary updates. Later in the discussion, he confirmed that membership reform is on the top of the TODO list of the new maintainer team (just after the rewrite of the nm.debian.org website).

What can be expected from this reform? The following answers are my own guesses based on my experience of Debian, but the project hasn’t decided anything yet.

  • First of all: a new status for contributors that are not packagers. The tricky part will be defining the process to follow and the rights granted.

  • Changes to the technical implementation of the DM status. The current implementation does not allow to give upload rights to a single DM if two are listed in the Uploaders field of a package (and both might not have the same experience for that package). Furthermore, it suffers from annoying restrictions like the inability to upload new binary packages.

  • A change of the Debian constitution to integrate those new statuses is almost unavoidable.

  • Other more invasive changes have been proposed like replacing the NM process by a simple designation by other DD, but it’s unlikely to happen. The NM process can already be greatly simplified by the application manager if the applicant can show good testimonials from other developers and if he has a track record of real contributions (e.g. as witnessed by changelog entries in Debian packages).

Almost two years have elapsed since the previous efforts in that direction, the new maintainer team has recruited new members and is in a general better shape. Hopefully, the next episode of this saga will have a better outcome.

This article was first published in Linux Weekly News. In a comment, Mark Shuttleworth tried to explain how the Ubuntu community is being setup.

Ubuntu default bookmarks pointing to Debian

Just noticed that the default bookmarks in Ubuntu include one entry pointing back to Debian. It even says that Ubuntu is based on Debian.

It’s a small thing but it made my day. It’s nice to see this kind of acknowledgement.

Click on the screenshot to see the picture in full size (and yes, most favicons suck with a dark background). For those wondering, the screenshot is taken on a default install of Ubuntu 10.04 Netbook Edition.

Joining Ubuntu as a Debian developer

I’m a Debian developer since 1998 and have been fostering cooperation between Debian and Ubuntu since its inception with various levels of success.

I recently decided to go one step further and join Ubuntu as an official member. I wanted to be able to vote and to promote best cooperation practices from the Ubuntu side as well (and not only in Debian). Joining as a Debian developer should be relatively easy, it is well known in the Debian community that Mark Shuttleworth once said:

Every Debian developer is also an Ubuntu developer, because one way to contribute to Ubuntu is to contribute to Debian.

So I sought to verify that too! Here’s the result.

I prepared my application page collecting 2 testimonials from Lucas Nussbaum and Colin Watson, I registered for the next EMEA membership meeting (which happened yesterday, irc logs here), and I attended. (I just followed the instructions on the wiki)

During the meeting, when it was my turn, I pasted my short introduction and got immediately +1 votes from several board members, they didn’t even want to hear from me, they were already convinced. :-) I still responded to a few questions and elaborated on my goals and was promptly accepted even if I made it clear that my technical contributions will continue to be done within Debian.

A few minutes later I was added to the ubuntumembers launchpad group and this morning I followed the instructions to add my feed to Planet Ubuntu and here I am…

The whole process has been very smooth and the meeting shows a very welcoming attitude. Even someone who was denied the Ubuntu member status was cheered for his work and encouraged to try again when he can present some testimonials from other members.

Hello Planet Ubuntu!