How to create Debian packages with alternative compression methods

While gzip is the standard Unix tool when it comes to compression, there are other tools available and some of them are performing better than gzip in terms of compression ratio. This article will explain where you can make use of them in your Debian packaging work.

In the source package

A source package is composed of multiple files. The .dsc file is always uncompressed and it’s fine since it’s a small textual file. The upstream tarballs can be compressed with gzip (orig.tar.gz), bzip2 (orig.tar.bz2), lzma (orig.tar.lzma) or xz (orig.tar.xz), so choose the one that you want if upstream provides the tarball compressed with multiple tools. Put it at the right place and dpkg-source will automatically use it. Note however that packages using source format “1.0″ are restricted to gzip, and the main Debian archive currently only allows gzip and bzip2 (xz might be allowed later) even if the source format “3.0 (quilt)” supports all of them.

The debian packaging files are provided either in a .diff.gz file for source format “1.0″ (again only gzip is supported) or in a .debian.tar file for source format “3.0 (quilt)”. The latter tarball can be compressed with the tool of your choice, you just have to tell dpkg-source which one to use (see below, note that gzip is the default).

In a native package, dpkg-source must generate the main tarball and you can instruct it to use another tool than gzip with the --compression option. That option is usually put in debian/source/options:

# Use bzip2 instead of gzip
compression = "bzip2"
compression-level = 9

For “3.0 (quilt)” source packages, this option is not very useful as the debian tarball that gets compressed is usually not very large. But some maintainers like to use the same compression tool for the upstream tarball and the debian tarball, so you can use this option to harmonize both.

In native packages, it’s much more interesting: for instance the size of dpkg’s source package has been reduced of 30% by switching to bzip2, saving 2Mb of disk.

In the binary packages

.deb files also contain compressed tar archives and by default they use gzip as well:

$ ar t dpkg_1.15.9_i386.deb 
debian-binary
control.tar.gz
data.tar.gz

data.tar.gz is the archive that contains all the files to be installed and it’s the one that you can compress with another tool if you want. Again this is mostly interesting for (very) large packages where the size difference clearly justifies deviating from the default compression tool. Try it out and see how many megabytes you can shove. Another aspect that you must keep in mind is that those alternative tools might use important amount of memory to do their job, both for compression and decompression. So if your package is meant to be installed on embedded platforms, or if you want to build your package on low-end hardware with few memory, you might want to stick with gzip.

Now how do you change the compression tool? Easy, dpkg-deb supports a -Z option, so you just have to pass “-Zbzip2″ for example. You can also pass “-z6″ for example to change the compression level to 6 (it’s interesting because a lower compression level might require less memory depending on the tool used). The dpkg-deb invocation is typically hidden behind the call to dh_builddeb in your debian/rules so you have to replace that invocation with “dh_builddeb -- -Zbzip2“.

If you are using a debhelper 7 tiny rules files, you have to add an override like in this example:

%:
	dh $@

override_dh_builddeb:
	dh_builddeb -- -Zbzip2

If you are using CDBS, you have to set the variable DEB_DH_BUILDDEB_ARGS:

include /usr/share/cdbs/1/rules/debhelper.mk
[...]
DEB_DH_BUILDDEB_ARGS = -- -Zbzip2

I hope you found this article helpful. Follow me on identi.ca or on twitter.

How to customize dpkg-source’s behaviour in your Debian source package

dpkg-source is the program that generates the Debian source package when a new package version is built. It offers many interesting command-line options but they are often not used because people don’t know how to ensure that they are used every time the package is built. Let’s fill that gap!

It is possible to forward some options to dpkg-source by typing them on the dpkg-buildpackage command line but you’d have to remember to type them every time. You could create a shell alias to avoid typing them but then you can’t have different options for different packages. Not very practical.

The proper solution has been implemented last year (in dpkg 1.15.5). It is now possible to put options in debian/source/options. Any long option (those starting with “--“) can be put in that file, one option per line with the leading “--” stripped.

Here’s an example:

# Bzip2 compression for debian.tar
compression = "bzip2"
compression-level = 7
# Do not generate diff for changes in config.(sub|guess)
extend-diff-ignore = "(^|/)config.(sub|guess)$"

Notice that spaces around the equal sign are possible contrary on the command line. You can use quotes around the value but it’s not required.

The debian/source/options file is part of the source package so if someone else grabs the resulting source package and rebuilds everything, they will use the options that you defined in that file.

You can also use debian/source/local-options but this time the file will not be included in the resulting source package. This is interesting for options that you want to use when you build from the VCS (Version Control Repository, aka git/svn/bzr/etc.) but that people downloading the resulting source package should not have. Some options (like --unapply-patches) are only allowed in that file to ensure a consistent experience for users of source packages.

You can learn more about the existing options in the dpkg-source manual page. Read it, I’m sure you’ll learn something. Did you know that you can tell dpkg-source to abort if you have upstream changes not managed by an existing patch in debian/patches? It’s --abort-on-upstream-changes and it’s only allowed in debian/source/local-options.

Be sure to subscribe to the RSS feed or to the email newsletter to not miss useful documentation for Debian contributors!

How to use multiple upstream tarballs in Debian source packages?

Since the introduction of the “3.0 (quilt)” source format, it is now possible to integrate multiple upstream tarballs in Debian source packages. This article will show you how to do the same with your own package shall you need it. It’s quite useful to easily integrate supplementary plugins, translations, or documentation that the upstream developers are providing in separate tarballs.

Step by step explanation

We’ll take the spamassassin source package as an example. The upstream version is 3.3.1. The main upstream tarball is named as usual (spamassassin_3.3.1.orig.tar.gz) and contains the top directory of our source package. We already have a debian directory because the package is not new.

Upstream provides spamassassin rules in a separate tarball named Mail-SpamAssassin-rules-3.3.1.r901671.tgz. We grab it, rename it to spamassassin_3.3.1.orig-pkgrules.tar.gz and put it next to the main tarball. The “pkgrules” part is the component name that we choose to identify the tarball, it’s also the name of the directory in which it will be extracted inside the source package. For now that directory doesn’t exist yet so we must create it.

$ mv Mail-SpamAssassin-rules-3.3.1.r901671.tgz spamassassin_3.3.1.orig-pkgrules.tar.gz
$ cd spamassassin-3.3.1
$ mkdir pkgrules
$ tar -C pkgrules -zxf ../spamassassin_3.3.1.orig-pkgrules.tar.gz

This is already enough, the next time that you will build the source package, the supplementary tarball will be automatically integrated in the generated source package.

$ dpkg-buildpackage -S
[...]
 dpkg-source -b spamassassin-3.3.1
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building spamassassin using existing ./spamassassin_3.3.1.orig-pkgrules.tar.gz ./spamassassin_3.3.1.orig.tar.gz
dpkg-source: info: building spamassassin in spamassassin_3.3.1-1.debian.tar.gz
dpkg-source: info: building spamassassin in spamassassin_3.3.1-1.dsc

The supplementary tarball is now part of the source package but we’re not making anything useful out of it. We have to modify debian/rules (or debian/spamassin.install) to install the new files in the binary package.

A special case: bundling related software

In very rare cases, you might want to create a bundle of several software (small perl modules for example) and you don’t have any main tarball, you only have several small tarballs. Rename all the tarballs following the same logic as above and when building the source package you can ask dpkg-source to create an empty (and fake) main archive for you with the option --create-empty-orig:

$ dpkg-buildpackage -S --source-option=--create-empty-orig

Use with care as the version number you give to the bundle is what users will see and it’s likely unrelated to the version number of each individual software.

Common mistakes

Forgetting to extract the supplementary tarball

If you forget to extract the content of the supplementary tarball in the pkgrules directory, dpkg-source will emit lots of warnings about those files being deleted. In fact, you did not delete them but you only forgot to create them in the first place.

dpkg-source: warning: ignoring deletion of directory pkgrules
dpkg-source: warning: ignoring deletion of file pkgrules/20_fake_helo_tests.cf
dpkg-source: warning: ignoring deletion of file pkgrules/60_shortcircuit.cf
[...]

Using a bad version number for the supplementary tarball

Sometimes the supplementary tarball has a version of its own that does not match the upstream version. You must still name the file in a way that matches the upstream version of the main tarball otherwise it will not be picked up by dpkg-source and it will generate a new patch in debian/patches/ containing the whole new directory.

It’s possible to encode the version number of the supplementary tarball in the component name (in our example above we could have picked “pkgrules-r901671″ as component name) but this means that the name of the associated directory will regularly change and you must adapt your packaging rules to cope with this.

However this last trick has the benefit of being able to update the additional tarball without bumping the upstream version. A sourceful upload of a new revision of the package will be accepted by the archive: the main tarball is ignored since it’s unchanged but the supplementary tarball is taken since it’s a new file for the archive (it has a different filename).

Be sure to move away old versions of the additional tarball when you do that if you don’t want to upload several versions of the same tarball by mistake!

Misextracting the supplementary tarball

dpkg-source is very smart when it extracts the supplementary tarball and you should be as well when you manually extract it.

If the tarball contains only a single top-level directory, that directory is extracted, renamed to match the component name and moved in the source package directory.
If the tarball contains several top-level files or directories, then the target directory is first created and the content of the archive is directly extracted into that directory.

Here are commands to install the files in both cases (we’re already in the source package directory):

$ mkdir pkgrules
# Archive contains a single top-level directory
$ tar -C pkgrules --strip-components=1 -zxf ../spamassassin_3.3.1.orig-pkgrules.tar.gz
# Archive contains many top-level entries
$ tar -C pkgrules -zxf ../spamassassin_3.3.1.orig-pkgrules.tar.gz

Quilt patch management with debhelper 7

Since quilt 0.46-7 (see #527255) you can automatically apply/unapply a quilt patch serie with debhelper’s dh command. It’s very practical in tiny rules files:

%:
	dh --with quilt $@

There’s also dh_quilt_patch and dh_quilt_unpatch for those who don’t use tiny rules files.

Enjoy !

Dell Latitude E4300 with Debian

So I replaced my Latitude D410 with a shiny new Latitude E4300 (Intel Core 2 Duo SP9400 2.4 Ghz with 4 Gb RAM). Here are some notes about this laptop that might be interesting for others.

SSD disk

I now use an SSD drive for my main disk (Dell Ultra Performance SSD, it’s the second generation of Samsung SSD) and I’m satisfied with that choice, I can boot (an unmodified Debian desktop install) from the SSD in less than 30 seconds while the same system booting from a traditional hard-disk takes more than 45 seconds.

X server

The Intel GM45 graphic card is not auto-recognized by Xorg 7.3 (or rather by xserver-xorg-video-intel 2.3.2 which is in lenny) so you end up with the vesa driver by default. It’s possible to force the usage of the intel driver by adding a “Driver “intel”” line in the device section of xorg.conf but I have opted to use Xorg 7.4 (available in experimental). With this version, I can successfully use the DVI output in the associated dock and I have working suspend/resume. It does create some interesting problems however since that version of the xserver relies on HAL to detect the keyboard layout and doesn’t use the Keyboard section of xorg.conf. You have to create /etc/hal/fdi/policy/10-keymap.fdi by using /usr/share/hal/fdi/policy/10osvendor/10-keymap.fdi as template and reload HAL then restart X.

Wifi support

The Intel 5100 Wifi chipset requires Linux 2.6.27 at least for the new iwlagn driver. This driver also needs a new firmware (the iwlwifi-5000 one) that is not yet integrated in the non-free package firmware-iwlwifi (see #497717).

Sound support

It works ok with alsa and the version integrated in linux 2.6.27 but it still has some rough edges when used in combination with the dock. Using the output jack connector on the dock doesn’t stop the output in the integrated loudspeakers and the volume on that connector is so low that you could think that it doesn’t work at all if you don’t pay attention. Using the microphone works fine.

For reference, if you play in the mixer, “Front mic” means the microphone connected on the dock while “Mic” means the one connected on the laptop. Each “Analog loopback X” option goes pairwise with the corresponding “Input source X” setting. In order for the recording to work, I have to set “Digital Input Source” to “Analog Input”, “Digital” must be activated and “Input source 1” defines the default input used for the recording.

Bluetooth support

Contrary to the previous laptop, Dell offered no choice on the bluetooth chipset, they only propose the “Dell 365 Bluetooth™ Card” so I took it but it doesn’t seem to work out of the box. In fact I can’t even see it with lspci or lsusb so I wonder if they did something wrong during the assembly. Googling on the topic didn’t gave me any good result, let me a comment if you know how to get this working.

Update: so apparently the bluetooth component is there (ID 0a5c:4500 Broadcom Corp.), it just appears as an USB hub so it’s somewhat difficult to guess that it’s effectively a bluetooth card.

Freezes, in particular with an amd64 installation

I first installed the system in 64 bits mode (amd64 architecture) but I had very regular freezes of the system (I couldn’t finish a single kernel compilation for example). Since I switched to an i386 installation, the system is more stable but I still get an occasional freeze every other day. It might be that a more recent kernel fixes this or maybe it will be fixed with a future Dell Bios update… we’ll see, but it’s my biggest complaint with this laptop so far.

Links

Lucas Nussbaum bought the same laptop, you might want to read his remarks as well.

More details

Load the full article only if you want to see the lspci and lsusb output on this laptop.
[Read more...]

Alioth documentation

Since people complained about lack of documentation concerning Alioth I decided to write down the most important things in the Debian Wiki.

Check http://wiki.debian.org/Alioth. Feel free to complete the page and to spread the infos whereever needed.