5 reasons why a Debian package is more than a simple file archive

Folder with gearsYou’re probably manipulating Debian packages everyday, but do you know what those files are? This article will show you their bowels… Surely they are more than file archives otherwise we would just use TAR archives (you know those files ending with .tar.gz). Let’s have a look!

1. It’s two TAR file archives in an AR file archive!

A .deb file is actually an archive using the AR format, you can manipulate it with the ar command. This archive contains 3 files, you can check it yourself, download any .deb file and run “ar t” on it:

$ ar t gwibber_2.31.91-1_all.deb
debian-binary
control.tar.gz
data.tar.gz

debian-binary is a text file indicating the version of the format of the .deb file, the current version is “2.0”.

$ ar p gwibber_2.31.91-1_all.deb debian-binary
2.0

data.tar.gz contains the real files of the package, the content of that archive gets installed in your root directory when you run “dpkg --unpack“.

But the most interesting part—which truly makes .deb files more than a file archive—is the last file. control.tar.gz contains meta-information used by the package manager. What are they?

$ ar p gwibber_2.31.91-1_all.deb control.tar.gz | tar tzf -
./
./postinst
./prerm
./preinst
./postrm
./conffiles
./md5sums
./control

2. It contains meta-information defining the package and its relationships

The control file within the control.tar.gz archive is the most fundamental file. It contains basic information about the package like its name, its version, its description, the architecture it runs on, who is maintaining it and so on. It also contains dependency fields so that the package manager can ensure that everything needed by the package is installed before-hand. If you want to learn more about those fields, you can check Binary control files in the Debian Policy.

Those information end up in /var/lib/dpkg/status once the package is installed.

3. It contains maintainer scripts so that everything can just work out of the box

At various steps of the installation/upgrade/removal process, dpkg is executing the maintainer scripts provided by the package:

  • postinst: after installation
  • preinst: before installation
  • postrm: after removal
  • prerm: before removal

Note that this description is largely simplified. In fact the scripts are executed on many other occasions with different parameters. There’s an entire chapter of the Debian Policy dedicated to this topic. But you might find this wiki page easier to grasp: http://wiki.debian.org/MaintainerScripts.

While this looks scary, it’s a very important feature. It’s required to cope with non-backwards compatible upgrades, to provide automatic configuration, to create system users on the fly, etc.

4. Configuration files are special files

Unpacking a file archive overwrites the previous version of the files. This is the desired behavior when you upgrade a package, except for configuration files. You prefer not to loose your customizations, don’t you?

That’s why packages can list configuration files in the conffiles file provided by control.tar.gz. That way dpkg will deal with them in a special way.

5. You can always add new meta-information

And in fact many tools already exploit the possibility to provide supplementary files in control.tar.gz:

  • debsums use the md5sums file to ensure no files were accidentally modified
  • dpkg-shlibdeps uses shlibs and symbols files to generate dependencies on libraries
  • debconf uses config scripts to collect configuration information from the user

Once installed, those files are kept by dpkg in /var/lib/dpkg/info/package.* along with maintainer scripts.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Understanding Debian’s release process

Currently, the main product of the Debian project is its stable release[1]. Those release come out approximately every 18-24 months. This article gives a short overview of the process leading to the next stable release.

Creating a new distribution

Immediately after a stable release, a new distribution is created in the Debian archive. Its initial content is a copy of the (just released) stable distribution. Its codename is decided by the release managers and there’s a tradition of picking a character’s name from the Toy Story movie.

As an example, the “wheezy” distribution will be created once “squeeze” (aka Debian 6.0) is out.

For simplicity there’s a generic name to refer to the distribution used to prepare the next stable release: it’s testing. In the Debian archive, testing is just a symbolic link pointing to the right directory (squeeze currently).

Updating packages, working on release goals

During most of the cycle, developers work on packaging new upstream versions and implementing release goals. They upload their packages in the unstable distribution.

From there packages trickle to the testing distribution once they satisfy some quality checks: they must not have new release-critical bugs, they must have been built on all architectures that were previously supported, they must not break any dependency in testing, and they must have spent at least 10 days in unstable.

This minimal period ensures the package has been tested and gives enough time to users to file bugs if the package is suffering from problems. If the problems are deemed release-critical, they will block the migration of the package to testing.

During this part of the release cycle, the main work of the release team is ensuring that updated packages flow from unstable to testing. It can be a tricky task: package dependencies frequently tie packages together so that they can only migrate to testing together. If only one of the tied packages is not ready (for example if a new revision has been uploaded and has not spent 10 days in unstable yet), then none of them can migrate.

Stabilizing, polishing, fixing release-critical issues

The constant churn of new packages makes it very difficult to build a very polished release. That’s why, at some point, release managers freeze the testing distribution: automatic updates are stopped and they vet every single update made to testing. They have strong requirements, the goal is to only allow updates fixing release critical bugs, or those which are low-risk and bring significant value to the user experience (like new translations, updated documentation, etc.).

During freeze, some packages are also removed because the current upstream version can’t be supported for the lifetime of the stable release.

The freeze tends to slow down the pace of changes in unstable. Many maintainers opt to push new upstream versions in experimental instead so that if they need to update their packages in testing, they can still do it through unstable. This procedure is recommended by the release managers because it means that updates that they unblock have been tested as usual. It’s not the case for updates uploaded directly to testing (through testing-proposed-updates).

This behavior is rather annoying for the bleeding-edge users that use testing or unstable like a rolling release.

Release time

Once release managers are satisfied of the quality of the new distribution, some last minute work is needed, like generating the CD images. In the Debian archive, the release is made official by pointing the “stable” symbolic link to the new distribution (and the “oldstable” one to the previous distribution).

Now it’s party time, the cycle is over, and a new one can start. :-)

[1] The Constantly Usable Testing project aims to make testing a first-class product like stable—but with a very different update policy.

Follow me on Identi.ca, Twitter and Facebook.

Everything you need to know about conffiles: configuration files managed by dpkg

The Debian policy dictates that package upgrades must take care of preserving user changes to configuration files. This article will explain you how most packages ensure this. This is important knowledge for anyone who has to manage upgrades: knowing how it works lets you easily automate most of it and deal correctly with the fallout.

How dpkg manages configuration files

Most packages rely on dpkg to properly install configuration files. Dpkg keeps a checksum of the last installed version of configuration file. When it must install a new version, it calculates the checksum of the currently installed file and if it doesn’t match anymore, it knows that the user has edited the file. In that case, instead of overwriting the configuration file, it asks the user what to do. You probably already have seen those questions, they look like this:

Configuration file `/etc/bash.bashrc'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** bash.bashrc (Y/I/N/O/D/Z) [default=N] ? 

In this specific example, if you answer “Y” or “I” (for “yes” or “install”), dpkg will install the new version of /etc/bash.bashrc but it will also backup the current version in /etc/bash.bashrc.dpkg-old. If you answer “N” or “O” (for “no” or “old”), dpkg will install the new version in /etc/bash.bashrc.dpkg-dist and /etc/bash.bashrc is left untouched. The two other answers allow you to examine the differences before taking a decision. Note that if you choose to start a shell, the new version is currently available as /etc/bash.bashrc.dpkg-new (and since Squeeze there are convenient environment variables $DPKG_CONFFILE_OLD and $DPKG_CONFFILE_NEW in case you want to create a custom review script).

All configurations files managed by dpkg are called “conffiles” because that’s the name of the field where they are recorded in the dpkg database. You can display the list of conffiles for any package:

$ dpkg --status bash
[...]
Conffiles:
 /etc/skel/.profile ecb6d3479ac3823f1da7f314d871989b
 /etc/skel/.bashrc 2afdd6c53990f2387a7ef9989af0bc07
 /etc/skel/.bash_logout 22bfb8c1dd94b5f3813a2b25da67463f
 /etc/bash.bashrc 5b3c3bc73d236e4e1b6f9b6c1ed5964e
[...]

The command “dpkg-query --showformat='${Conffiles}\n' --show bash” can give you the same information if you need to retrieve only that field. The 32 characters after the filename are the MD5 checksum of the original configuration file provided by the package.

Avoiding the conffile prompt

Every time that dpkg must install a new conffile that you have modified (and a removed file is only a particular case of a modified file in dpkg’s eyes), it will stop the upgrade and wait your answer. This can be particularly annoying for major upgrades. That’s why you can give predefined answers to dpkg with the help of multiple --force-conf* options:

  • --force-confold: do not modify the current configuration file, the new version is installed with a .dpkg-dist suffix. With this option alone, even configuration files that you have not modified are left untouched. You need to combine it with --force-confdef to let dpkg overwrite configuration files that you have not modified.
  • --force-confnew: always install the new version of the configuration file, the current version is kept in a file with the .dpkg-old suffix.
  • --force-confdef: ask dpkg to decide alone when it can and prompt otherwise. This is the default behavior of dpkg and this option is mainly useful in combination with --force-confold.
  • --force-confmiss: ask dpkg to install the configuration file if it’s currently missing (for example because you have removed the file by mistake).

If you use Apt, you can pass options to dpkg with a command-line like this:

$ apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

You can also make those options permanent by creating /etc/apt/apt.conf.d/local:

Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

Bringing up the conffile prompt at any time

The conffile prompt is only displayed when dpkg detects that the package provides an new version of the conffile. Thus reinstalling the same package will not bring up the prompt. But you can instruct dpkg to ask nevertheless with the --force-confask option. This is a new feature in Debian Squeeze. It will only ask for files that are locally modified.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Dell Latitude E4300 with Debian

So I replaced my Latitude D410 with a shiny new Latitude E4300 (Intel Core 2 Duo SP9400 2.4 Ghz with 4 Gb RAM). Here are some notes about this laptop that might be interesting for others.

SSD disk

I now use an SSD drive for my main disk (Dell Ultra Performance SSD, it’s the second generation of Samsung SSD) and I’m satisfied with that choice, I can boot (an unmodified Debian desktop install) from the SSD in less than 30 seconds while the same system booting from a traditional hard-disk takes more than 45 seconds.

X server

The Intel GM45 graphic card is not auto-recognized by Xorg 7.3 (or rather by xserver-xorg-video-intel 2.3.2 which is in lenny) so you end up with the vesa driver by default. It’s possible to force the usage of the intel driver by adding a “Driver “intel”” line in the device section of xorg.conf but I have opted to use Xorg 7.4 (available in experimental). With this version, I can successfully use the DVI output in the associated dock and I have working suspend/resume. It does create some interesting problems however since that version of the xserver relies on HAL to detect the keyboard layout and doesn’t use the Keyboard section of xorg.conf. You have to create /etc/hal/fdi/policy/10-keymap.fdi by using /usr/share/hal/fdi/policy/10osvendor/10-keymap.fdi as template and reload HAL then restart X.

Wifi support

The Intel 5100 Wifi chipset requires Linux 2.6.27 at least for the new iwlagn driver. This driver also needs a new firmware (the iwlwifi-5000 one) that is not yet integrated in the non-free package firmware-iwlwifi (see #497717).

Sound support

It works ok with alsa and the version integrated in linux 2.6.27 but it still has some rough edges when used in combination with the dock. Using the output jack connector on the dock doesn’t stop the output in the integrated loudspeakers and the volume on that connector is so low that you could think that it doesn’t work at all if you don’t pay attention. Using the microphone works fine.

For reference, if you play in the mixer, “Front mic” means the microphone connected on the dock while “Mic” means the one connected on the laptop. Each “Analog loopback X” option goes pairwise with the corresponding “Input source X” setting. In order for the recording to work, I have to set “Digital Input Source” to “Analog Input”, “Digital” must be activated and “Input source 1” defines the default input used for the recording.

Bluetooth support

Contrary to the previous laptop, Dell offered no choice on the bluetooth chipset, they only propose the “Dell 365 Bluetooth™ Card” so I took it but it doesn’t seem to work out of the box. In fact I can’t even see it with lspci or lsusb so I wonder if they did something wrong during the assembly. Googling on the topic didn’t gave me any good result, let me a comment if you know how to get this working.

Update: so apparently the bluetooth component is there (ID 0a5c:4500 Broadcom Corp.), it just appears as an USB hub so it’s somewhat difficult to guess that it’s effectively a bluetooth card.

Freezes, in particular with an amd64 installation

I first installed the system in 64 bits mode (amd64 architecture) but I had very regular freezes of the system (I couldn’t finish a single kernel compilation for example). Since I switched to an i386 installation, the system is more stable but I still get an occasional freeze every other day. It might be that a more recent kernel fixes this or maybe it will be fixed with a future Dell Bios update… we’ll see, but it’s my biggest complaint with this laptop so far.

Links

Lucas Nussbaum bought the same laptop, you might want to read his remarks as well.

More details

Load the full article only if you want to see the lspci and lsusb output on this laptop.
[Read more…]