Auto Mounting Windows Shares in GNOME with Gigolo and gvfs-fuse

The traditional way to mount Windows (or Samba) shares involves hardcoding the credentials in a plain-text file and some /etc/fstab entry to mount it automatically at boot time. If you don’t want to store a plain-text copy of your password, you’re bound to mount your shares interactively.

This is clearly sub-optimal and I thought that there must be a better way to handle this in the context of a GNOME desktop (which already supports connecting to such shares in the file browser). So I looked for a solution and after a bit of googling I found one.

Start by installing a few packages:

$ sudo apt-get install gigolo gvfs-fuse

Launch Gigolo, setup your shares as bookmarks and mark them as “Auto-Connect”.

During (first) connection, you will be prompted for your password and you have the possibility to store it in the GNOME keyring (and here it’s encrypted, not in plain-text!).

You should also configure the Gigolo preferences so that it starts minimized in the system tray (because we’re going to run it on session startup):

The last step is to ensure that Gigolo is executed at the start of each GNOME session. Unfortunately this GNOME feature is no longer accessible from the control center so you have to execute gnome-session-properties manually (from a terminal or the command line accessible via Alt+F2). Click on “Add” to add a new startup program:

You’re done!

The main limitation is that those shares are not real mounts, instead they are available within GNOME’s virtual file system (GVFS). If you use only 100% GNOME application, then it’s not a problem but otherwise it’s pretty annoying. You can’t “cd” in those shares from a terminal for example.

There’s a workaround though, it’s called “gvfs-fuse” and you installed it right at the start of this HOWTO. This service hooks into GVFS and exports all the virtual filesystem(s) in a real fuse-based mount that is automatically setup in ~/.gvfs/. However for this to work, the user must be in the “fuse” group. So you should run something like this:

$ sudo adduser $USER fuse

By the way, I haven’t found a way to use a non-hidden directory so if you want this directory to be more visible, I suggest that you create a symlink pointing to it.

Do you want to read more HOWTO like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Understand dpkg and don’t get stuck with a maintainer script failure

Continuing my series of articles on dpkg’s errors, this time I’ll cover a pretty common one which has several variations:

Setting up acpid (1:2.0.12-1) ...
rm: cannot remove `/etc/rc1.d/K20acpid': No such file or directory
dpkg: error processing acpid (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 acpid

Even if dpkg is failing and outputting the error message, the real problem is not in dpkg but in the installed package (acpid in the example above). As we already learned, a package contains not only files but also “maintainer scripts” that are executed at various points of the installation process (see some useful graphics to understand how they are called, thanks to Margarita Manterola).

Maintainer scripts in a package upgrade

In the introductory example it was acpid’s “post-installation script” that failed, and dpkg is only forwarding that failure back to the caller. The maintainer scripts are stored in /var/lib/dpkg/info/. You can thus inspect them and even modify them if you hit a bug and want to work around it (do this only if you understand what you do!).

One common modification is to add “set -x” at the start of the script and to retry the failing operation. That way you can see what’s executed exactly. Here’s what the output could look like after the addition of “set -x” to /var/lib/dpkg/info/acpid.postinst:

$ sudo dpkg --configure acpid
Setting up acpid (1:2.0.12-1) ...
+ dpkg --compare-versions 1:2.0.11-1 lt-nl 1.0.10-3
+ dpkg --compare-versions 1:2.0.11-1 lt-nl 1.0.6-16
+ dpkg --compare-versions 1:2.0.11-1 lt 1.0.6-6
+ rm /etc/rc1.d/K20acpid
rm: cannot remove `/etc/rc1.d/K20acpid': No such file or directory
dpkg: error processing acpid (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 acpid

This output helps you locate the command that is actually failing. Here’s it’s relatively easy since we have an error message from “rm”. And the fix is trivial too, we replace “rm” with “rm -f” so that it doesn’t fail when the file doesn’t exist (this is a fake bug I made up for this article—I just added a failing rm call—but it’s inspired by real bugs I experienced).

Maintainer scripts are supposed to be idempotent: we should be able to execute them several times in a row without bad consequences. It happens from time to time that the maintainer gets this wrong… on the first try it works, so he uploads his package and we discover the problem only later once someone ended up executing the same code twice for some reason.

Follow me on Identi.ca, Twitter, Facebook and Google+. Or subscribe to this blog by RSS or by email.

Understanding dpkg’s file overwrite error

This is probably one of the most common errors. You’re very likely to encounter it, in particular if you tend to mix packages from various origins/distributions, or if you’re using unstable. It looks like this:

Unpacking gbonds-data (from .../gbonds-data_2.0.3-2_all.deb) ...
dpkg: error processing /var/cache/apt/archives/gbonds-data_2.0.3-2_all.deb (--unpack):
 trying to overwrite '/usr/share/omf/gbonds/gbonds-C.omf', which is also in package gbonds 2.0.2-9
dpkg-deb: subprocess paste killed by signal (Broken pipe)

A given file can only be provided by a single package. So if you try to install a package that provides a file that is already part of another installed package, it will fail with a message similar to the above one.

Sometimes this failure will be meaningful because dpkg prevented you to install two unrelated packages that happen to have a real file conflict. In other cases, like in the example above, this failure is just the result of a mistake.

Folder with gears

The version 2.0.3-1 of gbonds split the architecture independent files in a separate package called gbonds-data but the maintainer forgot to add the required control field in gbonds-data (Replaces: gbonds (<< 2.0.3-1)). That field allows dpkg to take over files from the listed packages.

If you want to ignore the file conflict and let dpkg take over the file (even without the Replaces), you can pass the --force-overwrite command-line option.

But you’re not using dpkg directly, you’re probably using an APT frontend (like apt-get or aptitude). Don’t worry, there’s a simple way to define custom dpkg options to use:

# apt-get -o Dpkg::Options::="--force-overwrite" install gbonds-data

The syntax is a bit weird, but the “::” after “Options” is important, it’s the syntax that defines a list item value instead of a single value. And you can effectively pass multiple options to dpkg by putting multiple -o Dpkg::Options::="…".

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Deciphering one of dpkg’s weirdest errors: unable to open ‘/path/to/foo.dpkg-new’

We already studied one weird error of dpkg, let’s do another one:

Unpacking replacement libexo-common ...
dpkg: error processing /var/cache/apt/archives/libexo-common_0.6.1-1_all.deb (--unpack):
 unable to open '/usr/share/doc/exo/html/C/images/exo-preferred-applications-internet.png.dpkg-new': No such file or directory

Rather difficult to understand on the first look right? Let’s see in detail what’s usually happening when you get this error.

The first hint comes from the file extension “.dpkg-new”. This extension is used by dpkg to unpack the updated files near the old files. When everything has been unpacked, they are renamed over the old files.

The failure happens precisely when dpkg tries to rename the file (in fact when it tries to fsync() it before the rename)… but why does it fail?

Usually because there are unexpected symlinks (or bind mounts) involved that resulted in two different files being installed in the same directory. For example consider package that provides /dir1/a-file and /dir2/a-file. Now imagine that on the target system, /dir1 is a real directory but /dir2 is a symbolic link that points to /dir1.

When dpkg processes /dir1/a-file.dpkg-new everything is fine, but when it tries to process /dir2/a-file.dpkg-new it will fail because that file is the same than /dir1/a-file.dpkg-new which has already been renamed.

Diagnosing further the problem requires to understand why there’s a symlink instead of a real directory. It might be two packages that were badly coordinated, or a problem in the package itself because it lacks some code that drops the symlink in the preinst (so that dpkg installs the real directory instead of keeping the symlink).

There might be variations in the way two files end up sharing the same directory, but this simple example should have clarified the nature of the underlying problem.

Subscribe to this blog by RSS, by email or on Facebook.

7 tips to file useful Debian bug reports and get your problem solved

Filing bug reports is the most common way for users to contribute. Even if it’s not too difficult, I’ll give you some advice to improve the quality of your reports. After all, when you go out of your way to report a bug, it is in the hope to see it fixed… so let’s see how we can make this more likely.

1. Try to reproduce the bug

If you can’t reproduce the bug, it’s next to impossible to find the root cause and thus to fix it. In that case, I would suggest you to wait until you experienced the bug multiple times. Maybe you’ll be able to find something that triggers it (or that makes it more likely to encounter it). If the application has a debug/verbose mode, it might be a good idea to enable it until you experience the bug a second time. The generated log might be helpful for the developer to understand what happens exactly.

So don’t file bug reports straight away unless you can reproduce it. The exception to the rule is when the application gives some useful information like a core-dump, a back-trace or an error message.

Obviously if the bug happens during an upgrade, it’s difficult to reproduce it (unless you have multiple computers) but you should still report it. Be sure to include all the relevant information (versions of packages before and after the upgrade, logs of the upgrade, etc.).

2. Do your best to identify the faulty package

When you report a bug to Debian, you must assign it to a package. While there are pseudo-packages useful for problems which are not directly attributable to a real package, in most of the cases you should report a bug against the specific package that seems to be the cause of the problem you encountered.

In turn this often requires you to attribute the problem to a file (for example the executable of the application that triggers the bug). Once you have a filename you can use dpkg -S to identify the corresponding package.

$ dpkg -S /usr/bin/hamster-time-tracker
hamster-applet: /usr/bin/hamster-time-tracker

Note that reportbug accepts a filename as parameter and will do the above conversion for you.

If you only know the name of the application (but not the filename of the associated executable), you can use dpkg -S with a pattern to let it return all possible matches:

$ dpkg -S hamster
hamster-applet: /usr/share/applications/hamster-applet.desktop
hamster-applet: /usr/share/gnome/help/hamster-applet/es/statistics.page
[…]
hamster-applet: /usr/bin/hamster-time-tracker
[…]

Or you can also verify in the list of installed packages:

$ dpkg -l "*hamster*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version         Description
+++-===============-===============-==============================================
ii  hamster-applet  2.32.1-1        time tracking applet for GNOME

3. Verify that the bug is not already reported and/or fixed

If there’s a newer version of the software available, it’s a good idea to try to reproduce the problem with this version too. Because the developers tend to care only about the latest version, they will want to reproduce it with this version, and they will be annoyed if the problem that you reported is already fixed. That’s why bug reports of users of testing/unstable tend to be more useful than bug reports of stable users.

In any case, you want to verify that the bug has not yet been reported: filing a duplicate bug is useless and only generates more work for the developers to merge both bugs together. On the opposite, it’s highly appreciated to add supplementary information to an existing bug report, even a simple confirmation that the bug still exists on a newer version is useful.

Note that reportbug will automatically show you the list of open bugs before allowing you to submit a new one.

4. Use reportbug

While the Debian bug tracking system allows anyone to submit a new bug with a simple mail, you should really use a dedicated program like reportbug (or reportbug-ng) because it will automatically include lots of useful information in the generated report (version of dependencies, current architecture, etc.) and will assist you in all the steps.

5. Describe the problem so that the developer can reproduce it

Ideally your report should include everything required so that the developer can reproduce the problem on his system. If a given document triggers the bug, attach it.

Describe the steps required to reproduce the bug in great details just like you would explain it to your grand-ma. Explain how you expected the program to react and what happened instead.

You can learn much more on how to draft a good bug report in this article: How to report bugs effectively. It’s a bit long but well worth it if you intend to report bugs and thus interact with developers.

6. Be kind and willing to help

When you draft a bug report, keep in mind that you’re writing to a volunteer free software developer and not to a customer service. You should be respectful and follow the usual rules of courtesy. Developers’ attention is scarce and should not be wasted.

Be willing to help, if the developer starts investigating your problem, he might need your help to get supplementary information (in particular if he can’t reproduce it) and you should be ready to provide it. Thus it’s important to keep whatever you need to reproduce the problem.

In some cases, the Debian maintainer might be overworked and you can offer your help to forward the bug to the upstream bug tracker, it’s always appreciated. If you’re reasonably confident that the problem is not Debian-specific, you can do it straight away and set the forwarded field to the URL of the upstream bug report (for example with bts forwarded <bug> <url>).

7. Use the correct severity

The Debian bug tracking system lets you set the initial severity of the bug report (in decreasing severity: critical, grave, serious, important, normal, minor, wishlist). Pick the correct severity according to the official definitions but don’t misread them.

In particular, don’t over-inflate the severity: for instance if you lost some data due to a misuse of the software, it’s not “critical” (i.e. “rm -rf *” doesn’t warrant a critical bug against rm). If you use only a tiny part of a software, and that part doesn’t work, the package might be unusable for you but it’s not unusable for everybody, so it doesn’t warrant the “grave” severity. The “important” severity is often a good choice in those cases.

Do not under-estimate the severity either, if a problem is important enough that it must be fixed before the next stable release (for example a regression compared to the previous release), pick a release-critical severity (i.e. at least “serious”). The maintainer and the release manager can always lower the severity if they do not agree with your initial judgment.

And now, happy bug-reporting! You can refer to this article with this shorter URL: http://raphaelhertzog.com/go/bugreporting/

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

Deciphering one of dpkg’s weirdest errors: short read on buffer copy

As a Debian/Ubuntu user, you’re likely to be exposed at some point to an error reported by dpkg. In a series of articles, I’ll explain some of the errors that you might encounter.

Some error messages can be confusing at times. Most of the error strings do not appear very often and developers thus tend to use very terse description of the underlying problem. In other cases the architecture of the software makes it difficult to pin-point the real problem because the part that displays the error is several layers above the one that generated the initial error.

This is for example the case with this error of dpkg:

Unpacking replacement xulrunner-1.9.2 ...
dpkg-deb (subprocess): data: internal gzip read error: '<fd:0>: too many length or distance symbols'
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing /var/cache/apt/archives/xulrunner-1.9.2_1.9.2.17+build3+nobinonly-0ubuntu1_amd64.deb (--unpack):
 short read on buffer copy for backend dpkg-deb during `./usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so'

First, the decompression layer discovers something unexpected in the data read in the .deb file and dpkg-deb outputs the error message coming from zlib (“too many length or distance symbols”). This causes the premature end of dpkg-deb --fsys-tarfile that dpkg had executed to extract the .data.tar archive from the deb file. In turn, dpkg informs us that dpkg-deb did not send all the data that were announced (and hence the “short read” in the error message) and that were meant to be part of the file ‘/usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so’.

That’s all nice but it doesn’t help you much in general. What you must understand from the above is that the .deb file is corrupted (sometimes just truncated). In theory it should not happen since APT verifies the checksums of files when they are downloaded. But computers are not infallible and even if the downloaded data was good, it can have been corrupted when stored on disk (for example cheap SSD disks are known to not last very well).

Try removing the file (usually with apt-get clean since it’s stored in APT’s cache) and let APT download it again. Chances are that it will work on the second try. Otherwise consider doing a memory and HDD check as something is probably broken in your computer.

Join my free newsletter and learn more tips for users. Or click here to support my work on dpkg with Flattr, consider subscribing for a few months.

apt-get, aptitude, … pick the right Debian package manager for you

This is a frequently asked question: “What package manager shall I use?”. And my answer is “the one that suits your needs”. In my case, I even use different package managers depending on what I’m trying to do.

APT vs dpkg, which one is the package manager?

In the Debian world, we’re usually thinking of APT-based software when we’re referring to a “package manager”. But in truth, the real package manager is dpkg. It’s the low-level tool that takes a .deb file and extracts its content on the disk, or that takes the name of a package to remove the associated files, etc.

APT is better known because it’s the part of the packaging infrastructure that matters to the user. APT makes collection of software available to the user and does the dirty work of downloading all the required packages and installing them by calling dpkg in the correct order to respect the dependencies.

But APT is not a simple program, it’s a library and several different APT frontends have been developed on top of that library. The most widely known is apt-get since it’s the oldest one, and it’s provided by APT itself.

Graphical APT front-ends

update-manager is a simple frontend useful to install security updates and other trivial daily upgrades (if you’re using testing or sid). It’s the one that you get when you click in the desktop notification that tells you that updates are available. In cases, where the upgrade is too complicated for update-manager, it will suggest to run synaptic which is full featured package manager. You can browse the list on installed/available packages in numerous ways, you can mark packages for installation/upgrade/removal/purge and then run in one go all the recorded actions.

software-center aims to be an easy to use application installer, it will hide most of the packaging details and will only present installed/available applications (as defined by a .desktop file). It’s very user friendly and has been developed by Ubuntu.

Of the graphical front-ends, I use mainly synaptic and only when I’m reviewing what I have installed to trim the system down.

Console-based GUI APT front-ends

In this category, I’ll cite only aptitude. Run without parameter, it will start a powerful console-based GUI. Much like synaptic, you can have multiple views of the installed/available packages and mark packages for installation/upgrade/removal/purge before executing everything at once.

Command-line based package managers and APT front-ends

This is where the well known apt-get fits, but there are several other alternatives: aptitude, cupt, wajig. Wajig and cupt are special cases as they don’t use libapt: the former wraps several tools including apt-get, and the latter is a (partial) APT reimplementation (versions 1.x were in Perl, 2.x are now is C++).

You’re welcome to try them out and find out which one you prefer, but I have never felt the need to use something else than apt-get and aptitude.

apt-get or aptitude?

First I want to make it clear that you can use both and mix them without problems. It used to be annoying when apt-get did not track which packages were automatically installed while aptitude did, but now that both packages share this list, there’s no reason to avoid switching back and forth.

I would recommend apt-get for the big upgrades (i.e. dist-upgrade from one stable to the next) because it will always find quickly a relatively good solution while aptitude can find several convoluted solutions (or none) and it’s difficult to decide which one should be used.

On the opposite for regular upgrades in unstable (or testing), I would recommend “aptitude safe-upgrade“. It does a better job than apt-get at keeping on hold packages which are temporarily broken due to some not yet finished changes while still installing new packages when required. With aptitude it’s also possible to tweak dynamically the suggested operations while apt-get doesn’t allow this. And aptitude’s command line is probably more consistent: with apt-get you have to switch between apt-get and apt-cache depending on the operation that you want to do, aptitude on the other hand does everything by itself.

Take some time to read their respective documentation and to try them.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

Why you should always have a network connection when installing Debian

This is a simple tip but an important one: when you’re installing Debian, take the time required to ensure the machine is connected to the Internet with a wired connection. If you have DHCP available, the debian-installer will use it to configure the network.

Why not use the wireless connection?

Because debian-installer in Squeeze doesn’t support WPA encryption, but only WEP. So if you’re using WPA, picking the wireless connection will lead to no working network during the installation and this is to be avoided.

If you’re still using WEP, you can go ahead of course.

If you only have a wireless connection with WPA, your might want to help the debian-installer team and add the required support. Matthew Palmer did some work on it a few months ago (see this mail and his branch in the netcfg git repository) but he resigned from the d-i team in the mean time. So WPA support is still not available in the wheezy debian-installer.

Why is the network so important?

  1. The “tasks” that you select during the installation process might suggest installation of supplementary packages that are not available on your installation disc. If you install without network, the resulting system might differ from the expected one since it will be missing some packages that are available in the Debian repositories but not on your installation disc.
  2. Your installation media might be old and there are security updates that have been published. If you do your initial installation with network, the security updates will be installed before the reboot and thus before the services are exposed over the network.
  3. If you’re not installing a desktop with network-manager (Debian’s default GNOME Desktop provides it), the initial network configuration is important since this configuration is kept for the future. And you surely want network connectivity on your machine, don’t you?
  4. Without network, APT’s sources.list will not be properly configured to include an HTTP mirror of your country. And really, I prefer when apt-get install can work without the initial installation disc.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Download ISO images of Debian CD/DVD at light speed with Jigdo

Debian CD/DVD ISO images are huge files (4.7 GB for a DVD). It can take a long time to download if your connection with the server is not very fast. Jigdo can help you download them much more quickly if you have a high-speed connection to a normal Debian mirror.

If your download rate from your usual Debian mirror is not better than the download rate from the closest Debian CD mirror, then you should download your images straight from the Debian CD mirror. Otherwise read on and learn how you can save lots of time.

Jigdo is based on a simple idea: the CD images contain mainly thousands of .deb files (and source files too) that are already available on all Debian mirrors. So instead of downloading the ISO image in its entirety, you could reconstruct it with all the files that it contains, that you would download from your usual Debian mirror.

Thus when you download a .jigdo file, you only download a little text file (compressed with gzip) that contains enough information to reconstruct the complete image:

  • it contains a list of the individual files that are needed to rebuild the image;
  • it indicates a mirror where those files can be downloaded (it’s mainly a fallback URL in case our preferred mirror doesn’t have them anymore);
  • it contains an URL to a .template file which describes the layout of all files within the ISO image. It also contains the data which can’t be downloaded from the mirror.

The tool that we will use to regenerate the ISO images is called jigdo-lite, it’s available on Windows if you don’t have any Debian system yet. Otherwise just install the jigdo-file Debian package.

We just need the URL of a .jigdo file (get one from the Debian CD page) and we pass it to jidgo-lite on the command line. It asks for a path of “Files to scan”: if we have an older version of the same CD image, we can indicate a path where it is mounted and it will avoid redownloading the files that we already have, otherwise just press Enter. Then it asks for the Debian mirror to use and by default suggests the one used in /etc/apt/sources.list. Again pressing Enter is usually enough. You can even use --noask to avoid the prompts.

$ jigdo-lite http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo

Jigsaw Download "lite"
Copyright (C) 2001-2005  |  jigdo@
Richard Atterer          |  atterer.net
Getting mirror information from /etc/apt/sources.list

Downloading .jigdo file
--2011-03-22 09:08:11--  http://[2001:6b0:e:2018::163]/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo
[...]

-----------------------------------------------------------------
Images offered by `http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo':
  1: 'Debian GNU/Linux 6.0.1 "Squeeze" - Official Multi-architecture i386/amd64/source DVD #1 20110319-14:55 (20110319)' (debian-6.0.1-i386-amd64-source-DVD-1.iso)

Further information about `debian-6.0.1-i386-amd64-source-DVD-1.iso':
Generated on Sat, 19 Mar 2011 15:11:27 +0000

-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g. `/mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: 

-----------------------------------------------------------------
The jigdo file refers to files stored on Debian mirrors. Please
choose a Debian mirror as follows: Either enter a complete URL
pointing to a mirror (in the form
`ftp://ftp.debian.org/debian/'), or enter any regular expression
for searching through the list of mirrors: Try a two-letter
country code such as `de', or a country name like `United
States', or a server name like `sunsite'.
Debian mirror [http://ftp.fr.debian.org/debian]: 

Downloading .template file
--2011-03-22 09:12:41--  http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.template
[...]
Successfully created `debian-6.0.1-i386-amd64-source-DVD-1.iso'

-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that `debian-6.0.1-i386-amd64-source-DVD-1.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.

OK: Checksums match, image is good!

And you’re done, you have successfully downloaded an official Debian DVD with jigdo-lite. You can head over to the Debian CD page and find out the URL of the .jigdo files for the latest stable release.

One last thing, Bluray images are only available with Jigdo because they are so huge that keeping complete ISO images would be a waste of disk resources of the CD images server (and its mirrors).

Of course, if downloading ISO images and burning CD/DVD is not for you, you can always order some professionally printed CD/DVD. You will even help Debian by doing this.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Missing firmware in Debian? Learn how to deal with the problem

You know it already, since Debian 6.0 non-free firmware are no longer provided by a standard Debian installation. This will cause some troubles to users who need them. I’m thus going to do a small overview on the topic and teach you what you need to know to deal with the problem.

What are firmware and how are they used?

From the user’s point of view, a firmware is just some data that is needed by some piece of hardware in order to function properly. The driver for that hardware typically loads the firmware on the device as part of its initialization.

In the Linux kernel, the drivers are all using a standardized interface (request_firmware) to retrieve the firmware before sending it to the device. Thanks to this standardization, it’s possible to either embed the firmware in the kernel or to have it request the firmware from user-space when needed.

Debian (like most distributions) has selected the latter option. Thus when the kernel needs a firmware, it sends out a request to user-space. udev is getting the request with the name of the firmware, and in its default configuration (see /lib/udev/rules.d/80-drivers.rules) it executes /lib/udev/firmware.agent in response.

Where are firmware stored?

firmware.agent is a simple shell script that tries to locate a firmware before sending it back to the kernel through a sysfs entry. It looks into the following directories:

  • /lib/firmware/$(uname -r)
  • /lib/firmware
  • /usr/local/lib/firmware
  • /usr/lib/hotplug/firmware

Firmware provided by packages are thus usually in /lib/firmware and you can use /usr/local/lib/firmware for manually installed firmware.

How do I know whether I need a firmware?

First of, you can notice messages from the kernel telling you that it tried to load a firmware but it failed. They look like this:

e100: eth0: e100_request_firmware: Failed to load firmware "e100/d101m_ucode.bin": -2

But you might be informed sooner. When you install a new version of the Linux kernel with the official Debian packages, the post-installation script will go through all loaded modules (those listed by lsmod) and it will verify whether this module as provided by the newly installed kernel might require firmware files. This information can be retrieved with modinfo:

$ modinfo -F firmware /lib/modules/2.6.32-5-amd64/kernel/drivers/net/e100.ko
e100/d102e_ucode.bin
e100/d101s_ucode.bin
e100/d101m_ucode.bin

If one (or more) of those firmware is (are) not yet available on the system, you will get a warning message similar to this one:

update-initramfs will also generate a similar warning on the terminal:

update-initramfs: Generating /boot/initrd.img-2.6.32-5-amd64
W: Possible missing firmware /lib/firmware/e100/d102e_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101s_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101m_ucode.bin for module e100

The Debian installer also detects when you have hardware that might require a missing firmware file. You have the option to supply the missing files on a USB stick (either directly or through the corresponding package).

How do I find and install the missing firmware?

Now that you have the name of the firmware file that you want, it’s relatively easy to identify the package that provides the required file. You can use “apt-cache search <filename>” because the firmware packages embed the list of firmware files in their description. You can also use “apt-file” (provided by the package of the same name) or the web interface at packages.debian.org.

$ apt-cache search d101m_ucode.bin
firmware-linux-nonfree - Binary firmware for various drivers in the Linux kernel
$ apt-file search d101m_ucode.bin
firmware-linux-nonfree: /lib/firmware/e100/d101m_ucode.bin

If the above commands return nothing, you probably need to enable the “non-free” repository in your /etc/apt/sources.list (you can also enable it within synaptic). And you also want to run “sudo apt-file update” to have the latest information.

Now you can install the right package, in the example above it was firmware-linux-nonfree.

How do I install all firmware just to be sure I don’t miss any?

There’s no meta-package depending on all firmware packages so there’s no easy answer to this question. Furthermore not all firmware packages respect the naming convention “firmware-*” (there’s for example zd1211-firmware).

So your best bet is to look up all the packages with a generic search like this one:

$ apt-file --package-only search /lib/firmware/
atmel-firmware
[...]

And then install them all.

Are there DVD or CD images with non-free firmware?

Yes. Debian provides an unofficial netinst image for i386/amd64/powerpc with the non-free firmware, you can find it here: http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/.

I provide complete DVD sets with firmware, and Multi-Arch CD/DVD with firmware, it’s over there in my DVD shop: http://raphaelhertzog.com/go/debian-cd/ (i386/amd64 only)

When using those installation discs, the Debian-installer will be able to find the required firmware immediately. There’s no need for you to provide them on a USB stick.

Other questions?

I think I covered the most important things you have to know about firmware on Debian. But should you still have a question, feel free to share it in the comments so that I can improve this article.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.