How to prepare patches for Debian packages

You want to start contributing to Debian and/or Ubuntu, you decided to help a package maintainer and you’re now looking for how to change a source package and how to submit your changes.

1. Retrieve the source package and install build-dependencies

The first step is to retrieve the latest version of the source package and to install the required build-dependencies. I already covered how to do this with apt-get in the article explaining how to rebuild a source package.

If you prefer, you can use dget (from the devscripts package) to directly grab the source package. You can find the URL of the .dsc files in the Package Tracking System for example.

Sometimes apt-get will warn you that the source package is maintained in a VCS repository, like this:

$ apt-get source wordpress
[...]
NOTICE: 'wordpress' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/git/collab-maint/wordpress.git
[...]

In that case, you can use debcheckout to retrieve the VCS repository instead (provided that you have the corresponding VCS installed):

$ debcheckout wordpress
declared git repository at git://git.debian.org/git/collab-maint/wordpress.git
git clone git://git.debian.org/git/collab-maint/wordpress.git wordpress ...
Cloning into wordpress...

Note however that some maintainers use their VCS in a way that’s not really compatible with the explanations that I will give below.

It’s also a good idea to install the package “packaging-dev”. It’s a meta-package depending on the most common tools that are used for Debian packaging work.

2. Do the changes

Execute dch --nmu to record the fact that you’re working on an update prepared by someone who is not the maintainer (NMU means Non Maintainer Upload). This also ensures that if we build the package, we won’t overwrite the original source package that we downloaded, thus making it possible to generate a “diff” between both versions.

2.1. Modify Debian packaging files

Now fire your text editor and do the required changes in the “debian” sub-directory. You will probably run dch -a multiple times to document each subsequent change.

2.2. Modify upstream files

If you have to modify upstream files, the proper way to do it depends on the source package format (“1.0″ vs “3.0 (quilt)” vs “3.0 (native)”, see the debian/source/format file) and on the presence or not of a patch system (the what-patch can help you identify it). In this explanation, I’ll assume that the package is using the recommended format: “3.0 (quilt)”. (It also works for “1.0” if quilt is used and if you configured ~/.quiltrc as recommended by /usr/share/doc/quilt/README.source).

First you should ensure that all patches have been applied with quilt push -a. If there’s no patch yet, you want to create the debian/patches directory (with mkdir debian/patches). Note that you should better invoke quilt from the root of the source package (and the examples below assume this).

2.2.1. Import a patch

If the upstream changes are already in a patch file (say /tmp/patch that you downloaded from the upstream VCS repository) you can import that patch like this:

$ quilt import -P fix-foobar.patch /tmp/patch
Importing patch /tmp/patch (stored as fix-foobar.patch)
$ quilt push
Applying patch fix-foobar.patch
[...]
Now at patch fix-foobar.patch

The -P option allows to select the name of the patch file created in debian/patches/. As you see, the new patch file is recorded in debian/patches/series but not applied by default, we’re thus doing it with quilt push.

2.2.1. Create a new patch

If the upstream changes that you want to make are not in a patch yet, you should tell quilt that you’re going to create one:

$ quilt new fix-foobar.patch
Patch fix-foobar.patch is now on top

Then you must record every file that you’re going to modify with a quilt add invocation. quilt then makes a backup of those files so that it can generate the patch later on. If you’re going to modify the files with your text editor you can just do quilt edit file-to-modify, it’s the same than quilt add file-to-modify followed by sensible-editor file-to-modify.

$ quilt edit foobar.c
File foobar.c added to patch fix-foobar.patch

The last step is tell quilt to generate the patch:

$ quilt refresh
Refreshed patch fix-foobar.patch

3. Test your changes

You should build your modified package with “debuild -us -uc”. You can easily install the resulting package with “debi”. Verify that everything works as expected. If not, continue your modifications until you’re satisfied with the result.

4. Generate a patch and mail it

If you followed the instructions, you should have two .dsc files in the parent directory, like this:

$ cd ..
$ ls wordpress_*.dsc
../wordpress_3.0.5+dfsg-1.1.dsc
../wordpress_3.0.5+dfsg-1.dsc

Generating the patch to send to the maintainer is then just a matter of running debdiff:

$ debdiff wordpress_3.0.5+dfsg-1.dsc wordpress_3.0.5+dfsg-1.1.dsc >/tmp/wp-debdiff

You can send the /tmp/wp-debdiff file to the wordpress maintainer. Usually you send it via the bugreport that your update is fixing and you add the “patch” tag to the report.

This can be automated with the nmudiff utility. By default it assumes that you’re using mutt but it can also directly feed the resulting mail to sendmail. The default text that nmudiff proposes assumes that you’re actually performing an NMU and that the result has been uploaded. If that’s not the case, you should edit the text and make it clear that you’re just sending a patch.

If you have been working in a VCS repository, instead of using debdiff you can simply use the diff feature integrated in your VCS (git diff, svn diff, etc.). But note that with a distributed VCS (like git/bzr/mercurial, unlike svn) you should probably have committed all individual changes in separate changesets. And instead of sending a single patch, you’re probably going to send a series of patches (though it might be easier to just upload your branch in a public repository and give the corresponding URL to the maintainer).

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

How to start contributing to Debian?

I often get requests of persons who would like to contribute to Debian but who don’t know where to start. Let’s try to answer this question properly so that I can give out this URL the next time that I am asked.

The Debian website has a page explaining how to help Debian. While it provides no less than 10 suggestions in a daunting text-only list, it’s difficult to know what to do next once you picked up something that you could do.

I will try to fix this by providing concrete information for each cases in upcoming articles but in the mean time I propose you another approach to start with. Before answering your question (“what can I do for Debian”), we need to know some information about you.

What motivates you?

You’re a volunteer, you’re not doing stuff for Debian because someone told you so. You must have some intrinsic motivation and the ultimate motivation is usually that you’re enjoying what you’re doing.

So what are you enjoying and/or what are your motivations ?

  • Is there something that you would like to learn? A new programming language? Packaging? Coding? System administration? A specific software?
  • Do you want to interact with smart people?
  • Do you like to help users?
  • Do you like to fix software just so that it works for you?
  • Do you like to build something remarkable and useful for millions of people?

On the opposite, make sure to know what you hate and what you want to stay far away from. Maybe you dislike a programming language so much that you don’t want to be involved in a project where you would have to use it, etc.

Write down the answers to the questions, you might need them later when you’ll ask other Debian contributors how you can help.

What are your skills?

If you’re not interested in learning new skills, then you must obviously select a task where your current skills are sufficient. Again make a list of your skills and in particular of skills that you’d like to practice! Here’s a non-exhaustive list of skills to consider:

  • What languages are you fluent with? Are you confident to write documentation or translate documentation in those languages?
  • Are you a programmer? If yes, which languages do you know?
  • Can you diagnose problems? Can you debug problems with strace and/or gdb?
  • Can you triage bugs?
  • Do you know some Debian packaging?
  • Are you an artist and/or a web designer?
  • Do you know how to work with VCS (subversion, git, bzr, …)?

How much time can you spend on Debian?

This is the last important information that you need to communicate whenever you’re asking someone else what you could do for Debian. There’s no point giving you a big task if you can only spend 30 minutes every week. On the opposite if you can work on Debian full time during a week (because you’re between 2 contracts or because you’re in vacation), it’s equally important to know.

In general contributing to Debian requires time, you should be ready to spend at least several hours per week and possibly more at the start while you’re learning everything.

Find something to do

At this point you have a generic idea of what you’d like to do but you’re still missing a concrete objective. Let’s try to find one, we’ll explore several ways to do this.

Scratch your itch

The best objectives are those that satisfy your own needs. Here are some examples:

  • Did you notice a missing feature? Try to implement it.
  • Have you been annoyed by a bug? Try to fix it.
  • Did you lose too much time on something because there was no documentation? Write the missing documentation and submit it where appropriate.
  • File bug reports for the things that you can’t fix yourself. Even wishlist bug reports for new features.
  • Do you use software that are not packaged for Debian? Create the package(s) and maintain it/them.
  • Do you need a newer version of a package compared to what’s in Debian unstable? Contact the maintainer and propose your help to update the package.
  • Do you need a newer version of a package compared to what’s in Debian stable? Contact the maintainer and propose your help to create a backport.

If you’re a good Debian citizen, you have already filed bugs for issues that bugged you. Then you can browse http://bugs.debian.org/from:hertzog@debian.org to find out some ideas of stuff to do (obviously replace hertzog@debian.org by your own email address).

Join a team

If you don’t have a specific itch to scratch, you might want to focus your work on a specific team. Head over to wiki.debian.org/Teams and browse the list of teams.

You’ll surely find one that works in an area that you like. If you select a packaging team, pick one that works on packages that you’re actually using.

Some of the teams have instructions for newcomers, follow them when that is the case. Otherwise join the mailing list and the IRC channel, and get a feel of how the team works. See if it suits you, you can follow several teams at the same time and pick the one that you prefer after a few days/weeks.

Once you have lurked a bit, if you still don’t know how you could help, then just ask on the mailing list. Include all the answers you have collected to the 3 questions on your motivations, your skills and your time available.

Focus on a specific package

You can concentrate your work on a specific package even if you’re in a team, it’s often a good idea. But I list it separately because not all packages are team maintained and you might want to help maintain a package where there’s a single maintainer currently. I leave it up to you to find a way to select a package that interests you…

Then head over to the package tracking system: http://packages.qa.debian.org/dpkg (replace dpkg by the name of the package that interests you)

Fill in the form in the bottom-left corner with your email and select “opts” in the drop-down list, then click “go”. You get a new form where you can select the information that you’ll receive, I recommend you to keep everything except “upload-binary” and to validate the form.

From now on, you get the same mails than the maintainer (and a bit more actually), and it’s a good idea to inform the maintainer that you subscribed and that you’re going to help a bit. Maybe he’s willing to grant you commit rights immediately, maybe he will ask you to send patches for a start. The important thing is to create a good relationship. In any case (even if you did not get any answer from the maintainer), you should be free to help triage existing bugs and to help deal with the flow of incoming bugs (including forwarding bugs when appropriate).

Help a Debian developer

Paul Tagliamonte once blogged Hey, DDs, need help?. He offered his help to “overworked Debian developers”. His post was missing all the information required (cf motivations/skills/time) but the approach is a good one.

The best way to start contributing is to work with existing Debian developers. Even if you “just” want to be sponsored for your own pet package, you should consider that mentoring is a burden for many Debian developers and that you’re more likely to get a sponsor if you have an existing relationship with a Debian developer that you helped. Pick a developer that works in an area that is of interest, and offer him your help.

To simplify things even further, I have created a wiki page where you can find out how you can help me and thus build a relationship with me.

That’s it for now, I hope this article will help you to start contributing. Don’t forget to subscribe to my newsletter to not miss future articles for new contributors. Note that you can refer to this article with the following URL: http://raphaelhertzog.com/go/contributing/ (easier to type and remember).

PS: You might want to also check my Contributing to Debian page.

Deciphering one of dpkg’s weirdest errors: short read on buffer copy

As a Debian/Ubuntu user, you’re likely to be exposed at some point to an error reported by dpkg. In a series of articles, I’ll explain some of the errors that you might encounter.

Some error messages can be confusing at times. Most of the error strings do not appear very often and developers thus tend to use very terse description of the underlying problem. In other cases the architecture of the software makes it difficult to pin-point the real problem because the part that displays the error is several layers above the one that generated the initial error.

This is for example the case with this error of dpkg:

Unpacking replacement xulrunner-1.9.2 ...
dpkg-deb (subprocess): data: internal gzip read error: '<fd:0>: too many length or distance symbols'
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing /var/cache/apt/archives/xulrunner-1.9.2_1.9.2.17+build3+nobinonly-0ubuntu1_amd64.deb (--unpack):
 short read on buffer copy for backend dpkg-deb during `./usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so'

First, the decompression layer discovers something unexpected in the data read in the .deb file and dpkg-deb outputs the error message coming from zlib (“too many length or distance symbols”). This causes the premature end of dpkg-deb --fsys-tarfile that dpkg had executed to extract the .data.tar archive from the deb file. In turn, dpkg informs us that dpkg-deb did not send all the data that were announced (and hence the “short read” in the error message) and that were meant to be part of the file ‘/usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so’.

That’s all nice but it doesn’t help you much in general. What you must understand from the above is that the .deb file is corrupted (sometimes just truncated). In theory it should not happen since APT verifies the checksums of files when they are downloaded. But computers are not infallible and even if the downloaded data was good, it can have been corrupted when stored on disk (for example cheap SSD disks are known to not last very well).

Try removing the file (usually with apt-get clean since it’s stored in APT’s cache) and let APT download it again. Chances are that it will work on the second try. Otherwise consider doing a memory and HDD check as something is probably broken in your computer.

Join my free newsletter and learn more tips for users. Or click here to support my work on dpkg with Flattr, consider subscribing for a few months.

apt-get, aptitude, … pick the right Debian package manager for you

This is a frequently asked question: “What package manager shall I use?”. And my answer is “the one that suits your needs”. In my case, I even use different package managers depending on what I’m trying to do.

APT vs dpkg, which one is the package manager?

In the Debian world, we’re usually thinking of APT-based software when we’re referring to a “package manager”. But in truth, the real package manager is dpkg. It’s the low-level tool that takes a .deb file and extracts its content on the disk, or that takes the name of a package to remove the associated files, etc.

APT is better known because it’s the part of the packaging infrastructure that matters to the user. APT makes collection of software available to the user and does the dirty work of downloading all the required packages and installing them by calling dpkg in the correct order to respect the dependencies.

But APT is not a simple program, it’s a library and several different APT frontends have been developed on top of that library. The most widely known is apt-get since it’s the oldest one, and it’s provided by APT itself.

Graphical APT front-ends

update-manager is a simple frontend useful to install security updates and other trivial daily upgrades (if you’re using testing or sid). It’s the one that you get when you click in the desktop notification that tells you that updates are available. In cases, where the upgrade is too complicated for update-manager, it will suggest to run synaptic which is full featured package manager. You can browse the list on installed/available packages in numerous ways, you can mark packages for installation/upgrade/removal/purge and then run in one go all the recorded actions.

software-center aims to be an easy to use application installer, it will hide most of the packaging details and will only present installed/available applications (as defined by a .desktop file). It’s very user friendly and has been developed by Ubuntu.

Of the graphical front-ends, I use mainly synaptic and only when I’m reviewing what I have installed to trim the system down.

Console-based GUI APT front-ends

In this category, I’ll cite only aptitude. Run without parameter, it will start a powerful console-based GUI. Much like synaptic, you can have multiple views of the installed/available packages and mark packages for installation/upgrade/removal/purge before executing everything at once.

Command-line based package managers and APT front-ends

This is where the well known apt-get fits, but there are several other alternatives: aptitude, cupt, wajig. Wajig and cupt are special cases as they don’t use libapt: the former wraps several tools including apt-get, and the latter is a (partial) APT reimplementation (versions 1.x were in Perl, 2.x are now is C++).

You’re welcome to try them out and find out which one you prefer, but I have never felt the need to use something else than apt-get and aptitude.

apt-get or aptitude?

First I want to make it clear that you can use both and mix them without problems. It used to be annoying when apt-get did not track which packages were automatically installed while aptitude did, but now that both packages share this list, there’s no reason to avoid switching back and forth.

I would recommend apt-get for the big upgrades (i.e. dist-upgrade from one stable to the next) because it will always find quickly a relatively good solution while aptitude can find several convoluted solutions (or none) and it’s difficult to decide which one should be used.

On the opposite for regular upgrades in unstable (or testing), I would recommend “aptitude safe-upgrade“. It does a better job than apt-get at keeping on hold packages which are temporarily broken due to some not yet finished changes while still installing new packages when required. With aptitude it’s also possible to tweak dynamically the suggested operations while apt-get doesn’t allow this. And aptitude’s command line is probably more consistent: with apt-get you have to switch between apt-get and apt-cache depending on the operation that you want to do, aptitude on the other hand does everything by itself.

Take some time to read their respective documentation and to try them.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

Why you should always have a network connection when installing Debian

This is a simple tip but an important one: when you’re installing Debian, take the time required to ensure the machine is connected to the Internet with a wired connection. If you have DHCP available, the debian-installer will use it to configure the network.

Why not use the wireless connection?

Because debian-installer in Squeeze doesn’t support WPA encryption, but only WEP. So if you’re using WPA, picking the wireless connection will lead to no working network during the installation and this is to be avoided.

If you’re still using WEP, you can go ahead of course.

If you only have a wireless connection with WPA, your might want to help the debian-installer team and add the required support. Matthew Palmer did some work on it a few months ago (see this mail and his branch in the netcfg git repository) but he resigned from the d-i team in the mean time. So WPA support is still not available in the wheezy debian-installer.

Why is the network so important?

  1. The “tasks” that you select during the installation process might suggest installation of supplementary packages that are not available on your installation disc. If you install without network, the resulting system might differ from the expected one since it will be missing some packages that are available in the Debian repositories but not on your installation disc.
  2. Your installation media might be old and there are security updates that have been published. If you do your initial installation with network, the security updates will be installed before the reboot and thus before the services are exposed over the network.
  3. If you’re not installing a desktop with network-manager (Debian’s default GNOME Desktop provides it), the initial network configuration is important since this configuration is kept for the future. And you surely want network connectivity on your machine, don’t you?
  4. Without network, APT’s sources.list will not be properly configured to include an HTTP mirror of your country. And really, I prefer when apt-get install can work without the initial installation disc.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Download ISO images of Debian CD/DVD at light speed with Jigdo

Debian CD/DVD ISO images are huge files (4.7 GB for a DVD). It can take a long time to download if your connection with the server is not very fast. Jigdo can help you download them much more quickly if you have a high-speed connection to a normal Debian mirror.

If your download rate from your usual Debian mirror is not better than the download rate from the closest Debian CD mirror, then you should download your images straight from the Debian CD mirror. Otherwise read on and learn how you can save lots of time.

Jigdo is based on a simple idea: the CD images contain mainly thousands of .deb files (and source files too) that are already available on all Debian mirrors. So instead of downloading the ISO image in its entirety, you could reconstruct it with all the files that it contains, that you would download from your usual Debian mirror.

Thus when you download a .jigdo file, you only download a little text file (compressed with gzip) that contains enough information to reconstruct the complete image:

  • it contains a list of the individual files that are needed to rebuild the image;
  • it indicates a mirror where those files can be downloaded (it’s mainly a fallback URL in case our preferred mirror doesn’t have them anymore);
  • it contains an URL to a .template file which describes the layout of all files within the ISO image. It also contains the data which can’t be downloaded from the mirror.

The tool that we will use to regenerate the ISO images is called jigdo-lite, it’s available on Windows if you don’t have any Debian system yet. Otherwise just install the jigdo-file Debian package.

We just need the URL of a .jigdo file (get one from the Debian CD page) and we pass it to jidgo-lite on the command line. It asks for a path of “Files to scan”: if we have an older version of the same CD image, we can indicate a path where it is mounted and it will avoid redownloading the files that we already have, otherwise just press Enter. Then it asks for the Debian mirror to use and by default suggests the one used in /etc/apt/sources.list. Again pressing Enter is usually enough. You can even use --noask to avoid the prompts.

$ jigdo-lite http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo

Jigsaw Download "lite"
Copyright (C) 2001-2005  |  jigdo@
Richard Atterer          |  atterer.net
Getting mirror information from /etc/apt/sources.list

Downloading .jigdo file
--2011-03-22 09:08:11--  http://[2001:6b0:e:2018::163]/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo
[...]

-----------------------------------------------------------------
Images offered by `http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo':
  1: 'Debian GNU/Linux 6.0.1 "Squeeze" - Official Multi-architecture i386/amd64/source DVD #1 20110319-14:55 (20110319)' (debian-6.0.1-i386-amd64-source-DVD-1.iso)

Further information about `debian-6.0.1-i386-amd64-source-DVD-1.iso':
Generated on Sat, 19 Mar 2011 15:11:27 +0000

-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g. `/mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: 

-----------------------------------------------------------------
The jigdo file refers to files stored on Debian mirrors. Please
choose a Debian mirror as follows: Either enter a complete URL
pointing to a mirror (in the form
`ftp://ftp.debian.org/debian/'), or enter any regular expression
for searching through the list of mirrors: Try a two-letter
country code such as `de', or a country name like `United
States', or a server name like `sunsite'.
Debian mirror [http://ftp.fr.debian.org/debian]: 

Downloading .template file
--2011-03-22 09:12:41--  http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.template
[...]
Successfully created `debian-6.0.1-i386-amd64-source-DVD-1.iso'

-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that `debian-6.0.1-i386-amd64-source-DVD-1.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.

OK: Checksums match, image is good!

And you’re done, you have successfully downloaded an official Debian DVD with jigdo-lite. You can head over to the Debian CD page and find out the URL of the .jigdo files for the latest stable release.

One last thing, Bluray images are only available with Jigdo because they are so huge that keeping complete ISO images would be a waste of disk resources of the CD images server (and its mirrors).

Of course, if downloading ISO images and burning CD/DVD is not for you, you can always order some professionally printed CD/DVD. You will even help Debian by doing this.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

Missing firmware in Debian? Learn how to deal with the problem

You know it already, since Debian 6.0 non-free firmware are no longer provided by a standard Debian installation. This will cause some troubles to users who need them. I’m thus going to do a small overview on the topic and teach you what you need to know to deal with the problem.

What are firmware and how are they used?

From the user’s point of view, a firmware is just some data that is needed by some piece of hardware in order to function properly. The driver for that hardware typically loads the firmware on the device as part of its initialization.

In the Linux kernel, the drivers are all using a standardized interface (request_firmware) to retrieve the firmware before sending it to the device. Thanks to this standardization, it’s possible to either embed the firmware in the kernel or to have it request the firmware from user-space when needed.

Debian (like most distributions) has selected the latter option. Thus when the kernel needs a firmware, it sends out a request to user-space. udev is getting the request with the name of the firmware, and in its default configuration (see /lib/udev/rules.d/80-drivers.rules) it executes /lib/udev/firmware.agent in response.

Where are firmware stored?

firmware.agent is a simple shell script that tries to locate a firmware before sending it back to the kernel through a sysfs entry. It looks into the following directories:

  • /lib/firmware/$(uname -r)
  • /lib/firmware
  • /usr/local/lib/firmware
  • /usr/lib/hotplug/firmware

Firmware provided by packages are thus usually in /lib/firmware and you can use /usr/local/lib/firmware for manually installed firmware.

How do I know whether I need a firmware?

First of, you can notice messages from the kernel telling you that it tried to load a firmware but it failed. They look like this:

e100: eth0: e100_request_firmware: Failed to load firmware "e100/d101m_ucode.bin": -2

But you might be informed sooner. When you install a new version of the Linux kernel with the official Debian packages, the post-installation script will go through all loaded modules (those listed by lsmod) and it will verify whether this module as provided by the newly installed kernel might require firmware files. This information can be retrieved with modinfo:

$ modinfo -F firmware /lib/modules/2.6.32-5-amd64/kernel/drivers/net/e100.ko
e100/d102e_ucode.bin
e100/d101s_ucode.bin
e100/d101m_ucode.bin

If one (or more) of those firmware is (are) not yet available on the system, you will get a warning message similar to this one:

update-initramfs will also generate a similar warning on the terminal:

update-initramfs: Generating /boot/initrd.img-2.6.32-5-amd64
W: Possible missing firmware /lib/firmware/e100/d102e_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101s_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101m_ucode.bin for module e100

The Debian installer also detects when you have hardware that might require a missing firmware file. You have the option to supply the missing files on a USB stick (either directly or through the corresponding package).

How do I find and install the missing firmware?

Now that you have the name of the firmware file that you want, it’s relatively easy to identify the package that provides the required file. You can use “apt-cache search <filename>” because the firmware packages embed the list of firmware files in their description. You can also use “apt-file” (provided by the package of the same name) or the web interface at packages.debian.org.

$ apt-cache search d101m_ucode.bin
firmware-linux-nonfree - Binary firmware for various drivers in the Linux kernel
$ apt-file search d101m_ucode.bin
firmware-linux-nonfree: /lib/firmware/e100/d101m_ucode.bin

If the above commands return nothing, you probably need to enable the “non-free” repository in your /etc/apt/sources.list (you can also enable it within synaptic). And you also want to run “sudo apt-file update” to have the latest information.

Now you can install the right package, in the example above it was firmware-linux-nonfree.

How do I install all firmware just to be sure I don’t miss any?

There’s no meta-package depending on all firmware packages so there’s no easy answer to this question. Furthermore not all firmware packages respect the naming convention “firmware-*” (there’s for example zd1211-firmware).

So your best bet is to look up all the packages with a generic search like this one:

$ apt-file --package-only search /lib/firmware/
atmel-firmware
[...]

And then install them all.

Are there DVD or CD images with non-free firmware?

Yes. Debian provides an unofficial netinst image for i386/amd64/powerpc with the non-free firmware, you can find it here: http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/.

I provide complete DVD sets with firmware, and Multi-Arch CD/DVD with firmware, it’s over there in my DVD shop: http://raphaelhertzog.com/go/debian-cd/ (i386/amd64 only)

When using those installation discs, the Debian-installer will be able to find the required firmware immediately. There’s no need for you to provide them on a USB stick.

Other questions?

I think I covered the most important things you have to know about firmware on Debian. But should you still have a question, feel free to share it in the comments so that I can improve this article.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

My Debian package maintainer pledge

When coming back from DebConf9, I wrote this pledge:

As a package maintainer, I will do my best to help the Debian project release a stable version of our operating system. In particular, I will work together with the release team and I will keep all packages associated to my name free of release critical bugs.

To this effect, if I’m not registered as being busy or in vacation, I will start working on my release critical bugs as soon as possible (in less than 1 week in common cases). If I can’t deal with them in a timely fashion, I will state it clearly in the associated bug reports, tag them help and invite other contributors either to provide a patch or to do a non-maintainer upload.

If I do not manage to handle release critical bugs in the above described way, or if I almost never deal with any of my RC bugs by myself, I will:

  • not refuse help and even propose co-maintenance to good contributors
  • recognize my failure and actively try to find a new maintainer and/or co-maintainers
  • not complain if the quality assurance team decides to orphan the package

I recognize that my work is not limited to unstable. I will also work with the stable release team and the security team to provide updated packages for the stable and/or testing distribution when some issues deserve it.

I am aware of the limits of my skills and my available time and I will avoid packaging software that I would not be able to maintain properly.

It tries to sum up the minimal expectations that Debian should have towards the package maintainers. I tried first to get this pledge integrated into the NM/DM process but the discussion concluded that it would be best to document this in the developers-reference. There’s already a chapter called “Debian Developer’s Duties” where this can nicely fit.

I submitted this as a wishlist request for the developers-reference soon after and it’s still open. Now I went to the next step and prepared a patch that restructures the chapter about developer duties. See #548867 for details.

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

Debian Cleanup Tip #6: Remove automatically installed packages that are no longer needed

Last week we learned how to identify cruft on your Debian system. This week, for the last article in this series, we’ll learn more about automatically installed packages and how to get rid of them when you don’t need them any longer.

APT tracks automatically installed packages

When you install a new package with apt-get/aptitude/synaptic, it’s very common to end up installing many more packages: those are the dependencies of the installed package. Here’s an example:

$ sudo apt-get install pino
[...]
The following extra packages will be installed:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon
The following NEW packages will be installed:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon pino
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 478 kB of archives.
After this operation, 2531 kB of additional disk space will be used.
Do you want to continue [Y/n]? 

After this installation, the 5 “extra packages” will be marked as “automatically installed”. What does this mean? It means that you have not explicitly requested their installation and that the system should be free to remove them as soon as they are no longer needed.

You can verify that this is effectively the case with “apt-mark showauto” (it returns a list of the automatically installed packages).

$ apt-mark showauto |grep libdbusmenu
libdbusmenu-glib1
$ apt-mark showauto |grep pino
$

Aptitude shows this information with the “A” letter in its interactive interface and in the “aptitude search” output. “aptitude show” has a dedicated field for this:

$ aptitude show libdbusmenu-glib1
Package: libdbusmenu-glib1               
New: yes
State: installed
Automatically installed: yes
Version: 0.3.7-1
[...]

In Synaptic, it’s not very visible but once you have selected an installed package, you can verify in the “Package” menu whether “Automatically installed” is checked or not.

APT tells you which packages are no longer needed

Over time, some of those automatically installed packages become unnecessary because the packages that depended on them no longer do. It might be that they are using a newer version of the same library, or they switched to use something else, or they are able to do the task themselves.

Whatever the reason, the original dependency has vanished and the automatically installed package is no longer needed on the system.

Aptitude will automatically remove those unneeded packages the next time you run it but apt-get and synaptic do not.

Apt-get will inform you that some packages are no longer needed and will even tell you how you can get rid of them:

$ sudo apt-get remove pino
[...]
The following packages were automatically installed and are no longer required:
  notification-daemon libdbusmenu-glib1 libnotify1 libgee2 libindicate4
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  pino
0 upgraded, 0 newly installed, 1 to remove and 219 not upgraded.
After this operation, 1225 kB disk space will be freed.
Do you want to continue [Y/n]? 
[...]
$ sudo apt-get autoremove
[...]
The following packages will be REMOVED:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon
0 upgraded, 0 newly installed, 5 to remove and 219 not upgraded.
After this operation, 1307 kB disk space will be freed.
Do you want to continue [Y/n]? 
[...]

Synaptic will show you the packages that can be removed in a new section name “Installed (auto removable)” if you select the “Status” button in the bottom-left pane.

It’s thus a good habit to get rid of those unneeded package from time to time.

Use this feature to trim down your system

While APT usually sets the “Automatically installed” flag, you can also set it manually. It’s a very simple way to tell the system “I don’t need this package directly, feel free to remove it if nothing else requires it”.

# With apt-get
$ sudo apt-mark markauto libxml-simple-perl
# Or with aptitude
$ sudo aptitude markauto libxml-simple-perl

You can also do it in the interactive interface of aptitude with the key “M” (and “m” for unmarking). To do it in Synaptic, you have to use the menu entry “Package > Automatically installed”.

Many users like to have a minimal set of packages installed but they don’t really know which packages are really important and trying to remove every package to look what happens is cumbersome.

Thanks to this feature, you don’t try removing packages but you flag them as automatically installed. There is few risks in doing so when it concerns libraries (including python/perl modules). If the package is not indirectly needed by one of your important packages, it will be removed by apt-get autoremove, otherwise it’s kept for as long as it’s needed.

I would suggest to not mark as such packages of priority higher or equal to important to avoid nasty surprises (although I say this to not be blamed in case you remove too much, in theory the system should not remove essential components and all dependencies should be complete).

Also be aware of the consequences when you mark “task” packages like “gnome” as automatically installed… it will suggest you to remove your whole desktop. If you want to trim down the default desktop, you should “unmark” the desktop packages that you want to keep:

$ sudo apt-mark unmarkauto gnome-session gnome-panel

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Debian Cleanup Tip #5: identify cruft that can be removed from your Debian system

Last week we learned how to identify and restore packages whose files have been corrupted. This time we’ll concentrate ourselves on the non-packaged files…

Non-packaged files

They are files which are not provided by a Debian package, or in other words, files where dpkg --search finds no associated package:

$ dpkg --search /srv/cvs
dpkg-query: no path found matching pattern /srv/cvs

You always have such files on your system, at least all your own files in /home. But many daemons also create files as part of their work (and they are usually stored in /var): internal files for a database server, mail spool for a mail server, etc. Those are normal and you want to leave them alone.

But you might have non-packaged files in /usr and that should not be the case if you install everything from packages. It would thus be useful to be able to list those files in order to detect a software that has been manually installed.

Manually installed software is not a good idea

Such an installation might cause troubles for example by taking precedence over the same software provided in a Debian package. Over time the local installation will not be upgraded while the packaged one will.

The other packages which depend on this software will believe they have the latest version since their dependency is satisfied but in fact they are using the older version since it takes precedence.

So you want to get rid of those? Let’s see how we can find them.

Use cruft to identify non-packaged files

As I explained above, there are many non-packaged files that are legitimate and that you don’t want to remove. That’s why cruft does something more elaborated than a scan of the filesystem and a check of dpkg’s database.

It provides a way for packages to say which files they might legitimately create during run-time and that cruft should not report. And it knows of many such files. But it’s far from exhaustive and definitely not up-to-date.

So you should always take its output with suspicion and consider twice where the file came from. Do not trust it blindly to remove the files… you have been warned.

How to use cruft

You should give it a list of directories to ignore to reduce the noise in the output, for example like this:

$ sudo cruft -d / -r report --ignore /home --ignore /var --ignore /tmp
$ less report
cruft report: mercredi 23 février 2011, 15:45:34 (UTC+0100)

---- missing: ALTERNATIVES ----
        /etc/alternatives/cli-csc.1.gz
        /usr/share/man/man1/cli-csc.1.gz
---- missing: dpkg ----
        /etc/xdg/autostart/gnome-power-manager.desktop
        /usr/lib/libpython2.6_d.so.1.0-gdb.py
        /usr/share/fonts/X11/100dpi
        /usr/share/fonts/X11/75dpi
---- unexplained: / ----
        /boot
        /dev
        /etc/.java
        /etc/.java/.systemPrefs
[...]
        /usr/lib/pymodules/python2.6
        /usr/lib/pymodules/python2.6/.path
        /usr/lib/pymodules/python2.6/Brlapi-0.5.5.egg-info
[...]

Note that it doesn’t traverse filesystems so if your /usr is on another partition than /, you will need to use the option -d "/ /usr" to have it scan both.

Analyze the report

Now you can quietly go through the report that has been generated and decide which files need to be removed or not. The report also contains missing files (files which should exist according to the dpkg database but which are not there) but the bulk of the listing will be in the “unexplained” section: files which are not part of any package (and whose presence is not explained by any other explain script that packages can ship).

Again take this with great suspicion, and you should rather not delete a file if you don’t know it got there in the first place. For instance, on my system it lists many files below /usr/lib/pymodules/ and those are legitimate: they come from Debian packages but they are copied there dynamically from /usr/{lib,share}/pyshared in order to support multiple python versions. If you remove those files, you effectively break your system.

You will also find many .pyc files created by python packages, they are a byte-compiled version of the corresponding .py file. Removing them breaks nothing but you loose a bit of performance.

On the opposite, most of the files below /usr/local/ are likely the result of some manual software installation and those should be safe to remove (if you know that you are not using the corresponding software).

Conclusion: useful but needs work

In summary, you can use cruft to identify non-packaged files and maybe learn a bit more about what got manually installed on the system, but it requires some patience to go through the report as many of the files reported are false positives.

Yes, cruft badly needs supplementary volunteers to cope with the many ways packages legitimately generate non-packaged files. It’s not even complicated work: the package is mostly in shell and in Perl, and /usr/share/doc/cruft/README.gz explains how it all works.

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.