People behind Debian: Ben Hutchings, member of the kernel team

Ben Hutchings, photo by Andrew Mc Millan, license CC-BY-2.0

Ben Hutchings is a rather unassuming guy… but hiding behind his hat, there’s a real kernel hacker who backports new drivers for the kernel in Debian stable so that our flagship release supports very recent hardware.

Read on to learn more about Ben and the kernel team’s projects for Debian Wheezy!

Raphael: Who are you?

Ben: I’m a professional programmer, living in Cambridge, England with my long-suffering wife Nattie. In Debian, I mostly work on the Linux kernel and related packages.

Raphael: How did you start contributing to Debian?

Ben: I started using Debian in 1998 and at some point I subscribed to Debian Weekly News. So in 2003 I heard about the planned Debian 10th birthday party in Cambridge, and thought I would like to go to that. Somehow I persuaded Nattie that we should go, even though it was on the day of our wedding anniversary! We both enjoyed it; we made new friends and met some old ones (small world). From then on we have
both been socially involved in Debian UK.

In 2004 there was a bug-squashing party in Cambridge, and we attended that as well. That’s where I really started contributing – fixing bugs and learning about Debian packaging. Then in 2005 I made my first package (sgt-puzzles), attended DebConf, and was persuaded to enter the New Maintainer process.

NM involved a lot of waiting, but by the time I was given questions and tasks to do I had learned enough to get through quite quickly. In April 2006 I was approved as a Debian Developer.

Meanwhile, I looked at the videos from DebConf 5 and thought that it would be useful to distribute them on a DVD. That led me to start writing video software and to get involved in the video team for the next year’s DebConf.

Raphael: You have been one the main driver behind the removal of non-free firmwares from the kernel. Explain us what you did and what’s the status nowadays?

Ben: That’s giving me a bit more credit than I deserve.

For a long time the easy way for drivers to load ‘firmware’ programs was to include them as a ‘blob’ in their static data, but more recently the kernel has included a simple method for drivers to request a named blob at run-time. These requests are normally handled by udev by reading from files on disk, although there is a build-time option to include blobs in the kernel. Several upstream and distribution developers worked to convert the older drivers to use this method. I converted the last few of these drivers that Debian included in its binary packages.

In the upstream Linux source, those blobs have not actually been removed; they have been moved to a ‘firmware’ subdirectory. The long-term plan is to remove this while still allowing the inclusion of blobs at build-time from the separate ‘linux-firmware’ repository. For now, the Debian source package excludes this subdirectory from the upstream tarball, so it is all free software.

There are still a few drivers that have not been converted, and in Debian we just exclude the firmware from them (so they cannot be built). And from time to time a driver will be added to the ‘staging’ section of Linux that includes firmware in the old way. But it’s understood in the kernel community that it’s one of the bugs that will have to be fixed before the driver can move out of ‘staging’.

Raphael: Do you believe that Debian has done enough to make it easy for users to install the non-free firmwares that they need?

Ben: The installer, the Linux binary packages and initramfs-tools will warn about specific files that may be needed but are missing. Users who have enabled the non-free section should then be able to find the necessary package with apt-cache search, because each of the
binaries built from the firmware-nonfree source package includes driver and file names within its description. For the installer, there is a single tarball that provides everything.

We could make this easier, but I think we have gone about as far as we can while following the Debian Social Contract and Debian policy.

Raphael: At some point in the past, the Debian kernel team was not working very well. Did the situation improve?

Ben: Back in 2008 when I started working on the Linux kernel package to sort out the firmware issues, I think there were some problems of communication and coordination, and quite possibly some members were burned-out.

Since then, many of the most active kernel team members have been able to meet face-to-face to discuss future plans at LPC 2009 in Portland and the 2010 mini-DebConf in Paris. We generally seem to have productive discussions on the debian-kernel mailing list and elsewhere, and I think the team is working quite well. Several new contributors have joined after me.

I would say our biggest problem today is that we just don’t have enough time to do all we want to. Certainly, almost all my Debian time is now taken up with integrating upstream kernel releases and handling some fraction of the incoming bug reports. Occasionally I can take the time to work on actual features or the other packages I’m neglecting!

“Our biggest problem today is that we just don’t have enough time to do all we want to.”

Raphael: It is widely known that Linux is maintained in a git repository. But the Debian kernel team is using Subversion. I believe a switch is planned. Why was not git used from the start?

Ben: The linux-2.6 source package dates from the time when Linus made his first release using git. I wasn’t part of the team back then so I don’t know for sure why it was imported to Subversion. However, at that time hardly anyone knew how to use git, no-one had experience hosting public git repositories, and Alioth certainly didn’t offer that option.

Today there are no real blockers: everyone on the kernel team is familiar with using git; Alioth is ready to host it; we don’t have per-architecture patches that would require large numbers of branches. But it still takes time to plan such a conversion for what is a relatively complex source package (actually a small set of related source packages).

Raphael: What are your plans for Debian Wheezy?

Ben: Something I’ve already done, in conjunction with the installer team, is to start generating udebs from the linux-2.6 source package. The kernel and modules have to be repacked into lots of little udebs to avoid using too much memory during installation. The configuration for this used to be in a bunch of separate source packages; these could get out of step with the kernel build configuration and this would only be noticed some time later. Now we can update them both at the same time, they are effectively cross-checked on every upload, and the installer can always be built from the latest kernel version in testing or unstable.

I think that we should be encouraging PC users to install the 64-bit build (amd64), but many users will still use 32-bit (i386) for backward compatibility or out of habit. On i386, we’ve slightly reduced the variety of kernel flavours by getting rid of ‘686’ and making ‘686-pae’ the default (previously this was called ‘686-bigmem’). This means that the NX security feature will be used on all systems that support it. It should also mean that the first i386 CD can have suitable kernel packages for all systems.

I have been trying to work on providing a full choice of Linux Security Modules (LSMs). Despite their name, they cannot be built as kernel modules, so every enabled LSM is a waste of memory on the systems that don’t use it. This is a significant concern for smaller Debian systems. My intent is to allow all unused LSMs to be freed at boot time so that we can happily enable all of them.

I recently proposed to drop support for older x86 systems, starting with 486-class processors in wheezy. In general, this would allow the use of more compiler optimisations throughout userland and the kernel. However it seems that there isn’t that much to be gained unless we also drop 586-class processors, and there are still quite a few of those in use. So I think this will have to wait.

Uwe Kleine-König has been working to include real-time support (also known as PREEMPT_RT). This can provide low and very predictable I/O latency, which is useful for live audio synthesis, for example. It still requires a number of patches and a build configuration change, resulting in a separate binary package. We’re currently only building that for 64-bit PCs. (You may notice this is missing for Linux 3.1, because the real-time developers skipped this release.)

Raphael: What’s the biggest problem of Debian?

Ben: I think we try too hard to accommodate every possible option, without regard for the cost to developers and users in general. As an example, we now have sysvinit, file-rc, upstart and systemd all in testing. Daemon maintainers can’t rely on any advanced features of upstart or systemd because we refuse to choose between them. And the decision to support the FreeBSD kernel means that we cannot choose upstart or systemd as the only option. So all daemon maintainers will have to maintain those baroque init scripts for the indefinite future. We really should be able to decide as a distribution that when one option is technically good and popular then it can be made the only option. But no-one really has the authority to do that, so we muddle along with the pretence that all the options are equally valid and functional, while none of them is supported as well as they should be.

“We try too hard to accommodate every possible option, without regard for the cost to developers and users in general.”

We also try to build every package on every architecture, in general. I’m quite sure there are many (package, architecture) combinations that have no users, ever. But if at some point that combination FTBFS, developers will waste time investigating and fixing that – time that could have been spent working on bugs and features that users actually care about. Yes, sure, portability is good but you can’t prove portability just by making a package compile on every architecture. This also applies to the selection of drivers for the kernel, by the way.

Raphael: Is there someone in Debian that you admire for their contributions?

Well, there are many people, but I will pick out just a few:

Steve McIntyre, for his work as DPL to improve communication with the various Debian derivatives and to bring fresh blood into various core teams. Also for being a generous host for countless Debian social and bug-squashing events.

Stefano Zacchiroli, for improving further on communications with both downstream and upstream projects, and for regularly exercising his power to lead discussions to the benefit of the project.

Julien Cristau, for maintaining good humour while not only fighting against the tide of graphics driver regressions in X and the Linux kernel but also working on release management.

Jonathan Nieder, for taking on the unglamorous and frustrating task of kernel bug triage as a non-maintainer and developing it to a fine art.


Thank you to Ben for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook

.

Missing firmware in Debian? Learn how to deal with the problem

You know it already, since Debian 6.0 non-free firmware are no longer provided by a standard Debian installation. This will cause some troubles to users who need them. I’m thus going to do a small overview on the topic and teach you what you need to know to deal with the problem.

What are firmware and how are they used?

From the user’s point of view, a firmware is just some data that is needed by some piece of hardware in order to function properly. The driver for that hardware typically loads the firmware on the device as part of its initialization.

In the Linux kernel, the drivers are all using a standardized interface (request_firmware) to retrieve the firmware before sending it to the device. Thanks to this standardization, it’s possible to either embed the firmware in the kernel or to have it request the firmware from user-space when needed.

Debian (like most distributions) has selected the latter option. Thus when the kernel needs a firmware, it sends out a request to user-space. udev is getting the request with the name of the firmware, and in its default configuration (see /lib/udev/rules.d/80-drivers.rules) it executes /lib/udev/firmware.agent in response.

Where are firmware stored?

firmware.agent is a simple shell script that tries to locate a firmware before sending it back to the kernel through a sysfs entry. It looks into the following directories:

  • /lib/firmware/$(uname -r)
  • /lib/firmware
  • /usr/local/lib/firmware
  • /usr/lib/hotplug/firmware

Firmware provided by packages are thus usually in /lib/firmware and you can use /usr/local/lib/firmware for manually installed firmware.

How do I know whether I need a firmware?

First of, you can notice messages from the kernel telling you that it tried to load a firmware but it failed. They look like this:

e100: eth0: e100_request_firmware: Failed to load firmware "e100/d101m_ucode.bin": -2

But you might be informed sooner. When you install a new version of the Linux kernel with the official Debian packages, the post-installation script will go through all loaded modules (those listed by lsmod) and it will verify whether this module as provided by the newly installed kernel might require firmware files. This information can be retrieved with modinfo:

$ modinfo -F firmware /lib/modules/2.6.32-5-amd64/kernel/drivers/net/e100.ko
e100/d102e_ucode.bin
e100/d101s_ucode.bin
e100/d101m_ucode.bin

If one (or more) of those firmware is (are) not yet available on the system, you will get a warning message similar to this one:

update-initramfs will also generate a similar warning on the terminal:

update-initramfs: Generating /boot/initrd.img-2.6.32-5-amd64
W: Possible missing firmware /lib/firmware/e100/d102e_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101s_ucode.bin for module e100
W: Possible missing firmware /lib/firmware/e100/d101m_ucode.bin for module e100

The Debian installer also detects when you have hardware that might require a missing firmware file. You have the option to supply the missing files on a USB stick (either directly or through the corresponding package).

How do I find and install the missing firmware?

Now that you have the name of the firmware file that you want, it’s relatively easy to identify the package that provides the required file. You can use “apt-cache search <filename>” because the firmware packages embed the list of firmware files in their description. You can also use “apt-file” (provided by the package of the same name) or the web interface at packages.debian.org.

$ apt-cache search d101m_ucode.bin
firmware-linux-nonfree - Binary firmware for various drivers in the Linux kernel
$ apt-file search d101m_ucode.bin
firmware-linux-nonfree: /lib/firmware/e100/d101m_ucode.bin

If the above commands return nothing, you probably need to enable the “non-free” repository in your /etc/apt/sources.list (you can also enable it within synaptic). And you also want to run “sudo apt-file update” to have the latest information.

Now you can install the right package, in the example above it was firmware-linux-nonfree.

How do I install all firmware just to be sure I don’t miss any?

There’s no meta-package depending on all firmware packages so there’s no easy answer to this question. Furthermore not all firmware packages respect the naming convention “firmware-*” (there’s for example zd1211-firmware).

So your best bet is to look up all the packages with a generic search like this one:

$ apt-file --package-only search /lib/firmware/
atmel-firmware
[...]

And then install them all.

Are there DVD or CD images with non-free firmware?

Yes. Debian provides an unofficial netinst image for i386/amd64/powerpc with the non-free firmware, you can find it here: http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/.

I provide complete DVD sets with firmware, and Multi-Arch CD/DVD with firmware, it’s over there in my DVD shop: http://raphaelhertzog.com/go/debian-cd/ (i386/amd64 only)

When using those installation discs, the Debian-installer will be able to find the required firmware immediately. There’s no need for you to provide them on a USB stick.

Other questions?

I think I covered the most important things you have to know about firmware on Debian. But should you still have a question, feel free to share it in the comments so that I can improve this article.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

People behind Debian: Maximilian Attems, member of the kernel team

Maximilian, along with the other members of the Debian kernel team, has the overwhelming job of maintaining the Linux kernel in Debian. It’s one of the largest package and certainly one where dealing with bug reports is really difficult as most of them are hardware-specific, and thus difficult to reproduce.

He’s very enthusiastic and energetic, and does not fear criticizing when something doesn’t please him. You’ll see.

My questions are in bold, the rest is by Maximilian.

Who are you?

My name is Maximilian Attems. I am a theoretical physicist in my last year of PhD at the Technical University of Vienna. My main research area is the early phase of a Quark-Gluon Plasma as produced in heavy ion collisions at the LHC at CERN. I am developing simulations that take weeks on the Vienna Scientific Cluster (in the TOP 500 list). The rest of the lab is much less fancy and boils down to straight intel boxes without any binary blobs or external drivers (although lately we add radeon graphics for decent free 3D). Mathematica and Maple are the rare exceptions to the many dev tools of Debian (LaTeX, editors, git, IDE’s, Open MPI, ..) found at the institute, as those are unfortunately yet unmatched in Free Software for symbolic computations. The lab mostly runs a combination of Debian stable (testing starting from freeze) for desktops and oldstable/stable for servers. Debian is in use for more than 10 years. So people in the institute know some ups and downs of the project. Newcomers like my room neighbors are always surprised how functional a free Debian Desktop is. :)

What’s your biggest achievement within Debian?

Building lots and lots of kernels together with an growing uptake of the officially released linux images.

I joined the Debian kernel team shortly after Herbert Xu departed. I had been upstream Maintainer of the linux-2.6 janitor project for almost a year brewing hundreds of small cleanups with quilt in a tree named kjt for early linux-2.6. In Debian we had lots of fun in sorting out the troubles that the long 2.5 freeze had imposed: Meaning we were sitting on a huge diverging “monolithic” semi-good patchset. It was great fun to prepare 2.6.8 for Sarge with a huge team enthusiastic in shipping something real close to mainline (You have to imagine that back then you had no stable or longterm release nor any useful free tools like git. This involved passing patches around, hand editing them and seeing what the result does.)

From the Sarge install reports a common pattern emerged that the current Debian early userspace was causing lots of boot failures. This motivated me to develop an alternative using the new upstream initramfs features. So I got involved in early userspace. Thanks to large and active development team initramfs-tools got a nice ecosystem. It still tries to be as generic and flexible as possible and thus gains many nice features. Also H. Peter Anvin (hpa) gave me the official co-maintenance of klibc. klibc saw uptake and good patches from Google in the last 2 years. I am proud that the early userspace is working out fairly well these days, meaning you can shuffle discs around and see your box boot.

Later on we focused on 2.6.18 for Etch, which turned out to a be good release and picked up by several other distributions. Only very much later we would see such a “sync” again. With 2.6.26 for Lenny we got somehow unlucky as we just missed the new longterm release by one release. We also pushed for another update very late (during freeze) in the release cycle, which turned out to semi-work as too much things depend on linux-2.6.

For Squeeze 2.6.32 got picked thanks to discussions at Portland Linux Plumbers and it turned out to be a good release picked up by many distributions and external patchsets. The long-term support is going very well. Greg KH is doing a great job in collecting various needed fixes for it. Somehow we had hoped that the Squeeze freeze would start sooner and that the freeze duration would be shorter, since we were ready for a release starting from the actual freeze on. The only real big bastard on the cool 2.6.32 “sync” is Red Hat. Red Hat Enterprise 6.0 is shipping the linux-2.6 2.6.32 in obfuscated form. They released their linux-2.6 as one big tarball clashing with the spirit of the GPL. One can only mildly guess from the changelog which patches get applied. This is in sharp contrast to any previous Red Hat release and has not yet generated the sharp and snide comments in press it deserves. Red Hat should really step back and not make such stupid management moves. Next to them even the semi-maintained Oracle “Unbreakable” 2.6.32 branch looks better: It is git fetchable.

What are your plans and those of the kernel team for Debian Wheezy?

Since 2.6.32 many of the used patches landed upstream or are on the way (speakup, Kbuild Debian specific targets, ..). The proper vfs based unionfs is something we’d be looking forward. We haven’t yet picked the next upstream release we will base Wheezy on, so currently we can happily jump to the most recent ones. There are plans for better interaction with Debian Installer thanks to generating our udebs properly in linux-2.6 source itself. Also we are looking forward to using git as tool of maintenance. We’d hope that this will also allow for even better cross distribution collaboration.

Concerning early userspace I plan to release an initramfs-tools with more generic userspace for the default case and finally also a klibc only for embedded or tuning cases.

What do you like most in Debian?

For one thing I do like the 2 year release cycle. It is not too long to have completely outdated software and on the other hand it gives enough time to really see huge progress from release to release. Also at my institute the software is is recent enough without too much admin overhead. For servers the three years support are a bit short, but on the manageable side.

I do enjoy a lot the testing distribution. For my personal use it is very stable and thus I mainly run testing on my desktop and work boxes. (Occasionally mixing in things from sid for unbreaking transition or newer security fixes).

Debian is independent and not a commercial entity. I think this is its main force and even more important these days. I enjoy using the Debian platform a lot at work thus in return this motivates me to contribute to Debian itself. I also like the fact that we strive for technical correctness.

Is there some recurrent problem that hinders the progress of Debian?

The “New Maintainer process” is a strange way to discourage people to contribute to Debian. It is particularly bureaucratic and a huge waste of time both for the applicant and his manager. It should be completely thrown overboard.

One needs a more scalable approach for trust and credibility that also enhances the technical knowledge for coding and packaging of the applicant.

NM is currently set in stone as any outside critics is automatically rejected. Young and energetic people are crucial for Debian and the long-term viability of the project, this is the reason why I’d consider the “New Maintainer process” as Debian’s biggest problem.

Note from Raphaël Hertzog: I must say I do not share this point of view on the New Maintainer process, I have witnessed lots of improvements lately thanks to the addition of the Debian Maintainer status, and to the fact that a good history of contribution can easily subsume the annoying Tasks & Skills questionnaire.

Another thing I miss is professional graphics’ input both for the desktop theme and the website. I know that effort has been done there lately and it is good to see movement there, but the end result is still lacking.

Another trouble of Debian is its marketing capabilities. It should learn to better sell itself. It is the distribution users want to run and use—not the rebranded copies of itself with lock-in “sugar”. Debian is about choice and it offers plenty of it: it is a great default Desktop.

Linus Torvalds doesn’t find Debian (and/or Ubuntu) a good platform to hack on the kernel. Do you know why and what can we do about this?

The Fedora linux-2.6 receives contributions from several Red Hat employed upstream sub-Maintainers. Thus it typically carries huge patches which are not yet upstream. As a consequence eventual userland troubles get revealed quite quickly and are often seen there first. The cutting edge nature of Fedora rawhide is appealing for many developers.

The usual Debian package division of library development files and the library itself is traditionally an entry barrier for dev on Debian. Debian got pretty easily usable these days, although we could and should again improve a lot more in this sector. Personally I think that Linus hasn’t tried Debian for years.

I have the feeling that the implication of the Debian Kernel team in LKML has been on the rise. Is that true and how do you explain this?

Ben Hutchings is the Nr.1 contributor for 2.6.33. He also is top listed as author of patches on stable 2.6.32. Debian is not listed as organization as many send their linux-2.6 patches from their corporate or personal email address and thus it won’t be attributed to Debian.

There is currently no means to see how many patches get forwarded for the stable tree, but I certainly forwarded more then fifty patches. I was very happy when Greg KH personally thanked me in the 2.6.32.12 release.

In the Squeeze kernel, the firmwares have been stripped and moved into separate packages in the non-free section. What should a user do to ensure his system keeps working?

There is a debconf warning on linux-2.6 installation. It is quite clear that the free linux-2.6 can’t depend on the firmware of the non-free archive (also there is no strict dependency there technically).

On the terminal you’d also see warnings by update-initramfs on the initramfs generation for drivers included in the initramfs.

The debconf warning lists the filename(s) of the missing firmware(s). One can then “apt-cache search” for the firmware package name and install it via the non-free repository. The check runs against the current loaded modules. The match is not 100% accurate for special cases as the one where the device might be handled well by this driver without firmware, but is accurate enough to warrant the warning.

The set of virtualization technologies that the official Debian kernel supports seems to change regularly. Which of the currently available options would you recommend to users who want to build on something that will last?

KVM has been a smooth ride from day zero. It almost got included instantly upstream. The uptake it has is great as it sees both dev from Intel and AMD. Together with libvirt it’s management is easy. Also the performance of virtio is very good.

The linux containers are the thing we are looking forward for enhanced chroots in the Wheezy schedule. They are also manageable by libvirt.

Xen being the “bad outside boy” has an incredible shrinking patchset, thus is fair to expect to see it for Wheezy and beyond. For many it may come a bit late, but for old hardware without relevant CPU it is there.

Many tend to overstate the importance of the virtualization tech. I’d be much more looking forward to the better Desktop support in newer linux-2.6. The Desktop is important for linux and something that is in heavy use. The much better graphics support of the radeon and nouveau drivers:

  • For Nouveau in 32 with 33 drm we could only deliver a first taste meaning something better then rusty nv.
  • For Radeon the support for Evergreen and newer is only been shaping now.

The performance optimizations thanks to dcache scalability work and the neat automatic task-grouping for the CPU scheduler are very promising features for the usability of the linux desktop. Another nice to have feature is the online defrag of ext4 and its faster mkfs. Even cooler would be better scalability in ext4 (This side seems to have seen not enough effort lately).

Is there someone in Debian that you admire for their contributions?

Hans Peter Anvin and Ted Tso are a huge source of deep linux-2.6 knowledge and personal wisdom. I do enjoy all sorts of interactions with them.

Christoph Hellwig with Matthew Wilcox and also William Irwin for setting up the Debian kernel Team.

Several Debian leaders including the previous and the current one for their engagement, which very often happens behind the scene.

The Debian Gnome Team work is great, also the interactions have always been always easy and a pleasure.

Martin Michlmayr and previously Thiemo Seufer do an incredible job in porting Debian on funny and interesting ARM and MIPS boxes. Debian has a lot of upcoming potential on this area. I’m looking forward to other young enthusiastic people in that area.

Colin Watson is bridging Debian and Ubuntu, which is an immense task.

Michael Prokop bases on Debian an excellent recovery boot CD: http://www.grml.org. I’d be happy if any Debian Developer would work as carefully coding and working.


Thank you to Maximilian for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.