Debian – Page 58

My Debian activities in July 2011

August 5, 2011 by Raphaël Hertzog

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (170 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf.

Libre Software Meeting / RMLL

I attended “only” 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month.

During those 3 days I helped with the Debian booth that was already well taken care of by Frédéric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time.

One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world).

Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out… it’s far from being an international event and the number of talks doesn’t make for a better quality.

BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux?

dpkg-source improvements

During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to “3.0 (quilt)” source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there’s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the process you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option.

Build flags changes

Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly.

In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags.

Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly.

Makefile snippets provided by dpkg

At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere.

More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related).

dpkg-buildflags improvements

Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got.

I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements.

In the end, here are the main changes implemented:

new “prepend” directive to inject flags at the start (see commit);
new “strip” directive to strip flags from the result returned by dpkg-buildflags (see commit);
new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
new --export=configure command to inject build flags on the ./configure command line (see commit);
new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs.

DebConf: rolling and hardening build flags

I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it’s been a pleasure to meet again all my Debian friends. It’s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I’m pretty happy with the result.

I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688).

We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line.

Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It’s not ready to be merged but it’s working already (see my last update in the bug log).

A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel.

It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes.

More dpkg work

Lots of small things done:

I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we’re down to 154 bugs (from more than 300 a month ago!).
I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS

Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian.

I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we’re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great!

Misc packaging work

I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process.

I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there’s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)!

Thanks

See you next month for a new summary of my activities.

Understanding dpkg’s file overwrite error

August 1, 2011 by Raphaël Hertzog

This is probably one of the most common errors. You’re very likely to encounter it, in particular if you tend to mix packages from various origins/distributions, or if you’re using unstable. It looks like this:

Unpacking gbonds-data (from .../gbonds-data_2.0.3-2_all.deb) ...
dpkg: error processing /var/cache/apt/archives/gbonds-data_2.0.3-2_all.deb (--unpack):
 trying to overwrite '/usr/share/omf/gbonds/gbonds-C.omf', which is also in package gbonds 2.0.2-9
dpkg-deb: subprocess paste killed by signal (Broken pipe)

A given file can only be provided by a single package. So if you try to install a package that provides a file that is already part of another installed package, it will fail with a message similar to the above one.

Sometimes this failure will be meaningful because dpkg prevented you to install two unrelated packages that happen to have a real file conflict. In other cases, like in the example above, this failure is just the result of a mistake.

Folder with gears

The version 2.0.3-1 of gbonds split the architecture independent files in a separate package called gbonds-data but the maintainer forgot to add the required control field in gbonds-data (Replaces: gbonds (<< 2.0.3-1)). That field allows dpkg to take over files from the listed packages.

If you want to ignore the file conflict and let dpkg take over the file (even without the Replaces), you can pass the --force-overwrite command-line option.

But you’re not using dpkg directly, you’re probably using an APT frontend (like apt-get or aptitude). Don’t worry, there’s a simple way to define custom dpkg options to use:

# apt-get -o Dpkg::Options::="--force-overwrite" install gbonds-data

The syntax is a bit weird, but the “::” after “Options” is important, it’s the syntax that defines a list item value instead of a single value. And you can effectively pass multiple options to dpkg by putting multiple -o Dpkg::Options::="…".

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

People behind Debian: Martin Michlmayr, former Debian Project Leader

July 21, 2011 by Raphaël Hertzog

Martin Michlmayr is a Debian developer since 2000 and I share quite a few things with him, starting with his age and involvement in the quality assurance team. He managed to be elected Debian Project Leader in 2003 and 2004.

He’s no longer as active as he used to be but his input is always very valuable and he continues to do very interesting things in particular concerning the support of NAS devices. Read on for the details.

Raphael: Who are you?

Martin: I’m Martin Michlmayr. I’m 32, originally from Austria, and currently living in the UK.

I’ve contributed to various free software projects over the years but Debian is without doubt the one I’m most passionate about. I joined Debian in 2000 when I was a student. I worked on Debian more or less full time for a few years while I was pretending to study. Later I started a PhD to do research about quality and management aspects of volunteer free software projects. I investigated the release process in several free software projects, in particular looking at time-based releases. After finishing my PhD in 2007, I joined Hewlett-Packard. I’m part of HP’s Open Source Program Office and work on various free software and open source activities, both internally and within the community.

Raphael: How did you start contributing to Debian?

Martin: I first used Debian in the days of 0.93R6, some time around the end of 1995. The 0.93R6 release was still based on a.out but I needed an ELF-based system for some application, so I moved to Slackware. I then quickly moved to Red Hat Linux where I stayed for several years. I rediscovered Debian in 2000 and quickly decided to join the project. I cannot recall how I rediscovered Debian but when I did, it was clear to me that Debian was the ideal project for me: I could identify with its philosophy, I liked the volunteer nature of the project, and I found the size and diversity of Debian interesting since a large project offers a lot of different challenges and opportunities.

I remember how many new things there were to learn and back then the documentation and other resources for new contributors were nowhere as good as they are today. My application manager, Julian Gilbey, was a great help… he was incredibly friendly and passionate about Debian. I also remember meeting up with Peter Palfrader (weasel) for key signing when we were both in the New Maintainer queue. I was incredibly lucky with my New Maintainer process and soon became an official Debian Developer. Because there was a shortage of application managers, my first major contribution in Debian was to become an application manager myself and help other people join the project.

Debian is a large project with a long history and a rich culture, so new contributors should expect that it will take some time to become familiar with everything. Fortunately, there are many resources, such as documentation and the debian-mentors list, for new contributors. Another great way to become familiar with the way things are done in Debian is to subscribe to various Debian mailing lists and ideally to read some mailing list archives. It’s also a great idea to attend the Debian Conference or other conferences since meeting people in real life is a great way to integrate. I remember attending Debian Conference 1 in Bordeaux where I gave my first public talk.

Finally, new contributors should find an area where they can make a unique contribution. Most people in Debian maintain packages but there are so many other ways to contribute. For example, most of my contributions were not technical but were about coordination and other organizational activities.

Raphael: What’s your biggest achievement within Debian?

Martin: I’m particularly proud of a number of achievements:

New Maintainer: I helped a lot of people join Debian. It’s great to help someone join the project and then see how they contribute. Of course, some people join Debian and then quickly become inactive or retire… you never know in advance how it will work out. But I had the pleasure to help some truly outstanding contributors to join Debian.
Quality Assurance: I helped improve QA processes within Debian. In particular, I realized a few years ago that a lot of packages had maintainers who were inactive and that nobody did anything about it. I started to write to those maintainers to see what could be done. It’s hard because you don’t know the circumstance of someone… they may be inactive because of an illness or for other good reasons… so you have to be friendly, but yet persistent. Fortunately, most maintainers I contacted were truly inactive and so they couldn’t complain when I took their packages away.
DPL: I acted as the Debian Project Leader for two years. I’m particularly proud of this because Debian is a great project and it was an honour to represent it. I performed important organizational and coordination tasks. I also traveled to a lot of conferences and had the pleasure to meet many Debian Developers as well as users of Debian. It’s very motivating to meet users and to hear how they use Debian and how we can further improve it.
Debian on NAS: Debian is without the doubt the Linux distro with the best support for NAS (Network Attached Storage) devices. I was always impressed by what the OpenWRT folks have done to support wireless routers and wanted to do something similar for Debian. Unfortunately, wireless routers just don’t have enough storage for a full distro. But then NAS devices came along and they obviously have enough space since they are meant for storage.

Raphael: Speaking about NAS devices: what exactly are you doing on this topic and how can people help?

Martin: There are plenty of instructions on the Internet to install Linux distributions on NAS or various embedded devices by connecting a serial console and then typing in hundreds of commands. What I found is that such instructions significantly limit the user base because they are way too complicated for most users. There are just too many steps that can go wrong.

So instead, in Debian, we provide a solution that just works: usually, you download a firmware image for your NAS device from Debian and when you upgrade you get the Debian installer. You connect to the installer via SSH and perform a normal installation. The installer knows about the device and will prepare everything for you automatically… for example, it knows if the device has requirements for the partition layout and it will install the kernel where the device expects to find it; unfortunately, NAS devices are not like PCs, so the requirements are different for almost every device and therefore you need special code to support a new device. Finally, there are detailed installation guides and we provide help on our mailing lists.

There are a number of technical areas for improvement. The installation could be made even easier, and it would be nice to support new platforms and devices.

A bigger problem is that while we’ve implemented a great solution for NAS devices, we haven’t really extended this work to support other classes of devices. For example, tablets and mobile phones are getting incredibly popular and we don’t have a compelling solution for such devices, mostly because of the lack of an appropriate GUI.

Raphael: What are your plans for Debian Wheezy?

Martin: I’ve recently been asked by Stefano Zacchiroli, our current Debian Project Leader, to coordinate the care-taking of Debian finances. Debian, as a volunteer project, relies on donations and in-kind gifts (e.g. hardware) to maintain its infrastructure and to support various development efforts, such as funding sprints and other developer gatherings. Debian’s money and other assets are held by affiliate organizations around the world.

My responsibility will be to keep track of money and other assets (e.g. hardware and trademarks), work with the DPL to establish procedures related to the use of Debian’s assets, and make sure that the procedures are followed. Finally, we want to publish public statements so our donors know how we use their donations to further improve Debian. I just started working on this and this will be my main activity in Debian in the coming months.

Raphael: Speaking of money, I plan to run a fundraising to get the Debian book I wrote with Roland Mas translated (cf. https://debian-handbook.info). Is this something Debian should support?

Martin: First of all, I should make it clear that I don’t decide how Debian spends its money. This is up to the DPL to decide together with the project at a whole. I’ll just make sure that procedures are followed and expenses tracked and reported properly.

Having said that, in my opinion, it’s unlikely that Debian as a project will fund this effort. It would be inconsistent with the position of the project not to fund work directly (only some related expenses, such as travel costs to allow Debian teams to organize face-to-face meetings). Whether Debian should support the fundraising effort by helping to promote it is another question and that’s probably not as clear cut. It looks like a worthwhile effort, but on the other hand it would be unfair for authors of other Debian books for Debian to put its weight behind one… and there are many other efforts that are worth promoting… if you promote one, where do you stop? So while it sounds worthwhile, it’s probably better for Debian to stay out of it.

But somehow related to this, I sometimes worry about the fact that there are so few paid opportunities around Debian. If you contribute to the Linux kernel for a while, you have an excellent chance to get hired by someone and to work on the kernel full time. The kernel may be an extreme example but there are a lot of projects that have more paid opportunities than Debian, e.g. Mono, GNOME, OpenOffice/LibreOffice and KDE.

Obviously, there are some Debian Developers who can spend some time on Debian as part of their job. I know that some Canonical employees contribute to Debian, that support companies like credativ improve Debian as part of their work, and that system administrators fix bugs or package new software as they deploy Debian. But I think this is a minority of contributors and even they don’t work full time on Debian. Instead what I see is that a lot of people leave university, get a job and then no longer have time for Debian… or people start a family and no longer have time. I can take myself as an example since I don’t have nearly as much time as I did in the past when I was a student.

I guess there are different ways to deal with this problem… one would be to create more paid opportunities around Debian outside the project, another one might be to make it easier for new volunteers to join the project. I don’t have the answers to these questions… but it’s something I wonder about, and I also wonder whether pure volunteer projects can still keep up with projects with a lot of full time contributors.

Raphael: What motivates you to continue to contribute year after year?

Martin: Debian is a great project with a great mission, goals and people. I contribute to make Debian a better solution and to promote the free software philosophy. Finally, the community around Debian provides a lot of motivation. It’s amazing how much I’ve learned about other cultures because of my involvement in Debian and how many friends I’ve made over the years all around the world.

Raphael: Do you have wishes for Debian Wheezy?

Martin: Not really. I’m pretty happy with the way things are going at the moment. We have made a lot of organizational changes in the last few years from which the project has greatly benefited. I’m particularly pleased about the plans to adopt a time-based freeze.

Raphael: Is there someone in Debian that you admire for their contributions?

Martin: There are many people I admire greatly. I’d like to mention Joey Hess because he’s a great example to follow. He doesn’t get involved in politics, is easy to work with and does great technical work. In fact, he has made not one but several contributions that have completely changed Debian (debconf, debhelper, and debian-installer). Furthermore, Debian has a lot of contributors who have done great work over the years but who are not very vocal about it. People like Colin Watson or Peter Palfrader. Debian has many unique contributors and the list of people I admire is much longer than the few people I just mentioned.

Thank you to Martin for the time spent answering my questions. I hope you enjoyed reading his answers as I did. He raised some interesting questions.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

Deciphering one of dpkg’s weirdest errors: unable to open ‘/path/to/foo.dpkg-new’

July 18, 2011 by Raphaël Hertzog

We already studied one weird error of dpkg, let’s do another one:

Unpacking replacement libexo-common ...
dpkg: error processing /var/cache/apt/archives/libexo-common_0.6.1-1_all.deb (--unpack):
 unable to open '/usr/share/doc/exo/html/C/images/exo-preferred-applications-internet.png.dpkg-new': No such file or directory

Rather difficult to understand on the first look right? Let’s see in detail what’s usually happening when you get this error.

The first hint comes from the file extension “.dpkg-new”. This extension is used by dpkg to unpack the updated files near the old files. When everything has been unpacked, they are renamed over the old files.

The failure happens precisely when dpkg tries to rename the file (in fact when it tries to fsync() it before the rename)… but why does it fail?

Usually because there are unexpected symlinks (or bind mounts) involved that resulted in two different files being installed in the same directory. For example consider package that provides /dir1/a-file and /dir2/a-file. Now imagine that on the target system, /dir1 is a real directory but /dir2 is a symbolic link that points to /dir1.

When dpkg processes /dir1/a-file.dpkg-new everything is fine, but when it tries to process /dir2/a-file.dpkg-new it will fail because that file is the same than /dir1/a-file.dpkg-new which has already been renamed.

Diagnosing further the problem requires to understand why there’s a symlink instead of a real directory. It might be two packages that were badly coordinated, or a problem in the package itself because it lacks some code that drops the symlink in the preinst (so that dpkg installs the real directory instead of keeping the symlink).

There might be variations in the way two files end up sharing the same directory, but this simple example should have clarified the nature of the underlying problem.

Subscribe to this blog by RSS, by email or on Facebook.